---
title: "Data Source - TROCCO"
slug: "data-source-trocco"
updated: 2024-08-15T09:38:30Z
published: 2024-08-15T09:38:30Z
canonical: "documents.trocco.io/data-source-trocco"
---

> ## Documentation Index
> Fetch the complete documentation index at: https://documents.trocco.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Data Source - TROCCO

## summary

Help page for ETL Configuration of Data Settings held in TROCCO. Data Source TROCCO can transfer any of the following data

- **Data Catalog**
  - Metadata held in the Data Catalog can be transferred.
- **ETL Configuration**, **Data Mart Configuration**, **Workflow Definition**
  - Historical data on previously executed ETL Jobs can be ETLed.

## constraints

              About Data Catalog

              

In order to transfer data in the Data Catalog, the Data Catalog must be activated.

Data Catalog is **a paid option**. To request a trial or to sign up, please contact your sales representative or Customer Success.

              Period during which job execution history can be obtained

              

ETL Job can transfer historical data of jobs executed **up to one year in the past**, starting from the date and time when the job execution history is transferred. Historical data for jobs executed before that time cannot be ETL Job.

## Setting items

### STEP1 Basic settings

| item name | indispensable | Contents |
| --- | --- | --- |
| subject (of taxation, etc.) | Yes | Select the target to be transferred from the followingData CatalogETL ConfigurationData Mart ConfigurationWorkflow |
| data type | Yes | Select the data type to be transferred from the following - If **Data Catalog is** selected - User-defined metadata (BigQuery table information) - User-defined metadata (BigQuery Column Setting information) - column reference list - When **ETL Configuration**, **Data Mart Configuration**, or **Workflow Configuration** is selected - job execution history |

When **ETL Configuration**, **Data Mart Configuration**, or **Workflow Configuration** is selected

| item name | indispensable | Contents |
| --- | --- | --- |
| Data acquisition period | Yes | Enter the start and end dates and times of the data you wish to transfer. Please enter in `YYYYY-MM-DD` or `YYYYY-MM-DD HH:mm:ss` format. If `HH:mm:ss is` not specified, `00:00:00` is specified instead. |
| time zone | Yes | Select the time zone for the data acquisition period. |

              Criteria for data acquisition period

              

The data acquisition period is based on the execution date/time ( `created_at)` of each job. Therefore, the historical data of ETL Job whose `created_at` value is within the data acquisition period is subject to ETL Job.

## transfer item

Below is a list of items transferred for each target and each data type.

### Data Catalog

#### user defined metadata

There are two types of data types that can be transferred

- User-defined metadata (BigQuery table information):.
  - Transfers [table information](/docs/table-information-v1#%E3%83%86%E3%83%BC%E3%83%96%E3%83%AB%E6%83%85%E5%A0%B1) metadata.
- User-defined metadata (BigQuery Column Setting information)
  - Transfers [column information](/docs/table-information-v1#%E3%82%AB%E3%83%A9%E3%83%A0%E6%83%85%E5%A0%B1) metadata.

              Table for metadata transfer

              

Only metadata for table columns for which at least one of the **basic** or **user-defined metadata** has already been entered will be transferred.

| column name | type (e.g. of machine, goods, etc.) | Description. |
| --- | --- | --- |
| `bigquery_project_id` | `string` | Metadata **projects** related to BigQuery |
| `bigquery_dataset` | `string` | **Dataset of** metadata about BigQuery |
| `bigquery_table` | `string` | BigQuery table name |
| `bigquery_column` | `string` | BigQuery column name Only **user-defined metadata (BigQuery Column Setting information)** will be transferred if selected as the data type. |
| `trocco_metadata_logical_name` | `string` | **Logical name of** basic metadata |
| `trocco_metadata_description` | `string` | Basic Metadata **Description** |
| `trocco_metadata_last_updated_at` | `timestamp` | Last update date and time of basic metadata |
| (Fields defined in [the template for user-defined metadata](/docs/data-catalog-settings-v1#%E3%83%A6%E3%83%BC%E3%82%B6%E3%83%BC%E5%AE%9A%E7%BE%A9%E3%83%A1%E3%82%BF%E3%83%87%E3%83%BC%E3%82%BF%E7%94%A8%E3%83%86%E3%83%B3%E3%83%97%E3%83%AC%E3%83%BC%E3%83%88) ) | (see below for details) | User-defined metadata values |
| `user_defined_metadata_last_updated_at` | `timestamp` | Last modified date and time of user-defined metadata |

              Field defined in template for user-defined metadata

              

- About Data Types The data types defined in the template are converted to the following types
  - `String` -> `string`
  - `Text(Markdown)` -> `string`
  - `Integer` -> `long`
  - `Boolean` -> `boolean`
- About column names
  - If **non-alphanumeric characters** are used **in** the field names defined in the Template for User-Defined Metadata, some Data Destination Connectors may not be able to forward the data.
  - In the above case, change the column name to one consisting only of alphanumeric characters in Column Setting in ETL Configuration STEP 2.

#### column reference list

Forward the [column reference](/docs/column-reference-v1) list.

| column name | type (e.g. of machine, goods, etc.) | Description. |
| --- | --- | --- |
| `bigquery_src_project_id` | `string` | Metadata projects related to BigQuery (references) |
| `bigquery_src_dataset` | `string` | Dataset of metadata about BigQuery (references) |
| `bigquery_src_table` | `string` | BigQuery table name (reference source) |
| `bigquery_src_column` | `string` | BigQuery column name (reference source) |
| `bigquery_dst_project_id` | `string` | Metadata projects related to BigQuery (references) |
| `bigquery_dst_dataset` | `string` | Dataset of metadata about BigQuery (references) |
| `bigquery_dst_table` | `string` | BigQuery table names (references) |
| `bigquery_dst_column` | `string` | BigQuery column names (references) |
| `creation_type` | `string` | How to create a column referenceFor automatic definition: `trocco_data_source`If user-defined: `user` |
| `created_by` | `string` | Email address of the column reference creator |
| `last_updated_at` | `timestamp` | Last modified date and time of the column reference |

### ETL Configuration

#### job execution history

ETL Job transfers historical data about previously executed ETL Jobs.

| column name | type (e.g. of machine, goods, etc.) | Description. |
| --- | --- | --- |
| `job_id` | `long` | ETL Job ID |
| `job_url` | `string` | ETL Job URL |
| `job_definition_id` | `long` | ETL Job Source ETL Configuration ID |
| `job_definition_url` | `string` | ETL Job Source ETL Configuration URL |
| `job_definition_name` | `string` | Name of Data Source ETL Configuration from which the Job Source was created. |
| `executor_type` | `string` | How to execute an ETL Job`manual：Manual` execution`Scheduler`: Scheduled execution`workflow`: Executed as a workflow task`api`: Execution by API`job_dependency`: Execution by trigger job |
| `status` | `string` | Status of ETL Job`queued`: waiting for execution`setting_up`: preparing to run`executing`: Executing`interrupting`: execution is interrupted`Succeeded`: Execution completed (success)`error`: Execution completed (error)`canceled`: Execution completed (canceled)`skipped`: execution completed (skipped) |
| `transfer_records` | `long` | Number of records transferred |
| `transfer_bytes` | `long` | `null` |
| `skipped_records` | `long` | `null` |
| `started_at` | `timestamp` | `null` |
| `finished_at` | `timestamp` | `null` |
| `created_at` | `timestamp` | Date and time of execution of ETL Job ( `%Y-%m-%d %H:%M:%S %Z)` |

### Data Mart Configuration

#### job execution history

Transfers historical data on previously executed data mart jobs.

| column name | type (e.g. of machine, goods, etc.) | Description. |
| --- | --- | --- |
| `datamart_job_id` | `long` | DATAMART Job ID |
| `datamart_job_url` | `string` | Data Mart Job URL |
| `datamart_definition_id` | `long` | Data Mart Configuration ID from which the job originated |
| `datamart_definition_url` | `string` | Data Mart Configuration URL from which the job was created |
| `datamart_definition_name` | `string` | Name of the Data Mart Configuration from which the job originated. |
| `executor_type` | `string` | How to run a data mart job`manual：Manual` execution`Scheduler`: Scheduled execution`workflow`: Executed as a workflow task`job_dependency`: Execution by trigger job |
| `status` | `string` | Data Mart Job Status`queued`: waiting for execution`setting_up`: preparing to run`executing`: Executing`interrupting`: execution is interrupted`Succeeded`: Execution completed (success)`error`: Execution completed (error)`canceled`: Execution completed (canceled)`skipped`: execution completed (skipped) |
| `transfer_records` | `long` | Number of records transferred |
| `started_at` | `timestamp` | `null` |
| `finished_at` | `timestamp` | `null` |
| `created_at` | `timestamp` | Date and time of execution of the data mart job ( `%Y-%m-%d %H:%M:%S %Z)` |

### Workflow

#### job execution history

Transfers historical data about previously executed Workflow Jobs.

| column name | type (e.g. of machine, goods, etc.) | Description. |
| --- | --- | --- |
| `pipeline_job_item_id` | `long` | Unique ID associated with Workflow Job ID |
| `pipeline_job_id` | `long` | Workflow Job ID |
| `pipeline_job_url` | `string` | Workflow Job URL |
| `pipeline_definition_id` | `long` | Workflow Job's original Workflow Job ID |
| `pipeline_definition_url` | `string` | Workflow Job's original Workflow Job URL |
| `pipeline_definition_name` | `string` | The name of the Workflow Job's original Workflow definition |
| `executor_type` | `string` | How to Perform a Workflow Job`manual：Manual` execution`Scheduler`: Scheduled execution`workflow`: Executed as a workflow task`api`: Execution by API`retry`: Retry execution (reexecution from the stop position or reexecution by retry setting) |
| `canceled_type` | `string` | `null` |
| `status` | `string` | Workflow Job Status`queued`: waiting for execution`setting_up`: preparing to run`executing`: Executing`interrupting`: execution is interrupted`Succeeded`: Execution completed (success)`error`: Execution completed (error)`canceled`: Execution completed (canceled)`skipped`: execution completed (skipped)`retry_waiting`: Waiting state from execution completion (error) state to start retry (only if retry count is set) |
| `started_at` | `timestamp` | `null` |
| `finished_at` | `timestamp` | `null` |
| `created_at` | `timestamp` | Workflow Job execution date and time ( `%Y-%m-%d %H:%M:%S %Z)` |