Contents x
    Managed ETL Configuration
      • Print
      • PDF
      Contents

      Managed ETL Configuration

        • Print
        • PDF
        Article summary

        Help page on Managed ETL Configuration settings.

        Creating Managed ETL Configuration

        STEP1 ETL Configuration of Data Source and Data Destination

        common setting

        item nameindispensableContents
        Managed ETL Configuration Name✓Enter a name.
        The name of the associated ETL Configuration will be "Managed ETL Configuration Name_Data Source Name".
        Resource Group-You can only select this option if you are using the Team function.

        For Data Source Microsoft SQL Server

        For more information, see Data Source - Microsoft SQL Server Configuration section.

        item nameindispensabledefault valueContents
        Microsoft SQL Server Connection Configuration✓-From the pre-registered Microsoft SQL Server Connection Configuration, select the one that has the necessary permissions for this ETL Configuration.
        database✓-Enter the name of the Microsoft SQL Server database that contains the data you wish to transfer.
        schema✓-Select the name of the Microsoft SQL Server schema that contains the data you wish to transfer.
        Default time zone✓-Select the server-side time zone for Microsoft SQL Server.
        If the time zone is Japan, enter Asia/Tokyo.

        For Data Source MySQL

        For more information, see the ETL Source - Data Source - MySQL Configuration section.

        item nameindispensabledefault valueContents
        MySQL Connection Configuration✓-Select the preregistered MySQL Connection Configuration that has the necessary permissions for this ETL Configuration.
        database✓-Select the name of the MySQL database that contains the data you wish to transfer.
        time zone✓-Enter the server-side time zone for MySQL.
        If the time zone is Japan, enter Asia/Tokyo.

        For Data Source PostgreSQL

        For more information, see the Data Source - PostgreSQL Configuration section.

        item nameindispensabledefault valueContents
        PostgreSQL Connection Configuration✓-Select the previously registered PostgreSQL Connection Configuration that has the necessary permissions for this ETL Configuration.
        database✓-Enter the name of the PostgreSQL database that contains the data you wish to transfer.
        schema✓-Select the name of the PostgreSQL schema that contains the data you wish to transfer.
        Default time zone✓-Select the server-side time zone for PostgreSQL.
        If the time zone is Japan, enter Asia/Tokyo.

        For Data Source Salesforce

        For more information, see the Data Source - Salesforce Configuration section.

        item nameindispensabledefault valueContents
        Salesforce Connection Configuration✓-Select the previously registered Salesforce Connection Configuration that has the necessary permissions for this ETL Configuration.
        Extraction of deleted and archived records✓non-extractiveSelect whether records that have been deleted or archived in Salesforce are eligible for extraction.

        For Data Destination Google BigQuery

        For more information, see Data Destination - Google BigQuery Configuration section.

        item nameindispensabledefault valueContents
        Google BigQuery Connection Configuration✓-From the previously registered BigQuery Connection Configuration, select the one that has the necessary permissions for this ETL Configuration.
        Option for automatic creation of data sets-OFFIf ON is selected, a dataset will be automatically created if the specified dataset does not exist on BigQuery.
        data-set✓-Select the name of the Data Destination data set.
        Data set names must consist of only letters, numbers, and underscores.
        Dataset Location✓asia-northeast1 (Tokyo)Select the location of the Data Destination data set.
        To specify the Tokyo region, enter asia-northeast1 (Tokyo).

        For Data Destination Snowflake

        For more information, see Data Destination - E TL Configuration section of Snowflake.

        item nameindispensabledefault valueContents
        Snowflake Connection Configuration✓-Select the preregistered Snowflake Connection Configuration that has the necessary permissions for this ETL Configuration.
        warehouse✓-Select the Data Destination warehouse name.
        database✓-Select the Database Name for Data Destination.
        schema✓-Select the name of the Data Destination schema.
        Replace empty characters with NULL✓ONYou can choose whether or not to replace empty characters with NULL.
        If you want to ETL Configuration with empty characters, please turn off the setting.

        STEP2 Import settings

        You can choose whether to import data sources (e.g., RDBMS tables, Salesforce objects, etc.).
        In addition, the following can be set for each data source to be imported

        For Data Source Microsoft SQL Server

        item namedefault valueContents
        column (esp. in chemistry, file, equipment, etc.)-Displays a list of columns.
        Unchecked columns are not transferred.
        By default, all column check boxes are enabled.
        Data Source ModeFull LoadSelect either Full Load or Incremental Load.
        If Incremental Load is selected, an Incremental Data Transfer Key must be selected.
        For more information on Incremental Data Transfer Keys, see Data Source - "Columns to Determine Incremental Data" in Microsoft SQL Server.
        Data Destination Output ModeAll cases washed (REPLACE)Select the transfer mode.
        The items that can be selected in Data Destination mode and their behavior depend on the Connector. For details, please refer to the following, respectively.
      • Data Destination BigQuery - About the Transfer Mode
      • Data Destination Snowflake - About the Transfer Mode
        • Data Destination Table-Enter the name of the table after the transfer.
        partitionNo partitionsYou can only select this option if you have selected a Data Destination BigQuery.
        Select whether or not to partition the table.
        For more information, see Setting up Partitioning and Clustering.

        For Data Source MySQL

        item namedefault valueContents
        column (esp. in chemistry, file, equipment, etc.)-Displays a list of columns.
        Unchecked columns are not transferred.
        By default, all column check boxes are enabled.
        Data Source ModeFull LoadSelect either Full Load or Incremental Load.
        If Incremental Load is selected, an Incremental Data Transfer Key must be selected.
        For more information on Incremental Data Transfer Keys, see Data Source - "Columns to Determine Incremental Data" in MySQL.
        Data Destination Output ModeAll cases washed (REPLACE)Select the transfer mode.
        The items that can be selected in Data Destination mode and their behavior depend on the Connector. For details, please refer to the following, respectively.
      • Data Destination BigQuery - About the Transfer Mode
      • Data Destination Snowflake - About the Transfer Mode
        • Data Destination Table-Enter the name of the table after the transfer.
        partitionNo partitionsYou can only select this option if you have selected a Data Destination BigQuery.
        Select whether or not to partition the table.
        For more information, see Setting up Partitioning and Clustering.

        For Data Source PostgreSQL

        item namedefault valueContents
        column (esp. in chemistry, file, equipment, etc.)-Displays a list of columns.
        Unchecked columns are not transferred.
        By default, all column check boxes are enabled.
        Data Source ModeFull LoadSelect either Full Load or Incremental Load.
        If Incremental Load is selected, an Incremental Data Transfer Key must be selected.
        For more information on Incremental Data Transfer Keys, see Data Source - Columns for Determining Incremental Data in PostgreSQL.
        Data Destination Output ModeAll cases washed (REPLACE)Select the transfer mode.
        The items that can be selected in Data Destination mode and their behavior depend on the Connector. For details, please refer to the following, respectively.
      • Data Destination BigQuery - About the Transfer Mode
      • Data Destination Snowflake - About the Transfer Mode
        • Data Destination Table-Enter the name of the table after the transfer.
        partitionNo partitionsYou can only select this option if you have selected a Data Destination BigQuery.
        Select whether or not to partition the table.
        For more information, see Setting up Partitioning and Clustering.

        For Data Source Salesforce

        item namedefault valueContents
        column (esp. in chemistry, file, equipment, etc.)-Displays a list of columns.
        Unchecked columns are not transferred.
        By default, all column check boxes are enabled.
        Data Destination Output ModeAll cases washed (REPLACE)Select the transfer mode.
        The items that can be selected in Data Destination mode and their behavior depend on the Connector. For details, please refer to the following, respectively.
      • Data Destination BigQuery - About the Transfer Mode
      • Data Destination Snowflake - About the Transfer Mode
        • Data Destination Table-Enter the name of the table after the transfer.
        partitionNo partitionsYou can only select this option if you have selected a Data Destination BigQuery.
        Select whether or not to partition the table.
        For more information, see Setting up Partitioning and Clustering.

        STEP2 Job Setting

        The following are batch settings for the associated ETL Configuration.

        item namedefault value
        **OFFON** allows jobs to be started in parallel.
        01 or more, ETL Configuration starts the transfer process over from the beginning if a transfer fails in progress.
        If the transfer process is not powerful, duplicate records may occur.
        The maximum number of retries is 10.        		(1)

        STEP3 Notification and Label Settings

        Notification settings for Created/Dropped Tables detection results

        Data Setting can be configured whether or not to notify the results of Created/Dropped Tables of the Data Source (e.g., RDBMS tables, Salesforce objects, etc.) on the ETL Configuration side.
        When enabled, the Slack channel will be notified of any increase or decrease in the Data Source when Managed ETL Configuration is run from a workflow.

        Schema Change Detection Notification Settings

        For each ETL Configuration created, the Schema Change Detection settings can be changed at once.
        You can later change individual settings for each ETL Configuration individually.

        Label Setting

        Labels can be assigned to Managed ETL Configuration and to the associated ETL Configuration at one time.

        Edit Managed ETL Configuration

        The configuration items are the same as for creating a Managed ETL Configuration.

        Change Connection Configuration/Database

        Connection Configuration and database changes to the RDBMS-based Data Source Connector are deprecated.
        Table Created/Dropped Tables detection may cause all existing associated ETL Configurations to become Deleted Tables.

        Instead of making the change, we recommend creating a new Managed ETL Configuration.

        Common Settings for Registered ETL Configuration

        Editing the common settings of a registered ETL Configuration will overwrite any separate changes to the associated ETL Configuration.

        Check Created/Dropped Tables

        The detection can be viewed on the Managed ETL Configuration details screen or from the hamburger menu in the upper right corner of the screen.
        Detects and displays newly added and deleted data sources (e.g., RDBMS tables, Salesforce objects, etc.).

        STEP1 Newly added table

        Displays a list of newly added Data Sources at the Data Source.
        You can also choose whether or not to include the data source.
        Configuration items are, Creating Managed ETL Configuration The configuration items are the same as in STEP 2 of

        STEP2 Deletion table

        List of Data Sources deleted at the Data Source.
        You can also choose to delete the corresponding ETL Configuration.
        Note that the ETL Configuration on TROCCO will be deleted, but the Data Source at the Data Source will not be deleted.

        List of unadded tables

        You can view the list of unadded tables on the Managed ETL Configuration details page.
        List Data Sources (e.g., RDBMS tables, Salesforce objects, etc.) that are in the Data Source and have no ETL Configuration on the TROCCO side.
        You can also choose whether or not to include the data source.
        Configuration items are, Creating Managed ETL Configuration The configuration items are the same as in STEP 2 of

        Was this article helpful?
        What's Next