Contents x
    Data Destination - Snowflake
      • Print
      • PDF
      Contents

      Data Destination - Snowflake

        • Print
        • PDF
        Article summary

        summary

        Help Page for ETL Configuration of Data Setting to Snowflake.

        Case study on the use of Snowflake and TROCCO

        TROCCO's product website provides examples of customer installations.
        Please refer to this page together with this help page.

        Setting items

        STEP1 Basic settings

        item nameindispensabledefault valueContents
        Snowflake Connection ConfigurationYes-Select the preregistered Snowflake Connection Configuration that has the necessary permissions for this ETL Configuration.
        warehouseYes-Select the warehouse to be used as the query execution environment during transfer.
        databaseYes-Select the Database Name for Data Destination.
        schemaYes-Select the name of the Data Destination schema.
        tableYes-Enter the name of the Data Destination table.
        If the target table does not exist in the Data Destination database schema, the table will be created automatically.
        transfer modeYesAppend (INSERT)Select the transfer mode.
        For more information, see About Transfer Mode below.
        merge keyNo-This can be set when UPSERT (MERGE) is selected in ETL Configuration mode.
        Enter the name of the column to be treated as a merge key (primary key) that has no duplicate values and no null values.

        STEP1 Detailed settings

        Click on Advanced Settings to display the configuration items.

        item namedefault valueContents
        Replace empty characters with NULLONYou can choose whether or not to replace empty characters with NULL.
        If you want to ETL Configuration with empty characters, please turn off the setting.
        Delete temporary stage on errorOFFYou can choose to delete the temporary stage if the ETL Job fails.
        Even if this ETL Configuration is ON, the temporary stage will not be deleted if the ETL Job Setting is cancelled.
        Batch size (MB)50Enter the batch size.
        If an OutOfMemoryError occurs during the transfer, reduce the batch size accordingly.

        STEP2 Output Option

        item namedefault valueDetails
        Column Setting-Specify columns for creating temporary tables. The default values of the type are as follows
      • boolean: BOOLEAN
      • string:VARCHAR(65535)
      • long: BIGINT
      • double: DOUBLE PRECISION
      • timestamp: TIMESTAMP_NTZ(9)
      • json: VARIANT

        • Settings are required if you wish to use other than the above.
        Please refer to the official documentation for details on the types that can be specified.
        Maximum number of retries12Specifies the maximum number of retries to be performed.
        Retry waiting time1,000Specifies the waiting time for retries.
        Maximum retry wait time1,800,000Specifies the maximum time to wait for retries.
        Default time zoneUTCEnter the time zone to be specified in a column of type timestamp.
        Conditions under which schema-related settings apply

        The contents of the Column Settings in the STEP2 Output Option apply only when creating a new table.
        Specifically, it is applied when a job is executed in the following conditions

        • If the target table does not exist in Data Destination
        • When Full Data Transfer (REPLACE) is selected in the Transfer Mode
          • In this case, the schema of the Data Destination table is updated for each transfer, so the contents of the column configuration are applied each time.

        supplementary information

        About transfer mode

        All of the following explanations are for cases where the target table already exists in the Data Destination.
        If the target table does not exist in Data Destination, a new table is created in either mode.

        transfer modeSchema Changetransactionbehaviorsupplement
        Append (INSERT)NoYesThe following additions are made to the table.
        1. Create a temporary table and populate it with data
        2. INSERT data from a temporary table into a target table
        Postscript (INSERT DIRECT)NoNoThe table is appended as follows
        1. INSERT data directly into the target table
      • Compared to the append (INSERT ) mode, processing time is shorter, but transactions are not secured.
      • Therefore, if a transfer fails midway through, rows added along the way may remain in the target table.
        • TRUNCATE INSERTNoYesThe table will be rewashed as follows
        1. Create a temporary table and populate it with data
        2. DELETE the actual data in the target table and INSERT the data in the temporary table into the target table
        All cases washed (REPLACE)YesYesThe table will be rewashed as follows
        1. Create a temporary table and populate it with data
        2. DROP the target table itself and change the temporary table name to the target table name.
      • The schema of the existing table is deleted and updated to that of the new table.
        • UPSERT (MERGE)NoYesUPSERT to the table as follows
        1. Create a temporary table and populate it with data
        2. MERGE data from temporary table to target table
      • For each row of transferred data, if the value of the merge key exists in the corresponding column of the target table, the row is updated. If it does not exist, the line is appended.
      • If the merge key has duplicate values or contains null values, it will not UPSERT correctly.
        • Was this article helpful?
        What's Next