Schema Synchronization

summary

In Step 3 of the transfer setting edit screen, if there is a schema difference between the setting to be saved and the currently valid setting, you can select whether or not to reflect it on the BigQuery side.

When can the schema tracking feature be used?

• Available only if the transfer modeappend is one of the BigQuery,append_direct where the destination is BigQuery.delete_in_advance
• As for the transfer source, it can be used without restrictions.
• Since the RECORD type of BigQuery is not supported, it will be invalid if the table name that references schema information as a template is set in the output option, or if there is a column that sets the RECORD type as the data type of the column.
• If a column definition change is detected, that fact is displayed when you click Save and Apply in STEP3.

Screen image

detail

• As change deltas, you can detect:
  ・Add columns
  ・Delete columns
  ・ Change the column type (type on the BigQuery side)

• This is not detected via functions such as column rereading and schema change detection, but also when the column is manually changed in STEP2.

• Please note that changes to column descriptions and output option settings will not be detected.

• In addition, since difference detection is performed based on trocco settings, there is a risk that it will not work correctly if changes are made to the table on the BigQuery side without going through trocco.

• When it comes to adding columns, you can choose how you want to fill in the values.
  Fill with NULL: Fill with NULL values.
  ・ Fill with default value (literal): Fill with literal value (number, string, TRUE/FALSE) according to each type.
  Fill with default values (expressions): You can write SQL expressions and fill them. (Functions such as CURRENT_TIMESTAMP(), using the values of other columns by specifying other column names, calculation formulas, etc.)

Rerun

• If tracking fails, you can restart from the transfer setting details screen.
• If you fail repeatedly, please contact Customer Success.

supplement

• Since the total amount of SELECT is generated at the time of tracking, a scan on the BigQuery side occurs for that amount.
• Items such as column description, split table setting, partition lifetime, etc. that can be set in trocco are carried over to the following table, but other contents set directly to the BigQuery side may be lost at the time of follow-up. Please be careful.