Data Destination - Databricks

Overview

Help page for setting up data transfer to Databricks.

Available Cloud Environments

Data Source and Data Destination Databricks is available in any of the following cloud environments.

TROCCO uses Unity Catalog Volume as temporary file storage during data transfer, so it does not depend on storage services or interfaces specific to each cloud service.

Limitations

Unavailable Data Types

The following types are not supported by Databricks JDBC Driver and will cause an error if present in a destination table column.
- TIMESTAMP_NTZ （Currently TIMESTAMP_NTZ type is not yet supported in JDBC Drivers）
- INTERVAL （Delta Lake does not support the INTERVAL type）
The following types are unsupported and cannot transfer values to columns.
- BINARY
- ARRAY
- MAP
- STRUCT

Settings

Step 1: Basic settings

Field Name	Required	Default Value	Details
Databricks Connection Configuration	Yes	-	Select from a pre-set Databricks Connection Configuration that has the necessary permissions for this ETL Configuration.
Catalog	Yes	-	Select the name of the destination catalog.
Schema	Yes	-	Select the schema name of the destination.
Table	Yes	-	Select the destination table name. If the target table does not exist in the destination database schema, it will be created automatically.
Transfer Mode	Yes	`APPEND (INSERT)`	Select Transfer Mode. For details, see [About Transfer Modes](#About Transfer Modes) below.
Merge Key	No	-	Can be entered when `UPSERT (MERGE)` is selected in the Transfer Mode. If the primary key does not exist in the destination table, enter the name of the column to be treated as a merge key (primary key). For merge keys, enter columns that have no duplicate values and no null values.

Step 1: Advanced Settings

Field Name	Default Value	Details
Batch Size (MB)	50	Specify the batch size.
Default Time Zone	Etc/UTC	Specify the default time zone.

Additional Information

About Transfer Modes

Transfer Mode	Details
`APPEND (INSERT)`	Creates a temporary table before transferring the data. After all temporary tables have been created, inserts the data into the target table.
`APPEND (INSERT DIRECT)`	Insert rows directly into the target table. If the transfer fails, some data may have been inserted into the target table.
`REPLACE (TRUNCATE INSERT`	Creates a temporary table before transferring the data. After all temporary tables have been created, deletes the content of the target table and then insert data into the target table.
`REPLACE`	Creates a temporary table before transferring the data. Once the temporary table has been created, deletes the target table and rename the temporary table to the target name. If the transfer fails, the target table may be deleted.
`UPSERT (MERGE)`	Creates a temporary table before transferring the data. After all temporary tables have been created, the following process will be occur: For the target table, rows that match the merge key and value in the temporary table are updated, and rows that do not match are inserted.