Data Destination - Databricks

Overview

Help page for setting up data transfer to Databricks.

Available Cloud Environments

Data Source and Data Destination Databricks is available in any of the following cloud environments.

TROCCO uses Unity Catalog Volume as temporary file storage during data transfer, so it does not depend on storage services or interfaces specific to each cloud service.

Limitations

Unavailable Data Types

The following types are not supported by Databricks JDBC Driver and will cause an error if present in a destination table column.

TIMESTAMP_NTZ （Currently TIMESTAMP_NTZ type is not yet supported in JDBC Drivers）
INTERVAL （Delta Lake does not support the INTERVAL type）

The following types are unsupported and cannot transfer values to columns.

BINARY
ARRAY
MAP
STRUCT

Settings

Step 1: Basic settings

Field Name	Required	Default Value	Details
Databricks Connection Configuration	Yes	-	Select from a pre-set Databricks Connection Configuration that has the necessary permissions for this ETL Configuration.
Catalog	Yes	-	Select the name of the destination catalog.
Schema	Yes	-	Select the schema name of the destination.
Table	Yes	-	Select the destination table name. If the target table does not exist in the destination database schema, it will be created automatically.
Transfer Mode	Yes	`APPEND (INSERT)`	Select Transfer Mode. For details, see [About Transfer Modes](#About Transfer Modes) below.
Merge Key	No	-	Can be entered when `UPSERT (MERGE)` is selected in the Transfer Mode. If the primary key does not exist in the destination table, enter the name of the column to be treated as a merge key (primary key). For merge keys, enter columns that have no duplicate values and no null values.

Field Name

Required

Default Value

Details

Databricks Connection Configuration

Yes

Select from a pre-set Databricks Connection Configuration that has the necessary permissions for this ETL Configuration.

Catalog

Yes

Select the name of the destination catalog.

Schema

Yes

Select the schema name of the destination.

Table

Yes

Select the destination table name.
If the target table does not exist in the destination database schema, it will be created automatically.

Transfer Mode

Yes

APPEND (INSERT)

Select Transfer Mode.
For details, see [About Transfer Modes](#About Transfer Modes) below.

Merge Key

Can be entered when UPSERT (MERGE) is selected in the Transfer Mode.
If the primary key does not exist in the destination table, enter the name of the column to be treated as a merge key (primary key).
For merge keys, enter columns that have no duplicate values and no null values.

Step 1: Advanced Settings

Field Name	Default Value	Details
Batch Size (MB)	50	Specify the batch size.
Default Time Zone	Etc/UTC	Specify the default time zone.

Field Name

Default Value

Details

Batch Size (MB)

Specify the batch size.

Default Time Zone

Etc/UTC

Specify the default time zone.

Additional Information

About Transfer Modes

Transfer Mode	Details
`APPEND (INSERT)`	Creates a temporary table before transferring the data. After all temporary tables have been created, inserts the data into the target table.
`APPEND (INSERT DIRECT)`	Insert rows directly into the target table. If the transfer fails, some data may have been inserted into the target table.
`REPLACE (TRUNCATE INSERT`	Creates a temporary table before transferring the data. After all temporary tables have been created, deletes the content of the target table and then insert data into the target table.
`REPLACE`	Creates a temporary table before transferring the data. Once the temporary table has been created, deletes the target table and rename the temporary table to the target name. If the transfer fails, the target table may be deleted.
`UPSERT (MERGE)`	Creates a temporary table before transferring the data. After all temporary tables have been created, the following process will be occur: For the target table, rows that match the merge key and value in the temporary table are updated, and rows that do not match are inserted.

Transfer Mode

Details

APPEND (INSERT)

Creates a temporary table before transferring the data.
After all temporary tables have been created, inserts the data into the target table.

APPEND (INSERT DIRECT)

Insert rows directly into the target table.
If the transfer fails, some data may have been inserted into the target table.

REPLACE (TRUNCATE INSERT

Creates a temporary table before transferring the data.
After all temporary tables have been created, deletes the content of the target table and then insert data into the target table.

REPLACE

Creates a temporary table before transferring the data.
Once the temporary table has been created, deletes the target table and rename the temporary table to the target name.
If the transfer fails, the target table may be deleted.

UPSERT (MERGE)

Creates a temporary table before transferring the data.
After all temporary tables have been created, the following process will be occur:
For the target table, rows that match the merge key and value in the temporary table are updated, and rows that do not match are inserted.

Data Destination - Databricks

Overview

Limitations

Unavailable Data Types

Settings

Step 1: Basic settings

Step 1: Advanced Settings

Additional Information

About Transfer Modes

What's Next