Release Notes - August 2022

Prev Next

Hello! Release information for August 2022!

Data Catalog

Basic metadata is displayed on the column information screen

The basic metadata items "logical name" and "description" can now be selected whether or not to be displayed in the column list.
image

The above can be set from "TROCCO Basic Metadata Settings" in "Data Catalog Settings".
image

Download query execution results in CSV

Query Editor can now download query execution results in CSV format.
image

Query execution result download file size limit

The maximum file size of the CSV file that can be downloaded is 10 MB.
Lines that exceed the file size limit are truncated.

Edit metadata in the Markdown editor

The "Description" field of basic metadata and any field value of user-defined metadata can now be edited in the Markdown editor.
image

image

User-defined metadata fields can be configured in the "Template for User-defined Metadata" under "Data Catalog Setting".
Specify Text(Markdown) as the data type for the fields for which you want to use the Markdown editor.
image

Help Document Publication

Help documentation on Data Catalog is now available.
For more information, see About Data Catalog.

dbt linkage

dbt integration functionality has been released 🎉 .

  • We provide a runtime environment for the OSS version of dbt and GitHub integration.
  • The ETL to data modeling flow can be defined on the TROCCO Workflow function.
  • As of the end of August 2022, the following three dbt Git repository adapters are supported
    • Google BigQuery
    • Snowflake
    • Amazon Redshift
  • Please refer to the press release for details, including future release plans.

Managed ETL

Managed ETL is now available 🎉 .

Managed ETL is an evolution of the existing batch registration feature.
In addition to batch import of tables on the data source side, the new system automatically detects Created/Dropped Tables on the data source side and notifies Slack, and allows for easy batch execution of ETL Jobs on the workflow function.

Supported Connector for Managed ETL

As of the end of August 2022, only Data Source MySQL and Data Destination Google BigQuery are supported.

Adding Managed ETL Configuration

Created/Dropped Tables can be detected automatically.

In STEP 3 "Notification of Created/Dropped Tables Detection Results and Batch Label Settings", you can configure the settings for notification of Created/Dropped Tables detection results.
image

Managed ETL Configuration List

You can see the list of Managed ETL Configurations you added earlier.
In addition to the detailed screen for each Managed ETL Configuration, you can also go to the detailed screen for individual ETL Configurations under each Managed ETL Configuration with a single click.
image

Managed ETL Configuration Details Screen

For the relevant Managed ETL Configuration, you can manually add tables or delete individual ETL Configurations under your control.
image

A dialog box is also displayed when Created/Dropped Tables are detected.
image

Workflow

Batch Execution of ETL Jobs under Managed ETL Configuration

TROCCO Managed ETL Configuration" has been added as a task that can be added to the Flow Editor in Workflow Definitions.
This allows ETL Jobs under each Managed ETL Configuration to be executed in one go.
image

image

image

ETL Configuration

HubSpot added to Data Destination🎉.

A merge can be performed on a contact object.
image

TROCCO added to Data Source 🎉.

Data related to "User-Defined Metadata" and "Column Setting Reference List" held in TROCCO's Data Catalog can be transferred.
image

Data Source Marketo Target for Folders

Data Source Marketo can now retrieve the folder's data.
image

Added Ability to Support Custom Variables 🎉.

Custom Variables are now supported in the following functions

  • ETL Configuration Filter Settings in STEP2 "Data Preview and Advanced Settings" Filter Setting, Filter Value
    image

  • Keys and values in the "HTTP Header" of the Data Source HTTP (S)
    image

UI・UX

ETL Configuration List

Added "Author" to filter field.

ETL Configurations listed can now be narrowed down by "Author".
image

Filter item "name (partial match)" supports AND search.

AND searches can now be performed by entering entries separated by spaces.

  • When searching for a single word
    image
  • When an AND search is performed on two words
    image

Labels can now be added to selected ETL Configurations in batches.

By clicking the buttons in the order described within the screen below, the Add Label Bulk screen will appear.
image

Select the Label of your choice and click Add.
image

Account/User Screen

Displays the authentication method for each user

You can check if 2-step verification and SAML verification are enabled respectively.
image

In addition, the following modifications and others have been added

  • When line_item is selected as the report type in Data Source Twitter Ads, campaign_id is now included in the transfer content.

**The contents of this release are described above. **
**Please feel free to contact our Customer Success Representative if you have a release you are interested in. **
Happy Data Engineering!