Workflow data check

summary

What is Data Check?

This is one of the tasks that can be set up within the workflow.
This function checks the results of a query against the DWH against error conditions, and if the conditions are met, the corresponding task is marked as an error.

For example, if a particular string in a column is counted and the result is 2 or more, it is considered duplicate data and an error is generated.

DWH for data checking

• Google BigQuery
• Snowflake
• Amazon Redshift

Setup Method

The following is an example of how to set up Google BigQuery.

1. Specify the BigQuery Connection Configuration registered in advance.
   This Connection Configuration must be authorized to execute the query described.
   

2. Enter a query to check data.
   Write the query in a SELECT statement so that the query results in a single row and a single column of numbers.
   Custom Variables can be embedded in the query.
   Click Preview Run to see the results of the query you have written on the fly.
   

3. Specify error conditions.
   Specify the reference value and its condition. You can choose from six conditions.
   If the query result is null, you can also choose whether or not to mark the corresponding task as a success.