About Workflow Definitions

summary

Help page for Workflow definitions.

Each individual unit, such as an ETL Job or Data Mart Job, that is incorporated into a workflow is called a "task".
Workflow definition creates a series of flows by setting the order and combination of tasks.

Timeout setting

Maximum number of tasks to be executed simultaneously

Determines the maximum number of concurrent parallel tasks to be set in a flow.
Up to 10 tasks can be set to run concurrently.

Setting a task concurrency limit will increase the number of tasks running in parallel on a single Workflow when each task is defined in parallel, thereby reducing the overall execution time of the Workflow.

Timeout setting

If disabled, there will be no timeout by execution time in the Workflow definition. Execution proceeds until the configured workflow is completed.
If enabled, the Job will be canceled after a Job Setting time has elapsed since the Workflow Job started.
If a job is canceled due to a timeout, it can be rerun from the stop position or the Workflow Job can be aborted.
If a timeout is set outside of the Workflow Definition, such as in Data Source, it will take precedence over the timeout set in the Workflow Definition.
This function can be used when there are restrictions, such as a job schedule for a BI tool used after the workflow job, or when the workflow process must be completed by a specified time, and you want to force the workflow job to stop when the time is up.

Number of retries

Sets the number of times an automatic retry is performed when a workflow fails and the time interval before the next retry is executed.
Note that automatic retries are not performed in the following cases

• When the number of times is set to 0
• If all tasks are successful
• In case of cancellation

Duplicate Job Execution

If a Workflow Job is still running when it is time for the next schedule, choose whether to skip or duplicate the next schedule.
When skipping, the real-time nature of data is reduced, but there is no need to worry about duplicate data at Data Destination. If duplicate runs are performed, there are concerns about double reflection of data, but the data can be reflected as scheduled.

Task error handling

Selects whether or not to execute subsequent tasks when the previous task fails.
When OFF, if any task fails, the workflow is stopped without executing subsequent tasks.
If turned ON, subsequent tasks will be executed even if one of them fails.
In cases where there are tasks with back-and-forth dependencies in the workflow, error handling can be turned off to avoid unintended data updates.
If the configuration consists only of tasks with no dependencies, it is considered safe to execute all but the failed tasks.

Schedule and Notification Settings

• Schedule and manage workflow Job Settings.
  Select the timing for execution from four patterns: hourly, daily, weekly, and monthly.
  If a Workflow definition and each task had its own Schedule, all Schedules are to be executed. Whenever possible, it is recommended to operate with the Schedule set to either the Workflow definition or each task.

• You can set up Slack or email notifications and manage your alerts.
  Multiple notifications can be set up for a single Workflow Job, so they can be used for completion of the Workflow Job, errors, etc.

flow editing

• Create a flow by placing ETL Jobs, Data Mart Syncs, etc. on the Flow Edit screen.
• By selecting a range in the flow diagram, you can select multiple tasks at once and set their order by drawing a line from one starting point at a time.
• You can create parent-child relationships for workflows.
  When a parent workflow is stopped due to the failure of a task in a child workflow and the parent workflow is reexecuted, the reexecution starts from the job that caused the error in the child workflow.

Behavior when Workflow Job Fails

When a Workflow Job is rerun from a failed state, the rerun starts from the stopped task.

Example.)

• Flow diagram of ETL Job parallel 2 tasks + data mart + notification
• Allows up to 2 tasks to be executed in parallel
• If one of the ETL Jobs fails
• When task error handling is turned on, non-failed ETL Jobs, Data Mart Jobs, and Notification Tasks will be executed.
• If the task's error handling is turned off, Workflow Job will stop when an error is detected.
• If the ETL Job Settings are changed and rerun after an error is detected, the rerun will start from the modified task and subsequent jobs will be executed.