Workflow

summary

This page is a help page for workflow definitions.

Each unit, such as a transfer job or a data mart job, to be incorporated into a workflow is called a "task".
A workflow definition creates a set of flows by setting the order and combination of tasks.

Timeout settings

Task concurrency limit

Determine the maximum number of parallel tasks that you want to set up in your flow to run concurrently.
Up to 10 tasks can be set to run concurrently.

Setting the maximum number of concurrent tasks increases the number of tasks that run in parallel on one workflow when each task is defined in parallel, which leads to a reduction in the execution time of the entire workflow.

Timeout settings

If disabled, there is no timeout based on execution time in the workflow definition. Execution proceeds until the configured workflow is completed.
If enabled, the workflow job is canceled after a set amount of time has elapsed since it started.
If the job is canceled due to the timeout, you can restart it from the stop position or suspend the workflow job.
If a timeout is set in a place other than the workflow definition, such as the transfer source, that takes precedence over the timeout setting on the workflow definition.
For example, if there are restrictions such as the workflow process must be completed by the specified time, such as the relationship between the job schedule of the BI tool used later, or when you want to forcibly stop the workflow job when the time comes.

Number of retries

Sets the number of times an automatic retry occurs when a workflow fails, and the time interval before the next retry run.
Automatic retry will not be performed in the following cases.

• When the number of times is set to 0
• If all tasks were successful
• If there is a cancellation

Duplicate job execution

If the workflow job is still running at the time of the next schedule, choose whether to skip the next schedule or run in duplicate.
If you skip it, the real-time nature of the data will decrease, but there is no need to worry about data duplication at the transfer destination. In the case of duplicate execution, there are concerns such as double reflection of data, but data can be reflected as scheduled.

Task Error Handling

Select whether to run subsequent tasks when the previous task fails.
If OFF is set to OFF, if any task fails, the workflow stops without executing subsequent tasks.
If turned ON, even if any task fails, subsequent tasks will be executed.
If there are tasks in the workflow that have dependencies before and after, you can turn off error handling to avoid unintentional data updates.
If you configure only tasks with no dependencies, it is considered safe to run only failed tasks.

Schedule and notification settings

• You can schedule and manage job execution for workflows.
  Choose when to run from four patterns: hourly, daily, weekly, and monthly.
  If you have set a separate schedule for the workflow definition and each task, all schedules will be executed. Whenever possible, we recommend that you set a schedule for either the workflow definition or each task.

• You can set up notifications in Slack or email and manage alerts.
  Since multiple notifications can be set for one workflow, it is possible to use them according to the completion of the workflow job or when an error occurs.

Flow Edit

• On the flow edit screen, create a flow by placing a transfer job, data mart sink, etc.
• By selecting a range in the flow diagram, you can select multiple tasks at once and set the order by drawing a line at once from one starting point.
• You can create parent-child relationships for workflows.
  If the parent workflow stops because a task in the child workflow fails, and you rerun the parent workflow, the rerun starts with the job that failed the child workflow.

What happens when a workflow job fails

If you rerun a job from a failed state of a workflow job, the rerun starts from the stopped task.

e.g.)

• Flow diagram of transfer job parallel 2 tasks + data mart + notification
• Up to 2 parallel execution tasks are allowed.
• If one of the transfer jobs fails
• When task error handling is turned on, transfer jobs, data mart jobs, and notification tasks that have not failed are executed.
• If task error handling is turned off, the workflow job stops when an error is detected.
• If you change the settings of a transfer job where an error is detected and rerun it, the rerun starts from the corrected task and the subsequent job runs.