About Workflow Definitions

summary

Help page about workflow definitions.

Each transfer job, data mart job, or other one-by-one unit to be incorporated into a workflow is called a "task.
The workflow definition creates a series of flows by setting the order and combination of tasks.

Timeout setting

Maximum number of tasks to be executed simultaneously

Determines the maximum number of concurrent parallel tasks to be set in a flow.
Up to 10 tasks can be set to run concurrently.

Setting a task concurrency limit will increase the number of tasks running in parallel on a single workflow when each task is defined in parallel, thereby reducing the overall execution time of the workflow.

Timeout setting

If disabled, there will be no timeout by execution time in the workflow definition. Execution proceeds until the configured workflow is completed.
If enabled, the job is canceled after a set amount of time has elapsed since the workflow job started.
If a job is canceled due to a timeout, it can be rerun from the stop position or the workflow job can be aborted.
If a timeout is set somewhere other than in the workflow definition, such as the originator, it will take precedence over the timeout setting in the workflow definition.
This function can be used, for example, when there are restrictions such as a job schedule for a BI tool being used after the workflow, or when there are other restrictions such as workflow processing must be completed by a specified time, or when you want to forcibly stop a workflow job when the time is up.

Number of retries

Sets the number of times an automatic retry is performed when a workflow fails and the time interval before the next retry is executed.
Note that automatic retries are not performed in the following cases
- When the number of times is set to 0
- If all tasks are successful
- In case of cancellation

Duplicate Job Execution

If a workflow job is still running when it is time for the next schedule, choose whether to skip or duplicate the next schedule.
When skipping, the real-time nature of data is reduced, but there is no need to worry about duplication of data at the destination. If duplicate runs are performed, there are concerns about double reflection of data, but the data can be reflected as scheduled.

Task error handling

Selects whether or not to execute subsequent tasks when the previous task fails.
When OFF, if any task fails, the workflow is stopped without executing subsequent tasks.
If ON, subsequent tasks will be executed even if one of the tasks fails.
In cases where there are tasks with back-and-forth dependencies in the workflow, error handling can be turned off to avoid unintended data updates.
If the configuration consists only of tasks with no dependencies, it is considered safe to execute all but the failed tasks.

Schedule setting and notification setting

• Job execution in workflow can be scheduled and managed.
  Select the timing for execution from four patterns: hourly, daily, weekly, and monthly.
  If you had a workflow definition and a schedule set up for each task, all schedules shall be executed. Whenever possible, it is recommended that the scheduling be pulled to either the workflow definition or each task.

• You can set up Slack or Email notifications and manage alerts.
  Multiple notifications can be set up for a single workflow, so they can be used for completion of a workflow job, errors, etc.

flow editing

• Create a flow by placing transfer jobs, data mart syncs, etc. on the flow edit screen.
• By selecting a range in the flow diagram, you can select multiple tasks at once and set their order by drawing a line from one starting point at a time.
• You can create parent-child relationships for workflows.
  When a parent workflow is stopped due to the failure of a task in a child workflow and the parent workflow is reexecuted, the reexecution starts from the job that caused the error in the child workflow.

Behavior when a workflow job fails

When a workflow job is rerun from a failed state, the rerun starts from the stopped task.

Example.)
* Flow diagram of transfer job parallel 2 tasks + data mart + notification
* Allows up to 2 tasks to be executed in parallel
* If one of the transfer jobs fails
* If task error handling is turned on, non-failed transfer jobs, data mart jobs, and notification tasks will be executed.
* If task error handling is turned off, the workflow job will stop when an error is detected.
* If you change the settings of a transfer job for which an error is detected and rerun the job, the rerun will start from the modified task and subsequent jobs will be executed.