Data Source - S3
  • 16 Mar 2022
  • 1 Minute to read
  • Dark
    Light

Data Source - S3

  • Dark
    Light

Overview

Help page for setting up a data transfer from AWS S3.

Supported Protocol

Data Loader: Embulk
Using the embulk-input-s3 plugin.

Limitations

Transfer capacity is limited.

Settings

Step 1: General Settings

Field Name
Required
Default Value
Description
S3 Connection Configuration Yes - From the S3 Connection Configurations created in advance, select the one with the necessary permissions for this transfer.
If you have not previously created any Connection Configurations, click the Add Connection Configuration button to set up a new one.
Region Yes ap-northeast-1 Specify the region of the S3 bucket.
For further information regarding regions, please refer to the AWS Regions section in the following AWS documentation.
Bucket Yes - Enter the S3 bucket name.
Path Prefix No - Specify the path prefix for the data you wish to transfer.
If you specify up to the directory name, all files under the directory (matching the prefix) will be transferred.
If you specify up to a file name, only that file will be transferred.
In this field, you can use custom variables to embed a value that will be dynamically set when the job executes.
Path Regex No - In this field, you can specify a regular expression to determine whether or not to transfer certain files under the path prefix.
If left unspecified, all files under the prefix will be transferred.
Decompression Settings Yes Auto-detect Set the decompression method as either Auto-detect, or ZIP.
Input File Format Yes CSV/TSV Specify the input file format from the following options:

  • CSV/TSV
  • JSON Lines
  • JSONPath
  • LTSV
  • Transfer Method Yes Full Load Choose between either the Full Load or Incremental Load transfer methods.
    CSV/TSV File Format Yes CSV/TSV File With Header Rows Choose between the following options:

  • CSV/TSV File With Header Rows
  • CSV/TSV File Without Header Rows

  • When CSV/TSV File Without Header Rows is selected, trocco generates and assigns temporary header names.

    Advanced Settings

    Field Name
    Required
    Default Value
    Description
    Continue transfer when no file exists in the specified path? Yes Continue Transfer Choose whether or not to continue the transfer, when files under the set path (in the General Settings) cannot be found.
    When Continue Transfer is selected, the job will complete with 0 files transferred.
    When Stop Transfer with Error is selected, the job will stop with an error status.

    Was this article helpful?