Contents x
    Data Source - S3
    • 16 Mar 2022
    • Print
    • Dark
      Light
    • PDF
    Contents

    Data Source - S3

    • Updated on 16 Mar 2022
    • Print
    • Dark
      Light
    • PDF
    Article Summary

    Overview

    Help page for setting up a data transfer from AWS S3.

    Supported Protocol

    Data Loader: Embulk
    Using the embulk-input-s3 plugin.

    Limitations

    Transfer capacity is limited.

    Settings

    Step 1: General Settings

    Field Name
    Required
    Default Value
    Description
    S3 Connection ConfigurationYes-From the S3 Connection Configurations created in advance, select the one with the necessary permissions for this transfer.
    If you have not previously created any Connection Configurations, click the Add Connection Configuration button to set up a new one.
    RegionYesap-northeast-1Specify the region of the S3 bucket.
    For further information regarding regions, please refer to the AWS Regions section in the following AWS documentation.
    BucketYes-Enter the S3 bucket name.
    Path PrefixNo-Specify the path prefix for the data you wish to transfer.
    If you specify up to the directory name, all files under the directory (matching the prefix) will be transferred.
    If you specify up to a file name, only that file will be transferred.
    In this field, you can use custom variables to embed a value that will be dynamically set when the job executes.
    Path RegexNo-In this field, you can specify a regular expression to determine whether or not to transfer certain files under the path prefix.
    If left unspecified, all files under the prefix will be transferred.
    Decompression SettingsYesAuto-detectSet the decompression method as either Auto-detect, or ZIP.
    Input File FormatYesCSV/TSVSpecify the input file format from the following options:

  • CSV/TSV
  • JSON Lines
  • JSONPath
  • LTSV
    • Transfer MethodYesFull LoadChoose between either the Full Load or Incremental Load transfer methods.
    CSV/TSV File FormatYesCSV/TSV File With Header RowsChoose between the following options:

  • CSV/TSV File With Header Rows
  • CSV/TSV File Without Header Rows

    • When CSV/TSV File Without Header Rows is selected, trocco generates and assigns temporary header names.

    Advanced Settings

    Field Name
    Required
    Default Value
    Description
    Continue transfer when no file exists in the specified path?YesContinue TransferChoose whether or not to continue the transfer, when files under the set path (in the General Settings) cannot be found.
    When Continue Transfer is selected, the job will complete with 0 files transferred.
    When Stop Transfer with Error is selected, the job will stop with an error status.
    Was this article helpful?
    What's Next