Contents x
    Data Source - DynamoDB
    • 27 Dec 2022
    • Print
    • Dark
      Light
    • PDF
    Contents

    Data Source - DynamoDB

    • Updated on 27 Dec 2022
    • Print
    • Dark
      Light
    • PDF
    Article Summary

    Note

    This is a machine-translated version of the original Japanese article.
    Please understand that some of the information contained on this page may be inaccurate.

    Overview

    This is a help page for setting up transferring items from DynamoDB.

    Supported Protocols

    Settings

    STEP 1: General Settings

    Field NamesRequiredDefaultDescription
    DynamoDB connection informationYes-From the connection information registered in advance, select the one that has the necessary permissions for this transfer setting.
    Please refer to the separate page for how to set the connection information.
    RegionYes-Specify the AWS region.
    tableYes-Specify the table for which you want to retrieve items.
    Secondary IndexesNo-Specify a global/local secondary index.
    Please refer to the official documentation for details.
    Transfer ColumnYes-Specify the columns to be retrieved.
    At least one column must be set.
    The setting items differ depending on the type you select.

    • Column name (required)
    The name of the column to retrieve.
    • Type (required)
    The type of embulk to convert the retrieved value.
    • DynamoDB data type of value
    The data type set to the value of DynamoDB.
    Values of types other than the specified data type are treated as NULL.
    •format
    Type: Timestamp can be set at the time of selection.
    The format of the value to retrieve. (Default: %Y-%m-%d %H:%M: %S.%N %z)
    • Time zone
    Type: Timestamp can be set at the time of selection.
    The time zone to use if the value to be retrieved does not include a time zone. (Default: Asia/Tokyo)

    Supplement
    • About DynamoDB Types
    For example, in the image below, s_col  contains a string of A (data type: S(string)) and an array of {"A "} strings (data type:  SS (array of string)).
    If you specify S(string) as the data type, p_key:a, s_key:2  s_col are treated as NULL.
    image.png
    operationyes-Operations on DynamoDB.
    You can specify either of the following:
    • All Transfers (Scan)
    • Query
    Query expressionsNo-The conditional expression of the query.

    • Partition key (required)
    Specify the partition key set for the table.
    • Partition key value (required)
    Specify the value of the partition key to be filtered.
    • DynamoDB type of partition key (required)
    Specify the DynamoDB type of partition key.
    •Sort
    Specify the sort key set for the table.
    • Sort key operators
    Specify the sort key operator.
    • Sort key value 1
    Specify the sort key value to be filtered.
    • Sort key value 2
    Specify the sort key value to be filtered.
    It can be specified when between is selected as the sort key operator, and the query expression looks like this.
    {sort key} between {sort key value1} and {sort key value2}
    • DynamoDB Type of Sort Key
    Specify the DynamoDB type of the sort key.
    Number of segmentsNo-Operation: Can be specified when all transfers (scan) are selected.
    Splits a table or secondary index into a specified number of segments and performs a parallel scan.
    When retrieving a large number of items, you may encounter the StackOverflow error.
    It is possible to avoid StackOverflow errors by increasing the number of segments and performing parallel scans.

    For more information, see AWS official documentation1 and AWS official documentation2 .
    Number of items to be retrieved in a single operationNo-When transferring a large number of items, the OutOfMemory error can occur.
    It is possible to avoid OutOfMemory errors by adjusting the batch size.
    By default, there is no limit to the number of items, and 1MB worth of items are retrieved.

    For more information, see the official AWS documentation.
    Strongly consistent readingYesinvalidSpecify whether to enable or disable strongly consistent reads.
    If you specify a global secondary index for the secondary index, enabling strongly consistent reads will result in an error during transfer.

    Please refer to the official documentation for details.
    Was this article helpful?
    What's Next