Contents x
    Incremental Loading
    • 07 Dec 2022
    • Print
    • Dark
      Light
    • PDF
    Contents

    Incremental Loading

    • Updated on 07 Dec 2022
    • Print
    • Dark
      Light
    • PDF
    Article Summary

    Note

    This is a machine-translated version of the original Japanese article.
    Please understand that some of the information contained on this page may be inaccurate.

    summary

    This is the help page of "Differential Transfer Function" that can be specified by the transfer source in the transfer settings.

    What is a differential transfer?

    This mode transfers only incremental data from the previous transfer.
    If you enable incremental transfers, you will retain "how far you transferred" when transferring.
    Therefore, it is possible to identify new files added since the last transfer, and only the incremental files are transferred.

    Behavior during the first transfer

    Even if differential transfer is specified, all records are transferred at the time of the first transfer.
    Even at the time of the first transfer, it is possible to transfer files after any file by specifying the file path etc. as the last transferred record path.
    For details, please refer to the setting value item.

    Setting value

    If the transfer source is a database system

    If the transfer source is a database such as MySQL or PostgreSQL, a differential transfer is performed using columns.

    Item namedescription
    Table nameSpecifies the table from which the data is retrieved.
    Column nameSpecifies the column from which the delta transfer is based.
    If there is a unique and automatic ID column for the record, specify the column name.
    You can also specify multiple column names separated by commas.
    Last Transferred RecordNormally, you will not edit this form (TROCCO will update it automatically).
    Edit this form only if there is an abnormality in the execution of the job, or if you want to perform the transfer from any point during the first transfer.
    This form contains information about how far you transferred during the last transfer.

    If the source is File Storage

    If the source is file storage such as S3 or SFTP, the path prefix is used for incremental transfers.

    Item namedescription
    Last Transferred RecordNormally, you will not edit this form (TROCCO will update it automatically).
    Edit this form only if there is an abnormality in the execution of the job, or if you want to perform the transfer from any point during the first transfer.
    This form contains information about how far you transferred during the last transfer.

    example

    For example, if you transfer with the following files on S3, "3.csv" will be saved in the last transferred path.

    • 1.csv
    • 2.csv
    • 3.csv

    If you add 4 .csv in this state, only 4 .csv will be transferred, and "4.csv" will be stored in the last transferred path.

    About re-execution

    You can resume the transfer from any location by editing the last transferred record path.
    However, if you transfer a file that has already been transferred again, there is a possibility that the data will be duplicated at the destination, so please delete the data and try again.

    Was this article helpful?
    What's Next