Data Source - HTTP(S)
  • 24 Jan 2023
  • Dark
    Light
  • PDF

Data Source - HTTP(S)

  • Dark
    Light
  • PDF

Article Summary

Note

This is a machine-translated version of the original Japanese article.
Please understand that some of the information contained on this page may be inaccurate.

Overview

This is a help page for setting up data transfer from an HTTP(S) connection web page.

Supported Protocols

Limitations

  • Nothing in particular

Settings

STEP 1: General Settings

Field Name
Required
Default Value
Description
URLYes-Specify the URL of the transfer data acquisition source.
HTTP MethodsYesGETSpecifies the HTTP method to the data retriever.
User AgentNo-You can specify any user agent name in the request header.
Character codeNoutf8You can specify any character encoding in the request header.
Input file formatYesCSV/TSVSpecify the format of the input file.
For details, refer to About setting the input file format.
Paging functionYesinvalidIf enabled, paging requests can be included.
For details, please refer to the paging function below.
from/offset parameter name(Yes if paging function is enabled)-You can specify the from/offset parameter name for the paging request.
For details, please refer to the paging function below.
to parameter nameNo-You can specify the to parameter name of the paging request.
For details, please refer to the paging function below.
Number of requests(Yes if paging function is enabled)-You can specify the number of paging requests requested.
For details, please refer to the paging function below.
From/offset parameter initial valueNo0You can specify the initial value of the from/offset parameter of the paging request.
For details, please refer to the paging function below.
Number of from/offset parameters to advance in one requestNo1You can specify the number of from/offset parameters to advance in one request of a paging request.
For details, please refer to the paging function below.
parameterNo-You can set any key/value in the query parameters.
HTTP HeadersNo-You can set any key/value in the HTTP header.

STEP2: Advanced settings

Field Name
Default Value
Connection timeout (ms)2000
Read Timeout (ms)10000
Maximum number of retries5
Retry interval (ms)10000
Request interval (ms)0

About Pagination

If you enable the paging function and set items related to paging settings, you can include a paging request when retrieving transferred data.
For example, if you set the items as shown in the table, the following request parameters will be added. (Requests are issued for the number of requests.) )

Field Name
Default Value
from/offset parameter namefrom
to parameter nameto
Number of requests4
From/offset parameter initial value1
Number of from/offset parameters to advance in one request10
  1. ?from=1&to=10
  2. ?from=11&to=20
  3. ?from=21&to=30
  4. ?from=31&to=40

See the official documentation for more details.


Was this article helpful?