Data Source - HTTP(S)
- 24 Jan 2023
- Print
- DarkLight
- PDF
Data Source - HTTP(S)
- Updated on 24 Jan 2023
- Print
- DarkLight
- PDF
Article Summary
Share feedback
Thanks for sharing your feedback!
This is a machine-translated version of the original Japanese article.
Please understand that some of the information contained on this page may be inaccurate.
Overview
This is a help page for setting up data transfer from an HTTP(S) connection web page.
Supported Protocols
- Data Transfer (Embulk)
Using embulk-input-http
Limitations
- Nothing in particular
Settings
STEP 1: General Settings
Field Name | Required | Default Value | Description |
---|---|---|---|
URL | Yes | - | Specify the URL of the transfer data acquisition source. |
HTTP Methods | Yes | GET | Specifies the HTTP method to the data retriever. |
User Agent | No | - | You can specify any user agent name in the request header. |
Character code | No | utf8 | You can specify any character encoding in the request header. |
Input file format | Yes | CSV/TSV | Specify the format of the input file. For details, refer to About setting the input file format. |
Paging function | Yes | invalid | If enabled, paging requests can be included. For details, please refer to the paging function below. |
from/offset parameter name | (Yes if paging function is enabled) | - | You can specify the from/offset parameter name for the paging request. For details, please refer to the paging function below. |
to parameter name | No | - | You can specify the to parameter name of the paging request. For details, please refer to the paging function below. |
Number of requests | (Yes if paging function is enabled) | - | You can specify the number of paging requests requested. For details, please refer to the paging function below. |
From/offset parameter initial value | No | 0 | You can specify the initial value of the from/offset parameter of the paging request. For details, please refer to the paging function below. |
Number of from/offset parameters to advance in one request | No | 1 | You can specify the number of from/offset parameters to advance in one request of a paging request. For details, please refer to the paging function below. |
parameter | No | - | You can set any key/value in the query parameters. |
HTTP Headers | No | - | You can set any key/value in the HTTP header. |
STEP2: Advanced settings
Field Name | Default Value |
---|---|
Connection timeout (ms) | 2000 |
Read Timeout (ms) | 10000 |
Maximum number of retries | 5 |
Retry interval (ms) | 10000 |
Request interval (ms) | 0 |
About Pagination
If you enable the paging function and set items related to paging settings, you can include a paging request when retrieving transferred data.
For example, if you set the items as shown in the table, the following request parameters will be added. (Requests are issued for the number of requests.) )
Field Name | Default Value |
---|---|
from/offset parameter name | from |
to parameter name | to |
Number of requests | 4 |
From/offset parameter initial value | 1 |
Number of from/offset parameters to advance in one request | 10 |
- ?from=1&to=10
- ?from=11&to=20
- ?from=21&to=30
- ?from=31&to=40
See the official documentation for more details.
Was this article helpful?