Data Destination - Google Cloud Storage
- 07 Dec 2022
- Print
- DarkLight
- PDF
Data Destination - Google Cloud Storage
- Updated on 07 Dec 2022
- Print
- DarkLight
- PDF
Article Summary
Share feedback
Thanks for sharing your feedback!
This is a machine-translated version of the original Japanese article.
Please understand that some of the information contained on this page may be inaccurate.
summary
This is a help page for setting up data transfer to Google Cloud Storage on Google Cloud Platform.
Supported Protocols
- Data Transfer (Embulk)
Using embulk-output-gcs
constraint
- Nothing in particular
Setting items
STEP1 Basic settings
Item | namerequireddefault | valuecontent | |
---|---|---|---|
Google Cloud Strage connection information | Yes | - | From the connection information registered in advance, select the one that has the necessary permissions for this transfer setting. Please refer to the separate page for how to set the connection information. |
bucket | Yes | - | Specify the name of the bucket to which you want to transfer data. |
Path prefix | Yes | - | Specify the path prefix to which the data is transferred. trocco outputs multiple files to the destination bucket, starting with the path prefix. |
Output file extension | Yes | ..csv | Specify the extension of the output file. You can specify any extension, but it is common to specify according to .csv the file format. |
Output file format | Yes | .CSV | You can choose the format of the output file. Please see here for details. |
File compression | No | No compression | Here you can choose how to compress the file. Please see here for details. |
Transfer Mode | Yes | Parallel Transfer | Select the transfer mode from parallel transfer or transfer with a limited number of output files. If you select a transfer that suppresses the number of output files, file generation is performed internally with reduced parallelism. If the source is one file, one file will be generated on the destination. * Due to GCS restrictions, you can output up to 1GB per file. If the data exceeds 1GB, the job will fail as an error. |
Required Permissions
The permissions required to use this service are as follows.
- storage.buckets.create
- storage.buckets.get
- storage.buckets.list
- storage.objects.create
- storage.objects.delete
- storage.objects.get
- storage.objects.list
Was this article helpful?