Table of Contents

Search

  1. Preface
  2. Introduction to Data Engineering Streaming
  3. Data Engineering Streaming Administration
  4. Sources in a Streaming Mapping
  5. Targets in a Streaming Mapping
  6. Streaming Mappings
  7. Window Transformation
  8. Appendix A: Connections
  9. Appendix B: Monitoring REST API Reference
  10. Appendix C: Sample Files

Google Cloud Storage Data Objects

Google Cloud Storage Data Objects

A Google Cloud Storage data object is a physical data object that represents data in a Google Cloud Storage resource. After you configure a Google Cloud Storage connection, create a Google Cloud Storage data object to write to Google Cloud Storage targets.
You can configure the data object write operation properties that determine how data can be loaded to Google Cloud Storage targets. After you create a Google Cloud Storage data object, create a write operation. You can use the Google Cloud Storage data object write operation as a target in streaming mappings. You can create the data object write operation for the Google Cloud Storage data object automatically. Then, edit the advanced properties of the data object write operation and run a mapping.
When you configure the data operation properties, specify the format in which the data object writes data. You can specify Avro, JSON, or Parquet as format. When you specify Avro format, provide a sample Avro schema in an
.avsc
file. When you specify JSON, you must provide a sample file.
You can pass any payload format directly from source to target in streaming mappings. To successfully run a streaming mapping when you select multiple objects from different Google Cloud Storage buckets, ensure that all the Google Cloud Storage buckets belong to the same region and use the same credentials to access the Google Cloud Storage buckets. However, the cross-account service is not enabled for a Google Cloud Storage target.
When you use a Google Cloud Storage target in a streaming mapping, you can configure size-based rollover and time-based rollover while creating the mapping.
You cannot use the Mapping Flow option in a streaming mapping with Google Cloud Storage target.