Table of Contents


  1. Preface
  2. Introduction to Data Engineering Streaming
  3. Data Engineering Streaming Administration
  4. Sources in a Streaming Mapping
  5. Targets in a Streaming Mapping
  6. Streaming Mappings
  7. Transformation in Streaming Mappings
  8. Window Transformation
  9. Appendix A: Connections
  10. Appendix B: Monitoring REST API Reference
  11. Appendix C: Sample Files

Google PubSub Data Objects

Google PubSub Data Objects

A Google PubSub data object is a physical data object that represents data in a Google PubSub topic. After you create a Google PubSub connection, create a Google PubSub data object to read from Google PubSub. You can use Google PubSub Connector to read Google PubSub topics through subscriptions.
Google PubSub is a messaging service that decouples services that produce events from services that process events. You can use Google PubSub as a messaging-oriented middleware or for event ingestion and delivery for streaming analytics pipelines.
Google PubSub offers durable message storage and real-time message delivery with high availability and consistent performance at scale. You can run Google PubSub servers in all the available Google Cloud regions around the world.
Before you use Google PubSub Connector, you must complete the following prerequisite tasks:
  1. Ensure that your organization has the Google PubSub Connector license.
  2. Ensure that you have a Google service account to access Google PubSub.
  3. Ensure that you have the PROJECTID, CLIENTEMAIL, PRIVATE KEY, PRIVATEKEYID, and CLIENTID values for the Google service account. You will need to enter these details when you create a Google PubSub connection.
  4. Ensure that you have enabled the
    Google PubSub API
    for your Google Cloud project. Google PubSub Connector uses the Google APIs to integrate with Google PubSub.
For more information about the required permissions and the access control options available for a Google PubSub user, see the Google Cloud Pub/Sub documentation.
A Google PubSub topic can have multiple subscriptions. However, one subscription belongs to one single topic. To receive messages published to a topic, you must have a subscription to that topic. Messages are no longer visible in the subscription after they are acknowledged. The Google PubSub connector uses the synchronous pull mechanism to read the subscribed messages.
  • A streaming mapping with Google PubSub source cannot have multiple pipelines.
  • You cannot enable a streaming mapping with Google PubSub source to override the partition reads of the runtime attributes.
  • You cannot use a Java transformation in a streaming mapping with Google PubSub source.
  • You cannot reduce partition or change Google Pubsub subscription properties during warm start processing.


We’d like to hear from you!