Configure the Kafka cluster so that Big Data Streaming can produce and consume messages at the needed message ingestion rate.

To increase the rate of message consumption in Big Data Streaming, increase the number of Kafka brokers in the Kafka cluster and in the Kafka connection.

Increase the number of partitions on the Kafka topic. Ideally, the number of partitions can be equal to the number of CPU cores allocated to the executors. For example, if you set

spark.executor.instances

to 6 and

spark.executor.cores

to 3, there are 18 cores allocated. Then set the number of Kafka partitions to 18, so that there are 18 parallel tasks to read from the Kafka Source.

Ensure that the Kafka producer is publishing messages to every partition in a load balanced manner.

Reduce the number of network hops between Big Data Streaming and the Kafka cluster. Ideally the Kafka broker must be on the same machine as the data node or the Kafka cluster can run on its own machines with a zero latency network.

Configure the

batch.size

and

linger.ms

properties to increase throughput. For each partition, the producer maintains buffers of unsent records. The

batch.size

property specifies the size of the buffer. To accumulate as many messages as possible in the buffer, configure a high value for the

batch.size

property.

Kafka scalability depends on disk and network performance. The test setup included 12 disks per node on a 10 GBPS network with an open file limit of 65000.