AWS Kinesis Consumer performance improvement

Previous Topic Next Topic
classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view

AWS Kinesis Consumer performance improvement

I have narrowed down a performance bottleneck in the Processor I implemented in a SEDA route using the AWS Kinesis component.

I tested against an "empty" Processor (i.e. that doesn't do anything) and got the following metrics:

For the "empty" Processor:11:10| INFO | 107 | type=METER, name=kinesis-records-consumed, count=140, mean_rate=2.3331402604                                                                                                                                                       16133, m1=1.6458279659887582, m5=0.43339880218285814, m15=0.15173246327279963, rate_unit=events/second

For the "actual" Processor: 11:15| INFO | 107 | type=METER, name=kinesis-records-consumed, count=20, mean_rate=0.3333031357470416, m1=0.17909216046832732, m5=0.058841979331650814, m15=0.02131567762219331, rate_unit=events/second

So, the bottleneck definitely seems to be in the "actual" Processor.

I have increased the number of shards on the streamas well as tweaked a host of other parameters in order to improve performance with no real effect.

I now want to try using a custom ScheduledExecutorService to use multiple threads for the consumer.

Is there an example of how to do this?

Also, concerning the number of records in the metrics above, how does the Camel component determine how many records to read off the stream on the first read? I'd expect them to be similar and maybe different on subsequent reads?