I have narrowed down a performance bottleneck in the Processor I implemented in a SEDA route using the AWS Kinesis component.
I tested against an "empty" Processor (i.e. that doesn't do anything) and got the following metrics:
For the "empty" Processor:11:10| INFO | MarkerIgnoringBase.java 107 | type=METER, name=kinesis-records-consumed, count=140, mean_rate=2.3331402604 16133, m1=1.6458279659887582, m5=0.43339880218285814, m15=0.15173246327279963, rate_unit=events/second
For the "actual" Processor: 11:15| INFO | MarkerIgnoringBase.java 107 | type=METER, name=kinesis-records-consumed, count=20, mean_rate=0.3333031357470416, m1=0.17909216046832732, m5=0.058841979331650814, m15=0.02131567762219331, rate_unit=events/second
So, the bottleneck definitely seems to be in the "actual" Processor.
I have increased the number of shards on the streamas well as tweaked a host of other parameters in order to improve performance with no real effect.
I now want to try using a custom ScheduledExecutorService to use multiple threads for the consumer.
Is there an example of how to do this?
Also, concerning the number of records in the metrics above, how does the Camel component determine how many records to read off the stream on the first read? I'd expect them to be similar and maybe different on subsequent reads?