Camel Mybatis 2.13.1 BATCH of 10,000 records

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view

Camel Mybatis 2.13.1 BATCH of 10,000 records

I ran into an issue where I was  trying to insert a BATCH of roughly 3500 records into an Oracle 11g table of 13 columns from a CSV file using camel-bindy and camel-mybatis.

The BATCH was generating huge trace files on the database sever due to ORA-4031 errors. The SQL statement running in each session is over 80,000 lines long, taking as input over 42,000 bind variables.  Additionally, the process blocked over cursor contention.

I'm trying to satisfy the requirement: If invalid data, then rollback.

I did some research and tried the EIP Splitter on the collection in the Message Body. The sub routes to split and save the record to DB are placed in a DB Transaction policy with PROPAGATION_REQUIRED, it rolled back as expected. This worked absolutely fine. It took around 9 seconds to insert 5050 records.  

 Is Splitter a good choice for the above?   Does anyone have other ideas?