Split streaming and order of sub-messages

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view

Split streaming and order of sub-messages


I'm currently trying to get a Route going to read large csv files and writing their content to xml. The xml has to look like the following:
static content here
csv content here by line

Following an example of how my route operates:

// read files
.multicast("direct:header", "direct:data", "direct:footer");

// write XML header

// write data

// write closing XML tags

Now my question: http://camel.apache.org/splitter says that streaming will not guarantee the order of sub-messages but I need the xml to contain exactly the same order as the csv. As the intput files can be rather large, I need some form of streaming to read them one line/chunk at a time. Does streaming still cause lines to be out of order even if I'm not using parallel processing? In the end I will use 40+ Routes for different directories containing files, could that be a problem?

I tried writing my own splitter using an Iterator over a BufferedReader/LineIterator but it will still load the whole file.

What would be the best course of action to take?

Thanks for any help
Reply | Threaded
Open this post in threaded view

Re: Split streaming and order of sub-messages

I looked through the Splitter class and the streaming flag seems to decide between using an Iterable directly or saving all entries of it into a list. So as long as parallel processing is not active, it lookes like the sub messages should still be in order, regardless of stream or not. I don't know if streaming does some other hidden stuff though.