Robustly streaming messages to archive

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view

Robustly streaming messages to archive

We use camel in a standalone app which responds to a user request by sending some messages via JMS. To aid with troubleshooting/support, we also save these messages to disk using the 'file:' endpoint.

Unfortunately, since there are many tens of thousands of smallish (5-10KiB) messages per user request, the folders are quite unwieldy to deal with. e.g. transferring the saved messages from a network share is particularly slow compared to transferring a single larger file.

We tried aggregating all such messages for a user request and creating a zip archive on completion; however this has the drawback that nothing is logged if the process crashes or is forcibly terminated partway through a request.

Is there a camel-y solution to this problem? At the moment we are thinking of writing the messages into a single file via a tar stream, and possibly subsequently a gzip stream. My understanding is that, unlike .zip, both .tar and .gz support streaming (i.e. can tolerate partially complete archives), although I'm not an expert. I was wondering how the community has solved this problem?