Most efficenient way to zip or tar multiple zip files at source
This post has NOT been accepted by the mailing list yet.
We have a repository (source) with very tiny zip files. I need to ingest into hadoop using camel. Since there is a very large amount of zip files and with name node limitation. I am unable to ingest these tiny zip file.
What would be the best mechanism to ingest and how do I do it?
We started using Camel ZipAggrigator and that is too slow where by it was zipping the zip files and then moving to HDFS.
Is it possible to tar the zip using camel tools or bzip and then move it?