Most efficenient way to zip or tar multiple zip files at source

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Most efficenient way to zip or tar multiple zip files at source

hsachin
This post has NOT been accepted by the mailing list yet.
We have a repository (source) with very tiny zip files. I need to ingest into hadoop using camel.  Since there is a very large amount of zip files and with name node limitation. I am unable to ingest these tiny zip file.

What would be the best mechanism  to ingest and how do I do it?

We started using Camel ZipAggrigator and that is too slow where by it was zipping the zip files and then moving to HDFS.

Is it possible to tar the zip using camel tools or bzip and then move it?

Would be great if I can get ideas
Loading...