Camel › Camel - Users

Preventing reading of a large XML file in memory

Classic

List

Threaded

13 messages Options

vishal1981

Preventing reading of a large XML file in memory

Hi,
I am using Camel to transfer a file from my app server to an FTP server before converting it to a different format. The incoming file can be potentially huge (60 to 100 gbs) in size and I want to prevent Camel from reading the whole file in memory. I would like to call a bean which takes the input file location and an output file location as a parameter and not the message body.
Any pointers on how I can achieve this without reading the whole file in memory?
Thanks in advance.

Claus Ibsen-2

Re: Preventing reading of a large XML file in memory

Hi

Use the localWorkDirectory option on the JMS endpoint to stream
directly to a file when downloading the ftp file.

Then the message body is a GenericFile type (also a
org.apache.camel.WrappedFile from Camel 2.9 onwards) where you can
access the file using a stream etc.

So if you want to
from ftp
to bean (figure out file name)
to file

Then in the bean you need to set the header Exchange.FILE_NAME to
whatever you want the file to be saved as.

For that you can use the setHeader EIP and then use a method call
expression (eg like a method call on a bean).

On Mon, Jul 16, 2012 at 8:29 AM, vishal1981
<[hidden email]> wrote:

> Hi,
> I am using Camel to transfer a file from my app server to an FTP server
> before converting it to a different format. The incoming file can be
> potentially huge (60 to 100 gbs) in size and I want to prevent Camel from
> reading the whole file in memory. I would like to call a bean which takes
> the input file location and an output file location as a parameter and not
> the message body.
> Any pointers on how I can achieve this without reading the whole file in
> memory?
> Thanks in advance.
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Preventing-reading-of-a-large-XML-file-in-memory-tp5716064.html
> Sent from the Camel - Users mailing list archive at Nabble.com.

--
Claus Ibsen
-----------------
FuseSource
Email: [hidden email]
Web: http://fusesource.com
Twitter: davsclaus, fusenews
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen

vishal1981

Re: Preventing reading of a large XML file in memory

Thanks for the reply.
So is it compulsory to use a JMS Endpoint to prevent Camel from reading the whole message in memory?
What is to my bean I dont pass the message body as a parameter but just then filename? Will that prevent Camel from loading the whole message in memory before calling my bean method?
i.e. from(ftp.....).to(bean(MyBean.class, "doSomething(${header.CamelFileName})")
(am using Camel 2.9)

Claus Ibsen-2

Re: Preventing reading of a large XML file in memory

Sorry it was the ftp endpoint i meant

On Monday, July 16, 2012, vishal1981 wrote:

> Thanks for the reply.
> So is it compulsory to use a JMS Endpoint to prevent Camel from reading the
> whole message in memory?
> What is to my bean I dont pass the message body as a parameter but just
> then
> filename? Will that prevent Camel from loading the whole message in memory
> before calling my bean method?
> i.e. from(ftp.....).to(bean(MyBean.class,
> "doSomething(${header.CamelFileName})")
> (am using Camel 2.9)
>
>
> --
> View this message in context:
> http://camel.465427.n5.nabble.com/Preventing-reading-of-a-large-XML-file-in-memory-tp5716064p5716083.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>

vishal1981

RE: Preventing reading of a large XML file in memory

Thanks. I read about the working directory and thats exactly what I was looking for. thank you.

Yet to try it though.

However, what if I want the same behaviour for File. i.e. File ->Bean (no FTP). I dont want the file component to read the whole file just pass along the file path to the bean?

Is there something similiar for File component?

From: Claus Ibsen-2 [via Camel] [mailto:[hidden email]]
Sent: Monday, July 16, 2012 2:54 PM
To: Vishal Changrani
Subject: Re: Preventing reading of a large XML file in memory

Sorry it was the ftp endpoint i meant

On Monday, July 16, 2012, vishal1981 wrote:

If you reply to this email, your message will be added to the discussion below:

http://camel.465427.n5.nabble.com/Preventing-reading-of-a-large-XML-file-in-memory-tp5716064p5716115.html

To unsubscribe from Preventing reading of a large XML file in memory, click here.
NAML

Claus Ibsen-2

Re: Preventing reading of a large XML file in memory

On Tue, Jul 17, 2012 at 6:18 AM, vishal1981
<[hidden email]> wrote:
> Thanks. I read about the working directory and thats exactly what I was looking for. thank you.
> Yet to try it though.
> However, what if I want the same behaviour for File. i.e. File ->Bean (no FTP). I dont want the file component to read the whole file just pass along the file path to the bean?
> Is there something similiar for File component?
>

Yes as the file component only stores the handle to the file. And only
read it on demand.
So in your bean, just pass in java.io.File, or a FileInputStream,
InputStream etc.

>
> ________________________________
> From: Claus Ibsen-2 [via Camel] [mailto:[hidden email]]
> Sent: Monday, July 16, 2012 2:54 PM
> To: Vishal Changrani
> Subject: Re: Preventing reading of a large XML file in memory
>
> Sorry it was the ftp endpoint i meant
>
> On Monday, July 16, 2012, vishal1981 wrote:
>
>> Thanks for the reply.
>> So is it compulsory to use a JMS Endpoint to prevent Camel from reading the
>> whole message in memory?
>> What is to my bean I dont pass the message body as a parameter but just
>> then
>> filename? Will that prevent Camel from loading the whole message in memory
>> before calling my bean method?
>> i.e. from(ftp.....).to(bean(MyBean.class,
>> "doSomething(${header.CamelFileName})")
>> (am using Camel 2.9)
>>
>>
>> --
>> View this message in context:
>> http://camel.465427.n5.nabble.com/Preventing-reading-of-a-large-XML-file-in-memory-tp5716064p5716083.html
>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>
>
>
> --
> Claus Ibsen
> -----------------
> FuseSource
> Email: [hidden email]</user/SendEmail.jtp?type=node&node=5716115&i=0>
> Web: http://fusesource.com
> Twitter: davsclaus, fusenews
> Blog: http://davsclaus.com
> Author of Camel in Action: http://www.manning.com/ibsen
>
>
> ________________________________
> If you reply to this email, your message will be added to the discussion below:
>
> NAML<http://camel.465427.n5.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Preventing-reading-of-a-large-XML-file-in-memory-tp5716064p5716139.html
> Sent from the Camel - Users mailing list archive at Nabble.com.

vishal1981

RE: Preventing reading of a large XML file in memory

Thanks again.

I am using something like the following,

from(file.......).to(ejb:cms/CamelToPluginAdaptor/local?method=process(${header.CamelFileAbsolutePath},${properties:outbox}/${header.PROVIDER_ID}.xml,${header.PROVIDER_ID})

Where cms/CamelToPluginAdaptor /local is my EJB.

Will that prevent reading the whole file in memory?

From: Claus Ibsen-2 [via Camel] [mailto:[hidden email]]
Sent: Tuesday, July 17, 2012 12:43 AM
To: Vishal Changrani
Subject: Re: Preventing reading of a large XML file in memory

If you reply to this email, your message will be added to the discussion below:

http://camel.465427.n5.nabble.com/Preventing-reading-of-a-large-XML-file-in-memory-tp5716064p5716140.html

To unsubscribe from Preventing reading of a large XML file in memory, click here.
NAML

Claus Ibsen-2

Re: Preventing reading of a large XML file in memory

On Tue, Jul 17, 2012 at 6:47 AM, vishal1981
<[hidden email]> wrote:
> Thanks again.
> I am using something like the following,
>
> from(file.......).to(ejb:cms/CamelToPluginAdaptor/local?method=process(${header.CamelFileAbsolutePath},${properties:outbox}/${header.PROVIDER_ID}.xml,${header.PROVIDER_ID})
>
> Where cms/CamelToPluginAdaptor /local is my EJB.
>
> Will that prevent reading the whole file in memory?
>

Yes the parameters for the method call, does not require to content of
the file, so Camel will not load it.

>
>
> ________________________________
> From: Claus Ibsen-2 [via Camel] [mailto:[hidden email]]
> Sent: Tuesday, July 17, 2012 12:43 AM
> To: Vishal Changrani
> Subject: Re: Preventing reading of a large XML file in memory
>
> On Tue, Jul 17, 2012 at 6:18 AM, vishal1981
> <[hidden email]</user/SendEmail.jtp?type=node&node=5716140&i=0>> wrote:
>> Thanks. I read about the working directory and thats exactly what I was looking for. thank you.
>> Yet to try it though.
>> However, what if I want the same behaviour for File. i.e. File ->Bean (no FTP). I dont want the file component to read the whole file just pass along the file path to the bean?
>> Is there something similiar for File component?
>>
>
> Yes as the file component only stores the handle to the file. And only
> read it on demand.
> So in your bean, just pass in java.io.File, or a FileInputStream,
> InputStream etc.
>
>>
>> ________________________________
>> From: Claus Ibsen-2 [via Camel] [mailto:[hidden email]</user/SendEmail.jtp?type=node&node=5716140&i=1>]
>> Sent: Monday, July 16, 2012 2:54 PM
>> To: Vishal Changrani
>> Subject: Re: Preventing reading of a large XML file in memory
>>
>> Sorry it was the ftp endpoint i meant
>>
>> On Monday, July 16, 2012, vishal1981 wrote:
>>
>>> Thanks for the reply.
>>> So is it compulsory to use a JMS Endpoint to prevent Camel from reading the
>>> whole message in memory?
>>> What is to my bean I dont pass the message body as a parameter but just
>>> then
>>> filename? Will that prevent Camel from loading the whole message in memory
>>> before calling my bean method?
>>> i.e. from(ftp.....).to(bean(MyBean.class,
>>> "doSomething(${header.CamelFileName})")
>>> (am using Camel 2.9)
>>>
>>>
>>> --
>>> View this message in context:
>>> http://camel.465427.n5.nabble.com/Preventing-reading-of-a-large-XML-file-in-memory-tp5716064p5716083.html
>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>
>>
>>
>> --
>> Claus Ibsen
>> -----------------
>> FuseSource
>> Email: [hidden email]</user/SendEmail.jtp?type=node&node=5716115&i=0>
>> Web: http://fusesource.com
>> Twitter: davsclaus, fusenews
>> Blog: http://davsclaus.com
>> Author of Camel in Action: http://www.manning.com/ibsen
>>
>>
>> ________________________________
>> If you reply to this email, your message will be added to the discussion below:
>>
>> NAML<http://camel.465427.n5.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>
>>
>> --
>> View this message in context: http://camel.465427.n5.nabble.com/Preventing-reading-of-a-large-XML-file-in-memory-tp5716064p5716139.html
>> Sent from the Camel - Users mailing list archive at Nabble.com.
>
>
>
> --
> Claus Ibsen
> -----------------
> FuseSource
> Email: [hidden email]</user/SendEmail.jtp?type=node&node=5716140&i=2>
> Web: http://fusesource.com
> Twitter: davsclaus, fusenews
> Blog: http://davsclaus.com
> Author of Camel in Action: http://www.manning.com/ibsen
>
>
> ________________________________
> If you reply to this email, your message will be added to the discussion below:
>
> NAML<http://camel.465427.n5.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Preventing-reading-of-a-large-XML-file-in-memory-tp5716064p5716141.html
> Sent from the Camel - Users mailing list archive at Nabble.com.

vishal1981

RE: Preventing reading of a large XML file in memory

sweet..thanks a ton!

From: Claus Ibsen-2 [via Camel] [mailto:[hidden email]]
Sent: Tuesday, July 17, 2012 1:10 AM
To: Vishal Changrani
Subject: Re: Preventing reading of a large XML file in memory

If you reply to this email, your message will be added to the discussion below:

http://camel.465427.n5.nabble.com/Preventing-reading-of-a-large-XML-file-in-memory-tp5716064p5716143.html

To unsubscribe from Preventing reading of a large XML file in memory, click here.
NAML

vishal1981

RE: Preventing reading of a large XML file in memory

Hi,
Sorry to bug you again...but..
Now I have my route configuration as File -> MyBean ->Ftp
Now MyBean converts the input file to a different format. I want to avoid doing a exchange.getOut().setBody(<read new file>) since the new file is huge. But somehow I want to convey to the FTP producer that FTP over the new file and not the one via message.getIn().getBody().

I tried by just setting the message header .setHeader(Exchange.FILE_NAME, "new file path") but I think the FTP producer just does the getIn().getBody() irrespective of the file name.

Is there a way to do this?

Thanks for your help.

vishal1981

RE: Preventing reading of a large XML file in memory

In reply to this post by Claus Ibsen-2

From: Claus Ibsen-2 [via Camel] [mailto:[hidden email]]
Sent: Monday, July 16, 2012 2:54 PM
To: Vishal Changrani
Subject: Re: Preventing reading of a large XML file in memory

Sorry it was the ftp endpoint i meant

On Monday, July 16, 2012, vishal1981 wrote:

If you reply to this email, your message will be added to the discussion below:

http://camel.465427.n5.nabble.com/Preventing-reading-of-a-large-XML-file-in-memory-tp5716064p5716115.html

To unsubscribe from Preventing reading of a large XML file in memory, click here.
NAML

Willem.Jiang

Re: Preventing reading of a large XML file in memory

In reply to this post by vishal1981

Hi,

How did you process the old file?
Maybe you need some file cache to avoid loading the whole XML file into
the memory.

On 7/18/12 10:52 PM, vishal1981 wrote:

> Hi,
> Sorry to bug you again...but..
> Now I have my route configuration as File -> MyBean ->Ftp
> Now MyBean converts the input file to a different format. I want to avoid
> doing a exchange.getOut().setBody(<read new file>) since the new file is
> huge. But somehow I want to convey to the FTP producer that FTP over the new
> file and not the one via message.getIn().getBody().
>
> I tried by just setting the message header .setHeader(Exchange.FILE_NAME,
> "new file path") but I think the FTP producer just does the
> getIn().getBody() irrespective of the file name.
>
> Is there a way to do this?
>
> Thanks for your help.
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Preventing-reading-of-a-large-XML-file-in-memory-tp5716064p5716204.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
> .
>

--
Willem
----------------------------------
FuseSource
Web: http://www.fusesource.com
Blog: http://willemjiang.blogspot.com (English)
http://jnn.javaeye.com (Chinese)
Twitter: willemjiang
Weibo: willemjiang

vishal1981

RE: Preventing reading of a large XML file in memory

Hi,

Actually Camel already has support for what I was looking for. All I did was set the body of the message in my bean to a File object...when I pass this message down to a file endpoint it knows its a file handle and automatically reads from it. I am not sure which endpoints understand file object as message body and which dont but I suppose File and FTP atleast do. (by understand mean they realize that they are suppose to read from the File object and that the message is not just a string object).

thanks

From: Willem.Jiang [via Camel] [mailto:[hidden email]]
Sent: Wednesday, July 18, 2012 10:10 PM
To: Vishal Changrani
Subject: Re: Preventing reading of a large XML file in memory

Hi,

How did you process the old file?
Maybe you need some file cache to avoid loading the whole XML file into
the memory.

On 7/18/12 10:52 PM, vishal1981 wrote:

If you reply to this email, your message will be added to the discussion below:

http://camel.465427.n5.nabble.com/Preventing-reading-of-a-large-XML-file-in-memory-tp5716064p5716226.html

To unsubscribe from Preventing reading of a large XML file in memory, click here.
NAML