Re: memory leak in Camel

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: memory leak in Camel

Claus Ibsen-2
Hi

Its a bit hard to help when there is no more details about Camel and
what Camel routes you have etc.
Maybe you can build a sample project that can be used to demonstrate /
reproduce the issue you see.

Also there is some tips on support here in the bulleted list
http://camel.apache.org/support.html

On Wed, Jun 27, 2018 at 4:29 PM,  <[hidden email]> wrote:

> Hi there,
>
> I have the following problem: I’m running a Camel application that is
> triggered by a RESTful POST call. With each trigger, it contacts certain
> http addresses, downloads files to the file system, parses them, and sends
> some parsed informations to another service with a REST call.
>
> All this is executed on VERY large data amount with many files and so on. My
> problem is: I run into a severe memory leak.
>
> I’ve installed JProfiler and was trying to catch that leak. What is amazing
> to me is that there are after only a couple of minutes about 100 000 objects
> of the type “TreeMap$Entry”. I drilled down to them in JProbe and found that
> there are lots of message headers in main memory, for example things like
>
> Connection=keep alive
> CamelFileNameProduced=C:/path/to/the/produced/file
> cache-control=no-cache
> log=F
> getMethod=isStatisticsEnabled
>
>
> All these entries waste the main memory, and even garbage collection does
> not get rid of them. My expectation is that after the steps I’ve described
> in my first sentence have finished, the main memory should almost look as
> empty as it was before the first entry of the RESTful call.
>
> Please find attached an xml file that is exported from JProfiler:
> And this HTML file was also exported by JProfiler and may have a clearer
> view on what was going on:
>
>
> I’m having a hard time these days because I’m fighting with my colleagues to
> use Apache Camel. If they’ve found that memory leak, I’m going to lose that
> battle. So please help.
>
> Many thanks in advance!
>
> Kind regards,
> Christian Jacob
>
> Innogy SE
> Retail IT
> Integration and Digital Solutions (AFS-IGI)
> Rellinghauser Str. 37
> 45128 Essen
> T intern: 70-20581
> T extern: +49 (0)201 12 20582
> T mobil: +49 (0)1622843981
> Fax: +49 (0)201 12 24796
> mailto: [hidden email]
>
> ----------------------------------------------------------------
> innogy SE
> Vorsitzender des Aufsichtsrates: Dr. Erhard Schipporeit
> Vorstand: Uwe Tigges (Vorsitzender), Dr. Hans Buenting,
> Dr. Bernhard Guenther, Arno Hahn, Martin Herrmann, Hildegard Mueller
> Sitz der Gesellschaft: Essen, Eingetragen beim Amtsgericht Essen,
> Handelsregister-Nr. HRB 27091, USt-IdNr. DE304171711



--
Claus Ibsen
-----------------
http://davsclaus.com @davsclaus
Camel in Action 2: https://www.manning.com/ibsen2
Reply | Threaded
Open this post in threaded view
|

AW: memory leak in Camel

christian.jacob@innogy.com
Hi Claus,

thanks for your help. Unfortunately, the data with which I work are secret, and so is the connection to our data provider. For that reason, it might be tough to construct a reproducible example.

In the meantime, I threw my project into the waste and re-developed it from the scratch. And the result is, that there is no memory leak anymore. At least, the application is running since Friday and did not have any problems. The original software had run into the OutOfMemoryException after only a couple of minutes.

This proves that I must have done something wrong in my usage of Apache Camel.

Kind regards,
Christian

-----Ursprüngliche Nachricht-----
Von: Claus Ibsen [mailto:[hidden email]]
Gesendet: Dienstag, 3. Juli 2018 12:38
An: [hidden email]
Betreff: Re: memory leak in Camel

Hi

Its a bit hard to help when there is no more details about Camel and
what Camel routes you have etc.
Maybe you can build a sample project that can be used to demonstrate /
reproduce the issue you see.

Also there is some tips on support here in the bulleted list
http://camel.apache.org/support.html

On Wed, Jun 27, 2018 at 4:29 PM,  <[hidden email]> wrote:

> Hi there,
>
> I have the following problem: I’m running a Camel application that is
> triggered by a RESTful POST call. With each trigger, it contacts certain
> http addresses, downloads files to the file system, parses them, and sends
> some parsed informations to another service with a REST call.
>
> All this is executed on VERY large data amount with many files and so on. My
> problem is: I run into a severe memory leak.
>
> I’ve installed JProfiler and was trying to catch that leak. What is amazing
> to me is that there are after only a couple of minutes about 100 000 objects
> of the type “TreeMap$Entry”. I drilled down to them in JProbe and found that
> there are lots of message headers in main memory, for example things like
>
> Connection=keep alive
> CamelFileNameProduced=C:/path/to/the/produced/file
> cache-control=no-cache
> log=F
> getMethod=isStatisticsEnabled
>
>
> All these entries waste the main memory, and even garbage collection does
> not get rid of them. My expectation is that after the steps I’ve described
> in my first sentence have finished, the main memory should almost look as
> empty as it was before the first entry of the RESTful call.
>
> Please find attached an xml file that is exported from JProfiler:
> And this HTML file was also exported by JProfiler and may have a clearer
> view on what was going on:
>
>
> I’m having a hard time these days because I’m fighting with my colleagues to
> use Apache Camel. If they’ve found that memory leak, I’m going to lose that
> battle. So please help.
>
> Many thanks in advance!
>
> Kind regards,
> Christian Jacob
>
> Innogy SE
> Retail IT
> Integration and Digital Solutions (AFS-IGI)
> Rellinghauser Str. 37
> 45128 Essen
> T intern: 70-20581
> T extern: +49 (0)201 12 20582
> T mobil: +49 (0)1622843981
> Fax: +49 (0)201 12 24796
> mailto: [hidden email]
>
> ----------------------------------------------------------------
> innogy SE
> Vorsitzender des Aufsichtsrates: Dr. Erhard Schipporeit
> Vorstand: Uwe Tigges (Vorsitzender), Dr. Hans Buenting,
> Dr. Bernhard Guenther, Arno Hahn, Martin Herrmann, Hildegard Mueller
> Sitz der Gesellschaft: Essen, Eingetragen beim Amtsgericht Essen,
> Handelsregister-Nr. HRB 27091, USt-IdNr. DE304171711



--
Claus Ibsen
-----------------
http://davsclaus.com @davsclaus
Camel in Action 2: https://www.manning.com/ibsen2
----------------------------------------------------------------
innogy SE
Vorsitzender des Aufsichtsrates: Dr. Erhard Schipporeit
Vorstand: Uwe Tigges (Vorsitzender), Dr. Hans Buenting,
Dr. Bernhard Guenther, Arno Hahn, Martin Herrmann, Hildegard Mueller
Sitz der Gesellschaft: Essen, Eingetragen beim Amtsgericht Essen,
Handelsregister-Nr. HRB 27091, USt-IdNr. DE304171711
Reply | Threaded
Open this post in threaded view
|

Re: memory leak in Camel

Claus Ibsen-2
Hi

Ah okay, good to hear you were able to build a better solution that runs stable.

There are a lot of Camel applications over the world running stable in
production, so if we hear about memory leaks, we are keen on fixing /
looking into them.
But as you say they can sometimes be hard to track, whether its camel
code, user code, or 3rd party libraries that are the culprit, or a
combination of them.


On Tue, Jul 3, 2018 at 1:15 PM,  <[hidden email]> wrote:

> Hi Claus,
>
> thanks for your help. Unfortunately, the data with which I work are secret, and so is the connection to our data provider. For that reason, it might be tough to construct a reproducible example.
>
> In the meantime, I threw my project into the waste and re-developed it from the scratch. And the result is, that there is no memory leak anymore. At least, the application is running since Friday and did not have any problems. The original software had run into the OutOfMemoryException after only a couple of minutes.
>
> This proves that I must have done something wrong in my usage of Apache Camel.
>
> Kind regards,
> Christian
>
> -----Ursprüngliche Nachricht-----
> Von: Claus Ibsen [mailto:[hidden email]]
> Gesendet: Dienstag, 3. Juli 2018 12:38
> An: [hidden email]
> Betreff: Re: memory leak in Camel
>
> Hi
>
> Its a bit hard to help when there is no more details about Camel and
> what Camel routes you have etc.
> Maybe you can build a sample project that can be used to demonstrate /
> reproduce the issue you see.
>
> Also there is some tips on support here in the bulleted list
> http://camel.apache.org/support.html
>
> On Wed, Jun 27, 2018 at 4:29 PM,  <[hidden email]> wrote:
>> Hi there,
>>
>> I have the following problem: I’m running a Camel application that is
>> triggered by a RESTful POST call. With each trigger, it contacts certain
>> http addresses, downloads files to the file system, parses them, and sends
>> some parsed informations to another service with a REST call.
>>
>> All this is executed on VERY large data amount with many files and so on. My
>> problem is: I run into a severe memory leak.
>>
>> I’ve installed JProfiler and was trying to catch that leak. What is amazing
>> to me is that there are after only a couple of minutes about 100 000 objects
>> of the type “TreeMap$Entry”. I drilled down to them in JProbe and found that
>> there are lots of message headers in main memory, for example things like
>>
>> Connection=keep alive
>> CamelFileNameProduced=C:/path/to/the/produced/file
>> cache-control=no-cache
>> log=F
>> getMethod=isStatisticsEnabled
>>
>>
>> All these entries waste the main memory, and even garbage collection does
>> not get rid of them. My expectation is that after the steps I’ve described
>> in my first sentence have finished, the main memory should almost look as
>> empty as it was before the first entry of the RESTful call.
>>
>> Please find attached an xml file that is exported from JProfiler:
>> And this HTML file was also exported by JProfiler and may have a clearer
>> view on what was going on:
>>
>>
>> I’m having a hard time these days because I’m fighting with my colleagues to
>> use Apache Camel. If they’ve found that memory leak, I’m going to lose that
>> battle. So please help.
>>
>> Many thanks in advance!
>>
>> Kind regards,
>> Christian Jacob
>>
>> Innogy SE
>> Retail IT
>> Integration and Digital Solutions (AFS-IGI)
>> Rellinghauser Str. 37
>> 45128 Essen
>> T intern: 70-20581
>> T extern: +49 (0)201 12 20582
>> T mobil: +49 (0)1622843981
>> Fax: +49 (0)201 12 24796
>> mailto: [hidden email]
>>
>> ----------------------------------------------------------------
>> innogy SE
>> Vorsitzender des Aufsichtsrates: Dr. Erhard Schipporeit
>> Vorstand: Uwe Tigges (Vorsitzender), Dr. Hans Buenting,
>> Dr. Bernhard Guenther, Arno Hahn, Martin Herrmann, Hildegard Mueller
>> Sitz der Gesellschaft: Essen, Eingetragen beim Amtsgericht Essen,
>> Handelsregister-Nr. HRB 27091, USt-IdNr. DE304171711
>
>
>
> --
> Claus Ibsen
> -----------------
> http://davsclaus.com @davsclaus
> Camel in Action 2: https://www.manning.com/ibsen2
> ----------------------------------------------------------------
> innogy SE
> Vorsitzender des Aufsichtsrates: Dr. Erhard Schipporeit
> Vorstand: Uwe Tigges (Vorsitzender), Dr. Hans Buenting,
> Dr. Bernhard Guenther, Arno Hahn, Martin Herrmann, Hildegard Mueller
> Sitz der Gesellschaft: Essen, Eingetragen beim Amtsgericht Essen,
> Handelsregister-Nr. HRB 27091, USt-IdNr. DE304171711



--
Claus Ibsen
-----------------
http://davsclaus.com @davsclaus
Camel in Action 2: https://www.manning.com/ibsen2