[jira] Created: (CAMEL-1138) Memory leak in FileConsumer

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

[jira] Created: (CAMEL-1138) Memory leak in FileConsumer

JIRA jira@apache.org
Memory leak in FileConsumer
---------------------------

                 Key: CAMEL-1138
                 URL: https://issues.apache.org/activemq/browse/CAMEL-1138
             Project: Apache Camel
          Issue Type: Bug
          Components: camel-core
    Affects Versions: 1.5.0
            Reporter: Hadrian Zbarcea
            Priority: Critical
             Fix For: 1.5.1, 2.0.0


Using a memory profiler, we've identified what appears to be a substantial memory leak in FileConsumer in Camel 1.5.0.  It appears that the noopMap is constantly having items added to it, but nothing performs a remove on it when the file is consumed.  This causes a very large amount of string data to be accumulated in the heap.  In our application, this was a leak of several hundred megabytes and is a showstopper.   Considering the apparent severity of this issue, it would really be nice if a fix could be incorporated into a 1.5.1 version.

http://www.nabble.com/Memory-leak-in-FileConsumer-in-Camel-1.5.0-td20794405s22882.html#a20794405

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (CAMEL-1138) Memory leak in FileConsumer

JIRA jira@apache.org

    [ https://issues.apache.org/activemq/browse/CAMEL-1138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=47793#action_47793 ]

Claus Ibsen commented on CAMEL-1138:
------------------------------------

This is *fixed in 2.0* as we have removed all this troublesome code.

For 1.5.1 I suggest to replace the noopMap with the LRUCahce Map so it keeps up till 1000 files and nothing more.

> Memory leak in FileConsumer
> ---------------------------
>
>                 Key: CAMEL-1138
>                 URL: https://issues.apache.org/activemq/browse/CAMEL-1138
>             Project: Apache Camel
>          Issue Type: Bug
>          Components: camel-core
>    Affects Versions: 1.5.0
>            Reporter: Hadrian Zbarcea
>            Priority: Critical
>             Fix For: 1.5.1, 2.0.0
>
>
> Using a memory profiler, we've identified what appears to be a substantial memory leak in FileConsumer in Camel 1.5.0.  It appears that the noopMap is constantly having items added to it, but nothing performs a remove on it when the file is consumed.  This causes a very large amount of string data to be accumulated in the heap.  In our application, this was a leak of several hundred megabytes and is a showstopper.   Considering the apparent severity of this issue, it would really be nice if a fix could be incorporated into a 1.5.1 version.
> http://www.nabble.com/Memory-leak-in-FileConsumer-in-Camel-1.5.0-td20794405s22882.html#a20794405

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Reply | Threaded
Open this post in threaded view
|

[jira] Assigned: (CAMEL-1138) Memory leak in FileConsumer

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

     [ https://issues.apache.org/activemq/browse/CAMEL-1138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Claus Ibsen reassigned CAMEL-1138:
----------------------------------

    Assignee: Claus Ibsen

> Memory leak in FileConsumer
> ---------------------------
>
>                 Key: CAMEL-1138
>                 URL: https://issues.apache.org/activemq/browse/CAMEL-1138
>             Project: Apache Camel
>          Issue Type: Bug
>          Components: camel-core
>    Affects Versions: 1.5.0
>            Reporter: Hadrian Zbarcea
>            Assignee: Claus Ibsen
>            Priority: Critical
>             Fix For: 1.5.1, 2.0.0
>
>
> Using a memory profiler, we've identified what appears to be a substantial memory leak in FileConsumer in Camel 1.5.0.  It appears that the noopMap is constantly having items added to it, but nothing performs a remove on it when the file is consumed.  This causes a very large amount of string data to be accumulated in the heap.  In our application, this was a leak of several hundred megabytes and is a showstopper.   Considering the apparent severity of this issue, it would really be nice if a fix could be incorporated into a 1.5.1 version.
> http://www.nabble.com/Memory-leak-in-FileConsumer-in-Camel-1.5.0-td20794405s22882.html#a20794405

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (CAMEL-1138) Memory leak in FileConsumer

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

    [ https://issues.apache.org/activemq/browse/CAMEL-1138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=47809#action_47809 ]

Claus Ibsen commented on CAMEL-1138:
------------------------------------

These maps is part of some code logic to determine if the file has been changed using timestamp and filesize checks. All this code has been @deprecated and removed in 2.0.
It leads to unexpected behavior and is hard to test. And shouldn't generally be used.

If you use file connectivity then you should either delete or move files after they are processed and not keep then around.
As a fix for this in 1.5.1 I have added the LRUCache so the maps will contain at most 1000 elements.

> Memory leak in FileConsumer
> ---------------------------
>
>                 Key: CAMEL-1138
>                 URL: https://issues.apache.org/activemq/browse/CAMEL-1138
>             Project: Apache Camel
>          Issue Type: Bug
>          Components: camel-core
>    Affects Versions: 1.5.0
>            Reporter: Hadrian Zbarcea
>            Assignee: Claus Ibsen
>            Priority: Critical
>             Fix For: 1.5.1, 2.0.0
>
>
> Using a memory profiler, we've identified what appears to be a substantial memory leak in FileConsumer in Camel 1.5.0.  It appears that the noopMap is constantly having items added to it, but nothing performs a remove on it when the file is consumed.  This causes a very large amount of string data to be accumulated in the heap.  In our application, this was a leak of several hundred megabytes and is a showstopper.   Considering the apparent severity of this issue, it would really be nice if a fix could be incorporated into a 1.5.1 version.
> http://www.nabble.com/Memory-leak-in-FileConsumer-in-Camel-1.5.0-td20794405s22882.html#a20794405

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (CAMEL-1138) Memory leak in FileConsumer

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

    [ https://issues.apache.org/activemq/browse/CAMEL-1138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=47810#action_47810 ]

Claus Ibsen commented on CAMEL-1138:
------------------------------------

D:\project\camel-1x\camel-core>svn commit --message "CAMEL-1138: Quick fix for file consumer not freeing memory"
Sending        camel-core\src\main\java\org\apache\camel\component\file\FileConsumer.java
Transmitting file data .
Committed revision 722808.

> Memory leak in FileConsumer
> ---------------------------
>
>                 Key: CAMEL-1138
>                 URL: https://issues.apache.org/activemq/browse/CAMEL-1138
>             Project: Apache Camel
>          Issue Type: Bug
>          Components: camel-core
>    Affects Versions: 1.5.0
>            Reporter: Hadrian Zbarcea
>            Assignee: Claus Ibsen
>            Priority: Critical
>             Fix For: 1.5.1, 2.0.0
>
>
> Using a memory profiler, we've identified what appears to be a substantial memory leak in FileConsumer in Camel 1.5.0.  It appears that the noopMap is constantly having items added to it, but nothing performs a remove on it when the file is consumed.  This causes a very large amount of string data to be accumulated in the heap.  In our application, this was a leak of several hundred megabytes and is a showstopper.   Considering the apparent severity of this issue, it would really be nice if a fix could be incorporated into a 1.5.1 version.
> http://www.nabble.com/Memory-leak-in-FileConsumer-in-Camel-1.5.0-td20794405s22882.html#a20794405

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Reply | Threaded
Open this post in threaded view
|

[jira] Work started: (CAMEL-1138) Memory leak in FileConsumer

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

     [ https://issues.apache.org/activemq/browse/CAMEL-1138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Work on CAMEL-1138 started by Claus Ibsen.

> Memory leak in FileConsumer
> ---------------------------
>
>                 Key: CAMEL-1138
>                 URL: https://issues.apache.org/activemq/browse/CAMEL-1138
>             Project: Apache Camel
>          Issue Type: Bug
>          Components: camel-core
>    Affects Versions: 1.5.0
>            Reporter: Hadrian Zbarcea
>            Assignee: Claus Ibsen
>            Priority: Critical
>             Fix For: 1.5.1, 2.0.0
>
>
> Using a memory profiler, we've identified what appears to be a substantial memory leak in FileConsumer in Camel 1.5.0.  It appears that the noopMap is constantly having items added to it, but nothing performs a remove on it when the file is consumed.  This causes a very large amount of string data to be accumulated in the heap.  In our application, this was a leak of several hundred megabytes and is a showstopper.   Considering the apparent severity of this issue, it would really be nice if a fix could be incorporated into a 1.5.1 version.
> http://www.nabble.com/Memory-leak-in-FileConsumer-in-Camel-1.5.0-td20794405s22882.html#a20794405

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (CAMEL-1138) Memory leak in FileConsumer

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

    [ https://issues.apache.org/activemq/browse/CAMEL-1138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=47828#action_47828 ]

Claus Ibsen commented on CAMEL-1138:
------------------------------------

Hadrian is the current quick fix okay?

> Memory leak in FileConsumer
> ---------------------------
>
>                 Key: CAMEL-1138
>                 URL: https://issues.apache.org/activemq/browse/CAMEL-1138
>             Project: Apache Camel
>          Issue Type: Bug
>          Components: camel-core
>    Affects Versions: 1.5.0
>            Reporter: Hadrian Zbarcea
>            Assignee: Claus Ibsen
>            Priority: Critical
>             Fix For: 1.5.1, 2.0.0
>
>
> Using a memory profiler, we've identified what appears to be a substantial memory leak in FileConsumer in Camel 1.5.0.  It appears that the noopMap is constantly having items added to it, but nothing performs a remove on it when the file is consumed.  This causes a very large amount of string data to be accumulated in the heap.  In our application, this was a leak of several hundred megabytes and is a showstopper.   Considering the apparent severity of this issue, it would really be nice if a fix could be incorporated into a 1.5.1 version.
> http://www.nabble.com/Memory-leak-in-FileConsumer-in-Camel-1.5.0-td20794405s22882.html#a20794405

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (CAMEL-1138) Memory leak in FileConsumer

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

    [ https://issues.apache.org/activemq/browse/CAMEL-1138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=47834#action_47834 ]

Hadrian Zbarcea commented on CAMEL-1138:
----------------------------------------

I think so.  It actually solves the leak problem.

The side effect is that if a file is not moved/renamed after 1000 other entries/files it will disappear from the lru cache and look new again and get reprocessed, which may also lead to an infinite loop (that would make another one of those 1000+ files new/out of cache, and so on).  However I don't think this scenario is a valid one as it has other flaws, such as the fact that the state is lost after a camel restart, so files would get processed again in that case too (a comment in the code mentions that, actually).

I would add a warn in the wiki that not renaming/moving files is a dangerous scenario and should not be used.  

> Memory leak in FileConsumer
> ---------------------------
>
>                 Key: CAMEL-1138
>                 URL: https://issues.apache.org/activemq/browse/CAMEL-1138
>             Project: Apache Camel
>          Issue Type: Bug
>          Components: camel-core
>    Affects Versions: 1.5.0
>            Reporter: Hadrian Zbarcea
>            Assignee: Claus Ibsen
>            Priority: Critical
>             Fix For: 1.5.1, 2.0.0
>
>
> Using a memory profiler, we've identified what appears to be a substantial memory leak in FileConsumer in Camel 1.5.0.  It appears that the noopMap is constantly having items added to it, but nothing performs a remove on it when the file is consumed.  This causes a very large amount of string data to be accumulated in the heap.  In our application, this was a leak of several hundred megabytes and is a showstopper.   Considering the apparent severity of this issue, it would really be nice if a fix could be incorporated into a 1.5.1 version.
> http://www.nabble.com/Memory-leak-in-FileConsumer-in-Camel-1.5.0-td20794405s22882.html#a20794405

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (CAMEL-1138) Memory leak in FileConsumer

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

    [ https://issues.apache.org/activemq/browse/CAMEL-1138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=47835#action_47835 ]

Hadrian Zbarcea commented on CAMEL-1138:
----------------------------------------

I think we can close this issue.

> Memory leak in FileConsumer
> ---------------------------
>
>                 Key: CAMEL-1138
>                 URL: https://issues.apache.org/activemq/browse/CAMEL-1138
>             Project: Apache Camel
>          Issue Type: Bug
>          Components: camel-core
>    Affects Versions: 1.5.0
>            Reporter: Hadrian Zbarcea
>            Assignee: Claus Ibsen
>            Priority: Critical
>             Fix For: 1.5.1, 2.0.0
>
>
> Using a memory profiler, we've identified what appears to be a substantial memory leak in FileConsumer in Camel 1.5.0.  It appears that the noopMap is constantly having items added to it, but nothing performs a remove on it when the file is consumed.  This causes a very large amount of string data to be accumulated in the heap.  In our application, this was a leak of several hundred megabytes and is a showstopper.   Considering the apparent severity of this issue, it would really be nice if a fix could be incorporated into a 1.5.1 version.
> http://www.nabble.com/Memory-leak-in-FileConsumer-in-Camel-1.5.0-td20794405s22882.html#a20794405

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Reply | Threaded
Open this post in threaded view
|

[jira] Resolved: (CAMEL-1138) Memory leak in FileConsumer

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

     [ https://issues.apache.org/activemq/browse/CAMEL-1138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Claus Ibsen resolved CAMEL-1138.
--------------------------------

    Resolution: Fixed

> Memory leak in FileConsumer
> ---------------------------
>
>                 Key: CAMEL-1138
>                 URL: https://issues.apache.org/activemq/browse/CAMEL-1138
>             Project: Apache Camel
>          Issue Type: Bug
>          Components: camel-core
>    Affects Versions: 1.5.0
>            Reporter: Hadrian Zbarcea
>            Assignee: Claus Ibsen
>            Priority: Critical
>             Fix For: 1.5.1, 2.0.0
>
>
> Using a memory profiler, we've identified what appears to be a substantial memory leak in FileConsumer in Camel 1.5.0.  It appears that the noopMap is constantly having items added to it, but nothing performs a remove on it when the file is consumed.  This causes a very large amount of string data to be accumulated in the heap.  In our application, this was a leak of several hundred megabytes and is a showstopper.   Considering the apparent severity of this issue, it would really be nice if a fix could be incorporated into a 1.5.1 version.
> http://www.nabble.com/Memory-leak-in-FileConsumer-in-Camel-1.5.0-td20794405s22882.html#a20794405

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Reply | Threaded
Open this post in threaded view
|

[jira] Closed: (CAMEL-1138) Memory leak in FileConsumer

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

     [ https://issues.apache.org/activemq/browse/CAMEL-1138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Claus Ibsen closed CAMEL-1138.
------------------------------


> Memory leak in FileConsumer
> ---------------------------
>
>                 Key: CAMEL-1138
>                 URL: https://issues.apache.org/activemq/browse/CAMEL-1138
>             Project: Apache Camel
>          Issue Type: Bug
>          Components: camel-core
>    Affects Versions: 1.5.0
>            Reporter: Hadrian Zbarcea
>            Assignee: Claus Ibsen
>            Priority: Critical
>             Fix For: 1.5.1, 2.0.0
>
>
> Using a memory profiler, we've identified what appears to be a substantial memory leak in FileConsumer in Camel 1.5.0.  It appears that the noopMap is constantly having items added to it, but nothing performs a remove on it when the file is consumed.  This causes a very large amount of string data to be accumulated in the heap.  In our application, this was a leak of several hundred megabytes and is a showstopper.   Considering the apparent severity of this issue, it would really be nice if a fix could be incorporated into a 1.5.1 version.
> http://www.nabble.com/Memory-leak-in-FileConsumer-in-Camel-1.5.0-td20794405s22882.html#a20794405

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.