File Processor Not deleting the files

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
26 messages Options
12
Reply | Threaded
Open this post in threaded view
|

File Processor Not deleting the files

rdifrango
I have the following route:

context.addRoutes(new RouteBuilder {
    // Read the file(s) from the directory
    "file:perf?delete=true" ==> {
      // Split the file up at each line
      split(_.getIn().getBody(classOf[String]).split("\n")) {
        // The further split up the processing across multiple parsers
        loadbalance roundrobin {
        // Reference to my internal components
          to("direct:x")
          to("direct:y")
          to("direct:z")
        }
      // Join back the results from aboce
      // then split it out again turning off commit level changes
      }.loadbalance roundrobin {
        to("jdbc:dataSource?resetAutoCommit=false")
      to("jdbc:dataSource?resetAutoCommit=false")
      to("jdbc:dataSource?resetAutoCommit=false")
      }
    }

The behavior I'm see is that if I have multiple files, they aren't deleted once they are processed.  Any reason why not?
Reply | Threaded
Open this post in threaded view
|

Re: File Processor Not deleting the files

Claus Ibsen-2
What Camel version and OS are you using?

On Fri, Apr 27, 2012 at 3:32 PM, rdifrango <[hidden email]> wrote:

> I have the following route:
>
> context.addRoutes(new RouteBuilder {
>    // Read the file(s) from the directory
>    "file:perf?delete=true" ==> {
>      // Split the file up at each line
>      split(_.getIn().getBody(classOf[String]).split("\n")) {
>        // The further split up the processing across multiple parsers
>        loadbalance roundrobin {
>        // Reference to my internal components
>          to("direct:x")
>          to("direct:y")
>          to("direct:z")
>        }
>      // Join back the results from aboce
>      // then split it out again turning off commit level changes
>      }.loadbalance roundrobin {
>        to("jdbc:dataSource?resetAutoCommit=false")
>        to("jdbc:dataSource?resetAutoCommit=false")
>        to("jdbc:dataSource?resetAutoCommit=false")
>      }
>    }
>
> The behavior I'm see is that if I have multiple files, they aren't deleted
> once they are processed.  Any reason why not?
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/File-Processor-Not-deleting-the-files-tp5670301p5670301.html
> Sent from the Camel - Users mailing list archive at Nabble.com.



--
Claus Ibsen
-----------------
CamelOne 2012 Conference, May 15-16, 2012: http://camelone.com
FuseSource
Email: [hidden email]
Web: http://fusesource.com
Twitter: davsclaus, fusenews
Blog: http://davsclaus.blogspot.com/
Author of Camel in Action: http://www.manning.com/ibsen/
Reply | Threaded
Open this post in threaded view
|

Re: File Processor Not deleting the files

rdifrango
In reply to this post by rdifrango
Camel version 2.9.2 on a Windows 7 Machine.
Reply | Threaded
Open this post in threaded view
|

Re: File Processor Not deleting the files

Claus Ibsen-2
On Fri, Apr 27, 2012 at 5:01 PM, rdifrango <[hidden email]> wrote:
> Camel version 2.9.2 on a Windows 7 Machine.
>

Okay be careful on Windows OS to ensure you always closes file handles
and streams when you acces the files.
For example if you use a InputStream then make sure you close it etc.


> --
> View this message in context: http://camel.465427.n5.nabble.com/File-Processor-Not-deleting-the-files-tp5670301p5670530.html
> Sent from the Camel - Users mailing list archive at Nabble.com.



--
Claus Ibsen
-----------------
CamelOne 2012 Conference, May 15-16, 2012: http://camelone.com
FuseSource
Email: [hidden email]
Web: http://fusesource.com
Twitter: davsclaus, fusenews
Blog: http://davsclaus.blogspot.com/
Author of Camel in Action: http://www.manning.com/ibsen/
Reply | Threaded
Open this post in threaded view
|

Re: File Processor Not deleting the files

rdifrango
In reply to this post by rdifrango
I'm not using any streams directly in my code.  The complete code listing is as follows:

// Build the connection pool
  val ds = new BasicDataSource
  ds.setDriverClassName("oracle.jdbc.OracleDriver")
  ds.setUrl("jdbc:oracle:thin:@localhost:1521:xe")
  ds.setUsername("******")
  ds.setPassword("******")
  // Register it with Camel
  val reg = new SimpleRegistry
  reg.put("dataSource", ds)

  val format = "INSERT INTO EVT_PERF ( EVT_PERF_ID, CRLTN_ID, USER_ID, APPN_SYS_CD, HOST_NM, WEBLOGIC_INSTNC_NM, WEB_ANLYTCS_CRLTN_ID, LOGGER_CLASS_NM, EXEC_TIME ) values ( EVT_PERF_ID_SEQ.NEXTVAL, ''{0}'', ''{1}'', ''{2}'', ''{3}'', ''{4}'', ''{5}'', ''{6}'', {7} )"
  val processor = (exchange: Exchange) => {
    // Get the incoming line from the message exchange
    val line = exchange.getIn.getBody(classOf[String])
    // Remove the first part of the log message
    val lineSplit = line.split("\\[\\]\\: ")
    // Break it down into the individual components
    val split = lineSplit(1).split(" ")
    // Build the insert statement
    val x = MessageFormat.format(format, split(0), split(1), split(2), split(3), split(4), split(5), split(7), split(10))
    // Replace the incoming message with the insert statement
    exchange.getIn().setBody(x)
  }

  // Create the Camel Context
  val context = new DefaultCamelContext(reg)
  // Build the Route
  context.addRoutes(new RouteBuilder {
    // Read the file(s) from the directory
    "file:perf?delete=true&idempotent=true" ==> {
      // Split the file up at each line
      split(_.getIn().getBody(classOf[String]).split("\n")) {
        // The further split up the processing across multiple parsers
        loadbalance roundrobin {
          // Reference to my internal components
          to("direct:x")
          to("direct:y")
          to("direct:z")
        }
      }.loadbalance roundrobin {
        to("jdbc:dataSource?resetAutoCommit=false")
      to("jdbc:dataSource?resetAutoCommit=false")
      to("jdbc:dataSource?resetAutoCommit=false")
      }
    }

    // Setup my internal processors with references to my custom component
    "direct:x" process (processor)
    "direct:y" process (processor)
    "direct:z" process (processor)
  })

  // Start the camel process
  context.start
  // Wait for the process to execute
  Thread.sleep(120000)
  // Stop the camel process
  context.stop
Reply | Threaded
Open this post in threaded view
|

Re: File Processor Not deleting the files

rdifrango
In reply to this post by rdifrango
I guess this is implicitly touching the stream:

split(_.getIn().getBody(classOf[String]).split("\n"))

Could be the root?
Reply | Threaded
Open this post in threaded view
|

Re: File Processor Not deleting the files

Claus Ibsen-2
Yeah use the tokenize expression

In Java it would be

.split().tokenize("\n").streaming()


On Fri, Apr 27, 2012 at 5:13 PM, rdifrango <[hidden email]> wrote:
> I guess this is implicitly touching the stream:
>
> split(_.getIn().getBody(classOf[String]).split("\n"))
>
> Could be the root?
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/File-Processor-Not-deleting-the-files-tp5670301p5670589.html
> Sent from the Camel - Users mailing list archive at Nabble.com.



--
Claus Ibsen
-----------------
CamelOne 2012 Conference, May 15-16, 2012: http://camelone.com
FuseSource
Email: [hidden email]
Web: http://fusesource.com
Twitter: davsclaus, fusenews
Blog: http://davsclaus.blogspot.com/
Author of Camel in Action: http://www.manning.com/ibsen/
Reply | Threaded
Open this post in threaded view
|

Re: File Processor Not deleting the files

rdifrango
Yeah, in the Scala DSL, it doesn't seem to support that.  Is that a bug or feature request?
Reply | Threaded
Open this post in threaded view
|

Re: File Processor Not deleting the files

Claus Ibsen-2
On Fri, Apr 27, 2012 at 5:30 PM, rdifrango <[hidden email]> wrote:
> Yeah, in the Scala DSL, it doesn't seem to support that.  Is that a bug or
> feature request?
>

Fell free to log a JIRA. And as we love contributions you are welcome
to work on a patch to get that into the Scala DSL.
http://camel.apache.org/contributing.html

> --
> View this message in context: http://camel.465427.n5.nabble.com/File-Processor-Not-deleting-the-files-tp5670301p5670621.html
> Sent from the Camel - Users mailing list archive at Nabble.com.



--
Claus Ibsen
-----------------
CamelOne 2012 Conference, May 15-16, 2012: http://camelone.com
FuseSource
Email: [hidden email]
Web: http://fusesource.com
Twitter: davsclaus, fusenews
Blog: http://davsclaus.blogspot.com/
Author of Camel in Action: http://www.manning.com/ibsen/
Reply | Threaded
Open this post in threaded view
|

Re: File Processor Not deleting the files

rdifrango
So, I converted this over to Java and it is exhibiting the same behavior.  What did I do wrong:

public class JavaSplitPerfProcessor {
        static class LineSplitter implements Processor {
                static final String format = "INSERT INTO EVT_PERF ( EVT_PERF_ID, CRLTN_ID, USER_ID, APPN_SYS_CD, HOST_NM, WEBLOGIC_INSTNC_NM, WEB_ANLYTCS_CRLTN_ID, LOGGER_CLASS_NM, EXEC_TIME ) values ( EVT_PERF_ID_SEQ.NEXTVAL, ''{0}'', ''{1}'', ''{2}'', ''{3}'', ''{4}'', ''{5}'', ''{6}'', {7} )";

                @Override
                public void process(Exchange exchange) throws Exception {
                        // Get the incoming line from the message exchange
                        String line = exchange.getIn().getBody(String.class);
                        // Remove the first part of the log message
                        String[] lineSplit = line.split("\\[\\]\\: ");
                        // Break it down into the individual components
                        String[] split = lineSplit[1].split(" ");
                        // Build the insert statement
                        String x = MessageFormat
                                        .format(format, split[0], split[1], split[2], split[3],
                                                        split[4], split[5], split[7], split[10]);
                        // Replace the incoming message with the insert statement
                        exchange.getIn().setBody(x);
                }
        }

        public static void main(String[] args) throws Exception {
                // Build the connection pool
                BasicDataSource ds = new BasicDataSource();
                ds.setDriverClassName("oracle.jdbc.OracleDriver");
                ds.setUrl("jdbc:oracle:thin:@localhost:1521:xe");
                ds.setUsername("****");
                ds.setPassword("****");
                // Register it with Camel
                SimpleRegistry reg = new SimpleRegistry();
                reg.put("dataSource", ds);

                final LineSplitter lineSplitter = new LineSplitter();

                CamelContext context = new DefaultCamelContext(reg);
                context.addRoutes(new RouteBuilder() {
                        public void configure() {
                                from("file:perf?delete=true").split(body().tokenize("\n"))
                                                .streaming().process(lineSplitter)
                                                .to("jdbc:dataSource?resetAutoCommit=false").end();
                        }
                });

                context.start();
                Thread.sleep(60000);
                context.stop();
        }
}
Reply | Threaded
Open this post in threaded view
|

Re: File Processor Not deleting the files

Claus Ibsen-2
On Mon, Apr 30, 2012 at 3:27 PM, rdifrango <[hidden email]> wrote:

> So, I converted this over to Java and it is exhibiting the same behavior.
> What did I do wrong:
>
> public class JavaSplitPerfProcessor {
>        static class LineSplitter implements Processor {
>                static final String format = "INSERT INTO EVT_PERF ( EVT_PERF_ID,
> CRLTN_ID, USER_ID, APPN_SYS_CD, HOST_NM, WEBLOGIC_INSTNC_NM,
> WEB_ANLYTCS_CRLTN_ID, LOGGER_CLASS_NM, EXEC_TIME ) values (
> EVT_PERF_ID_SEQ.NEXTVAL, ''{0}'', ''{1}'', ''{2}'', ''{3}'', ''{4}'',
> ''{5}'', ''{6}'', {7} )";
>
>                @Override
>                public void process(Exchange exchange) throws Exception {
>                        // Get the incoming line from the message exchange
>                        String line = exchange.getIn().getBody(String.class);
>                        // Remove the first part of the log message
>                        String[] lineSplit = line.split("\\[\\]\\: ");
>                        // Break it down into the individual components
>                        String[] split = lineSplit[1].split(" ");
>                        // Build the insert statement
>                        String x = MessageFormat
>                                        .format(format, split[0], split[1], split[2], split[3],
>                                                        split[4], split[5], split[7], split[10]);
>                        // Replace the incoming message with the insert statement
>                        exchange.getIn().setBody(x);
>                }
>        }
>
>        public static void main(String[] args) throws Exception {
>                // Build the connection pool
>                BasicDataSource ds = new BasicDataSource();
>                ds.setDriverClassName("oracle.jdbc.OracleDriver");
>                ds.setUrl("jdbc:oracle:thin:@localhost:1521:xe");
>                ds.setUsername("****");
>                ds.setPassword("****");
>                // Register it with Camel
>                SimpleRegistry reg = new SimpleRegistry();
>                reg.put("dataSource", ds);
>
>                final LineSplitter lineSplitter = new LineSplitter();
>
>                CamelContext context = new DefaultCamelContext(reg);
>                context.addRoutes(new RouteBuilder() {
>                        public void configure() {
>                                from("file:perf?delete=true").split(body().tokenize("\n"))
>                                                .streaming().process(lineSplitter)
>                                                .to("jdbc:dataSource?resetAutoCommit=false").end();
>                        }
>                });
>
>                context.start();
>                Thread.sleep(60000);
>                context.stop();
>        }
> }
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/File-Processor-Not-deleting-the-files-tp5670301p5675961.html
> Sent from the Camel - Users mailing list archive at Nabble.com.

Windows is always a bit tricky with its file system.

I suspect this line
    String line = exchange.getIn().getBody(String.class);

Despite we close the buffers un the hood, the JDK may not close
wrapped buffers, which may lead
to the FileInputStream not being closed, and Windows not able to
delete the source file when the processing is done.

I got a XP box for testing. So let see what that old fella says.



--
Claus Ibsen
-----------------
CamelOne 2012 Conference, May 15-16, 2012: http://camelone.com
FuseSource
Email: [hidden email]
Web: http://fusesource.com
Twitter: davsclaus, fusenews
Blog: http://davsclaus.blogspot.com/
Author of Camel in Action: http://www.manning.com/ibsen/
Reply | Threaded
Open this post in threaded view
|

Re: File Processor Not deleting the files

rdifrango
Thanks, in the meantime I'm going to see if I can get this running on a Unix box.
Reply | Threaded
Open this post in threaded view
|

Re: File Processor Not deleting the files

rdifrango
I wonder if using one the Marshalling frameworks would solve this?
Reply | Threaded
Open this post in threaded view
|

Re: File Processor Not deleting the files

rdifrango
This post has NOT been accepted by the mailing list yet.
Actually that won't work because it loads everything into memory.  Streaming is the only rational thing to do because we are processing 50MB+ sized files. 

On Apr 30, 2012, at 2:58 PM, "rdifrango [via Camel]" <[hidden email]> wrote:

I wonder if using one the Marshalling frameworks would solve this?


If you reply to this email, your message will be added to the discussion below:
http://camel.465427.n5.nabble.com/File-Processor-Not-deleting-the-files-tp5670301p5676645.html
To unsubscribe from File Processor Not deleting the files, click here.
NAML
Reply | Threaded
Open this post in threaded view
|

Re: File Processor Not deleting the files

Claus Ibsen-2
In reply to this post by rdifrango
Hi

I committed a fix to trunk and 2.9 branches
https://issues.apache.org/jira/browse/CAMEL-5235

You are welcome to try on your windows system with a SNAPSHOT version.
Make sure to restart your windows box beforehand so it has no file
locks from the start.

Also if you do a full unit tests of camel-core, eg from the source code
cd camel-core
mvn clean install

Then afte the testing you should be able to do a
mvn clean

To delete all the temporary files created, to make sure Windows dont
have any locks.



On Mon, Apr 30, 2012 at 8:58 PM, rdifrango <[hidden email]> wrote:
> I wonder if using one the Marshalling frameworks would solve this?
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/File-Processor-Not-deleting-the-files-tp5670301p5676645.html
> Sent from the Camel - Users mailing list archive at Nabble.com.



--
Claus Ibsen
-----------------
CamelOne 2012 Conference, May 15-16, 2012: http://camelone.com
FuseSource
Email: [hidden email]
Web: http://fusesource.com
Twitter: davsclaus, fusenews
Blog: http://davsclaus.blogspot.com/
Author of Camel in Action: http://www.manning.com/ibsen/
Reply | Threaded
Open this post in threaded view
|

Re: File Processor Not deleting the files

rdifrango
Great, Thanks!  Will this fix also cover
the Scala DSL?

On May 1, 2012, at 3:08 AM, "Claus Ibsen-2 [via Camel]" <[hidden email]> wrote:

Hi

I committed a fix to trunk and 2.9 branches
https://issues.apache.org/jira/browse/CAMEL-5235

You are welcome to try on your windows system with a SNAPSHOT version.
Make sure to restart your windows box beforehand so it has no file
locks from the start.

Also if you do a full unit tests of camel-core, eg from the source code
cd camel-core
mvn clean install

Then afte the testing you should be able to do a
mvn clean

To delete all the temporary files created, to make sure Windows dont
have any locks.



On Mon, Apr 30, 2012 at 8:58 PM, rdifrango <[hidden email]> wrote:
> I wonder if using one the Marshalling frameworks would solve this?
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/File-Processor-Not-deleting-the-files-tp5670301p5676645.html
> Sent from the Camel - Users mailing list archive at Nabble.com.



--
Claus Ibsen
-----------------
CamelOne 2012 Conference, May 15-16, 2012: http://camelone.com
FuseSource
Email: [hidden email]
Web: http://fusesource.com
Twitter: davsclaus, fusenews
Blog: http://davsclaus.blogspot.com/
Author of Camel in Action: http://www.manning.com/ibsen/



If you reply to this email, your message will be added to the discussion below:
http://camel.465427.n5.nabble.com/File-Processor-Not-deleting-the-files-tp5670301p5677551.html
To unsubscribe from File Processor Not deleting the files, click here.
NAML
Reply | Threaded
Open this post in threaded view
|

Re: File Processor Not deleting the files

Claus Ibsen-2
On Tue, May 1, 2012 at 12:52 PM, rdifrango <[hidden email]> wrote:
> Great, Thanks!  Will this fix also cover
> the Scala DSL?
>

Yes the fix is in camel-core which also applies for Scala.


> On May 1, 2012, at 3:08 AM, "Claus Ibsen-2 [via Camel]" <
> [hidden email]> wrote:
>
> Hi
>
> I committed a fix to trunk and 2.9 branches
> https://issues.apache.org/jira/browse/CAMEL-5235
>
> You are welcome to try on your windows system with a SNAPSHOT version.
> Make sure to restart your windows box beforehand so it has no file
> locks from the start.
>
> Also if you do a full unit tests of camel-core, eg from the source code
> cd camel-core
> mvn clean install
>
> Then afte the testing you should be able to do a
> mvn clean
>
> To delete all the temporary files created, to make sure Windows dont
> have any locks.
>
>
>
> On Mon, Apr 30, 2012 at 8:58 PM, rdifrango <[hidden
> email]</user/SendEmail.jtp?type=node&node=5677551&i=0>>
> wrote:
>> I wonder if using one the Marshalling frameworks would solve this?
>>
>> --
>> View this message in context:
> http://camel.465427.n5.nabble.com/File-Processor-Not-deleting-the-files-tp5670301p5676645.html
>> Sent from the Camel - Users mailing list archive at Nabble.com.
>
>
>
> --
> Claus Ibsen
> -----------------
> CamelOne 2012 Conference, May 15-16, 2012: http://camelone.com
> FuseSource
> Email: [hidden email] </user/SendEmail.jtp?type=node&node=5677551&i=1>
> Web: http://fusesource.com
> Twitter: davsclaus, fusenews
> Blog: http://davsclaus.blogspot.com/
> Author of Camel in Action: http://www.manning.com/ibsen/
>
>
> ------------------------------
>  If you reply to this email, your message will be added to the discussion
> below:
> http://camel.465427.n5.nabble.com/File-Processor-Not-deleting-the-files-tp5670301p5677551.html
>  To unsubscribe from File Processor Not deleting the files, click
> here<
> .
> NAML<
http://camel.465427.n5.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/File-Processor-Not-deleting-the-files-tp5670301p5677821.html
> Sent from the Camel - Users mailing list archive at Nabble.com.



--
Claus Ibsen
-----------------
CamelOne 2012 Conference, May 15-16, 2012: http://camelone.com
FuseSource
Email: [hidden email]
Web: http://fusesource.com
Twitter: davsclaus, fusenews
Blog: http://davsclaus.blogspot.com/
Author of Camel in Action: http://www.manning.com/ibsen/
Reply | Threaded
Open this post in threaded view
|

Re: File Processor Not deleting the files

rdifrango
Has this been baked into a snapshot release that is available or source build only?
Reply | Threaded
Open this post in threaded view
|

Re: File Processor Not deleting the files

Claus Ibsen-2
On Fri, May 4, 2012 at 5:23 PM, rdifrango <[hidden email]> wrote:
> Has this been baked into a snapshot release that is available or source build
> only?
>

Yeah its in the trunk and 2.9 branches.
http://camel.apache.org/source.html

The 2.10 SNAPSHOT can be downloaded
http://camel.apache.org/download.html

Or you can build from source
http://camel.apache.org/building.html

> --
> View this message in context: http://camel.465427.n5.nabble.com/File-Processor-Not-deleting-the-files-tp5670301p5686206.html
> Sent from the Camel - Users mailing list archive at Nabble.com.



--
Claus Ibsen
-----------------
CamelOne 2012 Conference, May 15-16, 2012: http://camelone.com
FuseSource
Email: [hidden email]
Web: http://fusesource.com
Twitter: davsclaus, fusenews
Blog: http://davsclaus.blogspot.com/
Author of Camel in Action: http://www.manning.com/ibsen/
Reply | Threaded
Open this post in threaded view
|

Re: File Processor Not deleting the files

rdifrango
I downloaded the snapshot version today and ran both my Java and Scala versions and it still appears to have the same problem.  Here are the libraries that I used:

camel-core-2.10-SNAPSHOT.jar
camel-ftp-2.10-SNAPSHOT.jar
camel-jdbc-2.10-SNAPSHOT.jar
camel-jms-2.10-SNAPSHOT.jar
camel-scala-2.10-SNAPSHOT.jar
optional/log4j-1.2.16.jar
slf4j-api-1.6.1.jar
optional/slf4j-log4j12-1.6.4.jar
spring/spring-beans-3.0.7.RELEASE.jar
spring/spring-context-3.0.7.RELEASE.jar
spring/spring-core-3.0.7.RELEASE.jar
spring/spring-jdbc-3.0.7.RELEASE.jar
spring/spring-jms-3.0.7.RELEASE.jar
spring/spring-tx-3.0.7.RELEASE.jar
12