Using ConsumerTemplate to fetch files.

classic Classic list List threaded Threaded
21 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Using ConsumerTemplate to fetch files.

Andreas A.
Hi

I'm trying to do a set up where I use a Spring Trigger to schedule the trigging of a method that fetches a batch of files from ftp to local files and then stops.

I don't think I'm quite grasping the way ConsumerTemplate works. When I do as below the first run works, but the subsequent runs will fail, seemingly when files run again are of the same name. I get cannot rename exceptions. Am I experiencing competing consumers here or what? What is the correct way of achieving what I want?

Would it make more sense to just use the  FTPSClient manually combined with a Spring Trigger for this?

@Autowired
private ProducerTemplate producer;

@Autowired
private ConsumerTemplate consumer;

public void retrieveStatusFiles() throws Exception {
        String ftpsUri = "ftp:localhost:1981/inbox?consumer.delay=1h&username=camel&password=camel123&move=.done";
        String fileUri = "file:{{path.in}}";

        consumer.start();

        while(true) {
                Exchange ex = consumer.receive(ftpsUri, 5000);
                if(ex != null) {
                        logger.info("Fetching file: " + ex.getIn().getHeader("CamelFileName"));
                        producer.send(fileUri, ex);
                }
                else if (ex == null) {
                        consumer.stop();
                        break;
                }
        }
}
Reply | Threaded
Open this post in threaded view
|

Re: Using ConsumerTemplate to fetch files.

davsclaus
Administrator
Are you a subscriber of the mailing list? I haven't seen you mail before.

You need Camel 2.5 or better AFAIR for the consumer template to work with files, where it supports deleting / moving files when done.
Claus Ibsen
-----------------
FuseSource
Email: cibsen@fusesource.com
Web: http://fusesource.com
Twitter: davsclaus
Blog: http://davsclaus.blogspot.com/
Author of Camel in Action: http://www.manning.com/ibsen/
Reply | Threaded
Open this post in threaded view
|

Re: Using ConsumerTemplate to fetch files.

davsclaus
Administrator
Since you work with the low level Exchange, you need to done it if you want it to move the files.
exchange.getUnitOfWork().done(exchange);

This is done automatic for you in Camel 2.5 onwards if you use, eg receiveBody instead

Object body = consumer.redeveiveBody("uri", xxx);
Claus Ibsen
-----------------
FuseSource
Email: cibsen@fusesource.com
Web: http://fusesource.com
Twitter: davsclaus
Blog: http://davsclaus.blogspot.com/
Author of Camel in Action: http://www.manning.com/ibsen/
Reply | Threaded
Open this post in threaded view
|

Re: Using ConsumerTemplate to fetch files.

Andreas A.
In reply to this post by davsclaus
Hi

I should be a subscriber. I have made 100+ posts via nabble. I haven't changed anything regarding my account, are my posts not accepted on the mailinglist anymore or?

I've tried with both 2.5 and 2.6. I'm getting some really odd behaviour - why is it stacking the base dir "inbox"?:

-> ERROR Caused by: [org.apache.camel.component.file.GenericFileOperationFailedException - File operation failed: 550 CWD failed. "/inbox/inbox/inbox/.done": directory not found.
550 CWD failed. "/inbox/inbox/inbox": directory not found.
 Cannot change directory to: inbox. Code: 550]

-> DEBUG Changing directory: /
-> DEBUG Changing directory: inbox
-> DEBUG Changing directory: inbox
-> DEBUG Changing directory: inbox
-> DEBUG Renaming file: GenericFile[OPSIGE_2.edi] to: GenericFile[.done/OPSIGE_2.edi]
-> DEBUG Renaming file: inbox/OPSIGE_2.edi to: inbox/.done/OPSIGE_2.edi
-> ERROR Caused by: [org.apache.camel.component.file.GenericFileOperationFailedException - Cannot rename file: GenericFile[OPSIGE_2.edi] to: GenericFile[.done/OPSIGE_2.edi]]
org.apache.camel.component.file.GenericFileOperationFailedException: Cannot rename file: GenericFile[OPSIGE_2.edi] to: GenericFile[.done/OPSIGE_2.edi]
        at org.apache.camel.component.file.strategy.GenericFileProcessStrategySupport.renameFile(GenericFileProcessStrategySupport.java:100)
        at org.apache.camel.component.file.strategy.GenericFileRenameProcessStrategy.commit(GenericFileRenameProcessStrategy.java:83)
        at org.apache.camel.component.file.GenericFileOnCompletion.processStrategyCommit(GenericFileOnCompletion.java:121)
        at org.apache.camel.component.file.GenericFileOnCompletion.onCompletion(GenericFileOnCompletion.java:83)
Reply | Threaded
Open this post in threaded view
|

Re: Using ConsumerTemplate to fetch files.

Andreas A.
This post was updated on .
CONTENTS DELETED
The author has deleted this message.
Reply | Threaded
Open this post in threaded view
|

Re: Using ConsumerTemplate to fetch files.

Andreas A.
However if I use only receiveBody I miss the headers like the filename etc.
Reply | Threaded
Open this post in threaded view
|

Re: Using ConsumerTemplate to fetch files.

Claus Ibsen-2
On Tue, Dec 14, 2010 at 4:27 PM, Andreas A. <[hidden email]> wrote:
>
> However if I use only receiveBody I miss the headers like the filename etc.
> --

Then you need to done the Exchange yourself. See the doneUoW method in
DefaultConsumerTemplate

> View this message in context: http://camel.465427.n5.nabble.com/Using-ConsumerTemplate-to-fetch-files-tp3304671p3304747.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>



--
Claus Ibsen
-----------------
FuseSource
Email: [hidden email]
Web: http://fusesource.com
Twitter: davsclaus
Blog: http://davsclaus.blogspot.com/
Author of Camel in Action: http://www.manning.com/ibsen/
Reply | Threaded
Open this post in threaded view
|

Re: Using ConsumerTemplate to fetch files.

Andreas A.
Can't find anything along those lines. I tried doing

while(true) {
        Exchange ex = consumer.receive(ftpsUri, 5000);
        if(ex != null) {
                logger.info("Fetching file: " + ex.getIn().getHeader("CamelFileName"));
                producer.send(fileUri, ex);
                ex.getUnitOfWork().done(ex);
        }
        else if (ex == null) {
                consumer.stop();
                break;
        }
}

But this just throws nullpointer, I suppose the exchange doesn't have a unit of work.
Reply | Threaded
Open this post in threaded view
|

Re: Using ConsumerTemplate to fetch files.

Andreas A.
Oh it was a private method. Thought you meant in the api. I suppose I should do the manual handover then. Just pasting here for future reference:

if (exchange.getUnitOfWork() == null) {
        // handover completions and done them manually to ensure they are being executed
        List<Synchronization> synchronizations = exchange.handoverCompletions();
        UnitOfWorkHelper.doneSynchronizations(exchange, synchronizations, LOG);
} else {
        // done the unit of work
        exchange.getUnitOfWork().done(exchange);
}
Reply | Threaded
Open this post in threaded view
|

Re: Using ConsumerTemplate to fetch files.

Claus Ibsen-2
In reply to this post by Andreas A.
I have created a ticket to make this easier using the ConsumerTemplate API
https://issues.apache.org/jira/browse/CAMEL-3427

Check the source code in trunk, how to do it in your Camel version.


On Tue, Dec 14, 2010 at 4:39 PM, Andreas A. <[hidden email]> wrote:

>
> Can't find anything along those lines. I tried doing
>
> while(true) {
>        Exchange ex = consumer.receive(ftpsUri, 5000);
>        if(ex != null) {
>                logger.info("Fetching file: " + ex.getIn().getHeader("CamelFileName"));
>                producer.send(fileUri, ex);
>                ex.getUnitOfWork().done(ex);
>        }
>        else if (ex == null) {
>                consumer.stop();
>                break;
>        }
> }
>
> But this just throws nullpointer, I suppose the exchange doesn't have a unit
> of work.
> --
> View this message in context: http://camel.465427.n5.nabble.com/Using-ConsumerTemplate-to-fetch-files-tp3304671p3304768.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>



--
Claus Ibsen
-----------------
FuseSource
Email: [hidden email]
Web: http://fusesource.com
Twitter: davsclaus
Blog: http://davsclaus.blogspot.com/
Author of Camel in Action: http://www.manning.com/ibsen/
Reply | Threaded
Open this post in threaded view
|

Re: Using ConsumerTemplate to fetch files.

Andreas A.
Hi

Something fishy is still going on, this is the code now:

String ftpsUri = "ftp:localhost:1981/inbox?consumer.delay=1h&username=camel&password=camel123&move=.done";
String fileUri = "file:{{path.in}}";

consumer.start();

while(true) {
        Exchange exchange = consumer.receive(ftpsUri, 5000);
       
        if(exchange == null) {
                consumer.stop();
                break;
        }
       
        if (exchange.getUnitOfWork() == null) {
                // handover completions and done them manually to ensure they are being executed
                List<Synchronization> synchronizations = exchange.handoverCompletions();
                this.doneSynchronizations(exchange, synchronizations);
        } else {
                exchange.getUnitOfWork().done(exchange);
        }
        logger.info("Fetching file: " + exchange.getIn().getHeader("CamelFileName"));
        producer.send(fileUri, exchange);
}

-> ERROR Caused by: [org.apache.camel.component.file.GenericFileOperationFailedException - Cannot retrieve file: GenericFile[op06.txt] from: Endpoint[ftp://localhost:1981/inbox?consumer.delay=1h&move=.done&password=******&username=camel]]
org.apache.camel.component.file.GenericFileOperationFailedException: Cannot retrieve file: GenericFile[op06.txt] from: Endpoint[ftp://localhost:1981/inbox?consumer.delay=1h&move=.done&password=******&username=camel]
        at org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:308)
        at org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:155)
        at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:121)

---

-> ERROR Caused by: [org.apache.camel.component.file.GenericFileOperationFailedException - Cannot rename file: GenericFile[op01.txt] to: GenericFile[.done/op01.txt]]
org.apache.camel.component.file.GenericFileOperationFailedException: Cannot rename file: GenericFile[op01.txt] to: GenericFile[.done/op01.txt]
        at org.apache.camel.component.file.strategy.GenericFileProcessStrategySupport.renameFile(GenericFileProcessStrategySupport.java:100)

---

-> ERROR Caused by: [org.apache.camel.component.file.GenericFileOperationFailedException - File operation failed: 550 File not found
 Cannot change directory to: inbox. Code: 550]
org.apache.camel.component.file.GenericFileOperationFailedException: File operation failed: 550 File not found
 Cannot change directory to: inbox. Code: 550
        at org.apache.camel.component.file.remote.FtpOperations.doChangeDirectory(FtpOperations.java:611)
        at org.apache.camel.component.file.remote.FtpOperations.changeCurrentDirectory(FtpOperations.java:587)
        at org.apache.camel.component.file.remote.FtpOperations.retrieveFileToStreamInBody(FtpOperations.java:319)

---

I believe I am this the same way as the DefaultConsumerTemplate is doing it.
Could this have something to do with the stopping/starting of the consumer? Or is it still the "done"-ing that is not being performed correctly?
Reply | Threaded
Open this post in threaded view
|

Re: Using ConsumerTemplate to fetch files.

Claus Ibsen-2
Its probably best to receive without any timeout, using receiveNoWait.


On Tue, Dec 14, 2010 at 6:43 PM, Andreas A. <[hidden email]> wrote:

>
> Hi
>
> Something fishy is still going on, this is the code now:
>
> String ftpsUri =
> "ftp:localhost:1981/inbox?consumer.delay=1h&username=camel&password=camel123&move=.done";
> String fileUri = "file:{{path.in}}";
>
> consumer.start();
>
> while(true) {
>        Exchange exchange = consumer.receive(ftpsUri, 5000);
>
>        if(exchange == null) {
>                consumer.stop();
>                break;
>        }
>
>        if (exchange.getUnitOfWork() == null) {
>                // handover completions and done them manually to ensure they are being
> executed
>                List<Synchronization> synchronizations = exchange.handoverCompletions();
>                this.doneSynchronizations(exchange, synchronizations);
>        } else {
>                exchange.getUnitOfWork().done(exchange);
>        }
>        logger.info("Fetching file: " +
> exchange.getIn().getHeader("CamelFileName"));
>        producer.send(fileUri, exchange);
> }
>
> -> ERROR Caused by:
> [org.apache.camel.component.file.GenericFileOperationFailedException -
> Cannot retrieve file: GenericFile[op06.txt] from:
> Endpoint[ftp://localhost:1981/inbox?consumer.delay=1h&move=.done&password=******&username=camel]]
> org.apache.camel.component.file.GenericFileOperationFailedException: Cannot
> retrieve file: GenericFile[op06.txt] from:
> Endpoint[ftp://localhost:1981/inbox?consumer.delay=1h&move=.done&password=******&username=camel]
>        at
> org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:308)
>        at
> org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:155)
>        at
> org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:121)
>
> ---
>
> -> ERROR Caused by:
> [org.apache.camel.component.file.GenericFileOperationFailedException -
> Cannot rename file: GenericFile[op01.txt] to: GenericFile[.done/op01.txt]]
> org.apache.camel.component.file.GenericFileOperationFailedException: Cannot
> rename file: GenericFile[op01.txt] to: GenericFile[.done/op01.txt]
>        at
> org.apache.camel.component.file.strategy.GenericFileProcessStrategySupport.renameFile(GenericFileProcessStrategySupport.java:100)
>
> ---
>
> -> ERROR Caused by:
> [org.apache.camel.component.file.GenericFileOperationFailedException - File
> operation failed: 550 File not found
>  Cannot change directory to: inbox. Code: 550]
> org.apache.camel.component.file.GenericFileOperationFailedException: File
> operation failed: 550 File not found
>  Cannot change directory to: inbox. Code: 550
>        at
> org.apache.camel.component.file.remote.FtpOperations.doChangeDirectory(FtpOperations.java:611)
>        at
> org.apache.camel.component.file.remote.FtpOperations.changeCurrentDirectory(FtpOperations.java:587)
>        at
> org.apache.camel.component.file.remote.FtpOperations.retrieveFileToStreamInBody(FtpOperations.java:319)
>
> ---
>
> I believe I am this the same way as the DefaultConsumerTemplate is doing it.
> Could this have something to do with the stopping/starting of the consumer?
> Or is it still the "done"-ing that is not being performed correctly?
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Using-ConsumerTemplate-to-fetch-files-tp3304671p3304969.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>



--
Claus Ibsen
-----------------
FuseSource
Email: [hidden email]
Web: http://fusesource.com
Twitter: davsclaus
Blog: http://davsclaus.blogspot.com/
Author of Camel in Action: http://www.manning.com/ibsen/
Reply | Threaded
Open this post in threaded view
|

Re: Using ConsumerTemplate to fetch files.

Andreas A.
If I use receiveNoWait() no files will be downloaded from the ftp, maybe because the the ftp component doesn't have time to login to the ftp?
Reply | Threaded
Open this post in threaded view
|

Re: Using ConsumerTemplate to fetch files.

Claus Ibsen-2
On Thu, Dec 16, 2010 at 9:28 AM, Andreas A. <[hidden email]> wrote:
>
> If I use receiveNoWait() no files will be downloaded from the ftp, maybe
> because the the ftp component doesn't have time to login to the ftp?
> --

Ah yeah that is correct, there is no wait :)
That works better if you poll messages from a queue and the message is
already there waiting for you.



> View this message in context: http://camel.465427.n5.nabble.com/Using-ConsumerTemplate-to-fetch-files-tp3304671p3307566.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>



--
Claus Ibsen
-----------------
FuseSource
Email: [hidden email]
Web: http://fusesource.com
Twitter: davsclaus
Blog: http://davsclaus.blogspot.com/
Author of Camel in Action: http://www.manning.com/ibsen/
Reply | Threaded
Open this post in threaded view
|

Re: Using ConsumerTemplate to fetch files.

dilip.pashupathi
Hi,
I was working with something similar to this problem. But I am not able to capture the exceptions properly.

Setps:
1. Download the existing file from the FTP
2. Push the file to backup folder and append the file name with current date
3. Push a new file to the FTP which has a complete new file

Here is gist of what I am trying to achieve

in RouteBuilder:
 public void configure() throws Exception {

        onException(Exception.class)
                .log("Exception returned due: ${exception.message}")
                .to(INTERNAL_ERROR_HANDLER_ENDPOINT_URI)
                .to(logError(PACKAGE_NAME))
                .handled(true);

       from(INTERNAL_ERROR_HANDLER_ENDPOINT_URI)
                .log("File error route triggered due to : ${exception?.class}")
                .transform().simple("Error ${exception.message}")
                .bean(responseTranslator, "translateError")
                .end();

       from(ENDPOINT_URI)
                .to(log(PACKAGE_NAME))
                .process(contentBackupProcessor)
                .to(INTERNAL_ENDPOINT_URI)
                .end();

       from(INTERNAL_ENDPOINT_URI)
                .process(requestProcessor)
                .to(ftpHostConfiguration.ftpEndpointForPublishing())
                .bean(responseTranslator, "translate")
                .to(log(PACKAGE_NAME));
    }

ContentBackupProcessor:

       logger.debug("File backup processor called.");
        CamelContext context = exchange.getContext();
        Endpoint ftp = context.getEndpoint(ftpHostConfiguration.ftpConsumerEndpointForBackupProcessor());
        PollingConsumer consumer = ftp.createPollingConsumer();
        logger.debug("FTP Consumer created for backup");
        Endpoint producerFtp = context.getEndpoint(ftpHostConfiguration.ftpProducerEndpointForBackupProcessor());
        ProducerTemplate template = exchange.getContext().createProducerTemplate();
        logger.debug("FTP Producer created for backup");
        consumer.start();
        while(true){
            Exchange result = consumer.receive(5000);
            if(result == null){
                break;
            }
            template.send(producerFtp, result);
        }
        consumer.stop();
    }

And this is how my FTP URL Looks like
URL 1: Consumer URL for downloading the file
ftp://XXXXC@localhost:8899/?password=XXXXC&consumer.bridgeErrorHandler=true&throwExceptionOnConnectFailed=true&maximumReconnectAttempts=0&useList=true&delete=true

URL 2: Producer URL for moving to backup folder and adding current date to the fileName
ftp://XXXXC@localhost:8899/backup?password=XXXXC&consumer.bridgeErrorHandler=true&throwExceptionOnConnectFailed=true&maximumReconnectAttempts=0&useList=true&fileName=${file:name.noext}_${date:now:yyyyMMdd.hhmmss}.js


ftp://XXXX@localhost:8899/?password=XXXX&consumer.bridgeErrorHandler=true&throwExceptionOnConnectFailed=true&maximumReconnectAttempts=0

Since URL 1 and URL 2 is written inside the processor when exception happens like unable to connect to FTP, onException is not able to handle that properly. And the route ends with a lot of stack trace.

Kindly help. This is a bit urgent since we have a service that needs to be delivered urgently.

Thanks & Regards
Dilip H Pashupathi
Reply | Threaded
Open this post in threaded view
|

Re: Using ConsumerTemplate to fetch files.

dilip.pashupathi
Hi,
I was working with something similar to this problem. But I am not able to capture the exceptions properly.

Setps:
1. Download the existing file from the FTP
2. Push the file to backup folder and append the file name with current date
3. Push a new file to the FTP which has a complete new file

Here is gist of what I am trying to achieve

in RouteBuilder:
 public void configure() throws Exception {

        onException(Exception.class)
                .log("Exception returned due: ${exception.message}")
                .to(INTERNAL_ERROR_HANDLER_ENDPOINT_URI)
                .to(logError(PACKAGE_NAME))
                .handled(true);

       from(INTERNAL_ERROR_HANDLER_ENDPOINT_URI)
                .log("File error route triggered due to : ${exception?.class}")
                .transform().simple("Error ${exception.message}")
                .bean(responseTranslator, "translateError")
                .end();

       from(ENDPOINT_URI)
                .to(log(PACKAGE_NAME))
                .process(contentBackupProcessor)
                .to(INTERNAL_ENDPOINT_URI)
                .end();

       from(INTERNAL_ENDPOINT_URI)
                .process(requestProcessor)
                .to(ftpHostConfiguration.ftpEndpointForPublishing())
                .bean(responseTranslator, "translate")
                .to(log(PACKAGE_NAME));
    }

ContentBackupProcessor:

       logger.debug("File backup processor called.");
        CamelContext context = exchange.getContext();
        Endpoint ftp = context.getEndpoint(ftpHostConfiguration.ftpConsumerEndpointForBackupProcessor());
        PollingConsumer consumer = ftp.createPollingConsumer();
        logger.debug("FTP Consumer created for backup");
        Endpoint producerFtp = context.getEndpoint(ftpHostConfiguration.ftpProducerEndpointForBackupProcessor());
        ProducerTemplate template = exchange.getContext().createProducerTemplate();
        logger.debug("FTP Producer created for backup");
        consumer.start();
        while(true){
            Exchange result = consumer.receive(5000);
            if(result == null){
                break;
            }
            template.send(producerFtp, result);
        }
        consumer.stop();
    }

And this is how my FTP URL Looks like
URL 1: Consumer URL for downloading the file
ftp://XXXXC@localhost:8899/?password=XXXXC&consumer.bridgeErrorHandler=true&throwExceptionOnConnectFailed=true&maximumReconnectAttempts=0&useList=true&delete=true

URL 2: Producer URL for moving to backup folder and adding current date to the fileName
ftp://XXXXC@localhost:8899/backup?password=XXXXC&consumer.bridgeErrorHandler=true&throwExceptionOnConnectFailed=true&maximumReconnectAttempts=0&useList=true&fileName=${file:name.noext}_${date:now:yyyyMMdd.hhmmss}.js


ftp://XXXX@localhost:8899/?password=XXXX&consumer.bridgeErrorHandler=true&throwExceptionOnConnectFailed=true&maximumReconnectAttempts=0

Since URL 1 and URL 2 is written inside the processor when exception happens like unable to connect to FTP, onException is not able to handle that properly. And the route ends with a lot of stack trace.

Kindly help. This is a bit urgent since we have a service that needs to be delivered urgently.

Thanks & Regards
Dilip H Pashupathi
Reply | Threaded
Open this post in threaded view
|

Re: Using ConsumerTemplate to fetch files.

Zoran Regvart-2
Hi Dilip,
this to me seems to be an overly complicated solution to a simple
problem. If I understand your problem correctly: you gather a file
from ftp server1 and push to two different ftp servers, serverA and
serverB, the serverA gets the transformed file and serverB gets the
original file (backup)?

Why not do simply:

from("ftp:server1").multicast().to("direct:backup", "direct:transform");
from("direct:transform").bean(responseTranslator,
"translate").to("ftp:serverA");
from("direct:backup").to("ftp:serverB");

Might not be 100% your use case, but you can see it's simpler with
multicast EIP[1]

zoran

[1] https://camel.apache.org/multicast.html

On Fri, Mar 31, 2017 at 4:08 PM, dilip.pashupathi
<[hidden email]> wrote:

> Hi,
> I was working with something similar to this problem. But I am not able to
> capture the exceptions properly.
>
> Setps:
> 1. Download the existing file from the FTP
> 2. Push the file to backup folder and append the file name with current date
> 3. Push a new file to the FTP which has a complete new file
>
> Here is gist of what I am trying to achieve
>
> in RouteBuilder:
>  public void configure() throws Exception {
>
>         onException(Exception.class)
>                 .log("Exception returned due: ${exception.message}")
>                 .to(INTERNAL_ERROR_HANDLER_ENDPOINT_URI)
>                 .to(logError(PACKAGE_NAME))
>                 .handled(true);
>
>        from(INTERNAL_ERROR_HANDLER_ENDPOINT_URI)
>                 .log("File error route triggered due to :
> ${exception?.class}")
>                 .transform().simple("Error ${exception.message}")
>                 .bean(responseTranslator, "translateError")
>                 .end();
>
>        from(ENDPOINT_URI)
>                 .to(log(PACKAGE_NAME))
>                 .process(contentBackupProcessor)
>                 .to(INTERNAL_ENDPOINT_URI)
>                 .end();
>
>        from(INTERNAL_ENDPOINT_URI)
>                 .process(requestProcessor)
>                 .to(ftpHostConfiguration.ftpEndpointForPublishing())
>                 .bean(responseTranslator, "translate")
>                 .to(log(PACKAGE_NAME));
>     }
>
> ContentBackupProcessor:
>
>        logger.debug("File backup processor called.");
>         CamelContext context = exchange.getContext();
>         Endpoint ftp =
> context.getEndpoint(ftpHostConfiguration.ftpConsumerEndpointForBackupProcessor());
>         PollingConsumer consumer = ftp.createPollingConsumer();
>         logger.debug("FTP Consumer created for backup");
>         Endpoint producerFtp =
> context.getEndpoint(ftpHostConfiguration.ftpProducerEndpointForBackupProcessor());
>         ProducerTemplate template =
> exchange.getContext().createProducerTemplate();
>         logger.debug("FTP Producer created for backup");
>         consumer.start();
>         while(true){
>             Exchange result = consumer.receive(5000);
>             if(result == null){
>                 break;
>             }
>             template.send(producerFtp, result);
>         }
>         consumer.stop();
>     }
>
> And this is how my FTP URL Looks like
> URL 1: Consumer URL for downloading the file
> ftp://XXXXC@localhost:8899/?password=XXXXC&consumer.bridgeErrorHandler=true&throwExceptionOnConnectFailed=true&maximumReconnectAttempts=0&useList=true&delete=true
>
> URL 2: Producer URL for moving to backup folder and adding current date to
> the fileName
> ftp://XXXXC@localhost:8899/backup?password=XXXXC&consumer.bridgeErrorHandler=true&throwExceptionOnConnectFailed=true&maximumReconnectAttempts=0&useList=true&fileName=${file:name.noext}_${date:now:yyyyMMdd.hhmmss}.js
>
>
> ftp://XXXX@localhost:8899/?password=XXXX&consumer.bridgeErrorHandler=true&throwExceptionOnConnectFailed=true&maximumReconnectAttempts=0
>
> Since URL 1 and URL 2 is written inside the processor when exception happens
> like unable to connect to FTP, onException is not able to handle that
> properly. And the route ends with a lot of stack trace.
>
> Kindly help. This is a bit urgent since we have a service that needs to be
> delivered urgently.
>
> Thanks & Regards
> Dilip H Pashupathi
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Using-ConsumerTemplate-to-fetch-files-tp3304671p5796786.html
> Sent from the Camel - Users mailing list archive at Nabble.com.



--
Zoran Regvart
Reply | Threaded
Open this post in threaded view
|

Re: Using ConsumerTemplate to fetch files.

dilip.pashupathi
Hi, Thanks for the reply,
The thing is we don't have multiple FTP servers. We have only one server where all operations happens.

1. At the root folder of the FTP, there is file that needs to be deleted
2. Before deleting that file, the file should be taken as a backup and placed in one of the subdirectory.
3. After that from the request I need read the data and produce a file that should be placed in the ftp root directory.

For all the above only a single FTP server is available and there are three different calls happening.
First call is to download the file from
ftp_site/somefile.js
Pass this on to
ftp_site/backup
Read the file from request params and place it in
ftp_site/somefile_update_<date_time>.js

Reply | Threaded
Open this post in threaded view
|

Re: Using ConsumerTemplate to fetch files.

dilip.pashupathi
Hi,

I also tried your suggestions like below

 from("ftp://username@localhost:8899/...<other_default_params>")
     .mulitcast().to("direct:processes-backup","direct:upload-fresh");

from("direct:process-backup")
   .bean(responseTranslator,"translate").


But when I started my jetty server where the service is deployed it stopped with an error stating that "Connection Refused".

I was trying to simulate if any exception happens then that should be caught and thrown by onException handler.

Here is is the basics of how this service is configured. We have three different layers each separated by their own configurations.
1. Application or App layer
2. Business - where logical stuff happens
3. Integration services - where the connection with backend is established and request is sent to third party services for further processing.

And for all this we are only using camels "direct" component only since we don't have any use for other components. And these are deployed on "JBoss 7" servers for actual integration test. And for local test we use Jetty Server.

And since starting up the Jetty server will ping all the services and see if the routes are all connectable, if any of them throws error our servers are configured to stop there till the problem is resolved.

hence I cannot use the "from" at the very beginning  to do a multicast as well.
Reply | Threaded
Open this post in threaded view
|

Re: Using ConsumerTemplate to fetch files.

dilip.pashupathi
Hi,

Basic idea here is to capture any exception that may occur due to failure in FTP connections or any other form of exception. Since the process is happening inside the contentBackupProcessor, Camel is basically ignoring the exception and continuing with the next operation as if nothing wrong happened.

Thanks
Dilip
12