Quantcast

Flatpack and aggregation

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Flatpack and aggregation

Tom.Fornoville
This post was updated on .
I'm relatively new to Camel and guessing my problem can be solved with a combination of FlatPack and an aggregation but I'm not seeing how I can make it work without writing a lot of code.

It starts with a fixed width file with 2 columns (actually more but only 2 are relevant): action_id and organisation_id. The file contains a couple thousand lines and can contain duplicates.

2 things needed:

1. filter out lines for a specific action_id (only action_id 70 is of interest for us)
2. deduplicate based on organisation_id (the same organisation can have appear on multiple lines)

It will be greatly appreciated if anyone has a solution or can point me in the right direction .

Cheers,
 Tom
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Flatpack and aggregation

saltnlight5
Hello Tom,

I think you can try flatpack with one of the idempotent consumer (http://camel.apache.org/idempotent-consumer.html) to eliminate duplicated messages. For example, maybe something like this:

  from("file:inbox")
    .to("flatpack:fixed:yourdata.pzmap.xml")
    .filter(header("action_id"))
    .idempotentConsumer(header("organisation_id"), MemoryIdempotentRepository.memoryIdempotentRepository(200))

Also alternative to flatpack, you may also checkout Camel bindy: http://camel.apache.org/bindy.html to parse fixed length records.

Have fun,
Zemian
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Flatpack and aggregation

Tom.Fornoville
This post was updated on .
Hi Zemian,

Thanks for putting me on the right track with the idempotentConsumer!

Since Flatpack puts the records in a Java Map in the Body I had to do a
convertTo and ended up with the following:

<bean id="organisationMemoryRepository"
class="org.apache.camel.processor.idempotent.MemoryIdempotentRepository" />
...
<route id="process-line">
<from uri="flatpack:fixed:kbo.pzmap.xml" />
<convertBodyTo type="java.util.Map"/>
<filter>
<ognl>request.body["ACTION"] == "070"</ognl>
<idempotentConsumer messageIdRepositoryRef="organisationMemoryRepository">
<ognl>request.body["ORGANISATION"]</ognl>
<to uri="log:be.roots.organisation?level=INFO" />

</idempotentConsumer>
</filter>
</route>

This seems to work as expected :-)

Cheers,
Tom
Loading...