Aggregator and HawtDB - DB just seem to grow?

[mikaelfj]

Hi,

We are using the aggregator EIP to collect a set of files from which we are to generate other documents.

In order not to loose any of the aggregated documents, we are using the aggregator with the HawtDB as aggregation-repository.

This has been running flawless for a few months, but two weeks ago we suddenly began getting:
org.fusesource.hawtdb.api.IOPagingException: Invalid extent read request.  The requested page was not an extent: xxxx

This was "fixed" by extending the buffersize from the default 8Mb to 32Mb.

Now the problem has arisen again and we have the same kind of exceptions in the logs.

None of the file sets that are read are very big in size, so it is unlikely that a single file set would cause the HawtDB to go pass its limit.

Can it be that the HawtDB just grows?
... or did I miss something in the configuration of the route?
When reading the documentation in "The Book" it says that the Aggregator will issue a Commit, when the message has been processed successfully. This will cause the RecoverableAggregationRepository to Confirm the message, thus ensuring it will not be recovered.
But, will the Confirm remove the message or do I have to perform a cleanup of successfully processed messages in order to keep the HawtDB from growing?

Best regards
Mikael

[Claus Ibsen-2]

What version of hawtdb and Camel are you using?

There was a bug a while back, that caused it to grow. That was fixed
in a recent release of hawtdb.


On Mon, Mar 19, 2012 at 12:06 PM, mikaelfj <[hidden email]> wrote:

> Hi,
>
> We are using the aggregator EIP to collect a set of files from which we are
> to generate other documents.
>
> In order not to loose any of the aggregated documents, we are using the
> aggregator with the HawtDB as aggregation-repository.
>
> This has been running flawless for a few months, but two weeks ago we
> suddenly began getting:
> org.fusesource.hawtdb.api.IOPagingException: Invalid extent read request.
> The requested page was not an extent: xxxx
>
> This was "fixed" by extending the buffersize from the default 8Mb to 32Mb.
>
> Now the problem has arisen again and we have the same kind of exceptions in
> the logs.
>
> None of the file sets that are read are very big in size, so it is unlikely
> that a single file set would cause the HawtDB to go pass its limit.
>
> Can it be that the HawtDB just grows?
> ... or did I miss something in the configuration of the route?
> When reading the documentation in "The Book" it says that the Aggregator
> will issue a Commit, when the message has been processed successfully. This
> will cause the RecoverableAggregationRepository to Confirm the message, thus
> ensuring it will not be recovered.
> But, will the Confirm remove the message or do I have to perform a cleanup
> of successfully processed messages in order to keep the HawtDB from growing?
>
> Best regards
> Mikael
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Aggregator-and-HawtDB-DB-just-seem-to-grow-tp5576730p5576730.html
> Sent from the Camel - Users mailing list archive at Nabble.com.

--
Claus Ibsen
-----------------
CamelOne 2012 Conference, May 15-16, 2012: http://camelone.com
FuseSource
Email: [hidden email]
Web: http://fusesource.com
Twitter: davsclaus, fusenews
Blog: http://davsclaus.blogspot.com/
Author of Camel in Action: http://www.manning.com/ibsen/

[mikaelfj]

We are using Camel 2.6 and hawtdb-1.5.jar and hawtbuf-1.2.jar

[Claus Ibsen-2]

On Mon, Mar 19, 2012 at 12:45 PM, mikaelfj <[hidden email]> wrote:
> We are using Camel 2.6 and hawtdb-1.5.jar and hawtbuf-1.2.jar
>

There is newer releases of those.

And yes the confirm will cleanup the resources. See the source code in
camel-hawtdb.


>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Aggregator-and-HawtDB-DB-just-seem-to-grow-tp5576730p5576856.html
> Sent from the Camel - Users mailing list archive at Nabble.com.



--
Claus Ibsen
-----------------
CamelOne 2012 Conference, May 15-16, 2012: http://camelone.com
FuseSource
Email: [hidden email]
Web: http://fusesource.com
Twitter: davsclaus, fusenews
Blog: http://davsclaus.blogspot.com/
Author of Camel in Action: http://www.manning.com/ibsen/

[mikaelfj]

I'm stuck on Camel 2.6, but I can get hawtdb 1.6 and hawtbuf 1.4, which, judging from my unit-tests, seems to work with Camel 2.6.

Are you aware of any issues that are related to my problem, that is resolved in hawtdb 1.6?

Best regards
Mikael

[Claus Ibsen-2]

On Mon, Mar 19, 2012 at 1:36 PM, mikaelfj <[hidden email]> wrote:
> I'm stuck on Camel 2.6, but I can get hawtdb 1.6 and hawtbuf 1.4, which,
> judging from my unit-tests, seems to work with Camel 2.6.
>
> Are you aware of any issues that are related to my problem, that is resolved
> in hawtdb 1.6?
>

I am not aware of the changes between 1.5 and 1.6.
There is a user mailing list for hawtdb you can ask.

> Best regards
> Mikael
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Aggregator-and-HawtDB-DB-just-seem-to-grow-tp5576730p5576974.html
> Sent from the Camel - Users mailing list archive at Nabble.com.



--
Claus Ibsen
-----------------
CamelOne 2012 Conference, May 15-16, 2012: http://camelone.com
FuseSource
Email: [hidden email]
Web: http://fusesource.com
Twitter: davsclaus, fusenews
Blog: http://davsclaus.blogspot.com/
Author of Camel in Action: http://www.manning.com/ibsen/