looping http fetches of paginated data

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

looping http fetches of paginated data

Bob Jolliffe
I've got a problem which I can't quite figure out how to solve.  I
want to configure a route which fetches xml data (representing health
facilities) from a repository using a REST http get, transform the
data and load into another system.  Typically triggered from a quartz
endpoint running every 24 hours.

I have found that this is quite straightforward using quartz, http and
xslt components.  But ... the data coming from the repository is
paginated so I can't fetch it in a single request.

Is there anyway to create a route in camel DSL (xml or java) which
makes a series of requests to the http service, incrementing an http
parameter eg. Page=xx, until some test condition is true in the xml
result eg.  count('/Facilities/Facility')=0

Thanks for any pointers.

Regards
Bob
Reply | Threaded
Open this post in threaded view
|

Re: looping http fetches of paginated data

pontus.ullgren
Hello,

You can use loop (http://camel.apache.org/loop.html) and storing
information from each request in the exchange.

However I personally like to keep my camel routes as simple as
possible and avoid building logic such as this into the routes.
So I would recommend you to create your own processor that does all
the data collection and then enriches the message with the complete
message.

// Pontus



On Sun, Jul 8, 2012 at 3:40 PM, Bob Jolliffe <[hidden email]> wrote:

> I've got a problem which I can't quite figure out how to solve.  I
> want to configure a route which fetches xml data (representing health
> facilities) from a repository using a REST http get, transform the
> data and load into another system.  Typically triggered from a quartz
> endpoint running every 24 hours.
>
> I have found that this is quite straightforward using quartz, http and
> xslt components.  But ... the data coming from the repository is
> paginated so I can't fetch it in a single request.
>
> Is there anyway to create a route in camel DSL (xml or java) which
> makes a series of requests to the http service, incrementing an http
> parameter eg. Page=xx, until some test condition is true in the xml
> result eg.  count('/Facilities/Facility')=0
>
> Thanks for any pointers.
>
> Regards
> Bob
Reply | Threaded
Open this post in threaded view
|

Re: looping http fetches of paginated data

Sam (Stephen Samuel)
In reply to this post by Bob Jolliffe
What I would do is have a queue that triggers a HTTP fetch. The message on
this queue contains the pagination info, eg the start and limit. After each
request, it can then put another message back on the queue with updated
pagination info if there is more left to process. The initial quartz job
can put the "start" message onto the queue to begin the process. This queue
doesn't need to be anything fancy, it could just be a seda
http://camel.apache.org/seda.html

On Sun, Jul 8, 2012 at 2:40 PM, Bob Jolliffe <[hidden email]> wrote:

> I've got a problem which I can't quite figure out how to solve.  I
> want to configure a route which fetches xml data (representing health
> facilities) from a repository using a REST http get, transform the
> data and load into another system.  Typically triggered from a quartz
> endpoint running every 24 hours.
>
> I have found that this is quite straightforward using quartz, http and
> xslt components.  But ... the data coming from the repository is
> paginated so I can't fetch it in a single request.
>
> Is there anyway to create a route in camel DSL (xml or java) which
> makes a series of requests to the http service, incrementing an http
> parameter eg. Page=xx, until some test condition is true in the xml
> result eg.  count('/Facilities/Facility')=0
>
> Thanks for any pointers.
>
> Regards
> Bob
>



--
-Sam
Reply | Threaded
Open this post in threaded view
|

Re: looping http fetches of paginated data

pontus.ullgren
In reply to this post by pontus.ullgren
Just realised that you did not have the number of pages that you
needed to fetch before hand but wanted to loop based on a condition.
Then loop will not help you.

I would go with creating your own processor that does all the collection.

// Pontus

On Mon, Jul 9, 2012 at 1:07 PM, Pontus Ullgren <[hidden email]> wrote:

> Hello,
>
> You can use loop (http://camel.apache.org/loop.html) and storing
> information from each request in the exchange.
>
> However I personally like to keep my camel routes as simple as
> possible and avoid building logic such as this into the routes.
> So I would recommend you to create your own processor that does all
> the data collection and then enriches the message with the complete
> message.
>
> // Pontus
>
>
>
> On Sun, Jul 8, 2012 at 3:40 PM, Bob Jolliffe <[hidden email]> wrote:
>> I've got a problem which I can't quite figure out how to solve.  I
>> want to configure a route which fetches xml data (representing health
>> facilities) from a repository using a REST http get, transform the
>> data and load into another system.  Typically triggered from a quartz
>> endpoint running every 24 hours.
>>
>> I have found that this is quite straightforward using quartz, http and
>> xslt components.  But ... the data coming from the repository is
>> paginated so I can't fetch it in a single request.
>>
>> Is there anyway to create a route in camel DSL (xml or java) which
>> makes a series of requests to the http service, incrementing an http
>> parameter eg. Page=xx, until some test condition is true in the xml
>> result eg.  count('/Facilities/Facility')=0
>>
>> Thanks for any pointers.
>>
>> Regards
>> Bob
Reply | Threaded
Open this post in threaded view
|

Re: looping http fetches of paginated data

Claus Ibsen-2
On Mon, Jul 9, 2012 at 1:11 PM, Pontus Ullgren <[hidden email]> wrote:
> Just realised that you did not have the number of pages that you
> needed to fetch before hand but wanted to loop based on a condition.
> Then loop will not help you.
>

Yeah in the future we will improve loop to act like a while where you
can evaluate an expression.


> I would go with creating your own processor that does all the collection.
>

Yeah that would be fine. Or use the dynamic router EIP, where you can just keep
routing back to the http.
http://camel.apache.org/dynamic-router.html


> // Pontus
>
> On Mon, Jul 9, 2012 at 1:07 PM, Pontus Ullgren <[hidden email]> wrote:
>> Hello,
>>
>> You can use loop (http://camel.apache.org/loop.html) and storing
>> information from each request in the exchange.
>>
>> However I personally like to keep my camel routes as simple as
>> possible and avoid building logic such as this into the routes.
>> So I would recommend you to create your own processor that does all
>> the data collection and then enriches the message with the complete
>> message.
>>
>> // Pontus
>>
>>
>>
>> On Sun, Jul 8, 2012 at 3:40 PM, Bob Jolliffe <[hidden email]> wrote:
>>> I've got a problem which I can't quite figure out how to solve.  I
>>> want to configure a route which fetches xml data (representing health
>>> facilities) from a repository using a REST http get, transform the
>>> data and load into another system.  Typically triggered from a quartz
>>> endpoint running every 24 hours.
>>>
>>> I have found that this is quite straightforward using quartz, http and
>>> xslt components.  But ... the data coming from the repository is
>>> paginated so I can't fetch it in a single request.
>>>
>>> Is there anyway to create a route in camel DSL (xml or java) which
>>> makes a series of requests to the http service, incrementing an http
>>> parameter eg. Page=xx, until some test condition is true in the xml
>>> result eg.  count('/Facilities/Facility')=0
>>>
>>> Thanks for any pointers.
>>>
>>> Regards
>>> Bob



--
Claus Ibsen
-----------------
FuseSource
Email: [hidden email]
Web: http://fusesource.com
Twitter: davsclaus, fusenews
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen