Quantcast

Using Split with JSON File

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Using Split with JSON File

andcrow
This post was updated on .
Hello,

I have a route that triggers a SELECT in a database and creates a JSON file in a folder (this file which on average generates something around 100k to 150k of records).

Then another route that sends the data of this file to an endpoint using the HTTP4 component and performing a POST and REST API (the endpoint is already prepared to perform batch processing).

I noticed something curious, that in certain cases Camel sends the data, Web Services is processing correctly, but after a few seconds Camel returns me a DeadLetterChannel, with some exceptions:

 - "org.apache.http.wire" "http-outgoing-5" "end of stream"

 - "org.apache.http.nohttpresponseexception" "failed to respond"

 - "h.i.c.DefaultManagedHttpClientConnection" "http-outgoing-5" "Close connection"

 - "h.i.c.DefaultManagedHttpClientConnection" "http-outgoing-5" "Shutdown connection"

 - "o.a.http.impl.execchain.MainClientExec" "Connection discarded" "Connection released"

But the process continues normally, it tries to resend again, but gives the error again, but in AWS web services it is processing normally and I have verified that it returns the normal status codes.


Searching the Camel documentation I found the httpClient. I made the configuration by increasing the timeout, but nothing worked, it seems that it ignores these settings (httpClient.socketTimeout = 300000).

I'm thinking of splitting the JSON file, with some rule like every 1000 records generating 1 JSON file. So I will have several smaller files that will be processed faster and thus get the response from the server, avoiding this error.

My actual route thats find all records in database and create a json file is:

from("timer://query?fixedRate=true&delay=5s&period=10800s")
    .routeId("route-product")
    .setBody(constant(new ProductQuery().getProductSelect()))
.to("jdbc:myDataSource")
    .marshal()
        .json(JsonLibrary.Gson)
        .setHeader(Exchange.CONTENT_TYPE, constant("application/json"))
        .setHeader(Exchange.HTTP_CHARACTER_ENCODING, constant("UTF-8"))
        .setHeader("CamelFileName", constant("products.json"))
.to("file:products");

However, searching the documentation, I was able to perform the split:

from("timer://query?fixedRate=true&delay=5s&period=10800s")
    .routeId("route-product")
    .setBody(constant(new ProductQuery().getProductSelect()))
.to("jdbc:myDataSource?outputType=StreamList")
    .split(body()).streaming()
    .marshal()
        .json(JsonLibrary.Gson)
        .setHeader(Exchange.CONTENT_TYPE, constant("application/json"))
        .setHeader("CamelFileName", simple("${id}.json"))
.to("file:product");

In this way each record creates a JSON file, but in my case, I want to group these records, again in the documentation is suggested as follows:

.split().tokenize("\n", 1000).streaming()

But in this way the output of the log shows only 1 file with the following content:

"org.apache.camel.component.jdbc.ResultSetIterator@7be7d81f"

If I remove the option ?outputType=StreamList from the JDBC component's URI, then the following exception is thrown:

No converter found capable of converting from type [java.util.ArrayList<?>] to type [java.lang.String]
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Using Split with JSON File

andcrow
Still in the battle with this case ..... hehehehe

Viewing the documentation I found the component SQL Component:

In the options:

maxMessagesPerPoll: Camel 2.11: SQL consumer only: An integer value to define the maximum number of messages to gather per poll. By default, no maximum is set.

But also with this component he ignores this option:

from("timer://query?fixedRate=true&delay=5s&period=10800s")
    .routeId("route-product")
    .setBody(constant(new ProductQuery().getProductSelect()))
to("sql:" + new ProductQuery().getProductSelect() + "?maxMessagesPerPoll=1000&dataSource=myDataSource")
    .marshal()
        .json(JsonLibrary.Gson)
        .setHeader(Exchange.CONTENT_TYPE, constant("application/json"))
        .setHeader("CamelFileName", simple("${id}.json"))
.to("file:product");
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Using Split with JSON File

souciance
Couldn't you split the resultset from the database and for each split produce a json and send that batch to your web service?

On Mon, Apr 17, 2017 at 6:45 PM, andcrow [via Camel] <[hidden email]> wrote:
Still in the battle with this case ..... hehehehe

Viewing the documentation I found the component SQL Component:

In the options:

maxMessagesPerPoll: Camel 2.11: SQL consumer only: An integer value to define the maximum number of messages to gather per poll. By default, no maximum is set.

But also with this component he ignores this option:

from("timer://query?fixedRate=true&delay=5s&period=10800s")
    .routeId("route-product")
    .setBody(constant(new ProductQuery().getProductSelect()))
to("sql:" + new ProductQuery().getProductSelect() + "?maxMessagesPerPoll=1000&dataSource=myDataSource")
    .marshal()
        .json(JsonLibrary.Gson)
        .setHeader(Exchange.CONTENT_TYPE, constant("application/json"))
        .setHeader("CamelFileName", simple("${id}.json"))
.to("file:product");


If you reply to this email, your message will be added to the discussion below:
http://camel.465427.n5.nabble.com/Using-Split-with-JSON-File-tp5797508p5797770.html
To start a new topic under Camel - Users, email [hidden email]
To unsubscribe from Camel - Users, click here.
NAML

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Using Split with JSON File

andcrow
Did not work using maxMessagesPerPoll.

Does not group every 1000 records.

Creates 1 JSON file for each record.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Using Split with JSON File

souciance
I would suggest get the whole resultset and save the data on a queue or even as a file. Then you can parse it as a stream or every x number of rows and split.

On Mon, Apr 17, 2017 at 7:49 PM, andcrow [via Camel] <[hidden email]> wrote:
Did not work using maxMessagesPerPoll.

Does not group every 1000 records.

Creates 1 JSON file for each record.


If you reply to this email, your message will be added to the discussion below:
http://camel.465427.n5.nabble.com/Using-Split-with-JSON-File-tp5797508p5797772.html
To start a new topic under Camel - Users, email [hidden email]
To unsubscribe from Camel - Users, click here.
NAML

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Using Split with JSON File

andcrow
I'll try
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Using Split with JSON File

andcrow
In reply to this post by souciance
Does Apache Camel have any component that can perform auditing on a database? Or audit records that will be sent to a web service using a JSON file?
Loading...