Re: http4 not sending exchange data further in route for kubernetes API

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: http4 not sending exchange data further in route for kubernetes API

Andrea Cosentino-3
Why are you trying to watch a configmap with http component, when there is
a consumer for the events related to specific configmap?

Il sab 14 set 2019, 18:13 Imran Raza Khan <[hidden email]> ha
scritto:

> I am using [kubernetes rest api][1] API to watch configmap and calling it
> with http4 component, but http4 is not forwarding exchange further in
> route, from logs i observed that http4 keep getting updates from openshift
> on first open connection as its not closing connection so dont forward
> exchange further, how i can get this data down in route
>
>     from("timer://foo?fixedRate=true&period=60000")
>     .setHeader("Authorization", simple("Bearer xxx"))
>     .setHeader("Accept", simple("application/json"))
>     .setHeader("CamelHttpMethod", constant("GET"))
>
>
> .to("https4://localhost:8443/api/v1/watch/namespaces/my-qa/configmaps/my-config-map?httpClientConfigurer=#trustCert")
>     .to("log:my?showAll=true&multiline=true")
>     .to("mock:result");
>
> Trace:
>
>     [mel-1) thread #2 - timer://foo] headers                        DEBUG
> http-outgoing-0 >> GET
> /api/v1/watch/namespaces/my-qa/configmaps/my-config-map HTTP/1.1
>     [mel-1) thread #2 - timer://foo] headers                        DEBUG
> http-outgoing-0 >> Accept: application/json
>     [mel-1) thread #2 - timer://foo] headers                        DEBUG
> http-outgoing-0 >> Authorization: Bearer xxx
>     [mel-1) thread #2 - timer://foo] headers                        DEBUG
> http-outgoing-0 >> breadcrumbId: ID-NB-118-1568476509695-0-1
>     [mel-1) thread #2 - timer://foo] headers                        DEBUG
> http-outgoing-0 >> firedTime: Sat Sep 14 17:55:14 CEST 2019
>     [mel-1) thread #2 - timer://foo] headers                        DEBUG
> http-outgoing-0 >> Host: localhost:8443
>     [mel-1) thread #2 - timer://foo] headers                        DEBUG
> http-outgoing-0 >> Connection: Keep-Alive
>     [mel-1) thread #2 - timer://foo] headers                        DEBUG
> http-outgoing-0 >> User-Agent: Apache-HttpClient/4.5.6 (Java/1.8.0_162)
>     [mel-1) thread #2 - timer://foo] headers                        DEBUG
> http-outgoing-0 >> Accept-Encoding: gzip,deflate
>     [mel-1) thread #2 - timer://foo] wire                           DEBUG
> http-outgoing-0 >> "GET
> /api/v1/watch/namespaces/my-qa/configmaps/my-config-map HTTP/1.1[\r][\n]"
>     [mel-1) thread #2 - timer://foo] wire                           DEBUG
> http-outgoing-0 >> "Accept: application/json[\r][\n]"
>     [mel-1) thread #2 - timer://foo] wire                           DEBUG
> http-outgoing-0 >> "Authorization: Bearer xxx[\r][\n]"
>     [mel-1) thread #2 - timer://foo] wire                           DEBUG
> http-outgoing-0 >> "breadcrumbId: ID-NB-118-1568476509695-0-1[\r][\n]"
>     [mel-1) thread #2 - timer://foo] wire                           DEBUG
> http-outgoing-0 >> "firedTime: Sat Sep 14 17:55:14 CEST 2019[\r][\n]"
>     [mel-1) thread #2 - timer://foo] wire                           DEBUG
> http-outgoing-0 >> "Host: m.devokd.younicos.local:8443[\r][\n]"
>     [mel-1) thread #2 - timer://foo] wire                           DEBUG
> http-outgoing-0 >> "Connection: Keep-Alive[\r][\n]"
>     [mel-1) thread #2 - timer://foo] wire                           DEBUG
> http-outgoing-0 >> "User-Agent: Apache-HttpClient/4.5.6
> (Java/1.8.0_162)[\r][\n]"
>     [mel-1) thread #2 - timer://foo] wire                           DEBUG
> http-outgoing-0 >> "Accept-Encoding: gzip,deflate[\r][\n]"
>     [mel-1) thread #2 - timer://foo] wire                           DEBUG
> http-outgoing-0 >> "[\r][\n]"
>     [mel-1) thread #2 - timer://foo] wire                           DEBUG
> http-outgoing-0 << "HTTP/1.1 200 OK[\r][\n]"
>     [mel-1) thread #2 - timer://foo] wire                           DEBUG
> http-outgoing-0 << "Cache-Control: no-store[\r][\n]"
>     [mel-1) thread #2 - timer://foo] wire                           DEBUG
> http-outgoing-0 << "Content-Type: application/json[\r][\n]"
>     [mel-1) thread #2 - timer://foo] wire                           DEBUG
> http-outgoing-0 << "Date: Sat, 14 Sep 2019 15:55:14 GMT[\r][\n]"
>     [mel-1) thread #2 - timer://foo] wire                           DEBUG
> http-outgoing-0 << "Transfer-Encoding: chunked[\r][\n]"
>     [mel-1) thread #2 - timer://foo] wire                           DEBUG
> http-outgoing-0 << "[\r][\n]"
>     [mel-1) thread #2 - timer://foo] headers                        DEBUG
> http-outgoing-0 << HTTP/1.1 200 OK
>     [mel-1) thread #2 - timer://foo] headers                        DEBUG
> http-outgoing-0 << Cache-Control: no-store
>     [mel-1) thread #2 - timer://foo] headers                        DEBUG
> http-outgoing-0 << Content-Type: application/json
>     [mel-1) thread #2 - timer://foo] headers                        DEBUG
> http-outgoing-0 << Date: Sat, 14 Sep 2019 15:55:14 GMT
>     [mel-1) thread #2 - timer://foo] headers                        DEBUG
> http-outgoing-0 << Transfer-Encoding: chunked
>     [mel-1) thread #2 - timer://foo] MainClientExec                 DEBUG
> Connection can be kept alive indefinitely
>     [mel-1) thread #2 - timer://foo] HttpProducer                   DEBUG
> Http responseCode: 200
>     [mel-1) thread #2 - timer://foo] IOHelper                       TRACE
> Copying InputStream: org.apache.http.conn.EofSensorInputStream@3ab22477 ->
> OutputStream: CachedOutputStream[size: 0] with buffer: 4096 and flush on
> each write false
>     [mel-1) thread #2 - timer://foo] wire                           DEBUG
> http-outgoing-0 << "158[\r][\n]"
>     [mel-1) thread #2 - timer://foo] wire                           DEBUG
> http-outgoing-0 <<
>
> "{"type":"ADDED","object":{"kind":"ConfigMap","apiVersion":"v1","metadata":{"name":"my-config-map","namespace":"my-qa","selfLink":"/api/v1/namespaces/my-qa/configmaps/my-config-map","uid":"a8130369-d703-11e9-b37c-d2c8065c1507","resourceVersion":"33040706","creationTimestamp":"2019-09-14T15:23:51Z"},"data":{"test":"added
> first modification"}}}[\n]"
>     [mel-1) thread #2 - timer://foo] wire                           DEBUG
> http-outgoing-0 << "[\r][\n]"
>     [mel-1) thread #2 - timer://foo] wire                           DEBUG
> http-outgoing-0 << "156[\r][\n]"
>     [mel-1) thread #2 - timer://foo] wire                           DEBUG
> http-outgoing-0 <<
>
> "{"type":"MODIFIED","object":{"kind":"ConfigMap","apiVersion":"v1","metadata":{"name":"my-config-map","namespace":"yq-qa","selfLink":"/api/v1/namespaces/my-qa/configmaps/my-config-map","uid":"a8130369-d703-11e9-b37c-d2c8065c1507","resourceVersion":"33044977","creationTimestamp":"2019-09-14T15:23:51Z"},"data":{"test":"second
> modification"}}}[\n]"
>     [mel-1) thread #2 - timer://foo] wire                           DEBUG
> http-outgoing-0 << "[\r][\n]"
>
>   [1]:
>
> https://docs.openshift.com/container-platform/3.11/rest_api/api/v1.ConfigMap.html
>
Reply | Threaded
Open this post in threaded view
|

Re: http4 not sending exchange data further in route for kubernetes API

Andrea Cosentino-3
Use the Kubernetes component and consume events.

Il sab 14 set 2019, 19:25 Andrea Cosentino <[hidden email]> ha scritto:

> Why are you trying to watch a configmap with http component, when there is
> a consumer for the events related to specific configmap?
>
> Il sab 14 set 2019, 18:13 Imran Raza Khan <[hidden email]> ha
> scritto:
>
>> I am using [kubernetes rest api][1] API to watch configmap and calling it
>> with http4 component, but http4 is not forwarding exchange further in
>> route, from logs i observed that http4 keep getting updates from openshift
>> on first open connection as its not closing connection so dont forward
>> exchange further, how i can get this data down in route
>>
>>     from("timer://foo?fixedRate=true&period=60000")
>>     .setHeader("Authorization", simple("Bearer xxx"))
>>     .setHeader("Accept", simple("application/json"))
>>     .setHeader("CamelHttpMethod", constant("GET"))
>>
>>
>> .to("https4://localhost:8443/api/v1/watch/namespaces/my-qa/configmaps/my-config-map?httpClientConfigurer=#trustCert")
>>     .to("log:my?showAll=true&multiline=true")
>>     .to("mock:result");
>>
>> Trace:
>>
>>     [mel-1) thread #2 - timer://foo] headers                        DEBUG
>> http-outgoing-0 >> GET
>> /api/v1/watch/namespaces/my-qa/configmaps/my-config-map HTTP/1.1
>>     [mel-1) thread #2 - timer://foo] headers                        DEBUG
>> http-outgoing-0 >> Accept: application/json
>>     [mel-1) thread #2 - timer://foo] headers                        DEBUG
>> http-outgoing-0 >> Authorization: Bearer xxx
>>     [mel-1) thread #2 - timer://foo] headers                        DEBUG
>> http-outgoing-0 >> breadcrumbId: ID-NB-118-1568476509695-0-1
>>     [mel-1) thread #2 - timer://foo] headers                        DEBUG
>> http-outgoing-0 >> firedTime: Sat Sep 14 17:55:14 CEST 2019
>>     [mel-1) thread #2 - timer://foo] headers                        DEBUG
>> http-outgoing-0 >> Host: localhost:8443
>>     [mel-1) thread #2 - timer://foo] headers                        DEBUG
>> http-outgoing-0 >> Connection: Keep-Alive
>>     [mel-1) thread #2 - timer://foo] headers                        DEBUG
>> http-outgoing-0 >> User-Agent: Apache-HttpClient/4.5.6 (Java/1.8.0_162)
>>     [mel-1) thread #2 - timer://foo] headers                        DEBUG
>> http-outgoing-0 >> Accept-Encoding: gzip,deflate
>>     [mel-1) thread #2 - timer://foo] wire                           DEBUG
>> http-outgoing-0 >> "GET
>> /api/v1/watch/namespaces/my-qa/configmaps/my-config-map HTTP/1.1[\r][\n]"
>>     [mel-1) thread #2 - timer://foo] wire                           DEBUG
>> http-outgoing-0 >> "Accept: application/json[\r][\n]"
>>     [mel-1) thread #2 - timer://foo] wire                           DEBUG
>> http-outgoing-0 >> "Authorization: Bearer xxx[\r][\n]"
>>     [mel-1) thread #2 - timer://foo] wire                           DEBUG
>> http-outgoing-0 >> "breadcrumbId: ID-NB-118-1568476509695-0-1[\r][\n]"
>>     [mel-1) thread #2 - timer://foo] wire                           DEBUG
>> http-outgoing-0 >> "firedTime: Sat Sep 14 17:55:14 CEST 2019[\r][\n]"
>>     [mel-1) thread #2 - timer://foo] wire                           DEBUG
>> http-outgoing-0 >> "Host: m.devokd.younicos.local:8443[\r][\n]"
>>     [mel-1) thread #2 - timer://foo] wire                           DEBUG
>> http-outgoing-0 >> "Connection: Keep-Alive[\r][\n]"
>>     [mel-1) thread #2 - timer://foo] wire                           DEBUG
>> http-outgoing-0 >> "User-Agent: Apache-HttpClient/4.5.6
>> (Java/1.8.0_162)[\r][\n]"
>>     [mel-1) thread #2 - timer://foo] wire                           DEBUG
>> http-outgoing-0 >> "Accept-Encoding: gzip,deflate[\r][\n]"
>>     [mel-1) thread #2 - timer://foo] wire                           DEBUG
>> http-outgoing-0 >> "[\r][\n]"
>>     [mel-1) thread #2 - timer://foo] wire                           DEBUG
>> http-outgoing-0 << "HTTP/1.1 200 OK[\r][\n]"
>>     [mel-1) thread #2 - timer://foo] wire                           DEBUG
>> http-outgoing-0 << "Cache-Control: no-store[\r][\n]"
>>     [mel-1) thread #2 - timer://foo] wire                           DEBUG
>> http-outgoing-0 << "Content-Type: application/json[\r][\n]"
>>     [mel-1) thread #2 - timer://foo] wire                           DEBUG
>> http-outgoing-0 << "Date: Sat, 14 Sep 2019 15:55:14 GMT[\r][\n]"
>>     [mel-1) thread #2 - timer://foo] wire                           DEBUG
>> http-outgoing-0 << "Transfer-Encoding: chunked[\r][\n]"
>>     [mel-1) thread #2 - timer://foo] wire                           DEBUG
>> http-outgoing-0 << "[\r][\n]"
>>     [mel-1) thread #2 - timer://foo] headers                        DEBUG
>> http-outgoing-0 << HTTP/1.1 200 OK
>>     [mel-1) thread #2 - timer://foo] headers                        DEBUG
>> http-outgoing-0 << Cache-Control: no-store
>>     [mel-1) thread #2 - timer://foo] headers                        DEBUG
>> http-outgoing-0 << Content-Type: application/json
>>     [mel-1) thread #2 - timer://foo] headers                        DEBUG
>> http-outgoing-0 << Date: Sat, 14 Sep 2019 15:55:14 GMT
>>     [mel-1) thread #2 - timer://foo] headers                        DEBUG
>> http-outgoing-0 << Transfer-Encoding: chunked
>>     [mel-1) thread #2 - timer://foo] MainClientExec                 DEBUG
>> Connection can be kept alive indefinitely
>>     [mel-1) thread #2 - timer://foo] HttpProducer                   DEBUG
>> Http responseCode: 200
>>     [mel-1) thread #2 - timer://foo] IOHelper                       TRACE
>> Copying InputStream: org.apache.http.conn.EofSensorInputStream@3ab22477
>> ->
>> OutputStream: CachedOutputStream[size: 0] with buffer: 4096 and flush on
>> each write false
>>     [mel-1) thread #2 - timer://foo] wire                           DEBUG
>> http-outgoing-0 << "158[\r][\n]"
>>     [mel-1) thread #2 - timer://foo] wire                           DEBUG
>> http-outgoing-0 <<
>>
>> "{"type":"ADDED","object":{"kind":"ConfigMap","apiVersion":"v1","metadata":{"name":"my-config-map","namespace":"my-qa","selfLink":"/api/v1/namespaces/my-qa/configmaps/my-config-map","uid":"a8130369-d703-11e9-b37c-d2c8065c1507","resourceVersion":"33040706","creationTimestamp":"2019-09-14T15:23:51Z"},"data":{"test":"added
>> first modification"}}}[\n]"
>>     [mel-1) thread #2 - timer://foo] wire                           DEBUG
>> http-outgoing-0 << "[\r][\n]"
>>     [mel-1) thread #2 - timer://foo] wire                           DEBUG
>> http-outgoing-0 << "156[\r][\n]"
>>     [mel-1) thread #2 - timer://foo] wire                           DEBUG
>> http-outgoing-0 <<
>>
>> "{"type":"MODIFIED","object":{"kind":"ConfigMap","apiVersion":"v1","metadata":{"name":"my-config-map","namespace":"yq-qa","selfLink":"/api/v1/namespaces/my-qa/configmaps/my-config-map","uid":"a8130369-d703-11e9-b37c-d2c8065c1507","resourceVersion":"33044977","creationTimestamp":"2019-09-14T15:23:51Z"},"data":{"test":"second
>> modification"}}}[\n]"
>>     [mel-1) thread #2 - timer://foo] wire                           DEBUG
>> http-outgoing-0 << "[\r][\n]"
>>
>>   [1]:
>>
>> https://docs.openshift.com/container-platform/3.11/rest_api/api/v1.ConfigMap.html
>>
>
Reply | Threaded
Open this post in threaded view
|

Re: http4 not sending exchange data further in route for kubernetes API

imranrazakhan
In reply to this post by Andrea Cosentino-3
Please point me to kubernetes event example i tried to search but couldn't
find it.

On separate note can we read chunk data in route?
Reply | Threaded
Open this post in threaded view
|

Re: http4 not sending exchange data further in route for kubernetes API

Andrea Cosentino-3
You're right. We don't have a consumer for event related to config map. Can
you please open an enhancement?

Il sab 14 set 2019, 19:33 Imran Raza Khan <[hidden email]> ha
scritto:

> Please point me to kubernetes event example i tried to search but couldn't
> find it.
>
> On separate note can we read chunk data in route?
>
Reply | Threaded
Open this post in threaded view
|

Re: http4 not sending exchange data further in route for kubernetes API

Andrea Cosentino-3
I will work on it

Il sab 14 set 2019, 19:36 Andrea Cosentino <[hidden email]> ha scritto:

> You're right. We don't have a consumer for event related to config map.
> Can you please open an enhancement?
>
> Il sab 14 set 2019, 19:33 Imran Raza Khan <[hidden email]> ha
> scritto:
>
>> Please point me to kubernetes event example i tried to search but couldn't
>> find it.
>>
>> On separate note can we read chunk data in route?
>>
>
Reply | Threaded
Open this post in threaded view
|

Re: http4 not sending exchange data further in route for kubernetes API

imranrazakhan
In reply to this post by Andrea Cosentino-3
Till the time you are working on enhancement can i use http4 component to
listen for configmap events?

i require to read chunk data in route, its working fine with curl command
where i save output in file and monitor changes on file, but i am looking
for pure camel solution.
Reply | Threaded
Open this post in threaded view
|

Re: http4 not sending exchange data further in route for kubernetes API

Andrea Cosentino-3
I never tried that before. Can you create a Github project with a
reproducer? It would be useful

Il sab 14 set 2019, 19:45 Imran Raza Khan <[hidden email]> ha
scritto:

> Till the time you are working on enhancement can i use http4 component to
> listen for configmap events?
>
> i require to read chunk data in route, its working fine with curl command
> where i save output in file and monitor changes on file, but i am looking
> for pure camel solution.
>
Reply | Threaded
Open this post in threaded view
|

Re: http4 not sending exchange data further in route for kubernetes API

imranrazakhan
In reply to this post by Andrea Cosentino-3
Please find below issue for improvement

https://issues.apache.org/jira/browse/CAMEL-13978

Although i provided source code and debug trace of my route, i will still
try to provide you github project
Reply | Threaded
Open this post in threaded view
|

Re: http4 not sending exchange data further in route for kubernetes API

Jan Bednář
In reply to this post by imranrazakhan
Hi, you can try to switch to netty component, it is much more suitable
for HTTP streaming endpoints. Or experiment with http4 component
`connectionClose` option - This will add `Connection: close` header,
which is signal for server, that you dont support persistent connection.

Dne 14.9.2019 v 19:45 Imran Raza Khan napsal(a):
> Till the time you are working on enhancement can i use http4 component to
> listen for configmap events?
>
> i require to read chunk data in route, its working fine with curl command
> where i save output in file and monitor changes on file, but i am looking
> for pure camel solution.
>

Reply | Threaded
Open this post in threaded view
|

Re: http4 not sending exchange data further in route for kubernetes API

Jan Bednář
Also you cant try with http component option `chunked=false`, which
should efectively disable streaming.

Dne 14.9.2019 v 22:04 Jan Bednář napsal(a):

> Hi, you can try to switch to netty component, it is much more suitable
> for HTTP streaming endpoints. Or experiment with http4 component
> `connectionClose` option - This will add `Connection: close` header,
> which is signal for server, that you dont support persistent connection.
>
> Dne 14.9.2019 v 19:45 Imran Raza Khan napsal(a):
>> Till the time you are working on enhancement can i use http4
>> component to
>> listen for configmap events?
>>
>> i require to read chunk data in route, its working fine with curl
>> command
>> where i save output in file and monitor changes on file, but i am
>> looking
>> for pure camel solution.
>>
>

Reply | Threaded
Open this post in threaded view
|

Re: http4 not sending exchange data further in route for kubernetes API

imranrazakhan
This post was updated on .
In reply to this post by Andrea Cosentino-3
@ Jan Bednář

i have no issue with chunk data or close connection, As you can see from
debug logs that i am receiving all chunks of data, but http4 endpoint is
not forwarding further in route due to which i cant process it.

I want to process response as soon as i receive, but somehow http4 keep it
with itself.

Find HTTP trace on below gist

https://gist.github.com/imranrazakhan/9f97abb6080d1ab09825da7c08e53566
Reply | Threaded
Open this post in threaded view
|

Re: http4 not sending exchange data further in route for kubernetes API

imranrazakhan
In reply to this post by Andrea Cosentino-3
@Andrea

please find code on below link to test that http4 dont support processing
chunk data

https://github.com/imranrazakhan/KubernetesConfigMapWatcher

it look like http4 dont support reading of chunked data until it receive
all data, but in my case i want to receive/process data chunk by chunk.
similar behaviour working with curl, Below Added and Modified are two
chunks second got print after two minutes, in similar way i want camel
route should process chunk rather than whole message

    # curl -v -k -H "Authorization: Bearer xxx" -H 'Accept:
application/json' -H "Accept-Encoding: gzip"
https://localhost:8443/api/v1/watch/namespaces/my-qa/configmaps/my-config-map
    * About to connect() to localhost port 8443 (#0)
    *   Trying x.x.x.x...
    * Connected to localhost (x.x.x.x) port 8443 (#0)

    > GET /api/v1/watch/namespaces/my-qa/configmaps/my-config-map HTTP/1.1
    > User-Agent: curl/7.29.0
    > Host: localhost:8443
    > Authorization: Bearer xxx
    > Accept: application/json
    > Accept-Encoding: gzip
    >
    < HTTP/1.1 200 OK
    < Cache-Control: no-store
    < Content-Type: application/json
    < Date: Sun, 15 Sep 2019 11:48:14 GMT
    < Transfer-Encoding: chunked
    <

{"type":"ADDED","object":{"kind":"ConfigMap","apiVersion":"v1","metadata":{"name":"my-config-map","namespace":"my-qa","selfLink":"/api/v1/namespaces/yq-qa/configmaps/my-config-map","uid":"a8130369-d703-11e9-b37c-d2c8065c1507","resourceVersion":"33217315","creationTimestamp":"2019-09-14T15:23:51Z"},"data":{"test":"second
modification 20tefdsfsfsc"}}}


{"type":"MODIFIED","object":{"kind":"ConfigMap","apiVersion":"v1","metadata":{"name":"my-config-map","namespace":"my-qa","selfLink":"/api/v1/namespaces/my-qa/configmaps/my-config-map","uid":"a8130369-d703-11e9-b37c-d2c8065c1507","resourceVersion":"33217608","creationTimestamp":"2019-09-14T15:23:51Z"},"data":{"test":"second
modification 20tefdsfsfsc gzip"}}}