Content-Length Header being used previous HTTP Response

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Content-Length Header being used previous HTTP Response

Bob Paulin-2
Hi,

I have a camel rest route like the following

rest("/api/test")
        .post()
        .to("direct:test");

from("direct:test").

.toD("http4://localhost/test?bridgeEndpoint=true&throwExceptionOnFailure=false")

.toD("http4://localhost/test2?bridgeEndpoint=true&throwExceptionOnFailure=false");


Trouble occurs when the http4://localhost/test route sets a
content-length header it gets used for http4://localhost/test2.  There
are some web servers that use that header to determine request size
which if it never gets to the needed length will never close and after a
while timeout.  This happens despite content-length being in the
HttpHeaderFilterStrategy list because in the HttpProducer class there is
code like this.  So an input stream will still use the header.

// fallback as input stream
                    if (answer == null) {
                        // force the body as an input stream since this
is the fallback
                        InputStream is =
in.getMandatoryBody(InputStream.class);
                        String length =
in.getHeader(Exchange.CONTENT_LENGTH, String.class);
                        InputStreamEntity entity = null;
                        if (ObjectHelper.isEmpty(length) ) {
                            entity = new InputStreamEntity(is, -1);
                        } else {
                            entity = new InputStreamEntity(is,
Long.parseLong(length));
                        }
                        if (contentType != null) {
                            entity.setContentType(contentType.toString());
                        }
                        answer = entity;
                    }

I'm assuming there are cases where I might want to explicitly provide
the content-length to the request entity.  But I'd like to be able to
turn it off so it always falls back to -1 (chunked).

I'm working around the issue by doing this:

from("direct:test").

.toD("http4://localhost/test?bridgeEndpoint=true&throwExceptionOnFailure=false")

.setHeader("Content-Length", constant(""))

.toD("http4://localhost/test2?bridgeEndpoint=true&throwExceptionOnFailure=false");

But I'd like to suggest creating a more global way to ignore the
Content-Length header:

// fallback as input stream
                    if (answer == null) {
                        // force the body as an input stream since this
is the fallback
                        InputStream is =
in.getMandatoryBody(InputStream.class);
                        String length =
in.getHeader(Exchange.CONTENT_LENGTH, String.class);
                        InputStreamEntity entity = null;
                        if (ObjectHelper.isEmpty(length) ||
getEndpoint().getComponent().isForceChunkedRequestEntity()) {
                            entity = new InputStreamEntity(is, -1);
                        } else {
                            entity = new InputStreamEntity(is,
Long.parseLong(length));
                        }
                        if (contentType != null) {
                            entity.setContentType(contentType.toString());
                        }
                        answer = entity;
                    }


Thoughts on this approach?  How are other folks dealing with this? 
Happy to work on a contribution/patch but I wanted to discuss before
creating a jira. 


- Bob


signature.asc (836 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Content-Length Header being used previous HTTP Response

Claus Ibsen-2
You can remove the header before calling the http endpoints, with
removeHeader(Exchange.CONTENT_LENGTH)

On Sat, Sep 16, 2017 at 9:03 PM, Bob Paulin <[hidden email]> wrote:

> Hi,
>
> I have a camel rest route like the following
>
> rest("/api/test")
>         .post()
>         .to("direct:test");
>
> from("direct:test").
>
> .toD("http4://localhost/test?bridgeEndpoint=true&throwExceptionOnFailure=false")
>
> .toD("http4://localhost/test2?bridgeEndpoint=true&throwExceptionOnFailure=false");
>
>
> Trouble occurs when the http4://localhost/test route sets a
> content-length header it gets used for http4://localhost/test2.  There
> are some web servers that use that header to determine request size
> which if it never gets to the needed length will never close and after a
> while timeout.  This happens despite content-length being in the
> HttpHeaderFilterStrategy list because in the HttpProducer class there is
> code like this.  So an input stream will still use the header.
>
> // fallback as input stream
>                     if (answer == null) {
>                         // force the body as an input stream since this
> is the fallback
>                         InputStream is =
> in.getMandatoryBody(InputStream.class);
>                         String length =
> in.getHeader(Exchange.CONTENT_LENGTH, String.class);
>                         InputStreamEntity entity = null;
>                         if (ObjectHelper.isEmpty(length) ) {
>                             entity = new InputStreamEntity(is, -1);
>                         } else {
>                             entity = new InputStreamEntity(is,
> Long.parseLong(length));
>                         }
>                         if (contentType != null) {
>                             entity.setContentType(contentType.toString());
>                         }
>                         answer = entity;
>                     }
>
> I'm assuming there are cases where I might want to explicitly provide
> the content-length to the request entity.  But I'd like to be able to
> turn it off so it always falls back to -1 (chunked).
>
> I'm working around the issue by doing this:
>
> from("direct:test").
>
> .toD("http4://localhost/test?bridgeEndpoint=true&throwExceptionOnFailure=false")
>
> .setHeader("Content-Length", constant(""))
>
> .toD("http4://localhost/test2?bridgeEndpoint=true&throwExceptionOnFailure=false");
>
> But I'd like to suggest creating a more global way to ignore the
> Content-Length header:
>
> // fallback as input stream
>                     if (answer == null) {
>                         // force the body as an input stream since this
> is the fallback
>                         InputStream is =
> in.getMandatoryBody(InputStream.class);
>                         String length =
> in.getHeader(Exchange.CONTENT_LENGTH, String.class);
>                         InputStreamEntity entity = null;
>                         if (ObjectHelper.isEmpty(length) ||
> getEndpoint().getComponent().isForceChunkedRequestEntity()) {
>                             entity = new InputStreamEntity(is, -1);
>                         } else {
>                             entity = new InputStreamEntity(is,
> Long.parseLong(length));
>                         }
>                         if (contentType != null) {
>                             entity.setContentType(contentType.toString());
>                         }
>                         answer = entity;
>                     }
>
>
> Thoughts on this approach?  How are other folks dealing with this?
> Happy to work on a contribution/patch but I wanted to discuss before
> creating a jira.
>
>
> - Bob
>



--
Claus Ibsen
-----------------
http://davsclaus.com @davsclaus
Camel in Action 2: https://www.manning.com/ibsen2
Reply | Threaded
Open this post in threaded view
|

Re: Content-Length Header being used previous HTTP Response

Bob Paulin-2
Agree that removeHeader inbetween the routes fixes the problem.  It's
great experience having to go through all the code to figure out a
content-length header mismatch is the issue since the request just
hangs.  I ran into it again with the following scenerio using the
automatic json binding in the REST DSL with stream caching enabled.

this.getContext().setStreamCaching(true);

restConfiguration().component("netty4-http").port(8282).bindingMode(RestBindingMode.json);
               
                rest("/bind").post()
                    .type(InputDTO.class)
                    .outType(OutputDTO.class)
                    .to("direct:bind");

              from("direct:bind")
                    .setHeader(Exchange.HTTP_METHOD, constant("POST"))
                    .setHeader("Accept", constant("application/json"))
                    .setHeader(Exchange.HTTP_PATH, constant(""))
                    .setHeader(Exchange.CONTENT_TYPE,
constant("application/json"))
                     .process(new Processor() {
                       
                        public void process(Exchange exchange) throws
Exception {
                           InputDTO body =
exchange.getIn().getBody(InputDTO.class);
                           body.setDescription("SomethingElse");
                           
                        }
                    })
                    .marshal().json(JsonLibrary.Jackson)
                    .to("http4://localhost/realservice?bridgeEndpoint=true")
                    .unmarshal().json(JsonLibrary.Jackson,
FinalOutputDTO.class);

The json coming into /bind is pretty printed so includes spaces and
returns.  After it's marshaled to an object and the object is modified
the Content-length will change.  Seems like removing the content-length
header in the DataFormat used to do the marshal ling or perhaps
providing an option in the HTTP Client to ignore the content-header
would be useful.  Again willing to help collaborate on a patch but I'd
like some input on where it would fit best.  Currently thinking either
in the DataFormat marshal methods and/or the HttpProducer.

- Bob

On 2017/09/20 13:13:06, Claus Ibsen <[hidden email]> wrote:

> You can remove the header before calling the http endpoints, with>
> removeHeader(Exchange.CONTENT_LENGTH)>
>
> On Sat, Sep 16, 2017 at 9:03 PM, Bob Paulin <[hidden email]> wrote:>
> > Hi,>
> >>
> > I have a camel rest route like the following>
> >>
> > rest("/api/test")>
> > .post()>
> > .to("direct:test");>
> >>
> > from("direct:test").>
> >>
> >
.toD("http4://localhost/test?bridgeEndpoint=true&throwExceptionOnFailure=false")>

> >>
> >
.toD("http4://localhost/test2?bridgeEndpoint=true&throwExceptionOnFailure=false");>

> >>
> >>
> > Trouble occurs when the http4://localhost/test route sets a>
> > content-length header it gets used for http4://localhost/test2. There>
> > are some web servers that use that header to determine request size>
> > which if it never gets to the needed length will never close and
after a>
> > while timeout. This happens despite content-length being in the>
> > HttpHeaderFilterStrategy list because in the HttpProducer class
there is>

> > code like this. So an input stream will still use the header.>
> >>
> > // fallback as input stream>
> > if (answer == null) {>
> > // force the body as an input stream since this>
> > is the fallback>
> > InputStream is =>
> > in.getMandatoryBody(InputStream.class);>
> > String length =>
> > in.getHeader(Exchange.CONTENT_LENGTH, String.class);>
> > InputStreamEntity entity = null;>
> > if (ObjectHelper.isEmpty(length) ) {>
> > entity = new InputStreamEntity(is, -1);>
> > } else {>
> > entity = new InputStreamEntity(is,>
> > Long.parseLong(length));>
> > }>
> > if (contentType != null) {>
> > entity.setContentType(contentType.toString());>
> > }>
> > answer = entity;>
> > }>
> >>
> > I'm assuming there are cases where I might want to explicitly provide>
> > the content-length to the request entity. But I'd like to be able to>
> > turn it off so it always falls back to -1 (chunked).>
> >>
> > I'm working around the issue by doing this:>
> >>
> > from("direct:test").>
> >>
> >
.toD("http4://localhost/test?bridgeEndpoint=true&throwExceptionOnFailure=false")>

> >>
> > .setHeader("Content-Length", constant(""))>
> >>
> >
.toD("http4://localhost/test2?bridgeEndpoint=true&throwExceptionOnFailure=false");>

> >>
> > But I'd like to suggest creating a more global way to ignore the>
> > Content-Length header:>
> >>
> > // fallback as input stream>
> > if (answer == null) {>
> > // force the body as an input stream since this>
> > is the fallback>
> > InputStream is =>
> > in.getMandatoryBody(InputStream.class);>
> > String length =>
> > in.getHeader(Exchange.CONTENT_LENGTH, String.class);>
> > InputStreamEntity entity = null;>
> > if (ObjectHelper.isEmpty(length) ||>
> > getEndpoint().getComponent().isForceChunkedRequestEntity()) {>
> > entity = new InputStreamEntity(is, -1);>
> > } else {>
> > entity = new InputStreamEntity(is,>
> > Long.parseLong(length));>
> > }>
> > if (contentType != null) {>
> > entity.setContentType(contentType.toString());>
> > }>
> > answer = entity;>
> > }>
> >>
> >>
> > Thoughts on this approach? How are other folks dealing with this?>
> > Happy to work on a contribution/patch but I wanted to discuss before>
> > creating a jira.>
> >>
> >>
> > - Bob>
> >>
>
>
>
> -- >
> Claus Ibsen>
> ----------------->
> http://davsclaus.com @davsclaus>
> Camel in Action 2: https://www.manning.com/ibsen2>
>

signature.asc (836 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Content-Length Header being used previous HTTP Response

Bob Paulin-2
Sorry that should read "not a great experience".   I was not attempting
sarcasm for effect there.   Apologies for any misunderstand and the
extra noise on the list.


- Bob


On 8/16/2018 8:29 AM, Bob Paulin wrote:

> Agree that removeHeader inbetween the routes fixes the problem.  It's
> great experience having to go through all the code to figure out a
> content-length header mismatch is the issue since the request just
> hangs.  I ran into it again with the following scenerio using the
> automatic json binding in the REST DSL with stream caching enabled.
>
> this.getContext().setStreamCaching(true);
>
> restConfiguration().component("netty4-http").port(8282).bindingMode(RestBindingMode.json);
>                
>                 rest("/bind").post()
>                     .type(InputDTO.class)
>                     .outType(OutputDTO.class)
>                     .to("direct:bind");
>
>               from("direct:bind")
>                     .setHeader(Exchange.HTTP_METHOD, constant("POST"))
>                     .setHeader("Accept", constant("application/json"))
>                     .setHeader(Exchange.HTTP_PATH, constant(""))
>                     .setHeader(Exchange.CONTENT_TYPE,
> constant("application/json"))
>                      .process(new Processor() {
>                        
>                         public void process(Exchange exchange) throws
> Exception {
>                            InputDTO body =
> exchange.getIn().getBody(InputDTO.class);
>                            body.setDescription("SomethingElse");
>                            
>                         }
>                     })
>                     .marshal().json(JsonLibrary.Jackson)
>                     .to("http4://localhost/realservice?bridgeEndpoint=true")
>                     .unmarshal().json(JsonLibrary.Jackson,
> FinalOutputDTO.class);
>
> The json coming into /bind is pretty printed so includes spaces and
> returns.  After it's marshaled to an object and the object is modified
> the Content-length will change.  Seems like removing the content-length
> header in the DataFormat used to do the marshal ling or perhaps
> providing an option in the HTTP Client to ignore the content-header
> would be useful.  Again willing to help collaborate on a patch but I'd
> like some input on where it would fit best.  Currently thinking either
> in the DataFormat marshal methods and/or the HttpProducer.
>
> - Bob
>
> On 2017/09/20 13:13:06, Claus Ibsen <[hidden email]> wrote:
>> You can remove the header before calling the http endpoints, with>
>> removeHeader(Exchange.CONTENT_LENGTH)>
>>
>> On Sat, Sep 16, 2017 at 9:03 PM, Bob Paulin <[hidden email]> wrote:>
>>> Hi,>
>>> I have a camel rest route like the following>
>>> rest("/api/test")>
>>> .post()>
>>> .to("direct:test");>
>>> from("direct:test").>
> .toD("http4://localhost/test?bridgeEndpoint=true&throwExceptionOnFailure=false")>
>
> .toD("http4://localhost/test2?bridgeEndpoint=true&throwExceptionOnFailure=false");>
>
>>>>
>>> Trouble occurs when the http4://localhost/test route sets a>
>>> content-length header it gets used for http4://localhost/test2. There>
>>> are some web servers that use that header to determine request size>
>>> which if it never gets to the needed length will never close and
> after a>
>>> while timeout. This happens despite content-length being in the>
>>> HttpHeaderFilterStrategy list because in the HttpProducer class
> there is>
>>> code like this. So an input stream will still use the header.>
>>> // fallback as input stream>
>>> if (answer == null) {>
>>> // force the body as an input stream since this>
>>> is the fallback>
>>> InputStream is =>
>>> in.getMandatoryBody(InputStream.class);>
>>> String length =>
>>> in.getHeader(Exchange.CONTENT_LENGTH, String.class);>
>>> InputStreamEntity entity = null;>
>>> if (ObjectHelper.isEmpty(length) ) {>
>>> entity = new InputStreamEntity(is, -1);>
>>> } else {>
>>> entity = new InputStreamEntity(is,>
>>> Long.parseLong(length));>
>>> }>
>>> if (contentType != null) {>
>>> entity.setContentType(contentType.toString());>
>>> }>
>>> answer = entity;>
>>> }>
>>> I'm assuming there are cases where I might want to explicitly provide>
>>> the content-length to the request entity. But I'd like to be able to>
>>> turn it off so it always falls back to -1 (chunked).>
>>> I'm working around the issue by doing this:>
>>> from("direct:test").>
> .toD("http4://localhost/test?bridgeEndpoint=true&throwExceptionOnFailure=false")>
>
>>> .setHeader("Content-Length", constant(""))>
> .toD("http4://localhost/test2?bridgeEndpoint=true&throwExceptionOnFailure=false");>
>
>>> But I'd like to suggest creating a more global way to ignore the>
>>> Content-Length header:>
>>> // fallback as input stream>
>>> if (answer == null) {>
>>> // force the body as an input stream since this>
>>> is the fallback>
>>> InputStream is =>
>>> in.getMandatoryBody(InputStream.class);>
>>> String length =>
>>> in.getHeader(Exchange.CONTENT_LENGTH, String.class);>
>>> InputStreamEntity entity = null;>
>>> if (ObjectHelper.isEmpty(length) ||>
>>> getEndpoint().getComponent().isForceChunkedRequestEntity()) {>
>>> entity = new InputStreamEntity(is, -1);>
>>> } else {>
>>> entity = new InputStreamEntity(is,>
>>> Long.parseLong(length));>
>>> }>
>>> if (contentType != null) {>
>>> entity.setContentType(contentType.toString());>
>>> }>
>>> answer = entity;>
>>> }>
>>>>
>>> Thoughts on this approach? How are other folks dealing with this?>
>>> Happy to work on a contribution/patch but I wanted to discuss before>
>>> creating a jira.>
>>>>
>>> - Bob>
>>
>>
>> -- >
>> Claus Ibsen>
>> ----------------->
>> http://davsclaus.com @davsclaus>
>> Camel in Action 2: https://www.manning.com/ibsen2>
>>


signature.asc (836 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Content-Length Header being used previous HTTP Response

Claus Ibsen-2
In reply to this post by Bob Paulin-2
Hi

Yeah we should look at improving this. I have logged a ticket
https://issues.apache.org/jira/browse/CAMEL-12751

You are welcome to help around and provide unit tests / patches etc.
For example as github PRs. We love contributions
http://camel.apache.org/contributing
On Thu, Aug 16, 2018 at 3:29 PM Bob Paulin <[hidden email]> wrote:

>
> Agree that removeHeader inbetween the routes fixes the problem.  It's
> great experience having to go through all the code to figure out a
> content-length header mismatch is the issue since the request just
> hangs.  I ran into it again with the following scenerio using the
> automatic json binding in the REST DSL with stream caching enabled.
>
> this.getContext().setStreamCaching(true);
>
> restConfiguration().component("netty4-http").port(8282).bindingMode(RestBindingMode.json);
>
>                 rest("/bind").post()
>                     .type(InputDTO.class)
>                     .outType(OutputDTO.class)
>                     .to("direct:bind");
>
>               from("direct:bind")
>                     .setHeader(Exchange.HTTP_METHOD, constant("POST"))
>                     .setHeader("Accept", constant("application/json"))
>                     .setHeader(Exchange.HTTP_PATH, constant(""))
>                     .setHeader(Exchange.CONTENT_TYPE,
> constant("application/json"))
>                      .process(new Processor() {
>
>                         public void process(Exchange exchange) throws
> Exception {
>                            InputDTO body =
> exchange.getIn().getBody(InputDTO.class);
>                            body.setDescription("SomethingElse");
>
>                         }
>                     })
>                     .marshal().json(JsonLibrary.Jackson)
>                     .to("http4://localhost/realservice?bridgeEndpoint=true")
>                     .unmarshal().json(JsonLibrary.Jackson,
> FinalOutputDTO.class);
>
> The json coming into /bind is pretty printed so includes spaces and
> returns.  After it's marshaled to an object and the object is modified
> the Content-length will change.  Seems like removing the content-length
> header in the DataFormat used to do the marshal ling or perhaps
> providing an option in the HTTP Client to ignore the content-header
> would be useful.  Again willing to help collaborate on a patch but I'd
> like some input on where it would fit best.  Currently thinking either
> in the DataFormat marshal methods and/or the HttpProducer.
>
> - Bob
>
> On 2017/09/20 13:13:06, Claus Ibsen <[hidden email]> wrote:
> > You can remove the header before calling the http endpoints, with>
> > removeHeader(Exchange.CONTENT_LENGTH)>
> >
> > On Sat, Sep 16, 2017 at 9:03 PM, Bob Paulin <[hidden email]> wrote:>
> > > Hi,>
> > >>
> > > I have a camel rest route like the following>
> > >>
> > > rest("/api/test")>
> > > .post()>
> > > .to("direct:test");>
> > >>
> > > from("direct:test").>
> > >>
> > >
> .toD("http4://localhost/test?bridgeEndpoint=true&throwExceptionOnFailure=false")>
>
> > >>
> > >
> .toD("http4://localhost/test2?bridgeEndpoint=true&throwExceptionOnFailure=false");>
>
> > >>
> > >>
> > > Trouble occurs when the http4://localhost/test route sets a>
> > > content-length header it gets used for http4://localhost/test2. There>
> > > are some web servers that use that header to determine request size>
> > > which if it never gets to the needed length will never close and
> after a>
> > > while timeout. This happens despite content-length being in the>
> > > HttpHeaderFilterStrategy list because in the HttpProducer class
> there is>
> > > code like this. So an input stream will still use the header.>
> > >>
> > > // fallback as input stream>
> > > if (answer == null) {>
> > > // force the body as an input stream since this>
> > > is the fallback>
> > > InputStream is =>
> > > in.getMandatoryBody(InputStream.class);>
> > > String length =>
> > > in.getHeader(Exchange.CONTENT_LENGTH, String.class);>
> > > InputStreamEntity entity = null;>
> > > if (ObjectHelper.isEmpty(length) ) {>
> > > entity = new InputStreamEntity(is, -1);>
> > > } else {>
> > > entity = new InputStreamEntity(is,>
> > > Long.parseLong(length));>
> > > }>
> > > if (contentType != null) {>
> > > entity.setContentType(contentType.toString());>
> > > }>
> > > answer = entity;>
> > > }>
> > >>
> > > I'm assuming there are cases where I might want to explicitly provide>
> > > the content-length to the request entity. But I'd like to be able to>
> > > turn it off so it always falls back to -1 (chunked).>
> > >>
> > > I'm working around the issue by doing this:>
> > >>
> > > from("direct:test").>
> > >>
> > >
> .toD("http4://localhost/test?bridgeEndpoint=true&throwExceptionOnFailure=false")>
>
> > >>
> > > .setHeader("Content-Length", constant(""))>
> > >>
> > >
> .toD("http4://localhost/test2?bridgeEndpoint=true&throwExceptionOnFailure=false");>
>
> > >>
> > > But I'd like to suggest creating a more global way to ignore the>
> > > Content-Length header:>
> > >>
> > > // fallback as input stream>
> > > if (answer == null) {>
> > > // force the body as an input stream since this>
> > > is the fallback>
> > > InputStream is =>
> > > in.getMandatoryBody(InputStream.class);>
> > > String length =>
> > > in.getHeader(Exchange.CONTENT_LENGTH, String.class);>
> > > InputStreamEntity entity = null;>
> > > if (ObjectHelper.isEmpty(length) ||>
> > > getEndpoint().getComponent().isForceChunkedRequestEntity()) {>
> > > entity = new InputStreamEntity(is, -1);>
> > > } else {>
> > > entity = new InputStreamEntity(is,>
> > > Long.parseLong(length));>
> > > }>
> > > if (contentType != null) {>
> > > entity.setContentType(contentType.toString());>
> > > }>
> > > answer = entity;>
> > > }>
> > >>
> > >>
> > > Thoughts on this approach? How are other folks dealing with this?>
> > > Happy to work on a contribution/patch but I wanted to discuss before>
> > > creating a jira.>
> > >>
> > >>
> > > - Bob>
> > >>
> >
> >
> >
> > -- >
> > Claus Ibsen>
> > ----------------->
> > http://davsclaus.com @davsclaus>
> > Camel in Action 2: https://www.manning.com/ibsen2>
> >



--
Claus Ibsen
-----------------
http://davsclaus.com @davsclaus
Camel in Action 2: https://www.manning.com/ibsen2