Camel Stream caching unable to capture the entire stream

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Camel Stream caching unable to capture the entire stream

calyan.bandi
Hi,

I have a camel route which invokes a webservice, and tries to processes the response. I am using cxfrs component for the invocation. The response i am receiving is very huge and hence most of the content is removed from the end.

To remedy this, i am using stream caching - org.apache.camel.spi.StreamCachingStrategy. These options are not helping me in the route as i still see that the response is not fully captured and the end of the response is getting purged. Below are the settings i am using the camel context and the streamCache is enabled per route. I have tried to set the streamCache value per route and on the whole context, it didn't make any difference. Hence i am setting this value per route.

<bean id="myResponseStreamStrategy" class="org.apache.camel.impl.DefaultStreamCachingStrategy">
                <property name="spoolDirectory" value="${tempSpoolDirectory}"></property>
                <property name="spoolThreshold" value="131072"></property>
                <property name="spoolUsedHeapMemoryThreshold">
                        <value type="java.lang.Integer">30</value>
                </property>
                <property name="removeSpoolDirectoryWhenStopping" value="true"></property>
        </bean>

And in the logs i can see the settings being applied as:
[main] 2015-12-14 06:47:22 INFO  AbstractCamelContextFactoryBean:892 - Using custom StreamCachingStrategy: DefaultStreamCachingStrategy[spoolDirectory=/home/testing/tmp, spoolChiper=null, spoolThreshold=131072, spoolUsedHeapMemoryThreshold=30, bufferSize=4096, anySpoolRules=false]
[main] 2015-12-14 06:47:22 INFO  CamelContextFactoryBean:273 - Bridging Camel and Spring property placeholder configurer with id: bridgePropertyPlaceholder
[main] 2015-12-14 06:47:22 INFO  SpringCamelContext:1637 - Apache Camel 2.13.2 (CamelContext: context) is starting
[main] 2015-12-14 06:47:22 INFO  ManagedManagementStrategy:187 - JMX is enabled
[main] 2015-12-14 06:47:22 INFO  DefaultTypeConverter:50 - Loaded 206 type converters
[main] 2015-12-14 06:47:23 INFO  SpringCamelContext:1831 - AllowUseOriginalMessage is enabled. If access to the original message is not needed, then its recommended to turn this option off as it may improve performance.
[main] 2015-12-14 06:47:23 INFO  DefaultStreamCachingStrategy:314 - StreamCaching in use with spool directory: /home/testing/tmp and rules: [Spool > 128K body size, Spool > 30% used of Max heap memory]
 

I ran a java utility seperately to capture the entire response with which i captured the full JSON response and the size of the entire string is around 71172 bytes.

Any pointers around this issue, kindly suggest.

Thanks,
Kalyan
Reply | Threaded
Open this post in threaded view
|

Re: Camel Stream caching unable to capture the entire stream

calyan.bandi
Hi,

I found the anwer to the problem in apache cxf forums.

http://cxf.547215.n5.nabble.com/How-to-increase-the-max-size-of-a-REST-response-td563077.html

This is more of a issue in the CXF component than camel. But still for referene, i am sharing this information.

At the server side, we saw the below response headers for each of the requests.

ID: 2
Response-Code: 200
Content-Type: application/json
Headers: {Content-Type=[application/json], Date=[Wed, 16 Dec 2015 11:21:34 GMT]}
Messages: Outbound Message (saved to tmp file):
Filename: C:\Users\Temp\cxf-tmp-309797\cos8635606649006238176tmp
(message truncated to 65536 bytes)

And from the client logs, whatever the payload that is received from the server, we are seeing only the first chunk in the logs (calculated the response payload size from the logs and it is 65536 bytes). This is only a concern during logging and in my camel routes at the client side the entire response is getting unmarshalled from JSON and things are working fine.      

Thanks,
Kalyan
Reply | Threaded
Open this post in threaded view
|

Re: Camel Stream caching unable to capture the entire stream

Franz Paul Forsthofer
Hi Kalyan,

this issue is known. See
https://issues.apache.org/jira/browse/CAMEL-8419 . CAMEL-8419 contains
even a patch. But this patch is more a workaround, therefore this
patch was never merged.

Best Regards Franz

On Wed, Dec 16, 2015 at 12:35 PM, calyan.bandi <[hidden email]> wrote:

> Hi,
>
> I found the anwer to the problem in apache cxf forums.
>
> http://cxf.547215.n5.nabble.com/How-to-increase-the-max-size-of-a-REST-response-td563077.html
>
> This is more of a issue in the CXF component than camel. But still for
> referene, i am sharing this information.
>
> At the server side, we saw the below response headers for each of the
> requests.
>
> ID: 2
> Response-Code: 200
> Content-Type: application/json
> Headers: {Content-Type=[application/json], Date=[Wed, 16 Dec 2015 11:21:34
> GMT]}
> Messages: Outbound Message (saved to tmp file):
> Filename: C:\Users\Temp\cxf-tmp-309797\cos8635606649006238176tmp
> (message truncated to 65536 bytes)
>
> And from the client logs, whatever the payload that is received from the
> server, we are seeing only the first chunk in the logs (calculated the
> response payload size from the logs and it is 65536 bytes). This is only a
> concern during logging and in my camel routes at the client side the entire
> response is getting unmarshalled from JSON and things are working fine.
>
> Thanks,
> Kalyan
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Camel-Stream-caching-unable-to-capture-the-entire-stream-tp5775055p5775145.html
> Sent from the Camel - Users mailing list archive at Nabble.com.