[HEADS UP] - Optimzing routing engine to reduce stack frames in use during routing

classic Classic list List threaded Threaded
16 messages Options
Reply | Threaded
Open this post in threaded view
|

[HEADS UP] - Optimzing routing engine to reduce stack frames in use during routing

Claus Ibsen-2
Hi

Just to tell about the work I am currently doing
https://issues.apache.org/jira/browse/CAMEL-6377

We have room to optimize the routing engine in Camel to make it more
friendlier to end users in terms of
- reduced stacktraces
- faster and easier to debug the code if you single step through all
the routing engine logic
- potential optimization to skip over functionality that is turned off
(eg such as tracer etc)
- potential optimization to add new cross cutting functionality at
runtime (though not currently the primary goal)
- potential to skip using RedeliveryErrorHandler/DefaultErrorHandler
if end user have not configured to use any kind of redelivery (the
reason is that logic in this guy is excessive and is harder for people
to debug)
- reduce wrapping internal processors when not needed


There is a sample route in the ticket where a stacktrace is forced
being thrown. And the difference is 40 vs 28 lines in the stacktrace.
And there is room for further optimization.

The work is aimed at being non invasive on the current architecture
and also backwards compatible. So far I have noticed a problem with
the old behavior I have logged as:
https://issues.apache.org/jira/browse/CAMEL-6378




--
Claus Ibsen
-----------------
www.camelone.org: The open source integration conference.

Red Hat, Inc.
FuseSource is now part of Red Hat
Email: [hidden email]
Web: http://fusesource.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
Reply | Threaded
Open this post in threaded view
|

Re: [HEADS UP] - Optimzing routing engine to reduce stack frames in use during routing

Raul Kripalani
Hi Claus,

It's great that you're finding time for this. The need to lighten
stacktraces was already brought up within the Camel 3.0 context [1]. In
fact, there's also an entry in the roadmap page [2] which proposes moving
away from the recursive model into an iterative routing engine.

It's clear that the next step in that area is a prototype. I hope I can
start working on that soon, even though the whole Camel 3.0 conversation
seems to have cooled down at present.

Anyway, do you have a concrete work plan or list of processors you'd are
targeting with https://issues.apache.org/jira/browse/CAMEL-6377?

If so, could you create a few individual JIRA tasks? I'd happily take on
some of the simplification/rationalisation work!

Regards,

[1]
http://camel.465427.n5.nabble.com/DISCUSS-Camel-3-0-Core-of-the-routing-engine-td5727754.html
[2] http://camel.apache.org/camel-30-ideas.html

*Raúl Kripalani*
Enterprise Architect, Open Source Integration specialist, Program
Manager | Apache
Camel Committer
http://about.me/raulkripalani | http://www.linkedin.com/in/raulkripalani
http://blog.raulkr.net | twitter: @raulvk

On Sat, May 18, 2013 at 3:38 PM, Claus Ibsen <[hidden email]> wrote:

> Hi
>
> Just to tell about the work I am currently doing
> https://issues.apache.org/jira/browse/CAMEL-6377
>
> We have room to optimize the routing engine in Camel to make it more
> friendlier to end users in terms of
> - reduced stacktraces
> - faster and easier to debug the code if you single step through all
> the routing engine logic
> - potential optimization to skip over functionality that is turned off
> (eg such as tracer etc)
> - potential optimization to add new cross cutting functionality at
> runtime (though not currently the primary goal)
> - potential to skip using RedeliveryErrorHandler/DefaultErrorHandler
> if end user have not configured to use any kind of redelivery (the
> reason is that logic in this guy is excessive and is harder for people
> to debug)
> - reduce wrapping internal processors when not needed
>
>
> There is a sample route in the ticket where a stacktrace is forced
> being thrown. And the difference is 40 vs 28 lines in the stacktrace.
> And there is room for further optimization.
>
> The work is aimed at being non invasive on the current architecture
> and also backwards compatible. So far I have noticed a problem with
> the old behavior I have logged as:
> https://issues.apache.org/jira/browse/CAMEL-6378
>
>
>
>
> --
> Claus Ibsen
> -----------------
> www.camelone.org: The open source integration conference.
>
> Red Hat, Inc.
> FuseSource is now part of Red Hat
> Email: [hidden email]
> Web: http://fusesource.com
> Twitter: davsclaus
> Blog: http://davsclaus.com
> Author of Camel in Action: http://www.manning.com/ibsen
>
Reply | Threaded
Open this post in threaded view
|

Re: [HEADS UP] - Optimzing routing engine to reduce stack frames in use during routing

Claus Ibsen-2
Hi

I added some tasks as comments on
https://issues.apache.org/jira/browse/CAMEL-6377#comment-13661858

The goal of CAMEL-6377 is to do *internal* optimiztion changes only
with the goal of minimazing the stack-frames in use, as well reduce
the long stacktraces our end users see. And also for the end users who
go down the path of debugging the Camel routing, then there is less
"callback hell".

As Raul said we have ideas sketched for Camel 3.0, where we can do
bigger changes, and as well do some API changes (if it makes
value/sense) etc. And the ideas that Raul refers to is much bigger and
takes a lot longer to implement. And possible some prototype is
needed. And also care has to be taken as we have DSL in Scala / Groovy
/ Kotlin etc. that may be affected. And also a large user base on 2.x
API that relies on this being stable etc.

However as in CAMEL-6377 with fairly little effort (half a friday +
half a day on weekend + half toda) + the effort for the remainder
tasks (eg the comment on the JIRA) we could get this implemented
fairly soon.

Its IMHO important to keep the scope on the goals of "reducing
stack-frames, less long stacktraces, and easier debugging). All the
stuff about iterative routing engine etc, is also what is captured on
the Camel 3.0 ideas page, and also sketched in more details by Raul
with his idea of "decision" being returned from the processor(s). But
that is IMHO not the current scope of CAMEL-6377. The scope is to be
internal optimizations on the current 2.x architecture.

If people wanna help with CAMEL-6377, then check the JIRA ticket and
the list of bulleted tasks.

The code changes is about to be committed later this afternoon. Just
running one extra fully sanity unit tests of the entire code base, to
ensure no regressions or other gremlins introduced. Including all the
OSGi tests which you need to run specially.




On Sat, May 18, 2013 at 5:39 PM, Raul Kripalani <[hidden email]> wrote:

> Hi Claus,
>
> It's great that you're finding time for this. The need to lighten
> stacktraces was already brought up within the Camel 3.0 context [1]. In
> fact, there's also an entry in the roadmap page [2] which proposes moving
> away from the recursive model into an iterative routing engine.
>
> It's clear that the next step in that area is a prototype. I hope I can
> start working on that soon, even though the whole Camel 3.0 conversation
> seems to have cooled down at present.
>
> Anyway, do you have a concrete work plan or list of processors you'd are
> targeting with https://issues.apache.org/jira/browse/CAMEL-6377?
>
> If so, could you create a few individual JIRA tasks? I'd happily take on
> some of the simplification/rationalisation work!
>
> Regards,
>
> [1]
> http://camel.465427.n5.nabble.com/DISCUSS-Camel-3-0-Core-of-the-routing-engine-td5727754.html
> [2] http://camel.apache.org/camel-30-ideas.html
>
> *Raúl Kripalani*
> Enterprise Architect, Open Source Integration specialist, Program
> Manager | Apache
> Camel Committer
> http://about.me/raulkripalani | http://www.linkedin.com/in/raulkripalani
> http://blog.raulkr.net | twitter: @raulvk
>
> On Sat, May 18, 2013 at 3:38 PM, Claus Ibsen <[hidden email]> wrote:
>
>> Hi
>>
>> Just to tell about the work I am currently doing
>> https://issues.apache.org/jira/browse/CAMEL-6377
>>
>> We have room to optimize the routing engine in Camel to make it more
>> friendlier to end users in terms of
>> - reduced stacktraces
>> - faster and easier to debug the code if you single step through all
>> the routing engine logic
>> - potential optimization to skip over functionality that is turned off
>> (eg such as tracer etc)
>> - potential optimization to add new cross cutting functionality at
>> runtime (though not currently the primary goal)
>> - potential to skip using RedeliveryErrorHandler/DefaultErrorHandler
>> if end user have not configured to use any kind of redelivery (the
>> reason is that logic in this guy is excessive and is harder for people
>> to debug)
>> - reduce wrapping internal processors when not needed
>>
>>
>> There is a sample route in the ticket where a stacktrace is forced
>> being thrown. And the difference is 40 vs 28 lines in the stacktrace.
>> And there is room for further optimization.
>>
>> The work is aimed at being non invasive on the current architecture
>> and also backwards compatible. So far I have noticed a problem with
>> the old behavior I have logged as:
>> https://issues.apache.org/jira/browse/CAMEL-6378
>>
>>
>>
>>
>> --
>> Claus Ibsen
>> -----------------
>> www.camelone.org: The open source integration conference.
>>
>> Red Hat, Inc.
>> FuseSource is now part of Red Hat
>> Email: [hidden email]
>> Web: http://fusesource.com
>> Twitter: davsclaus
>> Blog: http://davsclaus.com
>> Author of Camel in Action: http://www.manning.com/ibsen
>>



--
Claus Ibsen
-----------------
www.camelone.org: The open source integration conference.

Red Hat, Inc.
FuseSource is now part of Red Hat
Email: [hidden email]
Web: http://fusesource.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
Reply | Threaded
Open this post in threaded view
|

Re: [HEADS UP] - Optimzing routing engine to reduce stack frames in use during routing

Claus Ibsen-2
Hi

Okay so we got some good headway now.

The sample use case on 2.11 has a stacktrace of about 40 lines (without JMX).
And on trunk we are down to 17 now (without JMX) and +2 if JMX enabled.

That is reducing the stack traces with by 3 fold in that given sample.



On Mon, May 20, 2013 at 11:16 AM, Claus Ibsen <[hidden email]> wrote:

> Hi
>
> I added some tasks as comments on
> https://issues.apache.org/jira/browse/CAMEL-6377#comment-13661858
>
> The goal of CAMEL-6377 is to do *internal* optimiztion changes only
> with the goal of minimazing the stack-frames in use, as well reduce
> the long stacktraces our end users see. And also for the end users who
> go down the path of debugging the Camel routing, then there is less
> "callback hell".
>
> As Raul said we have ideas sketched for Camel 3.0, where we can do
> bigger changes, and as well do some API changes (if it makes
> value/sense) etc. And the ideas that Raul refers to is much bigger and
> takes a lot longer to implement. And possible some prototype is
> needed. And also care has to be taken as we have DSL in Scala / Groovy
> / Kotlin etc. that may be affected. And also a large user base on 2.x
> API that relies on this being stable etc.
>
> However as in CAMEL-6377 with fairly little effort (half a friday +
> half a day on weekend + half toda) + the effort for the remainder
> tasks (eg the comment on the JIRA) we could get this implemented
> fairly soon.
>
> Its IMHO important to keep the scope on the goals of "reducing
> stack-frames, less long stacktraces, and easier debugging). All the
> stuff about iterative routing engine etc, is also what is captured on
> the Camel 3.0 ideas page, and also sketched in more details by Raul
> with his idea of "decision" being returned from the processor(s). But
> that is IMHO not the current scope of CAMEL-6377. The scope is to be
> internal optimizations on the current 2.x architecture.
>
> If people wanna help with CAMEL-6377, then check the JIRA ticket and
> the list of bulleted tasks.
>
> The code changes is about to be committed later this afternoon. Just
> running one extra fully sanity unit tests of the entire code base, to
> ensure no regressions or other gremlins introduced. Including all the
> OSGi tests which you need to run specially.
>
>
>
>
> On Sat, May 18, 2013 at 5:39 PM, Raul Kripalani <[hidden email]> wrote:
>> Hi Claus,
>>
>> It's great that you're finding time for this. The need to lighten
>> stacktraces was already brought up within the Camel 3.0 context [1]. In
>> fact, there's also an entry in the roadmap page [2] which proposes moving
>> away from the recursive model into an iterative routing engine.
>>
>> It's clear that the next step in that area is a prototype. I hope I can
>> start working on that soon, even though the whole Camel 3.0 conversation
>> seems to have cooled down at present.
>>
>> Anyway, do you have a concrete work plan or list of processors you'd are
>> targeting with https://issues.apache.org/jira/browse/CAMEL-6377?
>>
>> If so, could you create a few individual JIRA tasks? I'd happily take on
>> some of the simplification/rationalisation work!
>>
>> Regards,
>>
>> [1]
>> http://camel.465427.n5.nabble.com/DISCUSS-Camel-3-0-Core-of-the-routing-engine-td5727754.html
>> [2] http://camel.apache.org/camel-30-ideas.html
>>
>> *Raúl Kripalani*
>> Enterprise Architect, Open Source Integration specialist, Program
>> Manager | Apache
>> Camel Committer
>> http://about.me/raulkripalani | http://www.linkedin.com/in/raulkripalani
>> http://blog.raulkr.net | twitter: @raulvk
>>
>> On Sat, May 18, 2013 at 3:38 PM, Claus Ibsen <[hidden email]> wrote:
>>
>>> Hi
>>>
>>> Just to tell about the work I am currently doing
>>> https://issues.apache.org/jira/browse/CAMEL-6377
>>>
>>> We have room to optimize the routing engine in Camel to make it more
>>> friendlier to end users in terms of
>>> - reduced stacktraces
>>> - faster and easier to debug the code if you single step through all
>>> the routing engine logic
>>> - potential optimization to skip over functionality that is turned off
>>> (eg such as tracer etc)
>>> - potential optimization to add new cross cutting functionality at
>>> runtime (though not currently the primary goal)
>>> - potential to skip using RedeliveryErrorHandler/DefaultErrorHandler
>>> if end user have not configured to use any kind of redelivery (the
>>> reason is that logic in this guy is excessive and is harder for people
>>> to debug)
>>> - reduce wrapping internal processors when not needed
>>>
>>>
>>> There is a sample route in the ticket where a stacktrace is forced
>>> being thrown. And the difference is 40 vs 28 lines in the stacktrace.
>>> And there is room for further optimization.
>>>
>>> The work is aimed at being non invasive on the current architecture
>>> and also backwards compatible. So far I have noticed a problem with
>>> the old behavior I have logged as:
>>> https://issues.apache.org/jira/browse/CAMEL-6378
>>>
>>>
>>>
>>>
>>> --
>>> Claus Ibsen
>>> -----------------
>>> www.camelone.org: The open source integration conference.
>>>
>>> Red Hat, Inc.
>>> FuseSource is now part of Red Hat
>>> Email: [hidden email]
>>> Web: http://fusesource.com
>>> Twitter: davsclaus
>>> Blog: http://davsclaus.com
>>> Author of Camel in Action: http://www.manning.com/ibsen
>>>
>
>
>
> --
> Claus Ibsen
> -----------------
> www.camelone.org: The open source integration conference.
>
> Red Hat, Inc.
> FuseSource is now part of Red Hat
> Email: [hidden email]
> Web: http://fusesource.com
> Twitter: davsclaus
> Blog: http://davsclaus.com
> Author of Camel in Action: http://www.manning.com/ibsen



--
Claus Ibsen
-----------------
www.camelone.org: The open source integration conference.

Red Hat, Inc.
FuseSource is now part of Red Hat
Email: [hidden email]
Web: http://fusesource.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
Reply | Threaded
Open this post in threaded view
|

Re: [HEADS UP] - Optimzing routing engine to reduce stack frames in use during routing

Babak Vahdat


Am 21.05.13 14:17 schrieb "Claus Ibsen" unter <[hidden email]>:

>Hi
>
>Okay so we got some good headway now.
>
>The sample use case on 2.11 has a stacktrace of about 40 lines (without
>JMX).
>And on trunk we are down to 17 now (without JMX) and +2 if JMX enabled.
>
>That is reducing the stack traces with by 3 fold in that given sample.

Hi Claus

This's just awesome. Thanks!

Babak


>
>
>
>On Mon, May 20, 2013 at 11:16 AM, Claus Ibsen <[hidden email]>
>wrote:
>> Hi
>>
>> I added some tasks as comments on
>> https://issues.apache.org/jira/browse/CAMEL-6377#comment-13661858
>>
>> The goal of CAMEL-6377 is to do *internal* optimiztion changes only
>> with the goal of minimazing the stack-frames in use, as well reduce
>> the long stacktraces our end users see. And also for the end users who
>> go down the path of debugging the Camel routing, then there is less
>> "callback hell".
>>
>> As Raul said we have ideas sketched for Camel 3.0, where we can do
>> bigger changes, and as well do some API changes (if it makes
>> value/sense) etc. And the ideas that Raul refers to is much bigger and
>> takes a lot longer to implement. And possible some prototype is
>> needed. And also care has to be taken as we have DSL in Scala / Groovy
>> / Kotlin etc. that may be affected. And also a large user base on 2.x
>> API that relies on this being stable etc.
>>
>> However as in CAMEL-6377 with fairly little effort (half a friday +
>> half a day on weekend + half toda) + the effort for the remainder
>> tasks (eg the comment on the JIRA) we could get this implemented
>> fairly soon.
>>
>> Its IMHO important to keep the scope on the goals of "reducing
>> stack-frames, less long stacktraces, and easier debugging). All the
>> stuff about iterative routing engine etc, is also what is captured on
>> the Camel 3.0 ideas page, and also sketched in more details by Raul
>> with his idea of "decision" being returned from the processor(s). But
>> that is IMHO not the current scope of CAMEL-6377. The scope is to be
>> internal optimizations on the current 2.x architecture.
>>
>> If people wanna help with CAMEL-6377, then check the JIRA ticket and
>> the list of bulleted tasks.
>>
>> The code changes is about to be committed later this afternoon. Just
>> running one extra fully sanity unit tests of the entire code base, to
>> ensure no regressions or other gremlins introduced. Including all the
>> OSGi tests which you need to run specially.
>>
>>
>>
>>
>> On Sat, May 18, 2013 at 5:39 PM, Raul Kripalani <[hidden email]>
>>wrote:
>>> Hi Claus,
>>>
>>> It's great that you're finding time for this. The need to lighten
>>> stacktraces was already brought up within the Camel 3.0 context [1]. In
>>> fact, there's also an entry in the roadmap page [2] which proposes
>>>moving
>>> away from the recursive model into an iterative routing engine.
>>>
>>> It's clear that the next step in that area is a prototype. I hope I can
>>> start working on that soon, even though the whole Camel 3.0
>>>conversation
>>> seems to have cooled down at present.
>>>
>>> Anyway, do you have a concrete work plan or list of processors you'd
>>>are
>>> targeting with https://issues.apache.org/jira/browse/CAMEL-6377?
>>>
>>> If so, could you create a few individual JIRA tasks? I'd happily take
>>>on
>>> some of the simplification/rationalisation work!
>>>
>>> Regards,
>>>
>>> [1]
>>>
>>>http://camel.465427.n5.nabble.com/DISCUSS-Camel-3-0-Core-of-the-routing-
>>>engine-td5727754.html
>>> [2] http://camel.apache.org/camel-30-ideas.html
>>>
>>> *Raúl Kripalani*
>>> Enterprise Architect, Open Source Integration specialist, Program
>>> Manager | Apache
>>> Camel Committer
>>> http://about.me/raulkripalani |
>>>http://www.linkedin.com/in/raulkripalani
>>> http://blog.raulkr.net | twitter: @raulvk
>>>
>>> On Sat, May 18, 2013 at 3:38 PM, Claus Ibsen <[hidden email]>
>>>wrote:
>>>
>>>> Hi
>>>>
>>>> Just to tell about the work I am currently doing
>>>> https://issues.apache.org/jira/browse/CAMEL-6377
>>>>
>>>> We have room to optimize the routing engine in Camel to make it more
>>>> friendlier to end users in terms of
>>>> - reduced stacktraces
>>>> - faster and easier to debug the code if you single step through all
>>>> the routing engine logic
>>>> - potential optimization to skip over functionality that is turned off
>>>> (eg such as tracer etc)
>>>> - potential optimization to add new cross cutting functionality at
>>>> runtime (though not currently the primary goal)
>>>> - potential to skip using RedeliveryErrorHandler/DefaultErrorHandler
>>>> if end user have not configured to use any kind of redelivery (the
>>>> reason is that logic in this guy is excessive and is harder for people
>>>> to debug)
>>>> - reduce wrapping internal processors when not needed
>>>>
>>>>
>>>> There is a sample route in the ticket where a stacktrace is forced
>>>> being thrown. And the difference is 40 vs 28 lines in the stacktrace.
>>>> And there is room for further optimization.
>>>>
>>>> The work is aimed at being non invasive on the current architecture
>>>> and also backwards compatible. So far I have noticed a problem with
>>>> the old behavior I have logged as:
>>>> https://issues.apache.org/jira/browse/CAMEL-6378
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Claus Ibsen
>>>> -----------------
>>>> www.camelone.org: The open source integration conference.
>>>>
>>>> Red Hat, Inc.
>>>> FuseSource is now part of Red Hat
>>>> Email: [hidden email]
>>>> Web: http://fusesource.com
>>>> Twitter: davsclaus
>>>> Blog: http://davsclaus.com
>>>> Author of Camel in Action: http://www.manning.com/ibsen
>>>>
>>
>>
>>
>> --
>> Claus Ibsen
>> -----------------
>> www.camelone.org: The open source integration conference.
>>
>> Red Hat, Inc.
>> FuseSource is now part of Red Hat
>> Email: [hidden email]
>> Web: http://fusesource.com
>> Twitter: davsclaus
>> Blog: http://davsclaus.com
>> Author of Camel in Action: http://www.manning.com/ibsen
>
>
>
>--
>Claus Ibsen
>-----------------
>www.camelone.org: The open source integration conference.
>
>Red Hat, Inc.
>FuseSource is now part of Red Hat
>Email: [hidden email]
>Web: http://fusesource.com
>Twitter: davsclaus
>Blog: http://davsclaus.com
>Author of Camel in Action: http://www.manning.com/ibsen


Reply | Threaded
Open this post in threaded view
|

Re: [HEADS UP] - Optimzing routing engine to reduce stack frames in use during routing

Christian Mueller
Administrator
In reply to this post by Claus Ibsen-2
I like it!

Sent from a mobile device
Am 21.05.2013 14:18 schrieb "Claus Ibsen" <[hidden email]>:

> Hi
>
> Okay so we got some good headway now.
>
> The sample use case on 2.11 has a stacktrace of about 40 lines (without
> JMX).
> And on trunk we are down to 17 now (without JMX) and +2 if JMX enabled.
>
> That is reducing the stack traces with by 3 fold in that given sample.
>
>
>
> On Mon, May 20, 2013 at 11:16 AM, Claus Ibsen <[hidden email]>
> wrote:
> > Hi
> >
> > I added some tasks as comments on
> > https://issues.apache.org/jira/browse/CAMEL-6377#comment-13661858
> >
> > The goal of CAMEL-6377 is to do *internal* optimiztion changes only
> > with the goal of minimazing the stack-frames in use, as well reduce
> > the long stacktraces our end users see. And also for the end users who
> > go down the path of debugging the Camel routing, then there is less
> > "callback hell".
> >
> > As Raul said we have ideas sketched for Camel 3.0, where we can do
> > bigger changes, and as well do some API changes (if it makes
> > value/sense) etc. And the ideas that Raul refers to is much bigger and
> > takes a lot longer to implement. And possible some prototype is
> > needed. And also care has to be taken as we have DSL in Scala / Groovy
> > / Kotlin etc. that may be affected. And also a large user base on 2.x
> > API that relies on this being stable etc.
> >
> > However as in CAMEL-6377 with fairly little effort (half a friday +
> > half a day on weekend + half toda) + the effort for the remainder
> > tasks (eg the comment on the JIRA) we could get this implemented
> > fairly soon.
> >
> > Its IMHO important to keep the scope on the goals of "reducing
> > stack-frames, less long stacktraces, and easier debugging). All the
> > stuff about iterative routing engine etc, is also what is captured on
> > the Camel 3.0 ideas page, and also sketched in more details by Raul
> > with his idea of "decision" being returned from the processor(s). But
> > that is IMHO not the current scope of CAMEL-6377. The scope is to be
> > internal optimizations on the current 2.x architecture.
> >
> > If people wanna help with CAMEL-6377, then check the JIRA ticket and
> > the list of bulleted tasks.
> >
> > The code changes is about to be committed later this afternoon. Just
> > running one extra fully sanity unit tests of the entire code base, to
> > ensure no regressions or other gremlins introduced. Including all the
> > OSGi tests which you need to run specially.
> >
> >
> >
> >
> > On Sat, May 18, 2013 at 5:39 PM, Raul Kripalani <[hidden email]>
> wrote:
> >> Hi Claus,
> >>
> >> It's great that you're finding time for this. The need to lighten
> >> stacktraces was already brought up within the Camel 3.0 context [1]. In
> >> fact, there's also an entry in the roadmap page [2] which proposes
> moving
> >> away from the recursive model into an iterative routing engine.
> >>
> >> It's clear that the next step in that area is a prototype. I hope I can
> >> start working on that soon, even though the whole Camel 3.0 conversation
> >> seems to have cooled down at present.
> >>
> >> Anyway, do you have a concrete work plan or list of processors you'd are
> >> targeting with https://issues.apache.org/jira/browse/CAMEL-6377?
> >>
> >> If so, could you create a few individual JIRA tasks? I'd happily take on
> >> some of the simplification/rationalisation work!
> >>
> >> Regards,
> >>
> >> [1]
> >>
> http://camel.465427.n5.nabble.com/DISCUSS-Camel-3-0-Core-of-the-routing-engine-td5727754.html
> >> [2] http://camel.apache.org/camel-30-ideas.html
> >>
> >> *Raúl Kripalani*
> >> Enterprise Architect, Open Source Integration specialist, Program
> >> Manager | Apache
> >> Camel Committer
> >> http://about.me/raulkripalani |
> http://www.linkedin.com/in/raulkripalani
> >> http://blog.raulkr.net | twitter: @raulvk
> >>
> >> On Sat, May 18, 2013 at 3:38 PM, Claus Ibsen <[hidden email]>
> wrote:
> >>
> >>> Hi
> >>>
> >>> Just to tell about the work I am currently doing
> >>> https://issues.apache.org/jira/browse/CAMEL-6377
> >>>
> >>> We have room to optimize the routing engine in Camel to make it more
> >>> friendlier to end users in terms of
> >>> - reduced stacktraces
> >>> - faster and easier to debug the code if you single step through all
> >>> the routing engine logic
> >>> - potential optimization to skip over functionality that is turned off
> >>> (eg such as tracer etc)
> >>> - potential optimization to add new cross cutting functionality at
> >>> runtime (though not currently the primary goal)
> >>> - potential to skip using RedeliveryErrorHandler/DefaultErrorHandler
> >>> if end user have not configured to use any kind of redelivery (the
> >>> reason is that logic in this guy is excessive and is harder for people
> >>> to debug)
> >>> - reduce wrapping internal processors when not needed
> >>>
> >>>
> >>> There is a sample route in the ticket where a stacktrace is forced
> >>> being thrown. And the difference is 40 vs 28 lines in the stacktrace.
> >>> And there is room for further optimization.
> >>>
> >>> The work is aimed at being non invasive on the current architecture
> >>> and also backwards compatible. So far I have noticed a problem with
> >>> the old behavior I have logged as:
> >>> https://issues.apache.org/jira/browse/CAMEL-6378
> >>>
> >>>
> >>>
> >>>
> >>> --
> >>> Claus Ibsen
> >>> -----------------
> >>> www.camelone.org: The open source integration conference.
> >>>
> >>> Red Hat, Inc.
> >>> FuseSource is now part of Red Hat
> >>> Email: [hidden email]
> >>> Web: http://fusesource.com
> >>> Twitter: davsclaus
> >>> Blog: http://davsclaus.com
> >>> Author of Camel in Action: http://www.manning.com/ibsen
> >>>
> >
> >
> >
> > --
> > Claus Ibsen
> > -----------------
> > www.camelone.org: The open source integration conference.
> >
> > Red Hat, Inc.
> > FuseSource is now part of Red Hat
> > Email: [hidden email]
> > Web: http://fusesource.com
> > Twitter: davsclaus
> > Blog: http://davsclaus.com
> > Author of Camel in Action: http://www.manning.com/ibsen
>
>
>
> --
> Claus Ibsen
> -----------------
> www.camelone.org: The open source integration conference.
>
> Red Hat, Inc.
> FuseSource is now part of Red Hat
> Email: [hidden email]
> Web: http://fusesource.com
> Twitter: davsclaus
> Blog: http://davsclaus.com
> Author of Camel in Action: http://www.manning.com/ibsen
>
Reply | Threaded
Open this post in threaded view
|

Re: [HEADS UP] - Optimzing routing engine to reduce stack frames in use during routing

gnodet
In reply to this post by Claus Ibsen-2
Awesome work Claus ! This will make camel much easier to debug imho.



2013/5/20 Claus Ibsen <[hidden email]>

> Hi
>
> I added some tasks as comments on
> https://issues.apache.org/jira/browse/CAMEL-6377#comment-13661858
>
> The goal of CAMEL-6377 is to do *internal* optimiztion changes only
> with the goal of minimazing the stack-frames in use, as well reduce
> the long stacktraces our end users see. And also for the end users who
> go down the path of debugging the Camel routing, then there is less
> "callback hell".
>
> As Raul said we have ideas sketched for Camel 3.0, where we can do
> bigger changes, and as well do some API changes (if it makes
> value/sense) etc. And the ideas that Raul refers to is much bigger and
> takes a lot longer to implement. And possible some prototype is
> needed. And also care has to be taken as we have DSL in Scala / Groovy
> / Kotlin etc. that may be affected. And also a large user base on 2.x
> API that relies on this being stable etc.
>
> However as in CAMEL-6377 with fairly little effort (half a friday +
> half a day on weekend + half toda) + the effort for the remainder
> tasks (eg the comment on the JIRA) we could get this implemented
> fairly soon.
>
> Its IMHO important to keep the scope on the goals of "reducing
> stack-frames, less long stacktraces, and easier debugging). All the
> stuff about iterative routing engine etc, is also what is captured on
> the Camel 3.0 ideas page, and also sketched in more details by Raul
> with his idea of "decision" being returned from the processor(s). But
> that is IMHO not the current scope of CAMEL-6377. The scope is to be
> internal optimizations on the current 2.x architecture.
>
> If people wanna help with CAMEL-6377, then check the JIRA ticket and
> the list of bulleted tasks.
>
> The code changes is about to be committed later this afternoon. Just
> running one extra fully sanity unit tests of the entire code base, to
> ensure no regressions or other gremlins introduced. Including all the
> OSGi tests which you need to run specially.
>
>
>
>
> On Sat, May 18, 2013 at 5:39 PM, Raul Kripalani <[hidden email]> wrote:
> > Hi Claus,
> >
> > It's great that you're finding time for this. The need to lighten
> > stacktraces was already brought up within the Camel 3.0 context [1]. In
> > fact, there's also an entry in the roadmap page [2] which proposes moving
> > away from the recursive model into an iterative routing engine.
> >
> > It's clear that the next step in that area is a prototype. I hope I can
> > start working on that soon, even though the whole Camel 3.0 conversation
> > seems to have cooled down at present.
> >
> > Anyway, do you have a concrete work plan or list of processors you'd are
> > targeting with https://issues.apache.org/jira/browse/CAMEL-6377?
> >
> > If so, could you create a few individual JIRA tasks? I'd happily take on
> > some of the simplification/rationalisation work!
> >
> > Regards,
> >
> > [1]
> >
> http://camel.465427.n5.nabble.com/DISCUSS-Camel-3-0-Core-of-the-routing-engine-td5727754.html
> > [2] http://camel.apache.org/camel-30-ideas.html
> >
> > *Raúl Kripalani*
> > Enterprise Architect, Open Source Integration specialist, Program
> > Manager | Apache
> > Camel Committer
> > http://about.me/raulkripalani | http://www.linkedin.com/in/raulkripalani
> > http://blog.raulkr.net | twitter: @raulvk
> >
> > On Sat, May 18, 2013 at 3:38 PM, Claus Ibsen <[hidden email]>
> wrote:
> >
> >> Hi
> >>
> >> Just to tell about the work I am currently doing
> >> https://issues.apache.org/jira/browse/CAMEL-6377
> >>
> >> We have room to optimize the routing engine in Camel to make it more
> >> friendlier to end users in terms of
> >> - reduced stacktraces
> >> - faster and easier to debug the code if you single step through all
> >> the routing engine logic
> >> - potential optimization to skip over functionality that is turned off
> >> (eg such as tracer etc)
> >> - potential optimization to add new cross cutting functionality at
> >> runtime (though not currently the primary goal)
> >> - potential to skip using RedeliveryErrorHandler/DefaultErrorHandler
> >> if end user have not configured to use any kind of redelivery (the
> >> reason is that logic in this guy is excessive and is harder for people
> >> to debug)
> >> - reduce wrapping internal processors when not needed
> >>
> >>
> >> There is a sample route in the ticket where a stacktrace is forced
> >> being thrown. And the difference is 40 vs 28 lines in the stacktrace.
> >> And there is room for further optimization.
> >>
> >> The work is aimed at being non invasive on the current architecture
> >> and also backwards compatible. So far I have noticed a problem with
> >> the old behavior I have logged as:
> >> https://issues.apache.org/jira/browse/CAMEL-6378
> >>
> >>
> >>
> >>
> >> --
> >> Claus Ibsen
> >> -----------------
> >> www.camelone.org: The open source integration conference.
> >>
> >> Red Hat, Inc.
> >> FuseSource is now part of Red Hat
> >> Email: [hidden email]
> >> Web: http://fusesource.com
> >> Twitter: davsclaus
> >> Blog: http://davsclaus.com
> >> Author of Camel in Action: http://www.manning.com/ibsen
> >>
>
>
>
> --
> Claus Ibsen
> -----------------
> www.camelone.org: The open source integration conference.
>
> Red Hat, Inc.
> FuseSource is now part of Red Hat
> Email: [hidden email]
> Web: http://fusesource.com
> Twitter: davsclaus
> Blog: http://davsclaus.com
> Author of Camel in Action: http://www.manning.com/ibsen
>
Reply | Threaded
Open this post in threaded view
|

Re: [HEADS UP] - Optimzing routing engine to reduce stack frames in use during routing

Jon Anstey
In reply to this post by Claus Ibsen-2
Wow. Nice improvement!
On 2013-05-21 8:18 AM, "Claus Ibsen" <[hidden email]> wrote:

> Hi
>
> Okay so we got some good headway now.
>
> The sample use case on 2.11 has a stacktrace of about 40 lines (without
> JMX).
> And on trunk we are down to 17 now (without JMX) and +2 if JMX enabled.
>
> That is reducing the stack traces with by 3 fold in that given sample.
>
>
>
> On Mon, May 20, 2013 at 11:16 AM, Claus Ibsen <[hidden email]>
> wrote:
> > Hi
> >
> > I added some tasks as comments on
> > https://issues.apache.org/jira/browse/CAMEL-6377#comment-13661858
> >
> > The goal of CAMEL-6377 is to do *internal* optimiztion changes only
> > with the goal of minimazing the stack-frames in use, as well reduce
> > the long stacktraces our end users see. And also for the end users who
> > go down the path of debugging the Camel routing, then there is less
> > "callback hell".
> >
> > As Raul said we have ideas sketched for Camel 3.0, where we can do
> > bigger changes, and as well do some API changes (if it makes
> > value/sense) etc. And the ideas that Raul refers to is much bigger and
> > takes a lot longer to implement. And possible some prototype is
> > needed. And also care has to be taken as we have DSL in Scala / Groovy
> > / Kotlin etc. that may be affected. And also a large user base on 2.x
> > API that relies on this being stable etc.
> >
> > However as in CAMEL-6377 with fairly little effort (half a friday +
> > half a day on weekend + half toda) + the effort for the remainder
> > tasks (eg the comment on the JIRA) we could get this implemented
> > fairly soon.
> >
> > Its IMHO important to keep the scope on the goals of "reducing
> > stack-frames, less long stacktraces, and easier debugging). All the
> > stuff about iterative routing engine etc, is also what is captured on
> > the Camel 3.0 ideas page, and also sketched in more details by Raul
> > with his idea of "decision" being returned from the processor(s). But
> > that is IMHO not the current scope of CAMEL-6377. The scope is to be
> > internal optimizations on the current 2.x architecture.
> >
> > If people wanna help with CAMEL-6377, then check the JIRA ticket and
> > the list of bulleted tasks.
> >
> > The code changes is about to be committed later this afternoon. Just
> > running one extra fully sanity unit tests of the entire code base, to
> > ensure no regressions or other gremlins introduced. Including all the
> > OSGi tests which you need to run specially.
> >
> >
> >
> >
> > On Sat, May 18, 2013 at 5:39 PM, Raul Kripalani <[hidden email]>
> wrote:
> >> Hi Claus,
> >>
> >> It's great that you're finding time for this. The need to lighten
> >> stacktraces was already brought up within the Camel 3.0 context [1]. In
> >> fact, there's also an entry in the roadmap page [2] which proposes
> moving
> >> away from the recursive model into an iterative routing engine.
> >>
> >> It's clear that the next step in that area is a prototype. I hope I can
> >> start working on that soon, even though the whole Camel 3.0 conversation
> >> seems to have cooled down at present.
> >>
> >> Anyway, do you have a concrete work plan or list of processors you'd are
> >> targeting with https://issues.apache.org/jira/browse/CAMEL-6377?
> >>
> >> If so, could you create a few individual JIRA tasks? I'd happily take on
> >> some of the simplification/rationalisation work!
> >>
> >> Regards,
> >>
> >> [1]
> >>
> http://camel.465427.n5.nabble.com/DISCUSS-Camel-3-0-Core-of-the-routing-engine-td5727754.html
> >> [2] http://camel.apache.org/camel-30-ideas.html
> >>
> >> *Raúl Kripalani*
> >> Enterprise Architect, Open Source Integration specialist, Program
> >> Manager | Apache
> >> Camel Committer
> >> http://about.me/raulkripalani |
> http://www.linkedin.com/in/raulkripalani
> >> http://blog.raulkr.net | twitter: @raulvk
> >>
> >> On Sat, May 18, 2013 at 3:38 PM, Claus Ibsen <[hidden email]>
> wrote:
> >>
> >>> Hi
> >>>
> >>> Just to tell about the work I am currently doing
> >>> https://issues.apache.org/jira/browse/CAMEL-6377
> >>>
> >>> We have room to optimize the routing engine in Camel to make it more
> >>> friendlier to end users in terms of
> >>> - reduced stacktraces
> >>> - faster and easier to debug the code if you single step through all
> >>> the routing engine logic
> >>> - potential optimization to skip over functionality that is turned off
> >>> (eg such as tracer etc)
> >>> - potential optimization to add new cross cutting functionality at
> >>> runtime (though not currently the primary goal)
> >>> - potential to skip using RedeliveryErrorHandler/DefaultErrorHandler
> >>> if end user have not configured to use any kind of redelivery (the
> >>> reason is that logic in this guy is excessive and is harder for people
> >>> to debug)
> >>> - reduce wrapping internal processors when not needed
> >>>
> >>>
> >>> There is a sample route in the ticket where a stacktrace is forced
> >>> being thrown. And the difference is 40 vs 28 lines in the stacktrace.
> >>> And there is room for further optimization.
> >>>
> >>> The work is aimed at being non invasive on the current architecture
> >>> and also backwards compatible. So far I have noticed a problem with
> >>> the old behavior I have logged as:
> >>> https://issues.apache.org/jira/browse/CAMEL-6378
> >>>
> >>>
> >>>
> >>>
> >>> --
> >>> Claus Ibsen
> >>> -----------------
> >>> www.camelone.org: The open source integration conference.
> >>>
> >>> Red Hat, Inc.
> >>> FuseSource is now part of Red Hat
> >>> Email: [hidden email]
> >>> Web: http://fusesource.com
> >>> Twitter: davsclaus
> >>> Blog: http://davsclaus.com
> >>> Author of Camel in Action: http://www.manning.com/ibsen
> >>>
> >
> >
> >
> > --
> > Claus Ibsen
> > -----------------
> > www.camelone.org: The open source integration conference.
> >
> > Red Hat, Inc.
> > FuseSource is now part of Red Hat
> > Email: [hidden email]
> > Web: http://fusesource.com
> > Twitter: davsclaus
> > Blog: http://davsclaus.com
> > Author of Camel in Action: http://www.manning.com/ibsen
>
>
>
> --
> Claus Ibsen
> -----------------
> www.camelone.org: The open source integration conference.
>
> Red Hat, Inc.
> FuseSource is now part of Red Hat
> Email: [hidden email]
> Web: http://fusesource.com
> Twitter: davsclaus
> Blog: http://davsclaus.com
> Author of Camel in Action: http://www.manning.com/ibsen
>
Reply | Threaded
Open this post in threaded view
|

Re: [HEADS UP] - Optimzing routing engine to reduce stack frames in use during routing

Willem.Jiang
Administrator
In reply to this post by Claus Ibsen-2
Cool. This change makes Camel more green :)


--  
Willem Jiang

Red Hat, Inc.
FuseSource is now part of Red Hat
Web: http://www.fusesource.com | http://www.redhat.com
Blog: http://willemjiang.blogspot.com (http://willemjiang.blogspot.com/) (English)
          http://jnn.iteye.com (http://jnn.javaeye.com/) (Chinese)
Twitter: willemjiang  
Weibo: 姜宁willem





On Tuesday, May 21, 2013 at 8:17 PM, Claus Ibsen wrote:

> Hi
>  
> Okay so we got some good headway now.
>  
> The sample use case on 2.11 has a stacktrace of about 40 lines (without JMX).
> And on trunk we are down to 17 now (without JMX) and +2 if JMX enabled.
>  
> That is reducing the stack traces with by 3 fold in that given sample.
>  
>  
>  
> On Mon, May 20, 2013 at 11:16 AM, Claus Ibsen <[hidden email] (mailto:[hidden email])> wrote:
> > Hi
> >  
> > I added some tasks as comments on
> > https://issues.apache.org/jira/browse/CAMEL-6377#comment-13661858
> >  
> > The goal of CAMEL-6377 is to do *internal* optimiztion changes only
> > with the goal of minimazing the stack-frames in use, as well reduce
> > the long stacktraces our end users see. And also for the end users who
> > go down the path of debugging the Camel routing, then there is less
> > "callback hell".
> >  
> > As Raul said we have ideas sketched for Camel 3.0, where we can do
> > bigger changes, and as well do some API changes (if it makes
> > value/sense) etc. And the ideas that Raul refers to is much bigger and
> > takes a lot longer to implement. And possible some prototype is
> > needed. And also care has to be taken as we have DSL in Scala / Groovy
> > / Kotlin etc. that may be affected. And also a large user base on 2.x
> > API that relies on this being stable etc.
> >  
> > However as in CAMEL-6377 with fairly little effort (half a friday +
> > half a day on weekend + half toda) + the effort for the remainder
> > tasks (eg the comment on the JIRA) we could get this implemented
> > fairly soon.
> >  
> > Its IMHO important to keep the scope on the goals of "reducing
> > stack-frames, less long stacktraces, and easier debugging). All the
> > stuff about iterative routing engine etc, is also what is captured on
> > the Camel 3.0 ideas page, and also sketched in more details by Raul
> > with his idea of "decision" being returned from the processor(s). But
> > that is IMHO not the current scope of CAMEL-6377. The scope is to be
> > internal optimizations on the current 2.x architecture.
> >  
> > If people wanna help with CAMEL-6377, then check the JIRA ticket and
> > the list of bulleted tasks.
> >  
> > The code changes is about to be committed later this afternoon. Just
> > running one extra fully sanity unit tests of the entire code base, to
> > ensure no regressions or other gremlins introduced. Including all the
> > OSGi tests which you need to run specially.
> >  
> >  
> >  
> >  
> > On Sat, May 18, 2013 at 5:39 PM, Raul Kripalani <[hidden email] (mailto:[hidden email])> wrote:
> > > Hi Claus,
> > >  
> > > It's great that you're finding time for this. The need to lighten
> > > stacktraces was already brought up within the Camel 3.0 context [1]. In
> > > fact, there's also an entry in the roadmap page [2] which proposes moving
> > > away from the recursive model into an iterative routing engine.
> > >  
> > > It's clear that the next step in that area is a prototype. I hope I can
> > > start working on that soon, even though the whole Camel 3.0 conversation
> > > seems to have cooled down at present.
> > >  
> > > Anyway, do you have a concrete work plan or list of processors you'd are
> > > targeting with https://issues.apache.org/jira/browse/CAMEL-6377?
> > >  
> > > If so, could you create a few individual JIRA tasks? I'd happily take on
> > > some of the simplification/rationalisation work!
> > >  
> > > Regards,
> > >  
> > > [1]
> > > http://camel.465427.n5.nabble.com/DISCUSS-Camel-3-0-Core-of-the-routing-engine-td5727754.html
> > > [2] http://camel.apache.org/camel-30-ideas.html
> > >  
> > > *Raúl Kripalani*
> > > Enterprise Architect, Open Source Integration specialist, Program
> > > Manager | Apache
> > > Camel Committer
> > > http://about.me/raulkripalani | http://www.linkedin.com/in/raulkripalani
> > > http://blog.raulkr.net | twitter: @raulvk
> > >  
> > > On Sat, May 18, 2013 at 3:38 PM, Claus Ibsen <[hidden email] (mailto:[hidden email])> wrote:
> > >  
> > > > Hi
> > > >  
> > > > Just to tell about the work I am currently doing
> > > > https://issues.apache.org/jira/browse/CAMEL-6377
> > > >  
> > > > We have room to optimize the routing engine in Camel to make it more
> > > > friendlier to end users in terms of
> > > > - reduced stacktraces
> > > > - faster and easier to debug the code if you single step through all
> > > > the routing engine logic
> > > > - potential optimization to skip over functionality that is turned off
> > > > (eg such as tracer etc)
> > > > - potential optimization to add new cross cutting functionality at
> > > > runtime (though not currently the primary goal)
> > > > - potential to skip using RedeliveryErrorHandler/DefaultErrorHandler
> > > > if end user have not configured to use any kind of redelivery (the
> > > > reason is that logic in this guy is excessive and is harder for people
> > > > to debug)
> > > > - reduce wrapping internal processors when not needed
> > > >  
> > > >  
> > > > There is a sample route in the ticket where a stacktrace is forced
> > > > being thrown. And the difference is 40 vs 28 lines in the stacktrace.
> > > > And there is room for further optimization.
> > > >  
> > > > The work is aimed at being non invasive on the current architecture
> > > > and also backwards compatible. So far I have noticed a problem with
> > > > the old behavior I have logged as:
> > > > https://issues.apache.org/jira/browse/CAMEL-6378
> > > >  
> > > >  
> > > >  
> > > >  
> > > > --
> > > > Claus Ibsen
> > > > -----------------
> > > > www.camelone.org (http://www.camelone.org): The open source integration conference.
> > > >  
> > > > Red Hat, Inc.
> > > > FuseSource is now part of Red Hat
> > > > Email: [hidden email] (mailto:[hidden email])
> > > > Web: http://fusesource.com
> > > > Twitter: davsclaus
> > > > Blog: http://davsclaus.com
> > > > Author of Camel in Action: http://www.manning.com/ibsen
> > >  
> >  
> >  
> >  
> >  
> >  
> > --
> > Claus Ibsen
> > -----------------
> > www.camelone.org (http://www.camelone.org): The open source integration conference.
> >  
> > Red Hat, Inc.
> > FuseSource is now part of Red Hat
> > Email: [hidden email] (mailto:[hidden email])
> > Web: http://fusesource.com
> > Twitter: davsclaus
> > Blog: http://davsclaus.com
> > Author of Camel in Action: http://www.manning.com/ibsen
>  
>  
>  
>  
>  
> --  
> Claus Ibsen
> -----------------
> www.camelone.org (http://www.camelone.org): The open source integration conference.
>  
> Red Hat, Inc.
> FuseSource is now part of Red Hat
> Email: [hidden email] (mailto:[hidden email])
> Web: http://fusesource.com
> Twitter: davsclaus
> Blog: http://davsclaus.com
> Author of Camel in Action: http://www.manning.com/ibsen



Reply | Threaded
Open this post in threaded view
|

Re: [HEADS UP] - Optimzing routing engine to reduce stack frames in use during routing

Raul Kripalani
In reply to this post by Claus Ibsen-2
Yes, exactly. Completely agree with this approach.

I'm amazed with the amount of stack frame bloat you've been able to remove
already! Great work!

I'm planning to contribute too. Will sync up with you on IRC.

Regards,

*Raúl Kripalani*
Enterprise Architect, Open Source Integration specialist, Program
Manager | Apache
Camel Committer
http://about.me/raulkripalani | http://www.linkedin.com/in/raulkripalani
http://blog.raulkr.net | twitter: @raulvk

On Mon, May 20, 2013 at 10:16 AM, Claus Ibsen <[hidden email]> wrote:

> Hi
>
> I added some tasks as comments on
> https://issues.apache.org/jira/browse/CAMEL-6377#comment-13661858
>
> The goal of CAMEL-6377 is to do *internal* optimiztion changes only
> with the goal of minimazing the stack-frames in use, as well reduce
> the long stacktraces our end users see. And also for the end users who
> go down the path of debugging the Camel routing, then there is less
> "callback hell".
>
> As Raul said we have ideas sketched for Camel 3.0, where we can do
> bigger changes, and as well do some API changes (if it makes
> value/sense) etc. And the ideas that Raul refers to is much bigger and
> takes a lot longer to implement. And possible some prototype is
> needed. And also care has to be taken as we have DSL in Scala / Groovy
> / Kotlin etc. that may be affected. And also a large user base on 2.x
> API that relies on this being stable etc.
>
> However as in CAMEL-6377 with fairly little effort (half a friday +
> half a day on weekend + half toda) + the effort for the remainder
> tasks (eg the comment on the JIRA) we could get this implemented
> fairly soon.
>
> Its IMHO important to keep the scope on the goals of "reducing
> stack-frames, less long stacktraces, and easier debugging). All the
> stuff about iterative routing engine etc, is also what is captured on
> the Camel 3.0 ideas page, and also sketched in more details by Raul
> with his idea of "decision" being returned from the processor(s). But
> that is IMHO not the current scope of CAMEL-6377. The scope is to be
> internal optimizations on the current 2.x architecture.
>
> If people wanna help with CAMEL-6377, then check the JIRA ticket and
> the list of bulleted tasks.
>
> The code changes is about to be committed later this afternoon. Just
> running one extra fully sanity unit tests of the entire code base, to
> ensure no regressions or other gremlins introduced. Including all the
> OSGi tests which you need to run specially.
>
>
>
>
> On Sat, May 18, 2013 at 5:39 PM, Raul Kripalani <[hidden email]> wrote:
> > Hi Claus,
> >
> > It's great that you're finding time for this. The need to lighten
> > stacktraces was already brought up within the Camel 3.0 context [1]. In
> > fact, there's also an entry in the roadmap page [2] which proposes moving
> > away from the recursive model into an iterative routing engine.
> >
> > It's clear that the next step in that area is a prototype. I hope I can
> > start working on that soon, even though the whole Camel 3.0 conversation
> > seems to have cooled down at present.
> >
> > Anyway, do you have a concrete work plan or list of processors you'd are
> > targeting with https://issues.apache.org/jira/browse/CAMEL-6377?
> >
> > If so, could you create a few individual JIRA tasks? I'd happily take on
> > some of the simplification/rationalisation work!
> >
> > Regards,
> >
> > [1]
> >
> http://camel.465427.n5.nabble.com/DISCUSS-Camel-3-0-Core-of-the-routing-engine-td5727754.html
> > [2] http://camel.apache.org/camel-30-ideas.html
> >
> > *Raúl Kripalani*
> > Enterprise Architect, Open Source Integration specialist, Program
> > Manager | Apache
> > Camel Committer
> > http://about.me/raulkripalani | http://www.linkedin.com/in/raulkripalani
> > http://blog.raulkr.net | twitter: @raulvk
> >
> > On Sat, May 18, 2013 at 3:38 PM, Claus Ibsen <[hidden email]>
> wrote:
> >
> >> Hi
> >>
> >> Just to tell about the work I am currently doing
> >> https://issues.apache.org/jira/browse/CAMEL-6377
> >>
> >> We have room to optimize the routing engine in Camel to make it more
> >> friendlier to end users in terms of
> >> - reduced stacktraces
> >> - faster and easier to debug the code if you single step through all
> >> the routing engine logic
> >> - potential optimization to skip over functionality that is turned off
> >> (eg such as tracer etc)
> >> - potential optimization to add new cross cutting functionality at
> >> runtime (though not currently the primary goal)
> >> - potential to skip using RedeliveryErrorHandler/DefaultErrorHandler
> >> if end user have not configured to use any kind of redelivery (the
> >> reason is that logic in this guy is excessive and is harder for people
> >> to debug)
> >> - reduce wrapping internal processors when not needed
> >>
> >>
> >> There is a sample route in the ticket where a stacktrace is forced
> >> being thrown. And the difference is 40 vs 28 lines in the stacktrace.
> >> And there is room for further optimization.
> >>
> >> The work is aimed at being non invasive on the current architecture
> >> and also backwards compatible. So far I have noticed a problem with
> >> the old behavior I have logged as:
> >> https://issues.apache.org/jira/browse/CAMEL-6378
> >>
> >>
> >>
> >>
> >> --
> >> Claus Ibsen
> >> -----------------
> >> www.camelone.org: The open source integration conference.
> >>
> >> Red Hat, Inc.
> >> FuseSource is now part of Red Hat
> >> Email: [hidden email]
> >> Web: http://fusesource.com
> >> Twitter: davsclaus
> >> Blog: http://davsclaus.com
> >> Author of Camel in Action: http://www.manning.com/ibsen
> >>
>
>
>
> --
> Claus Ibsen
> -----------------
> www.camelone.org: The open source integration conference.
>
> Red Hat, Inc.
> FuseSource is now part of Red Hat
> Email: [hidden email]
> Web: http://fusesource.com
> Twitter: davsclaus
> Blog: http://davsclaus.com
> Author of Camel in Action: http://www.manning.com/ibsen
>
Reply | Threaded
Open this post in threaded view
|

Re: [HEADS UP] - Optimzing routing engine to reduce stack frames in use during routing

Claus Ibsen-2
In reply to this post by Claus Ibsen-2
Hi

So after a week we got good progress on this one.
We have been able to reduce the stack-frames with a factor of 2 - 3 times.

The sample we have been using for benchmark is down from 40 to 16.
And there is room for 1, 2 or 3 more to be shaved off.


Possible #1)
The JMX InstrumentationProcessor is harder to "reduce" as it wraps the
processor in the route to be executed, eg
- an EIP
- a custom bean
- a custom processor
- etc.

So it sits there and track utilization, how many calls, how long time,
how many success / failures / redeliveries etc.
And its this fine grained "redelivery" that is a challenge.

If we want to keep having fine grained redelivery tracking and
whatnot, then it has to sit just at the edge of the actual processor
being invoked, eg in between the error handler and the processor. So
when the error handler "kick in" and do a redelivery, we can keep
track of that.


Possible #2)
Just noticed these two which can be optimized. This ought to be easy and doable.

at org.apache.camel.processor.RedeliveryErrorHandler.processErrorHandler(RedeliveryErrorHandler.java:391)
at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:273)


Possible #3)
When calling a custom Processor that is only sync we could enhance
logic at places to call the sync processor without using the
ProcessorToAsyncProcessorBridge. This is also what gnodet have
suggested.













--
Claus Ibsen
-----------------
www.camelone.org: The open source integration conference.

Red Hat, Inc.
FuseSource is now part of Red Hat
Email: [hidden email]
Web: http://fusesource.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
Reply | Threaded
Open this post in threaded view
|

Re: [HEADS UP] - Optimzing routing engine to reduce stack frames in use during routing

gnodet
2013/5/28 Claus Ibsen <[hidden email]>

> Hi
>
> So after a week we got good progress on this one.
> We have been able to reduce the stack-frames with a factor of 2 - 3 times.
>
> The sample we have been using for benchmark is down from 40 to 16.
> And there is room for 1, 2 or 3 more to be shaved off.
>
>
> Possible #1)
> The JMX InstrumentationProcessor is harder to "reduce" as it wraps the
> processor in the route to be executed, eg
> - an EIP
> - a custom bean
> - a custom processor
> - etc.
>
> So it sits there and track utilization, how many calls, how long time,
> how many success / failures / redeliveries etc.
> And its this fine grained "redelivery" that is a challenge.
>
> If we want to keep having fine grained redelivery tracking and
> whatnot, then it has to sit just at the edge of the actual processor
> being invoked, eg in between the error handler and the processor. So
> when the error handler "kick in" and do a redelivery, we can keep
> track of that.
>

I don't think we should go too far and remove any feature.

>
>
> Possible #2)
> Just noticed these two which can be optimized. This ought to be easy and
> doable.
>
> at
> org.apache.camel.processor.RedeliveryErrorHandler.processErrorHandler(RedeliveryErrorHandler.java:391)
> at
> org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:273)
>
>
> Possible #3)
> When calling a custom Processor that is only sync we could enhance
> logic at places to call the sync processor without using the
> ProcessorToAsyncProcessorBridge. This is also what gnodet have
> suggested.


What I had in mind is rather the opposite, i.e. make sure we don't use any
Processor which does not implement AsyncProcessor, not calling the
processor synchronously.
I think it's easier to do and has the same result, i.e. not having to call
ProcessorToAsyncProcessorBridge.


>
>
>
>
>
>
>
>
>
>
>
> --
> Claus Ibsen
> -----------------
> www.camelone.org: The open source integration conference.
>
> Red Hat, Inc.
> FuseSource is now part of Red Hat
> Email: [hidden email]
> Web: http://fusesource.com
> Twitter: davsclaus
> Blog: http://davsclaus.com
> Author of Camel in Action: http://www.manning.com/ibsen
>
Reply | Threaded
Open this post in threaded view
|

Re: [HEADS UP] - Optimzing routing engine to reduce stack frames in use during routing

Claus Ibsen-2
On Tue, May 28, 2013 at 11:02 AM, Guillaume Nodet <[hidden email]> wrote:

> 2013/5/28 Claus Ibsen <[hidden email]>
>
>> Hi
>>
>> So after a week we got good progress on this one.
>> We have been able to reduce the stack-frames with a factor of 2 - 3 times.
>>
>> The sample we have been using for benchmark is down from 40 to 16.
>> And there is room for 1, 2 or 3 more to be shaved off.
>>
>>
>> Possible #1)
>> The JMX InstrumentationProcessor is harder to "reduce" as it wraps the
>> processor in the route to be executed, eg
>> - an EIP
>> - a custom bean
>> - a custom processor
>> - etc.
>>
>> So it sits there and track utilization, how many calls, how long time,
>> how many success / failures / redeliveries etc.
>> And its this fine grained "redelivery" that is a challenge.
>>
>> If we want to keep having fine grained redelivery tracking and
>> whatnot, then it has to sit just at the edge of the actual processor
>> being invoked, eg in between the error handler and the processor. So
>> when the error handler "kick in" and do a redelivery, we can keep
>> track of that.
>>
>
> I don't think we should go too far and remove any feature.
>
>>
>>
>> Possible #2)
>> Just noticed these two which can be optimized. This ought to be easy and
>> doable.
>>
>> at
>> org.apache.camel.processor.RedeliveryErrorHandler.processErrorHandler(RedeliveryErrorHandler.java:391)
>> at
>> org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:273)
>>
>>
>> Possible #3)
>> When calling a custom Processor that is only sync we could enhance
>> logic at places to call the sync processor without using the
>> ProcessorToAsyncProcessorBridge. This is also what gnodet have
>> suggested.
>
>
> What I had in mind is rather the opposite, i.e. make sure we don't use any
> Processor which does not implement AsyncProcessor, not calling the
> processor synchronously.
> I think it's easier to do and has the same result, i.e. not having to call
> ProcessorToAsyncProcessorBridge.
>

Yeah this would be possible for all the EIPs and processors we offer
out of the box in Camel.

But for end users who implement a custom Processor, eg just do
implements Processor, then we would need to use the bridge, or enhance
the logic in the routing engine, to not use the bridge and call the
sync method directly.

Though yeah if all the EIPs etc in Camel is AsyncProcessor then we are
all good. Its only the custom end user Processor that may be sync
only.

>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> --
>> Claus Ibsen
>> -----------------
>> www.camelone.org: The open source integration conference.
>>
>> Red Hat, Inc.
>> FuseSource is now part of Red Hat
>> Email: [hidden email]
>> Web: http://fusesource.com
>> Twitter: davsclaus
>> Blog: http://davsclaus.com
>> Author of Camel in Action: http://www.manning.com/ibsen
>>



--
Claus Ibsen
-----------------
www.camelone.org: The open source integration conference.

Red Hat, Inc.
FuseSource is now part of Red Hat
Email: [hidden email]
Web: http://fusesource.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
Reply | Threaded
Open this post in threaded view
|

Re: [HEADS UP] - Optimzing routing engine to reduce stack frames in use during routing

Claus Ibsen-2
In reply to this post by Claus Ibsen-2
Possible 2 + 3 implemented.


Raul did you by any chance had a chance to take a look also?

What remains is gnodets suggest which has been listed as task 18.
And then the naming of the new API and possible adding a bit more javadoc etc.




On Tue, May 28, 2013 at 10:26 AM, Claus Ibsen <[hidden email]> wrote:

> Hi
>
> So after a week we got good progress on this one.
> We have been able to reduce the stack-frames with a factor of 2 - 3 times.
>
> The sample we have been using for benchmark is down from 40 to 16.
> And there is room for 1, 2 or 3 more to be shaved off.
>
>
> Possible #1)
> The JMX InstrumentationProcessor is harder to "reduce" as it wraps the
> processor in the route to be executed, eg
> - an EIP
> - a custom bean
> - a custom processor
> - etc.
>
> So it sits there and track utilization, how many calls, how long time,
> how many success / failures / redeliveries etc.
> And its this fine grained "redelivery" that is a challenge.
>
> If we want to keep having fine grained redelivery tracking and
> whatnot, then it has to sit just at the edge of the actual processor
> being invoked, eg in between the error handler and the processor. So
> when the error handler "kick in" and do a redelivery, we can keep
> track of that.
>
>
> Possible #2)
> Just noticed these two which can be optimized. This ought to be easy and doable.
>
> at org.apache.camel.processor.RedeliveryErrorHandler.processErrorHandler(RedeliveryErrorHandler.java:391)
> at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:273)
>
>
> Possible #3)
> When calling a custom Processor that is only sync we could enhance
> logic at places to call the sync processor without using the
> ProcessorToAsyncProcessorBridge. This is also what gnodet have
> suggested.
>
>
>
>
>
>
>
>
>
>
>
>
>
> --
> Claus Ibsen
> -----------------
> www.camelone.org: The open source integration conference.
>
> Red Hat, Inc.
> FuseSource is now part of Red Hat
> Email: [hidden email]
> Web: http://fusesource.com
> Twitter: davsclaus
> Blog: http://davsclaus.com
> Author of Camel in Action: http://www.manning.com/ibsen



--
Claus Ibsen
-----------------
www.camelone.org: The open source integration conference.

Red Hat, Inc.
FuseSource is now part of Red Hat
Email: [hidden email]
Web: http://fusesource.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
Reply | Threaded
Open this post in threaded view
|

Re: [HEADS UP] - Optimzing routing engine to reduce stack frames in use during routing

Claus Ibsen-2
Hi

Have been working with the API while implementing all these
optimizations, I went for renaming Task -> Advice.
As its really an before + after advice we perform in the CamelInternalProcessor.

At this time I think the optimization is done. There is a potential
optimization in the SendProcessor because its used a lot, eg its the
(to) in the routes. Its using the ProducerCache to send the exchange
with the producer.

You only see longer stacktraces if the producer fails, then you have
2-3 strack-frames from the ProducerCache.

We have the potential to optimize this in SendProcessor if the
producer is singleton, and not pooled (which 95%+ are).

Though if the SendProcessor completes with no problem, then the
stack-frames is popped, and you wont see them in any subsequent
exception occurring later.


eg for example forcing an exception in a mock producer, yields

java.lang.IllegalArgumentException: Forced
at org.apache.camel.component.mock.MockEndpoint$1.process(MockEndpoint.java:267)
at org.apache.camel.processor.SendProcessor$2.doInAsyncProducer(SendProcessor.java:123)
at org.apache.camel.impl.ProducerCache.doInAsyncProducer(ProducerCache.java:298)
at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:118)
at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:72)
at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:388)

The potential optimization would be

at org.apache.camel.component.mock.MockEndpoint$1.process(MockEndpoint.java:267)
at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:118)
at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:72)
at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:388)



--
Claus Ibsen
-----------------
www.camelone.org: The open source integration conference.

Red Hat, Inc.
FuseSource is now part of Red Hat
Email: [hidden email]
Web: http://fusesource.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
Reply | Threaded
Open this post in threaded view
|

Re: [HEADS UP] - Optimzing routing engine to reduce stack frames in use during routing

Raul Kripalani
In reply to this post by Claus Ibsen-2
Hey Claus,

It's on my plate for this weekend.

Regards,
Raúl.

On Tue, May 28, 2013 at 12:18 PM, Claus Ibsen <[hidden email]> wrote:

> Possible 2 + 3 implemented.
>
>
> Raul did you by any chance had a chance to take a look also?
>
> What remains is gnodets suggest which has been listed as task 18.
> And then the naming of the new API and possible adding a bit more javadoc
> etc.
>
>
>
>
> On Tue, May 28, 2013 at 10:26 AM, Claus Ibsen <[hidden email]>
> wrote:
> > Hi
> >
> > So after a week we got good progress on this one.
> > We have been able to reduce the stack-frames with a factor of 2 - 3
> times.
> >
> > The sample we have been using for benchmark is down from 40 to 16.
> > And there is room for 1, 2 or 3 more to be shaved off.
> >
> >
> > Possible #1)
> > The JMX InstrumentationProcessor is harder to "reduce" as it wraps the
> > processor in the route to be executed, eg
> > - an EIP
> > - a custom bean
> > - a custom processor
> > - etc.
> >
> > So it sits there and track utilization, how many calls, how long time,
> > how many success / failures / redeliveries etc.
> > And its this fine grained "redelivery" that is a challenge.
> >
> > If we want to keep having fine grained redelivery tracking and
> > whatnot, then it has to sit just at the edge of the actual processor
> > being invoked, eg in between the error handler and the processor. So
> > when the error handler "kick in" and do a redelivery, we can keep
> > track of that.
> >
> >
> > Possible #2)
> > Just noticed these two which can be optimized. This ought to be easy and
> doable.
> >
> > at
> org.apache.camel.processor.RedeliveryErrorHandler.processErrorHandler(RedeliveryErrorHandler.java:391)
> > at
> org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:273)
> >
> >
> > Possible #3)
> > When calling a custom Processor that is only sync we could enhance
> > logic at places to call the sync processor without using the
> > ProcessorToAsyncProcessorBridge. This is also what gnodet have
> > suggested.
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > --
> > Claus Ibsen
> > -----------------
> > www.camelone.org: The open source integration conference.
> >
> > Red Hat, Inc.
> > FuseSource is now part of Red Hat
> > Email: [hidden email]
> > Web: http://fusesource.com
> > Twitter: davsclaus
> > Blog: http://davsclaus.com
> > Author of Camel in Action: http://www.manning.com/ibsen
>
>
>
> --
> Claus Ibsen
> -----------------
> www.camelone.org: The open source integration conference.
>
> Red Hat, Inc.
> FuseSource is now part of Red Hat
> Email: [hidden email]
> Web: http://fusesource.com
> Twitter: davsclaus
> Blog: http://davsclaus.com
> Author of Camel in Action: http://www.manning.com/ibsen
>