Subscription left once a route is removed programmatically

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Subscription left once a route is removed programmatically

Modanese, Riccardo
Hi all,

    I'm experimenting with programmatically routes creation and removal (Camel version 2.17).
The routes are created and removed at runtime through my custom RouteBuilder instance and are consuming from ActiveMQ 5 (with virtual topic enabled).
Once a route is removed (*) all the subscriptions (for this specific route) to the broker are removed except for one.
So one subscription to the broker still remains active.

Unfortunately I have no simple test code to reproduce the behavior I'm seeing.

Has someone any idea if I'm missing something or can point me to the right direction?

Thank you.

Riccardo


(*)  getContext().removeRouteDefinition(getContext().getRouteDefinition(routeId));
Reply | Threaded
Open this post in threaded view
|

Re: Subscription left once a route is removed programmatically

Claussnitzer, Ralf
Hi Riccardo,

what is happening in DEBUG log when you call the remove?

Maybe you are dealing with inflight exchanges here?

Can you try to instead just stop the routes?

-Ralf

On 3/27/20 2:20 PM, Modanese, Riccardo wrote:

> Hi all,
>
>      I'm experimenting with programmatically routes creation and removal (Camel version 2.17).
> The routes are created and removed at runtime through my custom RouteBuilder instance and are consuming from ActiveMQ 5 (with virtual topic enabled).
> Once a route is removed (*) all the subscriptions (for this specific route) to the broker are removed except for one.
> So one subscription to the broker still remains active.
>
> Unfortunately I have no simple test code to reproduce the behavior I'm seeing.
>
> Has someone any idea if I'm missing something or can point me to the right direction?
>
> Thank you.
>
> Riccardo
>
>
> (*)  getContext().removeRouteDefinition(getContext().getRouteDefinition(routeId));
Reply | Threaded
Open this post in threaded view
|

Re: Subscription left once a route is removed programmatically

Modanese, Riccardo
Hi
        I switched the log to DEBUG level and I simplified the test leaving only one route (and reducing the concurrent consumers from 2 to 4).

The issue still happens.

ActiveMQ has a subscription:

     queue://Consumer.my-route-subs-id:EXACTLY_ONCE.VirtualTopic.my-subs.>

with one consumer and no pending messages after the route is removed.
 

You can see the extracted log in attachment.






> Il giorno 30 mar 2020, alle ore 07:27, Ralf Claussnitzer <[hidden email]> ha scritto:
>
> Hi Riccardo,
>
> what is happening in DEBUG log when you call the remove?
>
> Maybe you are dealing with inflight exchanges here?
>
> Can you try to instead just stop the routes?
>
> -Ralf
>
> On 3/27/20 2:20 PM, Modanese, Riccardo wrote:
>> Hi all,
>>
>>     I'm experimenting with programmatically routes creation and removal (Camel version 2.17).
>> The routes are created and removed at runtime through my custom RouteBuilder instance and are consuming from ActiveMQ 5 (with virtual topic enabled).
>> Once a route is removed (*) all the subscriptions (for this specific route) to the broker are removed except for one.
>> So one subscription to the broker still remains active.
>>
>> Unfortunately I have no simple test code to reproduce the behavior I'm seeing.
>>
>> Has someone any idea if I'm missing something or can point me to the right direction?
>>
>> Thank you.
>>
>> Riccardo
>>
>>
>> (*)  getContext().removeRouteDefinition(getContext().getRouteDefinition(routeId));


camel_route_removal.txt (26K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Subscription left once a route is removed programmatically

Claussnitzer, Ralf
The route shutdown and removal appears to work.

Where do you see this active subscription?

Does it disappear when you GC the ActiveMQ component alltogether?

Can you show the result of `netstat -atp` on the Broker machine?

On 3/30/20 11:54 AM, Modanese, Riccardo wrote:

> Hi
>         I switched the log to DEBUG level and I simplified the test
> leaving only one route (and reducing the concurrent consumers from 2
> to 4).
>
> The issue still happens.
>
> ActiveMQ has a subscription:
>
> queue://Consumer.my-route-subs-id:EXACTLY_ONCE.VirtualTopic.my-subs.>
>
> with one consumer and no pending messages after the route is removed.
>
>
> You can see the extracted log in attachment.
>
>
>
>
>
>
> > Il giorno 30 mar 2020, alle ore 07:27, Ralf Claussnitzer
> <[hidden email]> ha scritto:
> >
> > Hi Riccardo,
> >
> > what is happening in DEBUG log when you call the remove?
> >
> > Maybe you are dealing with inflight exchanges here?
> >
> > Can you try to instead just stop the routes?
> >
> > -Ralf
> >
> > On 3/27/20 2:20 PM, Modanese, Riccardo wrote:
> >> Hi all,
> >>
> >>     I'm experimenting with programmatically routes creation and
> removal (Camel version 2.17).
> >> The routes are created and removed at runtime through my custom
> RouteBuilder instance and are consuming from ActiveMQ 5 (with virtual
> topic enabled).
> >> Once a route is removed (*) all the subscriptions (for this
> specific route) to the broker are removed except for one.
> >> So one subscription to the broker still remains active.
> >>
> >> Unfortunately I have no simple test code to reproduce the behavior
> I'm seeing.
> >>
> >> Has someone any idea if I'm missing something or can point me to
> the right direction?
> >>
> >> Thank you.
> >>
> >> Riccardo
> >>
> >>
> >> (*)
> getContext().removeRouteDefinition(getContext().getRouteDefinition(routeId));
>
Reply | Threaded
Open this post in threaded view
|

Re: Subscription left once a route is removed programmatically

Modanese, Riccardo
I have an ActiveMQ plugin to track subscriptions and other stuff.

Moreover, if I re-create the route after a while, the route processes the messages received once it is supposed to be removed.

The GC is done by a custom plugin since my use case doesn’t fit in any “regular” ActiveMQ GC.
Anyway, if I force the destination deletion, my plugin is able to delete the queue.
And when I re-create the route, the route doesn’t process any previous message.

To give you some more detail, I’m experimenting with the REST routes using the http4 endpoint but, the endpoint itself, is plugged to the context with a different schema like:

<bean id="my-route-id-one" class="org.apache.camel.component.http4.HttpComponent">
    ...
</bean>

you can see the netstat output in attachment.





> Il giorno 30 mar 2020, alle ore 16:19, Ralf Claussnitzer <[hidden email]> ha scritto:
>
> The route shutdown and removal appears to work.
>
> Where do you see this active subscription?
>
> Does it disappear when you GC the ActiveMQ component alltogether?
>
> Can you show the result of `netstat -atp` on the Broker machine?
>
> On 3/30/20 11:54 AM, Modanese, Riccardo wrote:
>> Hi
>>         I switched the log to DEBUG level and I simplified the test leaving only one route (and reducing the concurrent consumers from 2 to 4).
>>
>> The issue still happens.
>>
>> ActiveMQ has a subscription:
>>
>> queue://Consumer.my-route-subs-id:EXACTLY_ONCE.VirtualTopic.my-subs.>
>>
>> with one consumer and no pending messages after the route is removed.
>>
>>
>> You can see the extracted log in attachment.
>>
>>
>>
>>
>>
>>
>> > Il giorno 30 mar 2020, alle ore 07:27, Ralf Claussnitzer <[hidden email]> ha scritto:
>> >
>> > Hi Riccardo,
>> >
>> > what is happening in DEBUG log when you call the remove?
>> >
>> > Maybe you are dealing with inflight exchanges here?
>> >
>> > Can you try to instead just stop the routes?
>> >
>> > -Ralf
>> >
>> > On 3/27/20 2:20 PM, Modanese, Riccardo wrote:
>> >> Hi all,
>> >>
>> >>     I'm experimenting with programmatically routes creation and removal (Camel version 2.17).
>> >> The routes are created and removed at runtime through my custom RouteBuilder instance and are consuming from ActiveMQ 5 (with virtual topic enabled).
>> >> Once a route is removed (*) all the subscriptions (for this specific route) to the broker are removed except for one.
>> >> So one subscription to the broker still remains active.
>> >>
>> >> Unfortunately I have no simple test code to reproduce the behavior I'm seeing.
>> >>
>> >> Has someone any idea if I'm missing something or can point me to the right direction?
>> >>
>> >> Thank you.
>> >>
>> >> Riccardo
>> >>
>> >>
>> >> (*) getContext().removeRouteDefinition(getContext().getRouteDefinition(routeId));
>>


camel_route_removal_netstat.txt (3K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Subscription left once a route is removed programmatically

Modanese, Riccardo
I did several tests to try to understand the root cause of this weird behavior but I didn’t find any reason.
Anyway it happens only for the first route I remove. And it happens again if I create and remove again the same route.

In other word
create route_1
create route_2
create route_3
remove route_1 leave only one subscription active (regardless of the concurrent consumers count set)
remove route_2 completed fine (all consumers are removed and also the virtual queue created by the subscription)
remove route_3 completed fine (all consumers are removed and also the virtual queue created by the subscription)


Second test
In other word
create route_1
create route_2
create route_3
remove route_2 leave only one subscription active (regardless of the concurrent consumers count set)
remove route_1 completed fine (all consumers are removed and also the virtual queue created by the subscription)
remove route_3 completed fine (all consumers are removed and also the virtual queue created by the subscription)

I tried to remove the route in different way.
- remove the route definition as first step, and then the endpoints (checking if the route is correctly removed by the route definition remove)
- remove the route and then the route definition and the endpoints
- repeat the same steps invoking the route stop before
but the result is the same

Does someone have any idea about why it happens?

Il giorno 31 mar 2020, alle ore 10:27, Modanese, Riccardo <[hidden email]<mailto:[hidden email]>> ha scritto:

I have an ActiveMQ plugin to track subscriptions and other stuff.

Moreover, if I re-create the route after a while, the route processes the messages received once it is supposed to be removed.

The GC is done by a custom plugin since my use case doesn’t fit in any “regular” ActiveMQ GC.
Anyway, if I force the destination deletion, my plugin is able to delete the queue.
And when I re-create the route, the route doesn’t process any previous message.

To give you some more detail, I’m experimenting with the REST routes using the http4 endpoint but, the endpoint itself, is plugged to the context with a different schema like:

<bean id="my-route-id-one" class="org.apache.camel.component.http4.HttpComponent">
    ...
</bean>

you can see the netstat output in attachment.





> Il giorno 30 mar 2020, alle ore 16:19, Ralf Claussnitzer <[hidden email]<mailto:[hidden email]>> ha scritto:
>
> The route shutdown and removal appears to work.
>
> Where do you see this active subscription?
>
> Does it disappear when you GC the ActiveMQ component alltogether?
>
> Can you show the result of `netstat -atp` on the Broker machine?
>
> On 3/30/20 11:54 AM, Modanese, Riccardo wrote:
>> Hi
>>         I switched the log to DEBUG level and I simplified the test leaving only one route (and reducing the concurrent consumers from 2 to 4).
>>
>> The issue still happens.
>>
>> ActiveMQ has a subscription:
>>
>> queue://Consumer.my-route-subs-id:EXACTLY_ONCE.VirtualTopic.my-subs.>
>>
>> with one consumer and no pending messages after the route is removed.
>>
>>
>> You can see the extracted log in attachment.
>>
>>
>>
>>
>>
>>
>> > Il giorno 30 mar 2020, alle ore 07:27, Ralf Claussnitzer <[hidden email]<mailto:[hidden email]>> ha scritto:
>> >
>> > Hi Riccardo,
>> >
>> > what is happening in DEBUG log when you call the remove?
>> >
>> > Maybe you are dealing with inflight exchanges here?
>> >
>> > Can you try to instead just stop the routes?
>> >
>> > -Ralf
>> >
>> > On 3/27/20 2:20 PM, Modanese, Riccardo wrote:
>> >> Hi all,
>> >>
>> >>     I'm experimenting with programmatically routes creation and removal (Camel version 2.17).
>> >> The routes are created and removed at runtime through my custom RouteBuilder instance and are consuming from ActiveMQ 5 (with virtual topic enabled).
>> >> Once a route is removed (*) all the subscriptions (for this specific route) to the broker are removed except for one.
>> >> So one subscription to the broker still remains active.
>> >>
>> >> Unfortunately I have no simple test code to reproduce the behavior I'm seeing.
>> >>
>> >> Has someone any idea if I'm missing something or can point me to the right direction?
>> >>
>> >> Thank you.
>> >>
>> >> Riccardo
>> >>
>> >>
>> >> (*) getContext().removeRouteDefinition(getContext().getRouteDefinition(routeId));
>>

<camel_route_removal_netstat.txt>