Persistent MQTT Client Message Lost

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Persistent MQTT Client Message Lost

noone100
We're having difficulties with persistent clients (using
cleanSession=false). We would like to get messages that were sent while
our client was disconnected.

Steps to reproduce:
1. start route to create the initial subscription for the testclient
from("mqtt:bar?subscribeTopicName=testtopic&cleanSession=false&clientId=testclient&host=tcp://localhost:1883&qualityOfService=ExactlyOnce").transform(body().convertToString()).to("mock:test");

2. stop route (disconnect from broker)

3. send message to testtopic with qos 2 using another client id (e.g.
from cli)

4. repeat step 1. to connect and re-subscribe to testtopic

expected behaviour:
message delivered to mock endpoint (mock:test)

actual behaviour:
publish (delivery) to testclient is shown in broker log (mosquitto) and
in trace of
fusesource MQTTEndpoint but message is not delivered to mock endpoint.
The subscription and hence the callback is not setup at the connect time.

More or less same behaviour when paho client is used instead.
Reply | Threaded
Open this post in threaded view
|

Re: Persistent MQTT Client Message Lost

Willem.Jiang
Administrator
Hi

Can I know which version of Camel are you using?
I just checked the code of  camel-mqtt, there are some changes to fix
the connection related issues.
I doubt that the connection is not full released in your case.
Can you try to stop the camel application instead of stop the camel route?


Willem Jiang

Twitter: willemjiang
Weibo: 姜宁willem

On Wed, Aug 7, 2019 at 10:56 PM Michael Zaugg <[hidden email]> wrote:

>
> We're having difficulties with persistent clients (using
> cleanSession=false). We would like to get messages that were sent while
> our client was disconnected.
>
> Steps to reproduce:
> 1. start route to create the initial subscription for the testclient
> from("mqtt:bar?subscribeTopicName=testtopic&cleanSession=false&clientId=testclient&host=tcp://localhost:1883&qualityOfService=ExactlyOnce").transform(body().convertToString()).to("mock:test");
>
> 2. stop route (disconnect from broker)
>
> 3. send message to testtopic with qos 2 using another client id (e.g.
> from cli)
>
> 4. repeat step 1. to connect and re-subscribe to testtopic
>
> expected behaviour:
> message delivered to mock endpoint (mock:test)
>
> actual behaviour:
> publish (delivery) to testclient is shown in broker log (mosquitto) and
> in trace of
> fusesource MQTTEndpoint but message is not delivered to mock endpoint.
> The subscription and hence the callback is not setup at the connect time.
>
> More or less same behaviour when paho client is used instead.
Reply | Threaded
Open this post in threaded view
|

Re: Persistent MQTT Client Message Lost

noone100
Hi Willem

It's camel version 2.24.0

We gracefully stopped the whole java process with the same result.

"Sending PUBLISH to testclient" occurs before "Received SUBSCRIBE"
according to Mosquitto's log:

1565245897: New client connected from 172.17.0.1 as testclient (p1, c0,
k30).
1565245897: No will message specified.
1565245897: Sending CONNACK to testclient (0, 0)
1565245897: Sending PUBLISH to testclient (d0, q2, r0, m2, 'testtopic',
... (4 bytes))
1565245907: Received SUBSCRIBE from testclient
1565245907: testtopic (QoS 2)
1565245907: testclient 2 testtopic
1565245907: Sending SUBACK to testclient
1565245907: Received PUBREC from testclient (Mid: 2)
1565245907: Sending PUBREL to testclient (m2)
1565245907: Received PUBCOMP from testclient (Mid: 2, RC:0)

KR
Michael

On 08.08.19 03:13, Willem Jiang wrote:

> Hi
>
> Can I know which version of Camel are you using?
> I just checked the code of  camel-mqtt, there are some changes to fix
> the connection related issues.
> I doubt that the connection is not full released in your case.
> Can you try to stop the camel application instead of stop the camel route?
>
>
> Willem Jiang
>
> Twitter: willemjiang
> Weibo: 姜宁willem
>
> On Wed, Aug 7, 2019 at 10:56 PM Michael Zaugg <[hidden email]> wrote:
>>
>> We're having difficulties with persistent clients (using
>> cleanSession=false). We would like to get messages that were sent while
>> our client was disconnected.
>>
>> Steps to reproduce:
>> 1. start route to create the initial subscription for the testclient
>> from("mqtt:bar?subscribeTopicName=testtopic&cleanSession=false&clientId=testclient&host=tcp://localhost:1883&qualityOfService=ExactlyOnce").transform(body().convertToString()).to("mock:test");
>>
>> 2. stop route (disconnect from broker)
>>
>> 3. send message to testtopic with qos 2 using another client id (e.g.
>> from cli)
>>
>> 4. repeat step 1. to connect and re-subscribe to testtopic
>>
>> expected behaviour:
>> message delivered to mock endpoint (mock:test)
>>
>> actual behaviour:
>> publish (delivery) to testclient is shown in broker log (mosquitto) and
>> in trace of
>> fusesource MQTTEndpoint but message is not delivered to mock endpoint.
>> The subscription and hence the callback is not setup at the connect time.
>>
>> More or less same behaviour when paho client is used instead.

--
Michael Zaugg
Softwareingenieur

Puzzle ITC GmbH
www.puzzle.ch

Telefon +41 31 370 22 00
Direkt  +41 31 370 22 15
Mobile  +41 79 289 65 88
Fax     +41 31 370 22 01

Werfen Sie einen Blick in unseren Blog:
<http://www.puzzle.ch/blog>
Reply | Threaded
Open this post in threaded view
|

Re: Persistent MQTT Client Message Lost

Willem.Jiang
Administrator
I just checked the code of camel-mqtt code, and find a line[1] which
could let to the issue that you face.
If the endpoint is start and consumer is not start yet, the consumers
could be empty, camel cannot send the exchange to right consumer to
use. Maybe you can set a break or add some log to verify it.

[1]https://github.com/apache/camel/blob/24521870b81576b5caf9ff3951cff8a0c2c77ab2/components/camel-mqtt/src/main/java/org/apache/camel/component/mqtt/MQTTEndpoint.java#L256-L263

Willem Jiang

Twitter: willemjiang
Weibo: 姜宁willem

On Thu, Aug 8, 2019 at 2:45 PM Michael Zaugg <[hidden email]> wrote:

>
> Hi Willem
>
> It's camel version 2.24.0
>
> We gracefully stopped the whole java process with the same result.
>
> "Sending PUBLISH to testclient" occurs before "Received SUBSCRIBE"
> according to Mosquitto's log:
>
> 1565245897: New client connected from 172.17.0.1 as testclient (p1, c0,
> k30).
> 1565245897: No will message specified.
> 1565245897: Sending CONNACK to testclient (0, 0)
> 1565245897: Sending PUBLISH to testclient (d0, q2, r0, m2, 'testtopic',
> ... (4 bytes))
> 1565245907: Received SUBSCRIBE from testclient
> 1565245907:     testtopic (QoS 2)
> 1565245907: testclient 2 testtopic
> 1565245907: Sending SUBACK to testclient
> 1565245907: Received PUBREC from testclient (Mid: 2)
> 1565245907: Sending PUBREL to testclient (m2)
> 1565245907: Received PUBCOMP from testclient (Mid: 2, RC:0)
>
> KR
> Michael
>
> On 08.08.19 03:13, Willem Jiang wrote:
> > Hi
> >
> > Can I know which version of Camel are you using?
> > I just checked the code of  camel-mqtt, there are some changes to fix
> > the connection related issues.
> > I doubt that the connection is not full released in your case.
> > Can you try to stop the camel application instead of stop the camel route?
> >
> >
> > Willem Jiang
> >
> > Twitter: willemjiang
> > Weibo: 姜宁willem
> >
> > On Wed, Aug 7, 2019 at 10:56 PM Michael Zaugg <[hidden email]> wrote:
> >>
> >> We're having difficulties with persistent clients (using
> >> cleanSession=false). We would like to get messages that were sent while
> >> our client was disconnected.
> >>
> >> Steps to reproduce:
> >> 1. start route to create the initial subscription for the testclient
> >> from("mqtt:bar?subscribeTopicName=testtopic&cleanSession=false&clientId=testclient&host=tcp://localhost:1883&qualityOfService=ExactlyOnce").transform(body().convertToString()).to("mock:test");
> >>
> >> 2. stop route (disconnect from broker)
> >>
> >> 3. send message to testtopic with qos 2 using another client id (e.g.
> >> from cli)
> >>
> >> 4. repeat step 1. to connect and re-subscribe to testtopic
> >>
> >> expected behaviour:
> >> message delivered to mock endpoint (mock:test)
> >>
> >> actual behaviour:
> >> publish (delivery) to testclient is shown in broker log (mosquitto) and
> >> in trace of
> >> fusesource MQTTEndpoint but message is not delivered to mock endpoint.
> >> The subscription and hence the callback is not setup at the connect time.
> >>
> >> More or less same behaviour when paho client is used instead.
>
> --
> Michael Zaugg
> Softwareingenieur
>
> Puzzle ITC GmbH
> www.puzzle.ch
>
> Telefon +41 31 370 22 00
> Direkt  +41 31 370 22 15
> Mobile  +41 79 289 65 88
> Fax     +41 31 370 22 01
>
> Werfen Sie einen Blick in unseren Blog:
> <http://www.puzzle.ch/blog>
Reply | Threaded
Open this post in threaded view
|

Re: Persistent MQTT Client Message Lost

Claus Ibsen-2
Hi

Ah yeah well spotted Willem.

The endpoint should only call connection() when the 1st consumer is
being added, and then disconnect when the active consumer count hits
zero. So the endpoint needs to keep track on number of consumers
added/removed and react accordingly.

On Fri, Aug 9, 2019 at 8:00 AM Willem Jiang <[hidden email]> wrote:

>
> I just checked the code of camel-mqtt code, and find a line[1] which
> could let to the issue that you face.
> If the endpoint is start and consumer is not start yet, the consumers
> could be empty, camel cannot send the exchange to right consumer to
> use. Maybe you can set a break or add some log to verify it.
>
> [1]https://github.com/apache/camel/blob/24521870b81576b5caf9ff3951cff8a0c2c77ab2/components/camel-mqtt/src/main/java/org/apache/camel/component/mqtt/MQTTEndpoint.java#L256-L263
>
> Willem Jiang
>
> Twitter: willemjiang
> Weibo: 姜宁willem
>
> On Thu, Aug 8, 2019 at 2:45 PM Michael Zaugg <[hidden email]> wrote:
> >
> > Hi Willem
> >
> > It's camel version 2.24.0
> >
> > We gracefully stopped the whole java process with the same result.
> >
> > "Sending PUBLISH to testclient" occurs before "Received SUBSCRIBE"
> > according to Mosquitto's log:
> >
> > 1565245897: New client connected from 172.17.0.1 as testclient (p1, c0,
> > k30).
> > 1565245897: No will message specified.
> > 1565245897: Sending CONNACK to testclient (0, 0)
> > 1565245897: Sending PUBLISH to testclient (d0, q2, r0, m2, 'testtopic',
> > ... (4 bytes))
> > 1565245907: Received SUBSCRIBE from testclient
> > 1565245907:     testtopic (QoS 2)
> > 1565245907: testclient 2 testtopic
> > 1565245907: Sending SUBACK to testclient
> > 1565245907: Received PUBREC from testclient (Mid: 2)
> > 1565245907: Sending PUBREL to testclient (m2)
> > 1565245907: Received PUBCOMP from testclient (Mid: 2, RC:0)
> >
> > KR
> > Michael
> >
> > On 08.08.19 03:13, Willem Jiang wrote:
> > > Hi
> > >
> > > Can I know which version of Camel are you using?
> > > I just checked the code of  camel-mqtt, there are some changes to fix
> > > the connection related issues.
> > > I doubt that the connection is not full released in your case.
> > > Can you try to stop the camel application instead of stop the camel route?
> > >
> > >
> > > Willem Jiang
> > >
> > > Twitter: willemjiang
> > > Weibo: 姜宁willem
> > >
> > > On Wed, Aug 7, 2019 at 10:56 PM Michael Zaugg <[hidden email]> wrote:
> > >>
> > >> We're having difficulties with persistent clients (using
> > >> cleanSession=false). We would like to get messages that were sent while
> > >> our client was disconnected.
> > >>
> > >> Steps to reproduce:
> > >> 1. start route to create the initial subscription for the testclient
> > >> from("mqtt:bar?subscribeTopicName=testtopic&cleanSession=false&clientId=testclient&host=tcp://localhost:1883&qualityOfService=ExactlyOnce").transform(body().convertToString()).to("mock:test");
> > >>
> > >> 2. stop route (disconnect from broker)
> > >>
> > >> 3. send message to testtopic with qos 2 using another client id (e.g.
> > >> from cli)
> > >>
> > >> 4. repeat step 1. to connect and re-subscribe to testtopic
> > >>
> > >> expected behaviour:
> > >> message delivered to mock endpoint (mock:test)
> > >>
> > >> actual behaviour:
> > >> publish (delivery) to testclient is shown in broker log (mosquitto) and
> > >> in trace of
> > >> fusesource MQTTEndpoint but message is not delivered to mock endpoint.
> > >> The subscription and hence the callback is not setup at the connect time.
> > >>
> > >> More or less same behaviour when paho client is used instead.
> >
> > --
> > Michael Zaugg
> > Softwareingenieur
> >
> > Puzzle ITC GmbH
> > www.puzzle.ch
> >
> > Telefon +41 31 370 22 00
> > Direkt  +41 31 370 22 15
> > Mobile  +41 79 289 65 88
> > Fax     +41 31 370 22 01
> >
> > Werfen Sie einen Blick in unseren Blog:
> > <http://www.puzzle.ch/blog>



--
Claus Ibsen
-----------------
http://davsclaus.com @davsclaus
Camel in Action 2: https://www.manning.com/ibsen2
Reply | Threaded
Open this post in threaded view
|

Re: Persistent MQTT Client Message Lost

noone100
Hi

I would suggest to connect() after ALL Consumers are started in
doStart(). Is this approach feasible?

On 09.08.19 09:23, Claus Ibsen wrote:

> Hi
>
> Ah yeah well spotted Willem.
>
> The endpoint should only call connection() when the 1st consumer is
> being added, and then disconnect when the active consumer count hits
> zero. So the endpoint needs to keep track on number of consumers
> added/removed and react accordingly.
>
> On Fri, Aug 9, 2019 at 8:00 AM Willem Jiang <[hidden email]> wrote:
>>
>> I just checked the code of camel-mqtt code, and find a line[1] which
>> could let to the issue that you face.
>> If the endpoint is start and consumer is not start yet, the consumers
>> could be empty, camel cannot send the exchange to right consumer to
>> use. Maybe you can set a break or add some log to verify it.
>>
>> [1]https://github.com/apache/camel/blob/24521870b81576b5caf9ff3951cff8a0c2c77ab2/components/camel-mqtt/src/main/java/org/apache/camel/component/mqtt/MQTTEndpoint.java#L256-L263
>>
>> Willem Jiang
>>
>> Twitter: willemjiang
>> Weibo: 姜宁willem
>>
>> On Thu, Aug 8, 2019 at 2:45 PM Michael Zaugg <[hidden email]> wrote:
>>>
>>> Hi Willem
>>>
>>> It's camel version 2.24.0
>>>
>>> We gracefully stopped the whole java process with the same result.
>>>
>>> "Sending PUBLISH to testclient" occurs before "Received SUBSCRIBE"
>>> according to Mosquitto's log:
>>>
>>> 1565245897: New client connected from 172.17.0.1 as testclient (p1, c0,
>>> k30).
>>> 1565245897: No will message specified.
>>> 1565245897: Sending CONNACK to testclient (0, 0)
>>> 1565245897: Sending PUBLISH to testclient (d0, q2, r0, m2, 'testtopic',
>>> ... (4 bytes))
>>> 1565245907: Received SUBSCRIBE from testclient
>>> 1565245907:     testtopic (QoS 2)
>>> 1565245907: testclient 2 testtopic
>>> 1565245907: Sending SUBACK to testclient
>>> 1565245907: Received PUBREC from testclient (Mid: 2)
>>> 1565245907: Sending PUBREL to testclient (m2)
>>> 1565245907: Received PUBCOMP from testclient (Mid: 2, RC:0)
>>>
>>> KR
>>> Michael
>>>
>>> On 08.08.19 03:13, Willem Jiang wrote:
>>>> Hi
>>>>
>>>> Can I know which version of Camel are you using?
>>>> I just checked the code of  camel-mqtt, there are some changes to fix
>>>> the connection related issues.
>>>> I doubt that the connection is not full released in your case.
>>>> Can you try to stop the camel application instead of stop the camel route?
>>>>
>>>>
>>>> Willem Jiang
>>>>
>>>> Twitter: willemjiang
>>>> Weibo: 姜宁willem
>>>>
>>>> On Wed, Aug 7, 2019 at 10:56 PM Michael Zaugg <[hidden email]> wrote:
>>>>>
>>>>> We're having difficulties with persistent clients (using
>>>>> cleanSession=false). We would like to get messages that were sent while
>>>>> our client was disconnected.
>>>>>
>>>>> Steps to reproduce:
>>>>> 1. start route to create the initial subscription for the testclient
>>>>> from("mqtt:bar?subscribeTopicName=testtopic&cleanSession=false&clientId=testclient&host=tcp://localhost:1883&qualityOfService=ExactlyOnce").transform(body().convertToString()).to("mock:test");
>>>>>
>>>>> 2. stop route (disconnect from broker)
>>>>>
>>>>> 3. send message to testtopic with qos 2 using another client id (e.g.
>>>>> from cli)
>>>>>
>>>>> 4. repeat step 1. to connect and re-subscribe to testtopic
>>>>>
>>>>> expected behaviour:
>>>>> message delivered to mock endpoint (mock:test)
>>>>>
>>>>> actual behaviour:
>>>>> publish (delivery) to testclient is shown in broker log (mosquitto) and
>>>>> in trace of
>>>>> fusesource MQTTEndpoint but message is not delivered to mock endpoint.
>>>>> The subscription and hence the callback is not setup at the connect time.
>>>>>
>>>>> More or less same behaviour when paho client is used instead.
>>>
>>> --
>>> Michael Zaugg
>>> Softwareingenieur
>>>
>>> Puzzle ITC GmbH
>>> www.puzzle.ch
>>>
>>> Telefon +41 31 370 22 00
>>> Direkt  +41 31 370 22 15
>>> Mobile  +41 79 289 65 88
>>> Fax     +41 31 370 22 01
>>>
>>> Werfen Sie einen Blick in unseren Blog:
>>> <http://www.puzzle.ch/blog>
>
>
>

--
Michael Zaugg
Softwareingenieur

Puzzle ITC GmbH
www.puzzle.ch

Telefon +41 31 370 22 00
Direkt  +41 31 370 22 15
Mobile  +41 79 289 65 88
Fax     +41 31 370 22 01

Werfen Sie einen Blick in unseren Blog:
<http://www.puzzle.ch/blog>
Reply | Threaded
Open this post in threaded view
|

Re: Persistent MQTT Client Message Lost

Claus Ibsen-2
Hi

Yeah there is a listerne on the component level that triggers when all
routes are started, see how camel-quarz uses that etc.

On Mon, Aug 12, 2019 at 8:50 AM Michael Zaugg <[hidden email]> wrote:

>
> Hi
>
> I would suggest to connect() after ALL Consumers are started in
> doStart(). Is this approach feasible?
>
> On 09.08.19 09:23, Claus Ibsen wrote:
> > Hi
> >
> > Ah yeah well spotted Willem.
> >
> > The endpoint should only call connection() when the 1st consumer is
> > being added, and then disconnect when the active consumer count hits
> > zero. So the endpoint needs to keep track on number of consumers
> > added/removed and react accordingly.
> >
> > On Fri, Aug 9, 2019 at 8:00 AM Willem Jiang <[hidden email]> wrote:
> >>
> >> I just checked the code of camel-mqtt code, and find a line[1] which
> >> could let to the issue that you face.
> >> If the endpoint is start and consumer is not start yet, the consumers
> >> could be empty, camel cannot send the exchange to right consumer to
> >> use. Maybe you can set a break or add some log to verify it.
> >>
> >> [1]https://github.com/apache/camel/blob/24521870b81576b5caf9ff3951cff8a0c2c77ab2/components/camel-mqtt/src/main/java/org/apache/camel/component/mqtt/MQTTEndpoint.java#L256-L263
> >>
> >> Willem Jiang
> >>
> >> Twitter: willemjiang
> >> Weibo: 姜宁willem
> >>
> >> On Thu, Aug 8, 2019 at 2:45 PM Michael Zaugg <[hidden email]> wrote:
> >>>
> >>> Hi Willem
> >>>
> >>> It's camel version 2.24.0
> >>>
> >>> We gracefully stopped the whole java process with the same result.
> >>>
> >>> "Sending PUBLISH to testclient" occurs before "Received SUBSCRIBE"
> >>> according to Mosquitto's log:
> >>>
> >>> 1565245897: New client connected from 172.17.0.1 as testclient (p1, c0,
> >>> k30).
> >>> 1565245897: No will message specified.
> >>> 1565245897: Sending CONNACK to testclient (0, 0)
> >>> 1565245897: Sending PUBLISH to testclient (d0, q2, r0, m2, 'testtopic',
> >>> ... (4 bytes))
> >>> 1565245907: Received SUBSCRIBE from testclient
> >>> 1565245907:     testtopic (QoS 2)
> >>> 1565245907: testclient 2 testtopic
> >>> 1565245907: Sending SUBACK to testclient
> >>> 1565245907: Received PUBREC from testclient (Mid: 2)
> >>> 1565245907: Sending PUBREL to testclient (m2)
> >>> 1565245907: Received PUBCOMP from testclient (Mid: 2, RC:0)
> >>>
> >>> KR
> >>> Michael
> >>>
> >>> On 08.08.19 03:13, Willem Jiang wrote:
> >>>> Hi
> >>>>
> >>>> Can I know which version of Camel are you using?
> >>>> I just checked the code of  camel-mqtt, there are some changes to fix
> >>>> the connection related issues.
> >>>> I doubt that the connection is not full released in your case.
> >>>> Can you try to stop the camel application instead of stop the camel route?
> >>>>
> >>>>
> >>>> Willem Jiang
> >>>>
> >>>> Twitter: willemjiang
> >>>> Weibo: 姜宁willem
> >>>>
> >>>> On Wed, Aug 7, 2019 at 10:56 PM Michael Zaugg <[hidden email]> wrote:
> >>>>>
> >>>>> We're having difficulties with persistent clients (using
> >>>>> cleanSession=false). We would like to get messages that were sent while
> >>>>> our client was disconnected.
> >>>>>
> >>>>> Steps to reproduce:
> >>>>> 1. start route to create the initial subscription for the testclient
> >>>>> from("mqtt:bar?subscribeTopicName=testtopic&cleanSession=false&clientId=testclient&host=tcp://localhost:1883&qualityOfService=ExactlyOnce").transform(body().convertToString()).to("mock:test");
> >>>>>
> >>>>> 2. stop route (disconnect from broker)
> >>>>>
> >>>>> 3. send message to testtopic with qos 2 using another client id (e.g.
> >>>>> from cli)
> >>>>>
> >>>>> 4. repeat step 1. to connect and re-subscribe to testtopic
> >>>>>
> >>>>> expected behaviour:
> >>>>> message delivered to mock endpoint (mock:test)
> >>>>>
> >>>>> actual behaviour:
> >>>>> publish (delivery) to testclient is shown in broker log (mosquitto) and
> >>>>> in trace of
> >>>>> fusesource MQTTEndpoint but message is not delivered to mock endpoint.
> >>>>> The subscription and hence the callback is not setup at the connect time.
> >>>>>
> >>>>> More or less same behaviour when paho client is used instead.
> >>>
> >>> --
> >>> Michael Zaugg
> >>> Softwareingenieur
> >>>
> >>> Puzzle ITC GmbH
> >>> www.puzzle.ch
> >>>
> >>> Telefon +41 31 370 22 00
> >>> Direkt  +41 31 370 22 15
> >>> Mobile  +41 79 289 65 88
> >>> Fax     +41 31 370 22 01
> >>>
> >>> Werfen Sie einen Blick in unseren Blog:
> >>> <http://www.puzzle.ch/blog>
> >
> >
> >
>
> --
> Michael Zaugg
> Softwareingenieur
>
> Puzzle ITC GmbH
> www.puzzle.ch
>
> Telefon +41 31 370 22 00
> Direkt  +41 31 370 22 15
> Mobile  +41 79 289 65 88
> Fax     +41 31 370 22 01
>
> Werfen Sie einen Blick in unseren Blog:
> <http://www.puzzle.ch/blog>



--
Claus Ibsen
-----------------
http://davsclaus.com @davsclaus
Camel in Action 2: https://www.manning.com/ibsen2