Problem with netty4:udp and udpByteArrayCodec=true

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Problem with netty4:udp and udpByteArrayCodec=true

Ron Cecchini

First, a couple of thank yous several weeks in the making:

Thank you to Claus (as always) for help regarding replacing camel-rabbitmq with a custom class.  Indeed, I only needed to do a write.  So using 'to("bean:myBean?method=process)' worked perfectly.

Thank you to Zoran for advice on Spring injection vs. Component Scan.  I never was able to get beans in the Spring config auto-wired, so I had to do the manual steps of getting the Spring context, getting the bean, and then setting the bean where it needed to be.  A bit ugly, but only a few lines of code.  It works, so I'm leaving it alone for now.

Now on to my most recent hurdle...

I'm trying to write a protobuf over UDP to a server (a non-Camel app written in C++) expecting it serialized into a byte array.

The problem is that the only write that makes it to the server seems to be serialized incorrectly.  Conversely, when I think I've correctly serialized it, nothing seems to get written to the wire.

The route below is the one that makes it to the server.  When I do a tcpdump on the server's side, however, I can see that the payload is about 35-40 bytes bigger than what the protobuf says its serialized size should be:

from("direct:protobuf")
     .marshal().protobuf()
     .convertBodyTo(String.class, "UTF-8")  // needed, else to(netty4:udp) hangs
     .to("netty4:udp://host:port?udpConnectionlessSending=true&sync=false");

I immediately recognized this as probably being a "default codec" problem, similar to what I recently encountered when I was writing some JSON over UDP with netty4.

So I tried the following, but nothing was written over the wire:

     .to("netty4:udp://host:port?udpConnectionlessSending=true&sync=false&allowDefaultCodec=false");

I tried about a half dozen combinations: marshalling / not marshalling, convertBodyTo() / no convertBodyTo(), allowDefaultCode = false / true, etc.

Nothing was working.  And then I somehow stumbled upon 'udpByteArrayCodec' in the docs and thought I'd found it!

However, this didn't send anything to the wire either:

     .to("netty4:udp://host:post?udpConnectionlessSending=true&sync=false&udpByteArrayCodec=true");

Neither did this:

     .to("netty4:udp://host:port?udpConnectionlessSending=true&sync=false&udpByteArrayCodec=true&allowDefaultCodec=false");

Only the original route resulted in *anything* being written across the wire for the server to see:

     .to("netty4:udp://host:port?udpConnectionlessSending=true&sync=false");

Does anyone know what else I should try?

I have never had any luck trying to install a non-default encoder/decoder, but in this case I don't think I even need it.  I just need to serialize the protobuf to a byte array, and it seems that this 'udpByteArrayCodec' is exactly what I need.  So how should I be using it?  What could be preventing anything from appearing over the wire?

Thank you so much as always!

Ron
Reply | Threaded
Open this post in threaded view
|

Re: Problem with netty4:udp and udpByteArrayCodec=true

Zoran Regvart-2
Hi Ron,
I'm far from an expert on Netty, it seems to me like that
`convertBodyTo(String.class, "UTF-8")` will lead to issues with binary
data.

What you want (I think) is
`.to("netty4:udp://host:port?allowDefaultCodec=false&useByteBuf=true")`
without `convertBodyTo`

zoran


On Fri, Nov 9, 2018 at 6:12 AM, Ron Cecchini <[hidden email]> wrote:

>
> First, a couple of thank yous several weeks in the making:
>
> Thank you to Claus (as always) for help regarding replacing camel-rabbitmq with a custom class.  Indeed, I only needed to do a write.  So using 'to("bean:myBean?method=process)' worked perfectly.
>
> Thank you to Zoran for advice on Spring injection vs. Component Scan.  I never was able to get beans in the Spring config auto-wired, so I had to do the manual steps of getting the Spring context, getting the bean, and then setting the bean where it needed to be.  A bit ugly, but only a few lines of code.  It works, so I'm leaving it alone for now.
>
> Now on to my most recent hurdle...
>
> I'm trying to write a protobuf over UDP to a server (a non-Camel app written in C++) expecting it serialized into a byte array.
>
> The problem is that the only write that makes it to the server seems to be serialized incorrectly.  Conversely, when I think I've correctly serialized it, nothing seems to get written to the wire.
>
> The route below is the one that makes it to the server.  When I do a tcpdump on the server's side, however, I can see that the payload is about 35-40 bytes bigger than what the protobuf says its serialized size should be:
>
> from("direct:protobuf")
>      .marshal().protobuf()
>      .convertBodyTo(String.class, "UTF-8")  // needed, else to(netty4:udp) hangs
>      .to("netty4:udp://host:port?udpConnectionlessSending=true&sync=false");
>
> I immediately recognized this as probably being a "default codec" problem, similar to what I recently encountered when I was writing some JSON over UDP with netty4.
>
> So I tried the following, but nothing was written over the wire:
>
>      .to("netty4:udp://host:port?udpConnectionlessSending=true&sync=false&allowDefaultCodec=false");
>
> I tried about a half dozen combinations: marshalling / not marshalling, convertBodyTo() / no convertBodyTo(), allowDefaultCode = false / true, etc.
>
> Nothing was working.  And then I somehow stumbled upon 'udpByteArrayCodec' in the docs and thought I'd found it!
>
> However, this didn't send anything to the wire either:
>
>      .to("netty4:udp://host:post?udpConnectionlessSending=true&sync=false&udpByteArrayCodec=true");
>
> Neither did this:
>
>      .to("netty4:udp://host:port?udpConnectionlessSending=true&sync=false&udpByteArrayCodec=true&allowDefaultCodec=false");
>
> Only the original route resulted in *anything* being written across the wire for the server to see:
>
>      .to("netty4:udp://host:port?udpConnectionlessSending=true&sync=false");
>
> Does anyone know what else I should try?
>
> I have never had any luck trying to install a non-default encoder/decoder, but in this case I don't think I even need it.  I just need to serialize the protobuf to a byte array, and it seems that this 'udpByteArrayCodec' is exactly what I need.  So how should I be using it?  What could be preventing anything from appearing over the wire?
>
> Thank you so much as always!
>
> Ron



--
Zoran Regvart
Reply | Threaded
Open this post in threaded view
|

Re: Problem with netty4:udp and udpByteArrayCodec=true

Ron Cecchini
Bazinga!

Thanks, Zoran.  'useByteBuf' was right in front of my face - right above 'udpByteArrayCodec' - and I didn't see it.

At any rate, I tried what you said - and it didn't work right away - but that was my fault, not yours.

I first tried getting rid of the convertBodyTo(), as you mentioned.  But when that didn't work, and nothing was getting written over the wire, I went back to trying different things to use in convertBodyTo().  And when I eventually tried doing a convertBodyTo(ByteBuf.class), I got an exception that contained just enough information about "caching" that made me think that maybe I shouldn't be doing the 'getContext().setStreamCashing(true)' I've been doing by default.  So I commented that out - and it all finally worked just great.

Thank you so much again!

Ron

FWIW, here's what I did:

// getContext().setStreamCaching(true);

from("netty4:udp://host_in:port_in?networkInterface=lo&sync=false&allowDefaultCodec=false")
    .unmarshal(jaxbDataFormat)
    .process(new MyProcessor())  // converts POJO to protobuf and sets the Exchange
    .marshal().protobuf()
//    .convertBodyTo(ByteBuf.class) // XXX: not needed, 'useByteBuf=true' works just fine
    .to("netty4:udp://host_out:port_out?udpConnectionlessSending=true&sync=false&allowDefaultCodec=false&useByteBuf=true");

> On November 9, 2018 at 4:32 AM Zoran Regvart <[hidden email]> wrote:
>
>
> Hi Ron,
> I'm far from an expert on Netty, it seems to me like that
> `convertBodyTo(String.class, "UTF-8")` will lead to issues with binary
> data.
>
> What you want (I think) is
> `.to("netty4:udp://host:port?allowDefaultCodec=false&useByteBuf=true")`
> without `convertBodyTo`
>
> zoran
>
>
> On Fri, Nov 9, 2018 at 6:12 AM, Ron Cecchini <[hidden email]> wrote:
> >
> > First, a couple of thank yous several weeks in the making:
> >
> > Thank you to Claus (as always) for help regarding replacing camel-rabbitmq with a custom class.  Indeed, I only needed to do a write.  So using 'to("bean:myBean?method=process)' worked perfectly.
> >
> > Thank you to Zoran for advice on Spring injection vs. Component Scan.  I never was able to get beans in the Spring config auto-wired, so I had to do the manual steps of getting the Spring context, getting the bean, and then setting the bean where it needed to be.  A bit ugly, but only a few lines of code.  It works, so I'm leaving it alone for now.
> >
> > Now on to my most recent hurdle...
> >
> > I'm trying to write a protobuf over UDP to a server (a non-Camel app written in C++) expecting it serialized into a byte array.
> >
> > The problem is that the only write that makes it to the server seems to be serialized incorrectly.  Conversely, when I think I've correctly serialized it, nothing seems to get written to the wire.
> >
> > The route below is the one that makes it to the server.  When I do a tcpdump on the server's side, however, I can see that the payload is about 35-40 bytes bigger than what the protobuf says its serialized size should be:
> >
> > from("direct:protobuf")
> >      .marshal().protobuf()
> >      .convertBodyTo(String.class, "UTF-8")  // needed, else to(netty4:udp) hangs
> >      .to("netty4:udp://host:port?udpConnectionlessSending=true&sync=false");
> >
> > I immediately recognized this as probably being a "default codec" problem, similar to what I recently encountered when I was writing some JSON over UDP with netty4.
> >
> > So I tried the following, but nothing was written over the wire:
> >
> >      .to("netty4:udp://host:port?udpConnectionlessSending=true&sync=false&allowDefaultCodec=false");
> >
> > I tried about a half dozen combinations: marshalling / not marshalling, convertBodyTo() / no convertBodyTo(), allowDefaultCode = false / true, etc.
> >
> > Nothing was working.  And then I somehow stumbled upon 'udpByteArrayCodec' in the docs and thought I'd found it!
> >
> > However, this didn't send anything to the wire either:
> >
> >      .to("netty4:udp://host:post?udpConnectionlessSending=true&sync=false&udpByteArrayCodec=true");
> >
> > Neither did this:
> >
> >      .to("netty4:udp://host:port?udpConnectionlessSending=true&sync=false&udpByteArrayCodec=true&allowDefaultCodec=false");
> >
> > Only the original route resulted in *anything* being written across the wire for the server to see:
> >
> >      .to("netty4:udp://host:port?udpConnectionlessSending=true&sync=false");
> >
> > Does anyone know what else I should try?
> >
> > I have never had any luck trying to install a non-default encoder/decoder, but in this case I don't think I even need it.  I just need to serialize the protobuf to a byte array, and it seems that this 'udpByteArrayCodec' is exactly what I need.  So how should I be using it?  What could be preventing anything from appearing over the wire?
> >
> > Thank you so much as always!
> >
> > Ron
>
>
>
> --
> Zoran Regvart