Passwords in Camel endpoint URIs and limitations of RAW syntax

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

Passwords in Camel endpoint URIs and limitations of RAW syntax

Florian Patzl
Hello,
I'm trying to figure out the best way to handle passwords in Camel endpoint URIs in my application.
I know the topic has been cause for Stack Overflow posts, JIRA entries and mails but I'm still not sure I've got everything right.
Sorry for the big wall of text, but I think I should explain what exactly I've tried and found out on the topic.

The main problem is that the reserved URI characters '+' and '&' (plus and ampersand) cause parsing problems in Camel endpoint URIs.
'+' is replaced by a blank, and '&' is treated as the delimiter to the next parameter.
An example URI for the password "pwd2+2":
pop3://localhost:3110/?username=test2&password=pwd2%2B2

A relevant post is here:
https://stackoverflow.com/questions/11018987/camel-how-to-include-an-ampersand-as-data-in-a-uri-not-as-a-delimiter/34926623#34926623


Now, the solution in documentation is using the RAW(...) syntax:
https://camel.apache.org/manual/latest/faq/how-do-i-configure-endpoints.html#HowdoIconfigureendpoints-Configuringparametervaluesusingrawvalues
So for example:
pop3://localhost:3110/?username=test2&password=RAW(pwd2+2)

Using that feature means we can no longer treat Camel URIs as pure URIs in our application, because the '+' of the password must not be escaped for this to work.
But if there's no way around that, we can deal with that.

However, when trying the limits of the RAW(...) syntax, we noticed that it can not parse passwords that contain ')&'.
This was covered in the following JIRA issue, and since then there is support for an alternative syntax using curly braces: RAW{...}, that I didn't find in documentation:
https://issues.apache.org/jira/browse/CAMEL-12982
The last comment provides a pretty detailed summary of the options and limits.


The alternative RAW{...} syntax works fine, except for a minor flaw: It breaks URI sanitizing.
For example, Camel leaks the '&2' portion of the password 'pwd2&2' in log entries like:
pop3://localhost:3110/?password=xxxxxx&2%7D&username=test2

The same problem existed for the RAW(...) syntax:
https://issues.apache.org/jira/browse/CAMEL-11269
So the fix should be rather simple, I will check whether there's already an issue for that and might even be able to submit a PR for that.

But, more importantly: By checking the passwords for ')&' and '}&' and dynamically deciding the RAW syntax to use, we should be able to support any password *except* if they contain both ')&' and '}&'.
That is a weird constraint for passwords, but should be justifiable as technical limitation.


As an alternative to all of this, I sometimes saw the suggestion to configure the component with 'useRawUri':

  *   In DefaultComponent, useRawUri() is hardcoded to false. That means for applying that to built-in components (e.g. Mail, FTP) we'd have to subclass the components to override the method?
  *   Setting useRawUri on endpoint level does not seem to be supported: https://issues.apache.org/jira/browse/CAMEL-6230
I tried that for the Mail component and got an error for unknown parameter useRawUri.

So, is my conclusion correct that escaping passwords using RAW(...) or RAW{...} - depending on the input - is the best approach for handling passwords in endpoint URIs?
Or am I missing a different approach to configure the passwords on endpoints? I've briefly read up on using property placeholders in URIs and saw that we'd still have to use RAW(...) there. So I don't think that helps.
I _think_ completely moving away from endpoint URIs and instantiating endpoints in plain Java code would also solve the issue, but that would require a couple of major changes in our application.

Best regards,
Florian
Reply | Threaded
Open this post in threaded view
|

Re: Passwords in Camel endpoint URIs and limitations of RAW syntax

Claussnitzer, Ralf
Hi Florian,

I may have missed the answer to my questions in your detailed problem description. But how is this not solved by URL-Encoding?
There was once a bug with URL encodings in Camel. Does this bug still exist? What version of Camel are you using?

-Ralf
________________________________________
From: Florian Patzl <[hidden email]>
Sent: Wednesday, June 3, 2020 2:49 PM
To: [hidden email]
Subject: Passwords in Camel endpoint URIs and limitations of RAW syntax

Hello,
I'm trying to figure out the best way to handle passwords in Camel endpoint URIs in my application.
I know the topic has been cause for Stack Overflow posts, JIRA entries and mails but I'm still not sure I've got everything right.
Sorry for the big wall of text, but I think I should explain what exactly I've tried and found out on the topic.

The main problem is that the reserved URI characters '+' and '&' (plus and ampersand) cause parsing problems in Camel endpoint URIs.
'+' is replaced by a blank, and '&' is treated as the delimiter to the next parameter.
An example URI for the password "pwd2+2":
pop3://localhost:3110/?username=test2&password=pwd2%2B2

A relevant post is here:
https://stackoverflow.com/questions/11018987/camel-how-to-include-an-ampersand-as-data-in-a-uri-not-as-a-delimiter/34926623#34926623


Now, the solution in documentation is using the RAW(...) syntax:
https://camel.apache.org/manual/latest/faq/how-do-i-configure-endpoints.html#HowdoIconfigureendpoints-Configuringparametervaluesusingrawvalues
So for example:
pop3://localhost:3110/?username=test2&password=RAW(pwd2+2)

Using that feature means we can no longer treat Camel URIs as pure URIs in our application, because the '+' of the password must not be escaped for this to work.
But if there's no way around that, we can deal with that.

However, when trying the limits of the RAW(...) syntax, we noticed that it can not parse passwords that contain ')&'.
This was covered in the following JIRA issue, and since then there is support for an alternative syntax using curly braces: RAW{...}, that I didn't find in documentation:
https://issues.apache.org/jira/browse/CAMEL-12982
The last comment provides a pretty detailed summary of the options and limits.


The alternative RAW{...} syntax works fine, except for a minor flaw: It breaks URI sanitizing.
For example, Camel leaks the '&2' portion of the password 'pwd2&2' in log entries like:
pop3://localhost:3110/?password=xxxxxx&2%7D&username=test2

The same problem existed for the RAW(...) syntax:
https://issues.apache.org/jira/browse/CAMEL-11269
So the fix should be rather simple, I will check whether there's already an issue for that and might even be able to submit a PR for that.

But, more importantly: By checking the passwords for ')&' and '}&' and dynamically deciding the RAW syntax to use, we should be able to support any password *except* if they contain both ')&' and '}&'.
That is a weird constraint for passwords, but should be justifiable as technical limitation.


As an alternative to all of this, I sometimes saw the suggestion to configure the component with 'useRawUri':

  *   In DefaultComponent, useRawUri() is hardcoded to false. That means for applying that to built-in components (e.g. Mail, FTP) we'd have to subclass the components to override the method?
  *   Setting useRawUri on endpoint level does not seem to be supported: https://issues.apache.org/jira/browse/CAMEL-6230
I tried that for the Mail component and got an error for unknown parameter useRawUri.

So, is my conclusion correct that escaping passwords using RAW(...) or RAW{...} - depending on the input - is the best approach for handling passwords in endpoint URIs?
Or am I missing a different approach to configure the passwords on endpoints? I've briefly read up on using property placeholders in URIs and saw that we'd still have to use RAW(...) there. So I don't think that helps.
I _think_ completely moving away from endpoint URIs and instantiating endpoints in plain Java code would also solve the issue, but that would require a couple of major changes in our application.

Best regards,
Florian
Reply | Threaded
Open this post in threaded view
|

Re: Passwords in Camel endpoint URIs and limitations of RAW syntax

Florian Patzl
Hello Ralf,
thanks for your response. No, I didn't mention that in my description. :-)
URL encoding would be my preferred solution, too, but unfortunately that does not seem to prevent the problems with passwords containing ")&". Unless something about my encoding is wrong.

For example, given a password "pwd)&a=b", both with and without RAW(...) the result is wrong:

TRACE o.a.c.i.engine.AbstractCamelContext - Getting endpoint with raw uri: pop3://localhost:3110/?username=test2&password=RAW%28pwd%29%26a%3Db%29, normalized uri: pop3://localhost:3110/?a=b%29&password=RAW(pwd)&username=test2

TRACE o.a.c.i.engine.AbstractCamelContext - Getting endpoint with raw uri: pop3://localhost:3110/?username=test2&password=pwd%29%26a%3Db, normalized uri: pop3://localhost:3110/?a=b&password=pwd%29&username=test2

I'm currently testing on 3.3.0.

Best Regards,
Florian
________________________________
From: Claussnitzer, Ralf <[hidden email]>
Sent: Thursday, June 4, 2020 08:13
To: [hidden email] <[hidden email]>
Subject: Re: Passwords in Camel endpoint URIs and limitations of RAW syntax

Hi Florian,

I may have missed the answer to my questions in your detailed problem description. But how is this not solved by URL-Encoding?
There was once a bug with URL encodings in Camel. Does this bug still exist? What version of Camel are you using?

-Ralf
________________________________________
From: Florian Patzl <[hidden email]>
Sent: Wednesday, June 3, 2020 2:49 PM
To: [hidden email]
Subject: Passwords in Camel endpoint URIs and limitations of RAW syntax

Hello,
I'm trying to figure out the best way to handle passwords in Camel endpoint URIs in my application.
I know the topic has been cause for Stack Overflow posts, JIRA entries and mails but I'm still not sure I've got everything right.
Sorry for the big wall of text, but I think I should explain what exactly I've tried and found out on the topic.

The main problem is that the reserved URI characters '+' and '&' (plus and ampersand) cause parsing problems in Camel endpoint URIs.
'+' is replaced by a blank, and '&' is treated as the delimiter to the next parameter.
An example URI for the password "pwd2+2":
pop3://localhost:3110/?username=test2&password=pwd2%2B2

A relevant post is here:
https://stackoverflow.com/questions/11018987/camel-how-to-include-an-ampersand-as-data-in-a-uri-not-as-a-delimiter/34926623#34926623


Now, the solution in documentation is using the RAW(...) syntax:
https://camel.apache.org/manual/latest/faq/how-do-i-configure-endpoints.html#HowdoIconfigureendpoints-Configuringparametervaluesusingrawvalues
So for example:
pop3://localhost:3110/?username=test2&password=RAW(pwd2+2)

Using that feature means we can no longer treat Camel URIs as pure URIs in our application, because the '+' of the password must not be escaped for this to work.
But if there's no way around that, we can deal with that.

However, when trying the limits of the RAW(...) syntax, we noticed that it can not parse passwords that contain ')&'.
This was covered in the following JIRA issue, and since then there is support for an alternative syntax using curly braces: RAW{...}, that I didn't find in documentation:
https://issues.apache.org/jira/browse/CAMEL-12982
The last comment provides a pretty detailed summary of the options and limits.


The alternative RAW{...} syntax works fine, except for a minor flaw: It breaks URI sanitizing.
For example, Camel leaks the '&2' portion of the password 'pwd2&2' in log entries like:
pop3://localhost:3110/?password=xxxxxx&2%7D&username=test2

The same problem existed for the RAW(...) syntax:
https://issues.apache.org/jira/browse/CAMEL-11269
So the fix should be rather simple, I will check whether there's already an issue for that and might even be able to submit a PR for that.

But, more importantly: By checking the passwords for ')&' and '}&' and dynamically deciding the RAW syntax to use, we should be able to support any password *except* if they contain both ')&' and '}&'.
That is a weird constraint for passwords, but should be justifiable as technical limitation.


As an alternative to all of this, I sometimes saw the suggestion to configure the component with 'useRawUri':

  *   In DefaultComponent, useRawUri() is hardcoded to false. That means for applying that to built-in components (e.g. Mail, FTP) we'd have to subclass the components to override the method?
  *   Setting useRawUri on endpoint level does not seem to be supported: https://issues.apache.org/jira/browse/CAMEL-6230
I tried that for the Mail component and got an error for unknown parameter useRawUri.

So, is my conclusion correct that escaping passwords using RAW(...) or RAW{...} - depending on the input - is the best approach for handling passwords in endpoint URIs?
Or am I missing a different approach to configure the passwords on endpoints? I've briefly read up on using property placeholders in URIs and saw that we'd still have to use RAW(...) there. So I don't think that helps.
I _think_ completely moving away from endpoint URIs and instantiating endpoints in plain Java code would also solve the issue, but that would require a couple of major changes in our application.

Best regards,
Florian
Reply | Threaded
Open this post in threaded view
|

Re: Passwords in Camel endpoint URIs and limitations of RAW syntax

fabryprog
Hello Florian,

Did you try to save the password into a variable / parameter / config file
and lookup it into the URI?

Kind regards!

Il giorno gio 4 giu 2020 alle ore 10:50 Florian Patzl <
[hidden email]> ha scritto:

> Hello Ralf,
> thanks for your response. No, I didn't mention that in my description. :-)
> URL encoding would be my preferred solution, too, but unfortunately that
> does not seem to prevent the problems with passwords containing ")&".
> Unless something about my encoding is wrong.
>
> For example, given a password "pwd)&a=b", both with and without RAW(...)
> the result is wrong:
>
> TRACE o.a.c.i.engine.AbstractCamelContext - Getting endpoint with raw uri:
> pop3://localhost:3110/?username=test2&password=RAW%28pwd%29%26a%3Db%29,
> normalized uri:
> pop3://localhost:3110/?a=b%29&password=RAW(pwd)&username=test2
>
> TRACE o.a.c.i.engine.AbstractCamelContext - Getting endpoint with raw uri:
> pop3://localhost:3110/?username=test2&password=pwd%29%26a%3Db, normalized
> uri: pop3://localhost:3110/?a=b&password=pwd%29&username=test2
>
> I'm currently testing on 3.3.0.
>
> Best Regards,
> Florian
> ________________________________
> From: Claussnitzer, Ralf <[hidden email]>
> Sent: Thursday, June 4, 2020 08:13
> To: [hidden email] <[hidden email]>
> Subject: Re: Passwords in Camel endpoint URIs and limitations of RAW syntax
>
> Hi Florian,
>
> I may have missed the answer to my questions in your detailed problem
> description. But how is this not solved by URL-Encoding?
> There was once a bug with URL encodings in Camel. Does this bug still
> exist? What version of Camel are you using?
>
> -Ralf
> ________________________________________
> From: Florian Patzl <[hidden email]>
> Sent: Wednesday, June 3, 2020 2:49 PM
> To: [hidden email]
> Subject: Passwords in Camel endpoint URIs and limitations of RAW syntax
>
> Hello,
> I'm trying to figure out the best way to handle passwords in Camel
> endpoint URIs in my application.
> I know the topic has been cause for Stack Overflow posts, JIRA entries and
> mails but I'm still not sure I've got everything right.
> Sorry for the big wall of text, but I think I should explain what exactly
> I've tried and found out on the topic.
>
> The main problem is that the reserved URI characters '+' and '&' (plus and
> ampersand) cause parsing problems in Camel endpoint URIs.
> '+' is replaced by a blank, and '&' is treated as the delimiter to the
> next parameter.
> An example URI for the password "pwd2+2":
> pop3://localhost:3110/?username=test2&password=pwd2%2B2
>
> A relevant post is here:
>
> https://stackoverflow.com/questions/11018987/camel-how-to-include-an-ampersand-as-data-in-a-uri-not-as-a-delimiter/34926623#34926623
>
>
> Now, the solution in documentation is using the RAW(...) syntax:
>
> https://camel.apache.org/manual/latest/faq/how-do-i-configure-endpoints.html#HowdoIconfigureendpoints-Configuringparametervaluesusingrawvalues
> So for example:
> pop3://localhost:3110/?username=test2&password=RAW(pwd2+2)
>
> Using that feature means we can no longer treat Camel URIs as pure URIs in
> our application, because the '+' of the password must not be escaped for
> this to work.
> But if there's no way around that, we can deal with that.
>
> However, when trying the limits of the RAW(...) syntax, we noticed that it
> can not parse passwords that contain ')&'.
> This was covered in the following JIRA issue, and since then there is
> support for an alternative syntax using curly braces: RAW{...}, that I
> didn't find in documentation:
> https://issues.apache.org/jira/browse/CAMEL-12982
> The last comment provides a pretty detailed summary of the options and
> limits.
>
>
> The alternative RAW{...} syntax works fine, except for a minor flaw: It
> breaks URI sanitizing.
> For example, Camel leaks the '&2' portion of the password 'pwd2&2' in log
> entries like:
> pop3://localhost:3110/?password=xxxxxx&2%7D&username=test2
>
> The same problem existed for the RAW(...) syntax:
> https://issues.apache.org/jira/browse/CAMEL-11269
> So the fix should be rather simple, I will check whether there's already
> an issue for that and might even be able to submit a PR for that.
>
> But, more importantly: By checking the passwords for ')&' and '}&' and
> dynamically deciding the RAW syntax to use, we should be able to support
> any password *except* if they contain both ')&' and '}&'.
> That is a weird constraint for passwords, but should be justifiable as
> technical limitation.
>
>
> As an alternative to all of this, I sometimes saw the suggestion to
> configure the component with 'useRawUri':
>
>   *   In DefaultComponent, useRawUri() is hardcoded to false. That means
> for applying that to built-in components (e.g. Mail, FTP) we'd have to
> subclass the components to override the method?
>   *   Setting useRawUri on endpoint level does not seem to be supported:
> https://issues.apache.org/jira/browse/CAMEL-6230
> I tried that for the Mail component and got an error for unknown parameter
> useRawUri.
>
> So, is my conclusion correct that escaping passwords using RAW(...) or
> RAW{...} - depending on the input - is the best approach for handling
> passwords in endpoint URIs?
> Or am I missing a different approach to configure the passwords on
> endpoints? I've briefly read up on using property placeholders in URIs and
> saw that we'd still have to use RAW(...) there. So I don't think that helps.
> I _think_ completely moving away from endpoint URIs and instantiating
> endpoints in plain Java code would also solve the issue, but that would
> require a couple of major changes in our application.
>
> Best regards,
> Florian
>
Reply | Threaded
Open this post in threaded view
|

Re: Passwords in Camel endpoint URIs and limitations of RAW syntax

Omar Al-Safi
Hi Florian,

As Fabry mentioned, it would be worth checking to use config file to
achieve this. Here is an example:
https://github.com/apache/camel-examples/tree/master/examples/camel-example-debezium

Regards,
Omar

On Thu, Jun 4, 2020 at 10:56 AM FabryProg <[hidden email]> wrote:

> Hello Florian,
>
> Did you try to save the password into a variable / parameter / config file
> and lookup it into the URI?
>
> Kind regards!
>
> Il giorno gio 4 giu 2020 alle ore 10:50 Florian Patzl <
> [hidden email]> ha scritto:
>
> > Hello Ralf,
> > thanks for your response. No, I didn't mention that in my description.
> :-)
> > URL encoding would be my preferred solution, too, but unfortunately that
> > does not seem to prevent the problems with passwords containing ")&".
> > Unless something about my encoding is wrong.
> >
> > For example, given a password "pwd)&a=b", both with and without RAW(...)
> > the result is wrong:
> >
> > TRACE o.a.c.i.engine.AbstractCamelContext - Getting endpoint with raw
> uri:
> > pop3://localhost:3110/?username=test2&password=RAW%28pwd%29%26a%3Db%29,
> > normalized uri:
> > pop3://localhost:3110/?a=b%29&password=RAW(pwd)&username=test2
> >
> > TRACE o.a.c.i.engine.AbstractCamelContext - Getting endpoint with raw
> uri:
> > pop3://localhost:3110/?username=test2&password=pwd%29%26a%3Db, normalized
> > uri: pop3://localhost:3110/?a=b&password=pwd%29&username=test2
> >
> > I'm currently testing on 3.3.0.
> >
> > Best Regards,
> > Florian
> > ________________________________
> > From: Claussnitzer, Ralf <[hidden email]>
> > Sent: Thursday, June 4, 2020 08:13
> > To: [hidden email] <[hidden email]>
> > Subject: Re: Passwords in Camel endpoint URIs and limitations of RAW
> syntax
> >
> > Hi Florian,
> >
> > I may have missed the answer to my questions in your detailed problem
> > description. But how is this not solved by URL-Encoding?
> > There was once a bug with URL encodings in Camel. Does this bug still
> > exist? What version of Camel are you using?
> >
> > -Ralf
> > ________________________________________
> > From: Florian Patzl <[hidden email]>
> > Sent: Wednesday, June 3, 2020 2:49 PM
> > To: [hidden email]
> > Subject: Passwords in Camel endpoint URIs and limitations of RAW syntax
> >
> > Hello,
> > I'm trying to figure out the best way to handle passwords in Camel
> > endpoint URIs in my application.
> > I know the topic has been cause for Stack Overflow posts, JIRA entries
> and
> > mails but I'm still not sure I've got everything right.
> > Sorry for the big wall of text, but I think I should explain what exactly
> > I've tried and found out on the topic.
> >
> > The main problem is that the reserved URI characters '+' and '&' (plus
> and
> > ampersand) cause parsing problems in Camel endpoint URIs.
> > '+' is replaced by a blank, and '&' is treated as the delimiter to the
> > next parameter.
> > An example URI for the password "pwd2+2":
> > pop3://localhost:3110/?username=test2&password=pwd2%2B2
> >
> > A relevant post is here:
> >
> >
> https://stackoverflow.com/questions/11018987/camel-how-to-include-an-ampersand-as-data-in-a-uri-not-as-a-delimiter/34926623#34926623
> >
> >
> > Now, the solution in documentation is using the RAW(...) syntax:
> >
> >
> https://camel.apache.org/manual/latest/faq/how-do-i-configure-endpoints.html#HowdoIconfigureendpoints-Configuringparametervaluesusingrawvalues
> > So for example:
> > pop3://localhost:3110/?username=test2&password=RAW(pwd2+2)
> >
> > Using that feature means we can no longer treat Camel URIs as pure URIs
> in
> > our application, because the '+' of the password must not be escaped for
> > this to work.
> > But if there's no way around that, we can deal with that.
> >
> > However, when trying the limits of the RAW(...) syntax, we noticed that
> it
> > can not parse passwords that contain ')&'.
> > This was covered in the following JIRA issue, and since then there is
> > support for an alternative syntax using curly braces: RAW{...}, that I
> > didn't find in documentation:
> > https://issues.apache.org/jira/browse/CAMEL-12982
> > The last comment provides a pretty detailed summary of the options and
> > limits.
> >
> >
> > The alternative RAW{...} syntax works fine, except for a minor flaw: It
> > breaks URI sanitizing.
> > For example, Camel leaks the '&2' portion of the password 'pwd2&2' in log
> > entries like:
> > pop3://localhost:3110/?password=xxxxxx&2%7D&username=test2
> >
> > The same problem existed for the RAW(...) syntax:
> > https://issues.apache.org/jira/browse/CAMEL-11269
> > So the fix should be rather simple, I will check whether there's already
> > an issue for that and might even be able to submit a PR for that.
> >
> > But, more importantly: By checking the passwords for ')&' and '}&' and
> > dynamically deciding the RAW syntax to use, we should be able to support
> > any password *except* if they contain both ')&' and '}&'.
> > That is a weird constraint for passwords, but should be justifiable as
> > technical limitation.
> >
> >
> > As an alternative to all of this, I sometimes saw the suggestion to
> > configure the component with 'useRawUri':
> >
> >   *   In DefaultComponent, useRawUri() is hardcoded to false. That means
> > for applying that to built-in components (e.g. Mail, FTP) we'd have to
> > subclass the components to override the method?
> >   *   Setting useRawUri on endpoint level does not seem to be supported:
> > https://issues.apache.org/jira/browse/CAMEL-6230
> > I tried that for the Mail component and got an error for unknown
> parameter
> > useRawUri.
> >
> > So, is my conclusion correct that escaping passwords using RAW(...) or
> > RAW{...} - depending on the input - is the best approach for handling
> > passwords in endpoint URIs?
> > Or am I missing a different approach to configure the passwords on
> > endpoints? I've briefly read up on using property placeholders in URIs
> and
> > saw that we'd still have to use RAW(...) there. So I don't think that
> helps.
> > I _think_ completely moving away from endpoint URIs and instantiating
> > endpoints in plain Java code would also solve the issue, but that would
> > require a couple of major changes in our application.
> >
> > Best regards,
> > Florian
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Passwords in Camel endpoint URIs and limitations of RAW syntax

Florian Patzl
Hi,
thanks a lot, Fabry and Omar! Storing the password in a Properties file helps, I just tested with a really contrived example "++pwd2&)more)&}&}" and it worked.
I hadn't tried that before because the explanations for using RAW in properties files put me off:

> Notice we still define the RAW(value) style to ensure the password is used as is[...]
https://camel.apache.org/manual/latest/faq/how-do-i-configure-endpoints.html#HowdoIconfigureendpoints-Configuringparametervaluesusingrawvalues

We haven't used the properties mechanism before and storing them in a file might be problematic, but I think I've read that there are other options for providing properties as well.
I'll read up on that; your suggestions definitely helped, thanks!

Best regards,
Florian
________________________________
From: Omar Al-Safi <[hidden email]>
Sent: Thursday, June 4, 2020 10:59
To: [hidden email] <[hidden email]>
Subject: Re: Passwords in Camel endpoint URIs and limitations of RAW syntax

Hi Florian,

As Fabry mentioned, it would be worth checking to use config file to
achieve this. Here is an example:
https://github.com/apache/camel-examples/tree/master/examples/camel-example-debezium

Regards,
Omar

On Thu, Jun 4, 2020 at 10:56 AM FabryProg <[hidden email]> wrote:

> Hello Florian,
>
> Did you try to save the password into a variable / parameter / config file
> and lookup it into the URI?
>
> Kind regards!
>
> Il giorno gio 4 giu 2020 alle ore 10:50 Florian Patzl <
> [hidden email]> ha scritto:
>
> > Hello Ralf,
> > thanks for your response. No, I didn't mention that in my description.
> :-)
> > URL encoding would be my preferred solution, too, but unfortunately that
> > does not seem to prevent the problems with passwords containing ")&".
> > Unless something about my encoding is wrong.
> >
> > For example, given a password "pwd)&a=b", both with and without RAW(...)
> > the result is wrong:
> >
> > TRACE o.a.c.i.engine.AbstractCamelContext - Getting endpoint with raw
> uri:
> > pop3://localhost:3110/?username=test2&password=RAW%28pwd%29%26a%3Db%29,
> > normalized uri:
> > pop3://localhost:3110/?a=b%29&password=RAW(pwd)&username=test2
> >
> > TRACE o.a.c.i.engine.AbstractCamelContext - Getting endpoint with raw
> uri:
> > pop3://localhost:3110/?username=test2&password=pwd%29%26a%3Db, normalized
> > uri: pop3://localhost:3110/?a=b&password=pwd%29&username=test2
> >
> > I'm currently testing on 3.3.0.
> >
> > Best Regards,
> > Florian
> > ________________________________
> > From: Claussnitzer, Ralf <[hidden email]>
> > Sent: Thursday, June 4, 2020 08:13
> > To: [hidden email] <[hidden email]>
> > Subject: Re: Passwords in Camel endpoint URIs and limitations of RAW
> syntax
> >
> > Hi Florian,
> >
> > I may have missed the answer to my questions in your detailed problem
> > description. But how is this not solved by URL-Encoding?
> > There was once a bug with URL encodings in Camel. Does this bug still
> > exist? What version of Camel are you using?
> >
> > -Ralf
> > ________________________________________
> > From: Florian Patzl <[hidden email]>
> > Sent: Wednesday, June 3, 2020 2:49 PM
> > To: [hidden email]
> > Subject: Passwords in Camel endpoint URIs and limitations of RAW syntax
> >
> > Hello,
> > I'm trying to figure out the best way to handle passwords in Camel
> > endpoint URIs in my application.
> > I know the topic has been cause for Stack Overflow posts, JIRA entries
> and
> > mails but I'm still not sure I've got everything right.
> > Sorry for the big wall of text, but I think I should explain what exactly
> > I've tried and found out on the topic.
> >
> > The main problem is that the reserved URI characters '+' and '&' (plus
> and
> > ampersand) cause parsing problems in Camel endpoint URIs.
> > '+' is replaced by a blank, and '&' is treated as the delimiter to the
> > next parameter.
> > An example URI for the password "pwd2+2":
> > pop3://localhost:3110/?username=test2&password=pwd2%2B2
> >
> > A relevant post is here:
> >
> >
> https://stackoverflow.com/questions/11018987/camel-how-to-include-an-ampersand-as-data-in-a-uri-not-as-a-delimiter/34926623#34926623
> >
> >
> > Now, the solution in documentation is using the RAW(...) syntax:
> >
> >
> https://camel.apache.org/manual/latest/faq/how-do-i-configure-endpoints.html#HowdoIconfigureendpoints-Configuringparametervaluesusingrawvalues
> > So for example:
> > pop3://localhost:3110/?username=test2&password=RAW(pwd2+2)
> >
> > Using that feature means we can no longer treat Camel URIs as pure URIs
> in
> > our application, because the '+' of the password must not be escaped for
> > this to work.
> > But if there's no way around that, we can deal with that.
> >
> > However, when trying the limits of the RAW(...) syntax, we noticed that
> it
> > can not parse passwords that contain ')&'.
> > This was covered in the following JIRA issue, and since then there is
> > support for an alternative syntax using curly braces: RAW{...}, that I
> > didn't find in documentation:
> > https://issues.apache.org/jira/browse/CAMEL-12982
> > The last comment provides a pretty detailed summary of the options and
> > limits.
> >
> >
> > The alternative RAW{...} syntax works fine, except for a minor flaw: It
> > breaks URI sanitizing.
> > For example, Camel leaks the '&2' portion of the password 'pwd2&2' in log
> > entries like:
> > pop3://localhost:3110/?password=xxxxxx&2%7D&username=test2
> >
> > The same problem existed for the RAW(...) syntax:
> > https://issues.apache.org/jira/browse/CAMEL-11269
> > So the fix should be rather simple, I will check whether there's already
> > an issue for that and might even be able to submit a PR for that.
> >
> > But, more importantly: By checking the passwords for ')&' and '}&' and
> > dynamically deciding the RAW syntax to use, we should be able to support
> > any password *except* if they contain both ')&' and '}&'.
> > That is a weird constraint for passwords, but should be justifiable as
> > technical limitation.
> >
> >
> > As an alternative to all of this, I sometimes saw the suggestion to
> > configure the component with 'useRawUri':
> >
> >   *   In DefaultComponent, useRawUri() is hardcoded to false. That means
> > for applying that to built-in components (e.g. Mail, FTP) we'd have to
> > subclass the components to override the method?
> >   *   Setting useRawUri on endpoint level does not seem to be supported:
> > https://issues.apache.org/jira/browse/CAMEL-6230
> > I tried that for the Mail component and got an error for unknown
> parameter
> > useRawUri.
> >
> > So, is my conclusion correct that escaping passwords using RAW(...) or
> > RAW{...} - depending on the input - is the best approach for handling
> > passwords in endpoint URIs?
> > Or am I missing a different approach to configure the passwords on
> > endpoints? I've briefly read up on using property placeholders in URIs
> and
> > saw that we'd still have to use RAW(...) there. So I don't think that
> helps.
> > I _think_ completely moving away from endpoint URIs and instantiating
> > endpoints in plain Java code would also solve the issue, but that would
> > require a couple of major changes in our application.
> >
> > Best regards,
> > Florian
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Passwords in Camel endpoint URIs and limitations of RAW syntax

Roman Vottner
In reply to this post by Florian Patzl
Hi Florian,

if you're concerned about logging sensitive data, I'd recommend to
configure your logging framework to filter such sensitive information in
first place as the sensitive information might otherwise leak through
other means not in control of Camel itself, i.e. as logged directly from
within a bean.

I.e. in logback you could implement a CompositeConverter
(https://github.com/RovoMe/camel-rest-dsl-with-spring-security/blob/master/src/main/java/at/rovo/awsxray/utils/MaskingConverter.java)
where you perform masking of sensitive data yourself. In the logback
configuration you then need to register the custom implementation like
this
https://github.com/RovoMe/camel-rest-dsl-with-spring-security/blob/master/src/main/resources/logback.xml#L4
and use the conversionWord within your log pattern, i.e. %mask(%msg).

As this might be a costly operation on each log, it is probably
beneficial to define when to mask sensitive data by using markers. I.e.
in the linked implementation a CONFIDENTIAL marker is created which can
also be used in a Camel log
(https://github.com/RovoMe/camel-rest-dsl-with-spring-security/blob/master/src/main/java/at/rovo/awsxray/routes/HttpInvokerRoute.java#L61)
as well.

HTH,

Roman

On 04.06.2020 10:49, Florian Patzl wrote:

> Hello Ralf,
> thanks for your response. No, I didn't mention that in my description. :-)
> URL encoding would be my preferred solution, too, but unfortunately that does not seem to prevent the problems with passwords containing ")&". Unless something about my encoding is wrong.
>
> For example, given a password "pwd)&a=b", both with and without RAW(...) the result is wrong:
>
> TRACE o.a.c.i.engine.AbstractCamelContext - Getting endpoint with raw uri: pop3://localhost:3110/?username=test2&password=RAW%28pwd%29%26a%3Db%29, normalized uri: pop3://localhost:3110/?a=b%29&password=RAW(pwd)&username=test2
>
> TRACE o.a.c.i.engine.AbstractCamelContext - Getting endpoint with raw uri: pop3://localhost:3110/?username=test2&password=pwd%29%26a%3Db, normalized uri: pop3://localhost:3110/?a=b&password=pwd%29&username=test2
>
> I'm currently testing on 3.3.0.
>
> Best Regards,
> Florian
> ________________________________
> From: Claussnitzer, Ralf <[hidden email]>
> Sent: Thursday, June 4, 2020 08:13
> To: [hidden email] <[hidden email]>
> Subject: Re: Passwords in Camel endpoint URIs and limitations of RAW syntax
>
> Hi Florian,
>
> I may have missed the answer to my questions in your detailed problem description. But how is this not solved by URL-Encoding?
> There was once a bug with URL encodings in Camel. Does this bug still exist? What version of Camel are you using?
>
> -Ralf
> ________________________________________
> From: Florian Patzl <[hidden email]>
> Sent: Wednesday, June 3, 2020 2:49 PM
> To: [hidden email]
> Subject: Passwords in Camel endpoint URIs and limitations of RAW syntax
>
> Hello,
> I'm trying to figure out the best way to handle passwords in Camel endpoint URIs in my application.
> I know the topic has been cause for Stack Overflow posts, JIRA entries and mails but I'm still not sure I've got everything right.
> Sorry for the big wall of text, but I think I should explain what exactly I've tried and found out on the topic.
>
> The main problem is that the reserved URI characters '+' and '&' (plus and ampersand) cause parsing problems in Camel endpoint URIs.
> '+' is replaced by a blank, and '&' is treated as the delimiter to the next parameter.
> An example URI for the password "pwd2+2":
> pop3://localhost:3110/?username=test2&password=pwd2%2B2
>
> A relevant post is here:
> https://stackoverflow.com/questions/11018987/camel-how-to-include-an-ampersand-as-data-in-a-uri-not-as-a-delimiter/34926623#34926623
>
>
> Now, the solution in documentation is using the RAW(...) syntax:
> https://camel.apache.org/manual/latest/faq/how-do-i-configure-endpoints.html#HowdoIconfigureendpoints-Configuringparametervaluesusingrawvalues
> So for example:
> pop3://localhost:3110/?username=test2&password=RAW(pwd2+2)
>
> Using that feature means we can no longer treat Camel URIs as pure URIs in our application, because the '+' of the password must not be escaped for this to work.
> But if there's no way around that, we can deal with that.
>
> However, when trying the limits of the RAW(...) syntax, we noticed that it can not parse passwords that contain ')&'.
> This was covered in the following JIRA issue, and since then there is support for an alternative syntax using curly braces: RAW{...}, that I didn't find in documentation:
> https://issues.apache.org/jira/browse/CAMEL-12982
> The last comment provides a pretty detailed summary of the options and limits.
>
>
> The alternative RAW{...} syntax works fine, except for a minor flaw: It breaks URI sanitizing.
> For example, Camel leaks the '&2' portion of the password 'pwd2&2' in log entries like:
> pop3://localhost:3110/?password=xxxxxx&2%7D&username=test2
>
> The same problem existed for the RAW(...) syntax:
> https://issues.apache.org/jira/browse/CAMEL-11269
> So the fix should be rather simple, I will check whether there's already an issue for that and might even be able to submit a PR for that.
>
> But, more importantly: By checking the passwords for ')&' and '}&' and dynamically deciding the RAW syntax to use, we should be able to support any password *except* if they contain both ')&' and '}&'.
> That is a weird constraint for passwords, but should be justifiable as technical limitation.
>
>
> As an alternative to all of this, I sometimes saw the suggestion to configure the component with 'useRawUri':
>
>    *   In DefaultComponent, useRawUri() is hardcoded to false. That means for applying that to built-in components (e.g. Mail, FTP) we'd have to subclass the components to override the method?
>    *   Setting useRawUri on endpoint level does not seem to be supported: https://issues.apache.org/jira/browse/CAMEL-6230
> I tried that for the Mail component and got an error for unknown parameter useRawUri.
>
> So, is my conclusion correct that escaping passwords using RAW(...) or RAW{...} - depending on the input - is the best approach for handling passwords in endpoint URIs?
> Or am I missing a different approach to configure the passwords on endpoints? I've briefly read up on using property placeholders in URIs and saw that we'd still have to use RAW(...) there. So I don't think that helps.
> I _think_ completely moving away from endpoint URIs and instantiating endpoints in plain Java code would also solve the issue, but that would require a couple of major changes in our application.
>
> Best regards,
> Florian
>
Reply | Threaded
Open this post in threaded view
|

Re: Passwords in Camel endpoint URIs and limitations of RAW syntax

Vojtech Fried-2
In reply to this post by Florian Patzl
Hi Florian,

I also had this problem. Neither URI encoding nor RAW helped. My solution in 2.x Camel was to use #<name> notation in the URI with a String stored as <name> in a registry. Then Camel took the string and did not process it. It required some fiddling with Registries to make them writable, but it was doable. However, it seems that it does not work in Camel 3. Something like this is "${ref:<name>}, but when used with Strings, they are still somehow process by Camel and in the end it fails with some special characters.

/Vojtech

________________________________
From: Florian Patzl <[hidden email]>
Sent: Wednesday, June 3, 2020 14:49
To: [hidden email] <[hidden email]>
Subject: [EXTERNAL] - Passwords in Camel endpoint URIs and limitations of RAW syntax

Hello,
I'm trying to figure out the best way to handle passwords in Camel endpoint URIs in my application.
I know the topic has been cause for Stack Overflow posts, JIRA entries and mails but I'm still not sure I've got everything right.
Sorry for the big wall of text, but I think I should explain what exactly I've tried and found out on the topic.

The main problem is that the reserved URI characters '+' and '&' (plus and ampersand) cause parsing problems in Camel endpoint URIs.
'+' is replaced by a blank, and '&' is treated as the delimiter to the next parameter.
An example URI for the password "pwd2+2":
pop3://localhost:3110/?username=test2&password=pwd2%2B2

A relevant post is here:
https://urldefense.com/v3/__https://stackoverflow.com/questions/11018987/camel-how-to-include-an-ampersand-as-data-in-a-uri-not-as-a-delimiter/34926623*34926623__;Iw!!Obbck6kTJA!KGASSfrNiG19hrz3XFpkjkI-DfK5STncbGLPofkcoybTnb0mkrK4hbqAKtilg8Ud$


Now, the solution in documentation is using the RAW(...) syntax:
https://urldefense.com/v3/__https://camel.apache.org/manual/latest/faq/how-do-i-configure-endpoints.html*HowdoIconfigureendpoints-Configuringparametervaluesusingrawvalues__;Iw!!Obbck6kTJA!KGASSfrNiG19hrz3XFpkjkI-DfK5STncbGLPofkcoybTnb0mkrK4hbqAKrPfyljz$
So for example:
pop3://localhost:3110/?username=test2&password=RAW(pwd2+2)

Using that feature means we can no longer treat Camel URIs as pure URIs in our application, because the '+' of the password must not be escaped for this to work.
But if there's no way around that, we can deal with that.

However, when trying the limits of the RAW(...) syntax, we noticed that it can not parse passwords that contain ')&'.
This was covered in the following JIRA issue, and since then there is support for an alternative syntax using curly braces: RAW{...}, that I didn't find in documentation:
https://urldefense.com/v3/__https://issues.apache.org/jira/browse/CAMEL-12982__;!!Obbck6kTJA!KGASSfrNiG19hrz3XFpkjkI-DfK5STncbGLPofkcoybTnb0mkrK4hbqAKkO9zER_$
The last comment provides a pretty detailed summary of the options and limits.


The alternative RAW{...} syntax works fine, except for a minor flaw: It breaks URI sanitizing.
For example, Camel leaks the '&2' portion of the password 'pwd2&2' in log entries like:
pop3://localhost:3110/?password=xxxxxx&2%7D&username=test2

The same problem existed for the RAW(...) syntax:
https://urldefense.com/v3/__https://issues.apache.org/jira/browse/CAMEL-11269__;!!Obbck6kTJA!KGASSfrNiG19hrz3XFpkjkI-DfK5STncbGLPofkcoybTnb0mkrK4hbqAKinvp2KB$
So the fix should be rather simple, I will check whether there's already an issue for that and might even be able to submit a PR for that.

But, more importantly: By checking the passwords for ')&' and '}&' and dynamically deciding the RAW syntax to use, we should be able to support any password *except* if they contain both ')&' and '}&'.
That is a weird constraint for passwords, but should be justifiable as technical limitation.


As an alternative to all of this, I sometimes saw the suggestion to configure the component with 'useRawUri':

  *   In DefaultComponent, useRawUri() is hardcoded to false. That means for applying that to built-in components (e.g. Mail, FTP) we'd have to subclass the components to override the method?
  *   Setting useRawUri on endpoint level does not seem to be supported: https://urldefense.com/v3/__https://issues.apache.org/jira/browse/CAMEL-6230__;!!Obbck6kTJA!KGASSfrNiG19hrz3XFpkjkI-DfK5STncbGLPofkcoybTnb0mkrK4hbqAKiki1NMf$
I tried that for the Mail component and got an error for unknown parameter useRawUri.

So, is my conclusion correct that escaping passwords using RAW(...) or RAW{...} - depending on the input - is the best approach for handling passwords in endpoint URIs?
Or am I missing a different approach to configure the passwords on endpoints? I've briefly read up on using property placeholders in URIs and saw that we'd still have to use RAW(...) there. So I don't think that helps.
I _think_ completely moving away from endpoint URIs and instantiating endpoints in plain Java code would also solve the issue, but that would require a couple of major changes in our application.

Best regards,
Florian
Reply | Threaded
Open this post in threaded view
|

Re: Passwords in Camel endpoint URIs and limitations of RAW syntax

Vojtech Fried-2
As Claus pointed out, the correct syntax is now #bean:<name>.
/Vojtech

Reply | Threaded
Open this post in threaded view
|

Re: Passwords in Camel endpoint URIs and limitations of RAW syntax

Florian Patzl
Hello Vojtech,
thanks a lot for your input and updating your solution for 3.x.
I chose a slightly different approach by implementing a PropertiesFunction, so that we don't have to place every password in the registry.
The PropertiesFunction looks up the password in an pre-existing internal cache when Camel resolves the URI.

The only hurdle with that approach I've noticed so far is that the properties are resolved too early when the property-tag is not encoded - which the URI encoding class we've been using didn't.
For example, this works with special characters in the password:
pop3://localhost:3110/?username=test2&password=%7B%7Bmypw%3Apwd2%7D%7D
but this doesn't:
pop3://localhost:3110/?username=test2&password={{mypw:pwd2}}
That was a bit surprising, but I think completely encoding query parameter values is more correct anyway.

I've tested with the approach with the Camel Mail component and a custom component of ours and it works fine so far.
Best regards,
Florian
________________________________
From: Vojtech Fried <[hidden email]>
Sent: Monday, June 8, 2020 10:10
To: [hidden email] <[hidden email]>
Subject: Re: Passwords in Camel endpoint URIs and limitations of RAW syntax

As Claus pointed out, the correct syntax is now #bean:<name>.
/Vojtech