camel spring xml restructure

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

camel spring xml restructure

Claus Ibsen
Hi

I would suggest to restructure the camel spring xml format a bit.

The overall goal is:
====================
- ease of use

Current situations:
===================
- there are too many different types of xml tags possible that confuses end-users.
- most list types don't have a parent "holder" so they are mixed with all the others


IDE auto completion
===================
Using the IDE auto complete reveals many of these problems. For instance in the root camelContext tag, you get datatypes, endpoint, spring remoting, and other stuff and of course the routes etc is a bit of mixed content.

I would suggest to restructure the xml so you have some parent holder tags to ease end-users.


Motivation behind
=================
I would love the spring xml to be more firm and structured before letting loose my programming team on the spring xml instead of java DSL. Having IDE completion with less mixed type of elements would help a lot.

 

Camel 1.5
=========
We are nearly there I think the daatatypes must be restructure so they don't appear in the tag list from the root tag. Now you get articDS, xmlBeans, string, hl7, xstram and others. I think they should be child tags of a parent holder tag named dataTypes. This will help a lot.

I thing this change can be done in camel 1.5 without breaking to much. After all we have a bug in Camel so datatypes configured as camel tags isn't useable from the marshal and unmarshal tags. So it doesn't work (CAMEL-871).


Camel 2.0
=========
For Camel 2.0 we can consider having parent holders for
- endpoints
- routes
- the spring remote stuff
- jmx

And I also think the <package> tag should have a parent holder that better describes what it does. As now it's a bit of magic for new users reading the spring xml. Package doesn't hint that is the route builder and that is it a very important tag. Also we should consider making it possible to configure <package> as classnames or spring bean ids. So you better can get the link from camel context to what java class is the actual route builder.

We could consider having a restructure for this and have it something on the way as:

<routeBuilder>
   <package>com.acme</package>
</routeBuilder>

Or even better I think:

<routeBuilder>
   <builder package="com.acme"/>
</rouyteBuilder>

<routeBuilder>
   <builder class="com.acme.MyCheeseRoute"/>
   <builder class="com.acme.MyBeerRoute"/>
</rouyteBuilder>


And the spring ref variation is also very great:

<bean id="cheeseRoute" class="com.acme.MyCheeseRoute"/>

<routeBuilder>
   <builder ref="cheeseRoute/>
</rouyteBuilder>



Med venlig hilsen

Claus Ibsen
......................................
Silverbullet
Skovsgårdsvænget 21
8362 Hørning
Tlf. +45 2962 7576
Web: www.silverbullet.dk

Reply | Threaded
Open this post in threaded view
|

Re: camel spring xml restructure

Vadim Chekan
Claus,

I'm glad you see the problem. Arguably, difficulties with routing
configurations is one of the the biggest obstacles on wider Camel
adoption. There is nothing more frustrating than knowing exactly what
you want in terms of EIP but spending many hours with so minor issue as
syntax.

Interesting enough, I was thinking about it yesterday too and my
conclusion is such that it is very difficult to achieve smooth
configuration using xml. It is not the tool for the job. I mean it can
do it (it does it currently) and we can keep improving it but will we
get the easiest possible way of configuring camel routes?
I'll skip large philosophical discussion but in my opinion xml sometimes
becomes victim of its own success and programmers try to stretch its
application beyond the reasons because "xml is so great, I would add it
to my salad if I could" :)

Basically we are trying to define Abstract Syntax Tree in xml serialized
form. People don't usually think in terms of AST.

We can give a whirl an idea of Domain Specific Language (DSL). I have
some under my belt and it is not too difficult. The result would look
like this (syntax can be any, but I'm trying to be java script alike):

routes {
   route {
     from 'queue:Incoming'
     process {
       bean 'org.my.company.camel' 'MyFunction'
       aggregator()
     }
     if (header('myHeader') == 'value1') {
       to 'queue:Outgoing'
     } else {
       to 'queue:Alternative'
     }
   }
}

I thought about juel (or others evaluators).
I see a problem that those languagas will dictate syntax which may be
not optimal for camel route definition domain.
Another and bigger problem is that external language will not detect
camel invalid constructions. For example creating serializer but not
assigning it data format would be invalid in camel but from Juel point
of view it would be perfectly ok.
Having our own syntax parser will let us detect any syntax deviations
and provide user with very clear and informative error messages for
example " 'bean' or 'aggregator' are expected inside in 'process' ".

Another benefit we get is Backus-Naur Form (BNF) file which is human
readable and is very easy to use in documentation. As opposite to xml
schema which can be used for learning purposes but I would not risk
claiming it a recommended way.

Here are fragments of BNF as I see it:
route :
        ROUTE
        | from_list process_list to_list
        ;
...
process_list :
        /* can be empty*/
        | process
        | process process_list /* works as pipiline */
        ;
       
process :
        /* can be empty */
        | BEAN LITERAL
        | MULTICAST '{ process_list '}'
        | PIPELINE '{' process_list '}'
        | SPLIT LITERAL
        | AGGREGATE expression
        | AGGREGATE
        | resequence
        ;

resequence :
        BATCH RESEQUENCE
        | BATCH RESEQUENCE expression
        | BATCH RESEQUENCE NUMBER NUMBER
        | BATCH RESEQUENCE NUMBER NUMBER '(' expression_list ')'
        | STREAM RESEQUENCE resequence_options
        | STREAM RESEQUENCE resequence_options '(' expression ')'
        ;

to_list :
        to
        | to to_list
        ;
       
to :
        LITERAL
        | IF '(' bool_expression ')' THEN to
        | IF '(' bool_expression ')' THEN to ELSE to
        ;

If you guys think it worth a try I can mock up a prototype.
I think I would build an AST and then have another process which would
traverse AST and invoke RouteBuilder to build the actual route.
This way we preserve backward compatibility with existing ways of
building routes.

This route builder would be a separate plugin in order to be as little
invasive as possible.

AST can be used in future to build graph representation of camel routes
without building the actual routes. I recall James recent comment that
building route for display purpose only (without running it) added some
complexity.

Let me know what do you think folks.
Vadim.


Claus Ibsen wrote:

> Hi
>
> I would suggest to restructure the camel spring xml format a bit.
>
> The overall goal is:
> ====================
> - ease of use
>
> Current situations:
> ===================
> - there are too many different types of xml tags possible that confuses end-users.
> - most list types don't have a parent "holder" so they are mixed with all the others
>
>
> IDE auto completion
> ===================
> Using the IDE auto complete reveals many of these problems. For instance in the root camelContext tag, you get datatypes, endpoint, spring remoting, and other stuff and of course the routes etc is a bit of mixed content.
>
> I would suggest to restructure the xml so you have some parent holder tags to ease end-users.
>
>
> Motivation behind
> =================
> I would love the spring xml to be more firm and structured before letting loose my programming team on the spring xml instead of java DSL. Having IDE completion with less mixed type of elements would help a lot.
>
>  
>
> Camel 1.5
> =========
> We are nearly there I think the daatatypes must be restructure so they don't appear in the tag list from the root tag. Now you get articDS, xmlBeans, string, hl7, xstram and others. I think they should be child tags of a parent holder tag named dataTypes. This will help a lot.
>
> I thing this change can be done in camel 1.5 without breaking to much. After all we have a bug in Camel so datatypes configured as camel tags isn't useable from the marshal and unmarshal tags. So it doesn't work (CAMEL-871).
>
>
> Camel 2.0
> =========
> For Camel 2.0 we can consider having parent holders for
> - endpoints
> - routes
> - the spring remote stuff
> - jmx
>
> And I also think the <package> tag should have a parent holder that better describes what it does. As now it's a bit of magic for new users reading the spring xml. Package doesn't hint that is the route builder and that is it a very important tag. Also we should consider making it possible to configure <package> as classnames or spring bean ids. So you better can get the link from camel context to what java class is the actual route builder.
>
> We could consider having a restructure for this and have it something on the way as:
>
> <routeBuilder>
>    <package>com.acme</package>
> </routeBuilder>
>
> Or even better I think:
>
> <routeBuilder>
>    <builder package="com.acme"/>
> </rouyteBuilder>
>
> <routeBuilder>
>    <builder class="com.acme.MyCheeseRoute"/>
>    <builder class="com.acme.MyBeerRoute"/>
> </rouyteBuilder>
>
>
> And the spring ref variation is also very great:
>
> <bean id="cheeseRoute" class="com.acme.MyCheeseRoute"/>
>
> <routeBuilder>
>    <builder ref="cheeseRoute/>
> </rouyteBuilder>
>
>
>
> Med venlig hilsen
>
> Claus Ibsen
> ......................................
> Silverbullet
> Skovsgårdsvænget 21
> 8362 Hørning
> Tlf. +45 2962 7576
> Web: www.silverbullet.dk
>
>

Reply | Threaded
Open this post in threaded view
|

Re: camel spring xml restructure

jstrachan
In reply to this post by Claus Ibsen
Agreed with all that! For 2.0 I'd like to make the IDE completion for
Camel as good as we can make it!

Also worth mentioning is we should have nice XSDs for smart completion
for most of the common endpoint beans. e.g. so folks can do

<jms:endpoint transacted="true" queue="FOO.BAR" connectionFactory="foo"/>

or whatever - to make it easy to have nice smart IDE completion for
creating endpoints/components in Camel


2008/8/30 Claus Ibsen <[hidden email]>:

> Hi
>
> I would suggest to restructure the camel spring xml format a bit.
>
> The overall goal is:
> ====================
> - ease of use
>
> Current situations:
> ===================
> - there are too many different types of xml tags possible that confuses end-users.
> - most list types don't have a parent "holder" so they are mixed with all the others
>
>
> IDE auto completion
> ===================
> Using the IDE auto complete reveals many of these problems. For instance in the root camelContext tag, you get datatypes, endpoint, spring remoting, and other stuff and of course the routes etc is a bit of mixed content.
>
> I would suggest to restructure the xml so you have some parent holder tags to ease end-users.
>
>
> Motivation behind
> =================
> I would love the spring xml to be more firm and structured before letting loose my programming team on the spring xml instead of java DSL. Having IDE completion with less mixed type of elements would help a lot.
>
>
>
> Camel 1.5
> =========
> We are nearly there I think the daatatypes must be restructure so they don't appear in the tag list from the root tag. Now you get articDS, xmlBeans, string, hl7, xstram and others. I think they should be child tags of a parent holder tag named dataTypes. This will help a lot.
>
> I thing this change can be done in camel 1.5 without breaking to much. After all we have a bug in Camel so datatypes configured as camel tags isn't useable from the marshal and unmarshal tags. So it doesn't work (CAMEL-871).
>
>
> Camel 2.0
> =========
> For Camel 2.0 we can consider having parent holders for
> - endpoints
> - routes
> - the spring remote stuff
> - jmx
>
> And I also think the <package> tag should have a parent holder that better describes what it does. As now it's a bit of magic for new users reading the spring xml. Package doesn't hint that is the route builder and that is it a very important tag. Also we should consider making it possible to configure <package> as classnames or spring bean ids. So you better can get the link from camel context to what java class is the actual route builder.
>
> We could consider having a restructure for this and have it something on the way as:
>
> <routeBuilder>
>   <package>com.acme</package>
> </routeBuilder>
>
> Or even better I think:
>
> <routeBuilder>
>   <builder package="com.acme"/>
> </rouyteBuilder>
>
> <routeBuilder>
>   <builder class="com.acme.MyCheeseRoute"/>
>   <builder class="com.acme.MyBeerRoute"/>
> </rouyteBuilder>
>
>
> And the spring ref variation is also very great:
>
> <bean id="cheeseRoute" class="com.acme.MyCheeseRoute"/>
>
> <routeBuilder>
>   <builder ref="cheeseRoute/>
> </rouyteBuilder>
>
>
>
> Med venlig hilsen
>
> Claus Ibsen
> ......................................
> Silverbullet
> Skovsgårdsvænget 21
> 8362 Hørning
> Tlf. +45 2962 7576
> Web: www.silverbullet.dk
>
>



--
James
-------
http://macstrac.blogspot.com/

Open Source Integration
http://open.iona.com
Reply | Threaded
Open this post in threaded view
|

Re: camel spring xml restructure

jstrachan
In reply to this post by Vadim Chekan
2008/8/30 Vadim Chekan <[hidden email]>:

> Claus,
>
> I'm glad you see the problem. Arguably, difficulties with routing
> configurations is one of the the biggest obstacles on wider Camel adoption.
> There is nothing more frustrating than knowing exactly what you want in
> terms of EIP but spending many hours with so minor issue as syntax.
>
> Interesting enough, I was thinking about it yesterday too and my conclusion
> is such that it is very difficult to achieve smooth configuration using xml.
> It is not the tool for the job. I mean it can do it (it does it currently)
> and we can keep improving it but will we get the easiest possible way of
> configuring camel routes?
> I'll skip large philosophical discussion but in my opinion xml sometimes
> becomes victim of its own success and programmers try to stretch its
> application beyond the reasons because "xml is so great, I would add it to
> my salad if I could" :)
>
> Basically we are trying to define Abstract Syntax Tree in xml serialized
> form. People don't usually think in terms of AST.

Totally agree BTW. FWIW I'd always envisioned we'd have multiple DSLs;
Java, XML, Groovy, Ruby, Scala and even a real DSL - hopefully all
using the same AST underneath...
http://activemq.apache.org/camel/maven/camel-core/apidocs/org/apache/camel/model/package-summary.html

allowing folks to create routes using one language and save them in
another DSL if folks wanted to etc.


> We can give a whirl an idea of Domain Specific Language (DSL). I have some
> under my belt and it is not too difficult. The result would look like this
> (syntax can be any, but I'm trying to be java script alike):
>
> routes {
>  route {
>    from 'queue:Incoming'
>    process {
>      bean 'org.my.company.camel' 'MyFunction'
>      aggregator()
>    }
>    if (header('myHeader') == 'value1') {
>      to 'queue:Outgoing'
>    } else {
>      to 'queue:Alternative'
>    }
>  }
> }

Have you looked at the Scala DSL? Its pretty close to this already! :)
http://activemq.apache.org/camel/scala-dsl.html
http://activemq.apache.org/camel/scala-dsl-eip.html

> I thought about juel (or others evaluators).
> I see a problem that those languagas will dictate syntax which may be not
> optimal for camel route definition domain.
> Another and bigger problem is that external language will not detect camel
> invalid constructions. For example creating serializer but not assigning it
> data format would be invalid in camel but from Juel point of view it would
> be perfectly ok.

Yeah - we need to make sure we validate as much of the AST up front as
we can along with giving great error messages.


> Having our own syntax parser will let us detect any syntax deviations and
> provide user with very clear and informative error messages for example "
> 'bean' or 'aggregator' are expected inside in 'process' ".
>
> Another benefit we get is Backus-Naur Form (BNF) file which is human
> readable and is very easy to use in documentation. As opposite to xml schema
> which can be used for learning purposes but I would not risk claiming it a
> recommended way.
>
> Here are fragments of BNF as I see it:
> route :
>        ROUTE
>        | from_list process_list to_list
>        ;
> ...
> process_list :
>        /* can be empty*/
>        | process
>        | process process_list /* works as pipiline */
>        ;
>
> process :
>        /* can be empty */
>        | BEAN LITERAL
>        | MULTICAST '{ process_list '}'
>        | PIPELINE '{' process_list '}'
>        | SPLIT LITERAL
>        | AGGREGATE expression
>        | AGGREGATE
>        | resequence
>        ;
>
> resequence :
>        BATCH RESEQUENCE
>        | BATCH RESEQUENCE expression
>        | BATCH RESEQUENCE NUMBER NUMBER
>        | BATCH RESEQUENCE NUMBER NUMBER '(' expression_list ')'
>        | STREAM RESEQUENCE resequence_options
>        | STREAM RESEQUENCE resequence_options '(' expression ')'
>        ;
>
> to_list :
>        to
>        | to to_list
>        ;
>
> to :
>        LITERAL
>        | IF '(' bool_expression ')' THEN to
>        | IF '(' bool_expression ')' THEN to ELSE to
>        ;
>
> If you guys think it worth a try I can mock up a prototype.
> I think I would build an AST and then have another process which would
> traverse AST and invoke RouteBuilder to build the actual route.
> This way we preserve backward compatibility with existing ways of building
> routes.
>
> This route builder would be a separate plugin in order to be as little
> invasive as possible.
>
> AST can be used in future to build graph representation of camel routes
> without building the actual routes. I recall James recent comment that
> building route for display purpose only (without running it) added some
> complexity.
>
> Let me know what do you think folks.

I've always fancied a real DSL as well :). As you mention - the
hardest part of any real DSL is what language is used for
predicates/expressions. e.g. do we let any language be used...

xpath( /foo/bar )

or something?

But it'd be great to try make one. Incidentally if you do fancy
experimenting with this; have a look at the xtext library in eclipse.
I created a little prototype project here...

https://svn.apache.org/repos/asf/activemq/camel/ide/camel-eclipse/

which attempts to take an Antlr grammar for the DSL and auto-create
both a parser but also an eclipse based editor for the language. It
takes an Antlr grammer here...

https://svn.apache.org/repos/asf/activemq/camel/ide/camel-eclipse/camel.routing/src/camel.xtxt

which as you can see is kinda basic :)

Having both a nice textual DSL along with an IDE to edit it would rock :)

Its well worth looking at the Scala DSL which looks great; maybe they
could both look and feel kinda similar? FWIW the only thing I don't
like with the scala DSL is the use of _ which is a bit odd - but thats
a scala thing

--
James
-------
http://macstrac.blogspot.com/

Open Source Integration
http://open.iona.com
Reply | Threaded
Open this post in threaded view
|

Re: camel spring xml restructure

gertv
James,

The _ is indeed a Scala specific construct: it represents a parameter in
a function literal -- in our case this often represents the Camel
Exchange.  You only need them if you are using Scala function literals
though, I managed to get rid of them using partially applied functions
for the other supported languages like xpath and jxpath a while ago.

You can find an xpath example in
http://svn.eu.apache.org/repos/asf/activemq/camel/trunk/components/camel-scala/src/test/scala/org/apache/camel/scala/dsl/SplitterRouteBuilderTest.scala 
and a jxpath one in
http://svn.eu.apache.org/repos/asf/activemq/camel/trunk/components/camel-scala/src/test/scala/org/apache/camel/scala/dsl/RecipientListRouteTest.scala

Regards,

Gert



James Strachan wrote:
> Its well worth looking at the Scala DSL which looks great; maybe they
> could both look and feel kinda similar? FWIW the only thing I don't
> like with the scala DSL is the use of _ which is a bit odd - but thats
> a scala thing
>  

Reply | Threaded
Open this post in threaded view
|

Re: camel spring xml restructure

jstrachan
2008/9/1 Gert Vanthienen <[hidden email]>:

> James,
>
> The _ is indeed a Scala specific construct: it represents a parameter in a
> function literal -- in our case this often represents the Camel Exchange.
>  You only need them if you are using Scala function literals though, I
> managed to get rid of them using partially applied functions for the other
> supported languages like xpath and jxpath a while ago.
> You can find an xpath example in
> http://svn.eu.apache.org/repos/asf/activemq/camel/trunk/components/camel-scala/src/test/scala/org/apache/camel/scala/dsl/SplitterRouteBuilderTest.scala
> and a jxpath one in
> http://svn.eu.apache.org/repos/asf/activemq/camel/trunk/components/camel-scala/src/test/scala/org/apache/camel/scala/dsl/RecipientListRouteTest.scala

Awesome stuff - I'd not realised you'd added neato language functions
too :) Nice!

--
James
-------
http://macstrac.blogspot.com/

Open Source Integration
http://open.iona.com
Reply | Threaded
Open this post in threaded view
|

Re: camel spring xml restructure

Vadim Chekan
In reply to this post by jstrachan
Thanks for your comments James,
It is good to know that BNF is supported in eclipse out of the box.
I'll try not to concentrate on the grammar too much and implement
from-to syntax. And when it works, we can discuss the final syntax.

Vadim.

James Strachan wrote:

> 2008/8/30 Vadim Chekan <[hidden email]>:
>> Claus,
>>
>> I'm glad you see the problem. Arguably, difficulties with routing
>> configurations is one of the the biggest obstacles on wider Camel adoption.
>> There is nothing more frustrating than knowing exactly what you want in
>> terms of EIP but spending many hours with so minor issue as syntax.
>>
>> Interesting enough, I was thinking about it yesterday too and my conclusion
>> is such that it is very difficult to achieve smooth configuration using xml.
>> It is not the tool for the job. I mean it can do it (it does it currently)
>> and we can keep improving it but will we get the easiest possible way of
>> configuring camel routes?
>> I'll skip large philosophical discussion but in my opinion xml sometimes
>> becomes victim of its own success and programmers try to stretch its
>> application beyond the reasons because "xml is so great, I would add it to
>> my salad if I could" :)
>>
>> Basically we are trying to define Abstract Syntax Tree in xml serialized
>> form. People don't usually think in terms of AST.
>
> Totally agree BTW. FWIW I'd always envisioned we'd have multiple DSLs;
> Java, XML, Groovy, Ruby, Scala and even a real DSL - hopefully all
> using the same AST underneath...
> http://activemq.apache.org/camel/maven/camel-core/apidocs/org/apache/camel/model/package-summary.html
>
> allowing folks to create routes using one language and save them in
> another DSL if folks wanted to etc.
>
>
>> We can give a whirl an idea of Domain Specific Language (DSL). I have some
>> under my belt and it is not too difficult. The result would look like this
>> (syntax can be any, but I'm trying to be java script alike):
>>
>> routes {
>>  route {
>>    from 'queue:Incoming'
>>    process {
>>      bean 'org.my.company.camel' 'MyFunction'
>>      aggregator()
>>    }
>>    if (header('myHeader') == 'value1') {
>>      to 'queue:Outgoing'
>>    } else {
>>      to 'queue:Alternative'
>>    }
>>  }
>> }
>
> Have you looked at the Scala DSL? Its pretty close to this already! :)
> http://activemq.apache.org/camel/scala-dsl.html
> http://activemq.apache.org/camel/scala-dsl-eip.html
>
>> I thought about juel (or others evaluators).
>> I see a problem that those languagas will dictate syntax which may be not
>> optimal for camel route definition domain.
>> Another and bigger problem is that external language will not detect camel
>> invalid constructions. For example creating serializer but not assigning it
>> data format would be invalid in camel but from Juel point of view it would
>> be perfectly ok.
>
> Yeah - we need to make sure we validate as much of the AST up front as
> we can along with giving great error messages.
>
>
>> Having our own syntax parser will let us detect any syntax deviations and
>> provide user with very clear and informative error messages for example "
>> 'bean' or 'aggregator' are expected inside in 'process' ".
>>
>> Another benefit we get is Backus-Naur Form (BNF) file which is human
>> readable and is very easy to use in documentation. As opposite to xml schema
>> which can be used for learning purposes but I would not risk claiming it a
>> recommended way.
>>
>> Here are fragments of BNF as I see it:
>> route :
>>        ROUTE
>>        | from_list process_list to_list
>>        ;
>> ...
>> process_list :
>>        /* can be empty*/
>>        | process
>>        | process process_list /* works as pipiline */
>>        ;
>>
>> process :
>>        /* can be empty */
>>        | BEAN LITERAL
>>        | MULTICAST '{ process_list '}'
>>        | PIPELINE '{' process_list '}'
>>        | SPLIT LITERAL
>>        | AGGREGATE expression
>>        | AGGREGATE
>>        | resequence
>>        ;
>>
>> resequence :
>>        BATCH RESEQUENCE
>>        | BATCH RESEQUENCE expression
>>        | BATCH RESEQUENCE NUMBER NUMBER
>>        | BATCH RESEQUENCE NUMBER NUMBER '(' expression_list ')'
>>        | STREAM RESEQUENCE resequence_options
>>        | STREAM RESEQUENCE resequence_options '(' expression ')'
>>        ;
>>
>> to_list :
>>        to
>>        | to to_list
>>        ;
>>
>> to :
>>        LITERAL
>>        | IF '(' bool_expression ')' THEN to
>>        | IF '(' bool_expression ')' THEN to ELSE to
>>        ;
>>
>> If you guys think it worth a try I can mock up a prototype.
>> I think I would build an AST and then have another process which would
>> traverse AST and invoke RouteBuilder to build the actual route.
>> This way we preserve backward compatibility with existing ways of building
>> routes.
>>
>> This route builder would be a separate plugin in order to be as little
>> invasive as possible.
>>
>> AST can be used in future to build graph representation of camel routes
>> without building the actual routes. I recall James recent comment that
>> building route for display purpose only (without running it) added some
>> complexity.
>>
>> Let me know what do you think folks.
>
> I've always fancied a real DSL as well :). As you mention - the
> hardest part of any real DSL is what language is used for
> predicates/expressions. e.g. do we let any language be used...
>
> xpath( /foo/bar )
>
> or something?
>
> But it'd be great to try make one. Incidentally if you do fancy
> experimenting with this; have a look at the xtext library in eclipse.
> I created a little prototype project here...
>
> https://svn.apache.org/repos/asf/activemq/camel/ide/camel-eclipse/
>
> which attempts to take an Antlr grammar for the DSL and auto-create
> both a parser but also an eclipse based editor for the language. It
> takes an Antlr grammer here...
>
> https://svn.apache.org/repos/asf/activemq/camel/ide/camel-eclipse/camel.routing/src/camel.xtxt
>
> which as you can see is kinda basic :)
>
> Having both a nice textual DSL along with an IDE to edit it would rock :)
>
> Its well worth looking at the Scala DSL which looks great; maybe they
> could both look and feel kinda similar? FWIW the only thing I don't
> like with the scala DSL is the use of _ which is a bit odd - but thats
> a scala thing
>

Reply | Threaded
Open this post in threaded view
|

Re: camel spring xml restructure

jstrachan
2008/9/2 Vadim Chekan <[hidden email]>:
> Thanks for your comments James,
> It is good to know that BNF is supported in eclipse out of the box.
> I'll try not to concentrate on the grammar too much and implement from-to
> syntax. And when it works, we can discuss the final syntax.

Great stuff! :)
--
James
-------
http://macstrac.blogspot.com/

Open Source Integration
http://open.iona.com
Reply | Threaded
Open this post in threaded view
|

Re: camel spring xml restructure

hadrian
In reply to this post by Vadim Chekan
Hi Vadim,

This has been on my radar (on and off) for almost a year now.  I still  
don't have a good solution.

We definitely want code completion in the ui, and we'd like the  
language to be extensible.  One of the biggest issue is that once you  
add support for a new pattern, it has to go in ProcessorType, which is  
why it grew like this.  It is quite easy now for a component to add  
support for new TypeConverters, Components, even Expression language,  
but not for DSL.  Ideally the DSL would be pluggable in a non  
intrusive way.

The biggest issue, I think is that we have to define the dsl in terms  
of not yet defined structures (actually I think there's exactly one:  
the pipeline, which in your example is called process).  That is,  
whatever the pipeline is a choice would be:

when_clause:
        WHEN pipeline;

otherwise_clause:
        OTHERWISE pipeline;

choice_pattern:
        (when_clause) +
        [otherwise_clause;

pattern:
        ...
        | choice_pattern
        ...
        /* should be extensible */

pipeline:
        (pattern)+


Well, there is some recursive definition there.  One should be able to  
define a new pattern in a separate BNF description, but still be added  
to the language, even if the price for this would be to rebuild the  
whole language again (which i don't see yet how one could avoid).

Lastly, one thing that would be nice to solve and is a bit messed up  
now, is not supporting constructs like:
from().choice().when().when() (notice the second when)
from
().choice().when().pipeline().otherwise().pipeline().when().pipeline()  
(notice a when after otherwise)
from
().choice
().when
().choice().when().pipeline().otherwise().pipeline().when().pipeline()  
(but this would be legal, as the last when would belong to the first  
choice)

Things could get pretty complicated because java is not a dynamic  
language, but I feel is should be possible.  Even in one programming  
language we could/should be able to support multiple syntaxes.  For  
instance a good subset of bpel could be supported, not just spring  
xml.  BAM does a bit of that, but it's built on top of the existing  
camel dsl, it does not add to it.

Cheers
Hadrian



>>> process_list :
>>>       /* can be empty*/
>>>       | process
>>>       | process process_list /* works as pipiline */
>>>       ;
>>



On Sep 2, 2008, at 4:24 AM, Vadim Chekan wrote:

> Thanks for your comments James,
> It is good to know that BNF is supported in eclipse out of the box.
> I'll try not to concentrate on the grammar too much and implement  
> from-to syntax. And when it works, we can discuss the final syntax.
>
> Vadim.
>
> James Strachan wrote:
>> 2008/8/30 Vadim Chekan <[hidden email]>:
>>> Claus,
>>>
>>> I'm glad you see the problem. Arguably, difficulties with routing
>>> configurations is one of the the biggest obstacles on wider Camel  
>>> adoption.
>>> There is nothing more frustrating than knowing exactly what you  
>>> want in
>>> terms of EIP but spending many hours with so minor issue as syntax.
>>>
>>> Interesting enough, I was thinking about it yesterday too and my  
>>> conclusion
>>> is such that it is very difficult to achieve smooth configuration  
>>> using xml.
>>> It is not the tool for the job. I mean it can do it (it does it  
>>> currently)
>>> and we can keep improving it but will we get the easiest possible  
>>> way of
>>> configuring camel routes?
>>> I'll skip large philosophical discussion but in my opinion xml  
>>> sometimes
>>> becomes victim of its own success and programmers try to stretch its
>>> application beyond the reasons because "xml is so great, I would  
>>> add it to
>>> my salad if I could" :)
>>>
>>> Basically we are trying to define Abstract Syntax Tree in xml  
>>> serialized
>>> form. People don't usually think in terms of AST.
>> Totally agree BTW. FWIW I'd always envisioned we'd have multiple  
>> DSLs;
>> Java, XML, Groovy, Ruby, Scala and even a real DSL - hopefully all
>> using the same AST underneath...
>> http://activemq.apache.org/camel/maven/camel-core/apidocs/org/apache/camel/model/package-summary.html
>> allowing folks to create routes using one language and save them in
>> another DSL if folks wanted to etc.
>>> We can give a whirl an idea of Domain Specific Language (DSL). I  
>>> have some
>>> under my belt and it is not too difficult. The result would look  
>>> like this
>>> (syntax can be any, but I'm trying to be java script alike):
>>>
>>> routes {
>>> route {
>>>   from 'queue:Incoming'
>>>   process {
>>>     bean 'org.my.company.camel' 'MyFunction'
>>>     aggregator()
>>>   }
>>>   if (header('myHeader') == 'value1') {
>>>     to 'queue:Outgoing'
>>>   } else {
>>>     to 'queue:Alternative'
>>>   }
>>> }
>>> }
>> Have you looked at the Scala DSL? Its pretty close to this  
>> already! :)
>> http://activemq.apache.org/camel/scala-dsl.html
>> http://activemq.apache.org/camel/scala-dsl-eip.html
>>> I thought about juel (or others evaluators).
>>> I see a problem that those languagas will dictate syntax which may  
>>> be not
>>> optimal for camel route definition domain.
>>> Another and bigger problem is that external language will not  
>>> detect camel
>>> invalid constructions. For example creating serializer but not  
>>> assigning it
>>> data format would be invalid in camel but from Juel point of view  
>>> it would
>>> be perfectly ok.
>> Yeah - we need to make sure we validate as much of the AST up front  
>> as
>> we can along with giving great error messages.
>>> Having our own syntax parser will let us detect any syntax  
>>> deviations and
>>> provide user with very clear and informative error messages for  
>>> example "
>>> 'bean' or 'aggregator' are expected inside in 'process' ".
>>>
>>> Another benefit we get is Backus-Naur Form (BNF) file which is human
>>> readable and is very easy to use in documentation. As opposite to  
>>> xml schema
>>> which can be used for learning purposes but I would not risk  
>>> claiming it a
>>> recommended way.
>>>
>>> Here are fragments of BNF as I see it:
>>> route :
>>>       ROUTE
>>>       | from_list process_list to_list
>>>       ;
>>> ...
>>> process_list :
>>>       /* can be empty*/
>>>       | process
>>>       | process process_list /* works as pipiline */
>>>       ;
>>>
>>> process :
>>>       /* can be empty */
>>>       | BEAN LITERAL
>>>       | MULTICAST '{ process_list '}'
>>>       | PIPELINE '{' process_list '}'
>>>       | SPLIT LITERAL
>>>       | AGGREGATE expression
>>>       | AGGREGATE
>>>       | resequence
>>>       ;
>>>
>>> resequence :
>>>       BATCH RESEQUENCE
>>>       | BATCH RESEQUENCE expression
>>>       | BATCH RESEQUENCE NUMBER NUMBER
>>>       | BATCH RESEQUENCE NUMBER NUMBER '(' expression_list ')'
>>>       | STREAM RESEQUENCE resequence_options
>>>       | STREAM RESEQUENCE resequence_options '(' expression ')'
>>>       ;
>>>
>>> to_list :
>>>       to
>>>       | to to_list
>>>       ;
>>>
>>> to :
>>>       LITERAL
>>>       | IF '(' bool_expression ')' THEN to
>>>       | IF '(' bool_expression ')' THEN to ELSE to
>>>       ;
>>>
>>> If you guys think it worth a try I can mock up a prototype.
>>> I think I would build an AST and then have another process which  
>>> would
>>> traverse AST and invoke RouteBuilder to build the actual route.
>>> This way we preserve backward compatibility with existing ways of  
>>> building
>>> routes.
>>>
>>> This route builder would be a separate plugin in order to be as  
>>> little
>>> invasive as possible.
>>>
>>> AST can be used in future to build graph representation of camel  
>>> routes
>>> without building the actual routes. I recall James recent comment  
>>> that
>>> building route for display purpose only (without running it) added  
>>> some
>>> complexity.
>>>
>>> Let me know what do you think folks.
>> I've always fancied a real DSL as well :). As you mention - the
>> hardest part of any real DSL is what language is used for
>> predicates/expressions. e.g. do we let any language be used...
>> xpath( /foo/bar )
>> or something?
>> But it'd be great to try make one. Incidentally if you do fancy
>> experimenting with this; have a look at the xtext library in eclipse.
>> I created a little prototype project here...
>> https://svn.apache.org/repos/asf/activemq/camel/ide/camel-eclipse/
>> which attempts to take an Antlr grammar for the DSL and auto-create
>> both a parser but also an eclipse based editor for the language. It
>> takes an Antlr grammer here...
>> https://svn.apache.org/repos/asf/activemq/camel/ide/camel-eclipse/camel.routing/src/camel.xtxt
>> which as you can see is kinda basic :)
>> Having both a nice textual DSL along with an IDE to edit it would  
>> rock :)
>> Its well worth looking at the Scala DSL which looks great; maybe they
>> could both look and feel kinda similar? FWIW the only thing I don't
>> like with the scala DSL is the use of _ which is a bit odd - but  
>> thats
>> a scala thing
>

Reply | Threaded
Open this post in threaded view
|

Re: camel spring xml restructure

Vadim Chekan
Here is my plan:

+ Write route from-to parser
++ Implement AST generation
~ invoke parser from camel
. Inject AST into camel "AST/builder"
. Implement syntax error reporting
. Implement comprehensive grammar
. Integrate full grammar into camel
. Test suite
. Documentation.
. Eclipse support.
. Commit & release

At the moment I implemented camel integration as:
   <camelContext id="camel"
xmlns="http://activemq.apache.org/camel/schema/spring">
    <routeBuilderRef id="routeBuilder" ref="dsl"/>
   </camelContext>

   <bean id="dsl" class="org.mycompany.test.Main">
    <property name="DSLRoute">
                <value>
    route {
    from 'a'
    to 'b'
    }
    </value>
    </property>
   </bean>

It is good enough for prototype purposes. But I'd be glad to hear back
what the real solution should look like.

I think we are fine in terms of extensibility of the DSL. Majority of
configurations will use existing patterns. We can keep DSL open ended by
enabling something like "bean" operator which can instantiate a class,
using provided class name, or spring ref. That would allow using any non
standard types of filters, etc. Moreover. depending on the language
context this "bean" operator can expect the class to implement different
interfaces. For example "bean" keyword in "to" clause can cast the
objects as IEndpoint while "bean" declared in pipeline would cast
parameter as IProces.

Vadim.

Hadrian Zbarcea wrote:

> Hi Vadim,
>
> This has been on my radar (on and off) for almost a year now.  I still
> don't have a good solution.
>
> We definitely want code completion in the ui, and we'd like the language
> to be extensible.  One of the biggest issue is that once you add support
> for a new pattern, it has to go in ProcessorType, which is why it grew
> like this.  It is quite easy now for a component to add support for new
> TypeConverters, Components, even Expression language, but not for DSL.  
> Ideally the DSL would be pluggable in a non intrusive way.
>
> The biggest issue, I think is that we have to define the dsl in terms of
> not yet defined structures (actually I think there's exactly one: the
> pipeline, which in your example is called process).  That is, whatever
> the pipeline is a choice would be:
>
> when_clause:
>     WHEN pipeline;
>
> otherwise_clause:
>     OTHERWISE pipeline;
>
> choice_pattern:
>     (when_clause) +
>     [otherwise_clause;
>
> pattern:
>     ...
>     | choice_pattern
>     ...
>     /* should be extensible */
>
> pipeline:
>     (pattern)+
>
>
> Well, there is some recursive definition there.  One should be able to
> define a new pattern in a separate BNF description, but still be added
> to the language, even if the price for this would be to rebuild the
> whole language again (which i don't see yet how one could avoid).
>
> Lastly, one thing that would be nice to solve and is a bit messed up
> now, is not supporting constructs like:
> from().choice().when().when() (notice the second when)
> from().choice().when().pipeline().otherwise().pipeline().when().pipeline()
> (notice a when after otherwise)
> from().choice().when().choice().when().pipeline().otherwise().pipeline().when().pipeline()
> (but this would be legal, as the last when would belong to the first
> choice)
>
> Things could get pretty complicated because java is not a dynamic
> language, but I feel is should be possible.  Even in one programming
> language we could/should be able to support multiple syntaxes.  For
> instance a good subset of bpel could be supported, not just spring xml.  
> BAM does a bit of that, but it's built on top of the existing camel dsl,
> it does not add to it.
>
> Cheers
> Hadrian  
>
>
>
>>>> process_list :
>>>>       /* can be empty*/
>>>>       | process
>>>>       | process process_list /* works as pipiline */
>>>>       ;
>>>
>
>
>
> On Sep 2, 2008, at 4:24 AM, Vadim Chekan wrote:
>
>> Thanks for your comments James,
>> It is good to know that BNF is supported in eclipse out of the box.
>> I'll try not to concentrate on the grammar too much and implement
>> from-to syntax. And when it works, we can discuss the final syntax.
>>
>> Vadim.
>>
>> James Strachan wrote:
>>> 2008/8/30 Vadim Chekan <[hidden email]>:
>>>> Claus,
>>>>
>>>> I'm glad you see the problem. Arguably, difficulties with routing
>>>> configurations is one of the the biggest obstacles on wider Camel
>>>> adoption.
>>>> There is nothing more frustrating than knowing exactly what you want in
>>>> terms of EIP but spending many hours with so minor issue as syntax.
>>>>
>>>> Interesting enough, I was thinking about it yesterday too and my
>>>> conclusion
>>>> is such that it is very difficult to achieve smooth configuration
>>>> using xml.
>>>> It is not the tool for the job. I mean it can do it (it does it
>>>> currently)
>>>> and we can keep improving it but will we get the easiest possible
>>>> way of
>>>> configuring camel routes?
>>>> I'll skip large philosophical discussion but in my opinion xml
>>>> sometimes
>>>> becomes victim of its own success and programmers try to stretch its
>>>> application beyond the reasons because "xml is so great, I would add
>>>> it to
>>>> my salad if I could" :)
>>>>
>>>> Basically we are trying to define Abstract Syntax Tree in xml
>>>> serialized
>>>> form. People don't usually think in terms of AST.
>>> Totally agree BTW. FWIW I'd always envisioned we'd have multiple DSLs;
>>> Java, XML, Groovy, Ruby, Scala and even a real DSL - hopefully all
>>> using the same AST underneath...
>>> http://activemq.apache.org/camel/maven/camel-core/apidocs/org/apache/camel/model/package-summary.html 
>>>
>>> allowing folks to create routes using one language and save them in
>>> another DSL if folks wanted to etc.
>>>> We can give a whirl an idea of Domain Specific Language (DSL). I
>>>> have some
>>>> under my belt and it is not too difficult. The result would look
>>>> like this
>>>> (syntax can be any, but I'm trying to be java script alike):
>>>>
>>>> routes {
>>>> route {
>>>>   from 'queue:Incoming'
>>>>   process {
>>>>     bean 'org.my.company.camel' 'MyFunction'
>>>>     aggregator()
>>>>   }
>>>>   if (header('myHeader') == 'value1') {
>>>>     to 'queue:Outgoing'
>>>>   } else {
>>>>     to 'queue:Alternative'
>>>>   }
>>>> }
>>>> }
>>> Have you looked at the Scala DSL? Its pretty close to this already! :)
>>> http://activemq.apache.org/camel/scala-dsl.html
>>> http://activemq.apache.org/camel/scala-dsl-eip.html
>>>> I thought about juel (or others evaluators).
>>>> I see a problem that those languagas will dictate syntax which may
>>>> be not
>>>> optimal for camel route definition domain.
>>>> Another and bigger problem is that external language will not detect
>>>> camel
>>>> invalid constructions. For example creating serializer but not
>>>> assigning it
>>>> data format would be invalid in camel but from Juel point of view it
>>>> would
>>>> be perfectly ok.
>>> Yeah - we need to make sure we validate as much of the AST up front as
>>> we can along with giving great error messages.
>>>> Having our own syntax parser will let us detect any syntax
>>>> deviations and
>>>> provide user with very clear and informative error messages for
>>>> example "
>>>> 'bean' or 'aggregator' are expected inside in 'process' ".
>>>>
>>>> Another benefit we get is Backus-Naur Form (BNF) file which is human
>>>> readable and is very easy to use in documentation. As opposite to
>>>> xml schema
>>>> which can be used for learning purposes but I would not risk
>>>> claiming it a
>>>> recommended way.
>>>>
>>>> Here are fragments of BNF as I see it:
>>>> route :
>>>>       ROUTE
>>>>       | from_list process_list to_list
>>>>       ;
>>>> ...
>>>> process_list :
>>>>       /* can be empty*/
>>>>       | process
>>>>       | process process_list /* works as pipiline */
>>>>       ;
>>>>
>>>> process :
>>>>       /* can be empty */
>>>>       | BEAN LITERAL
>>>>       | MULTICAST '{ process_list '}'
>>>>       | PIPELINE '{' process_list '}'
>>>>       | SPLIT LITERAL
>>>>       | AGGREGATE expression
>>>>       | AGGREGATE
>>>>       | resequence
>>>>       ;
>>>>
>>>> resequence :
>>>>       BATCH RESEQUENCE
>>>>       | BATCH RESEQUENCE expression
>>>>       | BATCH RESEQUENCE NUMBER NUMBER
>>>>       | BATCH RESEQUENCE NUMBER NUMBER '(' expression_list ')'
>>>>       | STREAM RESEQUENCE resequence_options
>>>>       | STREAM RESEQUENCE resequence_options '(' expression ')'
>>>>       ;
>>>>
>>>> to_list :
>>>>       to
>>>>       | to to_list
>>>>       ;
>>>>
>>>> to :
>>>>       LITERAL
>>>>       | IF '(' bool_expression ')' THEN to
>>>>       | IF '(' bool_expression ')' THEN to ELSE to
>>>>       ;
>>>>
>>>> If you guys think it worth a try I can mock up a prototype.
>>>> I think I would build an AST and then have another process which would
>>>> traverse AST and invoke RouteBuilder to build the actual route.
>>>> This way we preserve backward compatibility with existing ways of
>>>> building
>>>> routes.
>>>>
>>>> This route builder would be a separate plugin in order to be as little
>>>> invasive as possible.
>>>>
>>>> AST can be used in future to build graph representation of camel routes
>>>> without building the actual routes. I recall James recent comment that
>>>> building route for display purpose only (without running it) added some
>>>> complexity.
>>>>
>>>> Let me know what do you think folks.
>>> I've always fancied a real DSL as well :). As you mention - the
>>> hardest part of any real DSL is what language is used for
>>> predicates/expressions. e.g. do we let any language be used...
>>> xpath( /foo/bar )
>>> or something?
>>> But it'd be great to try make one. Incidentally if you do fancy
>>> experimenting with this; have a look at the xtext library in eclipse.
>>> I created a little prototype project here...
>>> https://svn.apache.org/repos/asf/activemq/camel/ide/camel-eclipse/
>>> which attempts to take an Antlr grammar for the DSL and auto-create
>>> both a parser but also an eclipse based editor for the language. It
>>> takes an Antlr grammer here...
>>> https://svn.apache.org/repos/asf/activemq/camel/ide/camel-eclipse/camel.routing/src/camel.xtxt 
>>>
>>> which as you can see is kinda basic :)
>>> Having both a nice textual DSL along with an IDE to edit it would
>>> rock :)
>>> Its well worth looking at the Scala DSL which looks great; maybe they
>>> could both look and feel kinda similar? FWIW the only thing I don't
>>> like with the scala DSL is the use of _ which is a bit odd - but thats
>>> a scala thing
>>
>
>