[DISCUSS] Meta model

classic Classic list List threaded Threaded
14 messages Options
Reply | Threaded
Open this post in threaded view
|

[DISCUSS] Meta model

gnodet
Hey everyone !

The last weeks, I've spend quite some time working on a camel metamodel.
The idea is to invert the way things are built in camel so instead of
generating the metamodel from the classes, the metamodel would be
maintained manually and used to generate a bunch of things.

This would bring the following benefits:
  - the metamodel would necessarily be up to date
  - generate the model classes (XyzDefinition classes) with an homogeneous
fluent api (similar to the new endpoint DSL)
  - get rid of some round tripping between java -> json -> java, copying
json files everywhere , so great simplification of the build process
  - the DSL allows type-safe + property placeholders / references everywhere
  - generate xml readers / parsers without relying on JAXB
  - brings extensibility of the DSL as it would be much easier to create
DSLs based on other languages from the metamodel (yaml for example)

I've pushed a branch that shows my experiments.  It's not fully working
yet, but it gives a good idea. It's available at
  https://github.com/gnodet/camel/tree/model
I'm progressing a bit slowly as there are lots of step by step adjustements
to do in order to keep compatibility of the DSL.

Currently the model is generated from the json files, but the idea is to
reverse this process and have the metamodel the real primary input for all
generation.
The model currently looks like:
  https://gist.github.com/gnodet/75457febcca9a893c3c1a2b8499189b2

The current JAXB model will need to be moved into a separate module so that
it can be kept for compatibility without interfering with the new generated
java DSL.

So, still quite some work left, but I wanted to bring it to the community
sooner rather than later, especially before I go on PTO for a few weeks
where I'll be mostly offline.
Happy to discuss anything or provide more infos.

Cheers,
Guillaume Nodet
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] Meta model

lburgazzoli
I Guillaume,

I think it is a little bit hard to figure out what's happening looking at
the code changes as it is quite a huge list of changed file :)
Maybe having some concrete examples or an examples of what things will look
like could help to have a better understanding and giove more options to
suggest enhancements or raising concerns.


---
Luca Burgazzoli


On Wed, Jul 24, 2019 at 4:45 PM Guillaume Nodet <[hidden email]> wrote:

> Hey everyone !
>
> The last weeks, I've spend quite some time working on a camel metamodel.
> The idea is to invert the way things are built in camel so instead of
> generating the metamodel from the classes, the metamodel would be
> maintained manually and used to generate a bunch of things.
>
> This would bring the following benefits:
>   - the metamodel would necessarily be up to date
>   - generate the model classes (XyzDefinition classes) with an homogeneous
> fluent api (similar to the new endpoint DSL)
>   - get rid of some round tripping between java -> json -> java, copying
> json files everywhere , so great simplification of the build process
>   - the DSL allows type-safe + property placeholders / references
> everywhere
>   - generate xml readers / parsers without relying on JAXB
>   - brings extensibility of the DSL as it would be much easier to create
> DSLs based on other languages from the metamodel (yaml for example)
>
> I've pushed a branch that shows my experiments.  It's not fully working
> yet, but it gives a good idea. It's available at
>   https://github.com/gnodet/camel/tree/model
> I'm progressing a bit slowly as there are lots of step by step adjustements
> to do in order to keep compatibility of the DSL.
>
> Currently the model is generated from the json files, but the idea is to
> reverse this process and have the metamodel the real primary input for all
> generation.
> The model currently looks like:
>   https://gist.github.com/gnodet/75457febcca9a893c3c1a2b8499189b2
>
> The current JAXB model will need to be moved into a separate module so that
> it can be kept for compatibility without interfering with the new generated
> java DSL.
>
> So, still quite some work left, but I wanted to bring it to the community
> sooner rather than later, especially before I go on PTO for a few weeks
> where I'll be mostly offline.
> Happy to discuss anything or provide more infos.
>
> Cheers,
> Guillaume Nodet
>
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] Meta model

Andrea Cosentino-3
Yeah, a concrete example of what has been done could be useful to better
understand

Il mer 24 lug 2019, 17:11 Luca Burgazzoli <[hidden email]> ha
scritto:

> I Guillaume,
>
> I think it is a little bit hard to figure out what's happening looking at
> the code changes as it is quite a huge list of changed file :)
> Maybe having some concrete examples or an examples of what things will look
> like could help to have a better understanding and giove more options to
> suggest enhancements or raising concerns.
>
>
> ---
> Luca Burgazzoli
>
>
> On Wed, Jul 24, 2019 at 4:45 PM Guillaume Nodet <[hidden email]> wrote:
>
> > Hey everyone !
> >
> > The last weeks, I've spend quite some time working on a camel metamodel.
> > The idea is to invert the way things are built in camel so instead of
> > generating the metamodel from the classes, the metamodel would be
> > maintained manually and used to generate a bunch of things.
> >
> > This would bring the following benefits:
> >   - the metamodel would necessarily be up to date
> >   - generate the model classes (XyzDefinition classes) with an
> homogeneous
> > fluent api (similar to the new endpoint DSL)
> >   - get rid of some round tripping between java -> json -> java, copying
> > json files everywhere , so great simplification of the build process
> >   - the DSL allows type-safe + property placeholders / references
> > everywhere
> >   - generate xml readers / parsers without relying on JAXB
> >   - brings extensibility of the DSL as it would be much easier to create
> > DSLs based on other languages from the metamodel (yaml for example)
> >
> > I've pushed a branch that shows my experiments.  It's not fully working
> > yet, but it gives a good idea. It's available at
> >   https://github.com/gnodet/camel/tree/model
> > I'm progressing a bit slowly as there are lots of step by step
> adjustements
> > to do in order to keep compatibility of the DSL.
> >
> > Currently the model is generated from the json files, but the idea is to
> > reverse this process and have the metamodel the real primary input for
> all
> > generation.
> > The model currently looks like:
> >   https://gist.github.com/gnodet/75457febcca9a893c3c1a2b8499189b2
> >
> > The current JAXB model will need to be moved into a separate module so
> that
> > it can be kept for compatibility without interfering with the new
> generated
> > java DSL.
> >
> > So, still quite some work left, but I wanted to bring it to the community
> > sooner rather than later, especially before I go on PTO for a few weeks
> > where I'll be mostly offline.
> > Happy to discuss anything or provide more infos.
> >
> > Cheers,
> > Guillaume Nodet
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] Meta model

Onder SEZGIN
Hi,

yes some examples to describe these new features would be great.
There are tones of changes which is a bit hard to follow at this stage (at
least for me)


On Wed, Jul 24, 2019 at 4:17 PM Andrea Cosentino <[hidden email]> wrote:

> Yeah, a concrete example of what has been done could be useful to better
> understand
>
> Il mer 24 lug 2019, 17:11 Luca Burgazzoli <[hidden email]> ha
> scritto:
>
> > I Guillaume,
> >
> > I think it is a little bit hard to figure out what's happening looking at
> > the code changes as it is quite a huge list of changed file :)
> > Maybe having some concrete examples or an examples of what things will
> look
> > like could help to have a better understanding and giove more options to
> > suggest enhancements or raising concerns.
> >
> >
> > ---
> > Luca Burgazzoli
> >
> >
> > On Wed, Jul 24, 2019 at 4:45 PM Guillaume Nodet <[hidden email]>
> wrote:
> >
> > > Hey everyone !
> > >
> > > The last weeks, I've spend quite some time working on a camel
> metamodel.
> > > The idea is to invert the way things are built in camel so instead of
> > > generating the metamodel from the classes, the metamodel would be
> > > maintained manually and used to generate a bunch of things.
> > >
> > > This would bring the following benefits:
> > >   - the metamodel would necessarily be up to date
> > >   - generate the model classes (XyzDefinition classes) with an
> > homogeneous
> > > fluent api (similar to the new endpoint DSL)
> > >   - get rid of some round tripping between java -> json -> java,
> copying
> > > json files everywhere , so great simplification of the build process
> > >   - the DSL allows type-safe + property placeholders / references
> > > everywhere
> > >   - generate xml readers / parsers without relying on JAXB
> > >   - brings extensibility of the DSL as it would be much easier to
> create
> > > DSLs based on other languages from the metamodel (yaml for example)
> > >
> > > I've pushed a branch that shows my experiments.  It's not fully working
> > > yet, but it gives a good idea. It's available at
> > >   https://github.com/gnodet/camel/tree/model
> > > I'm progressing a bit slowly as there are lots of step by step
> > adjustements
> > > to do in order to keep compatibility of the DSL.
> > >
> > > Currently the model is generated from the json files, but the idea is
> to
> > > reverse this process and have the metamodel the real primary input for
> > all
> > > generation.
> > > The model currently looks like:
> > >   https://gist.github.com/gnodet/75457febcca9a893c3c1a2b8499189b2
> > >
> > > The current JAXB model will need to be moved into a separate module so
> > that
> > > it can be kept for compatibility without interfering with the new
> > generated
> > > java DSL.
> > >
> > > So, still quite some work left, but I wanted to bring it to the
> community
> > > sooner rather than later, especially before I go on PTO for a few weeks
> > > where I'll be mostly offline.
> > > Happy to discuss anything or provide more infos.
> > >
> > > Cheers,
> > > Guillaume Nodet
> > >
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] Meta model

Claus Ibsen-2
In reply to this post by gnodet
Hi

Good to see more experiments, but as others have said in this mail
thread, its too overwhelming to dive into and understand.

So what I can see is that the model is now more "coded" in velocity vm
and xml xslt files, than what we had before with the java model
classes with JAXB annotations.
Also I fail to see how it would understand if we add a new EIP today -
how would you go about doing that?

I do like that if we can have a lighter XML parser that is stax based than JAXB.
And also if we can generate the XML XSD without JAXB at all.

And the meta-model that you link to is a XML file which seems like an
aggregation of what we have today in camel-catalog in the various json
files.

As you are going on PTO for a while I think we should maybe keep this
in mind that this work will not get completed or finished in the near
future.
We may consider getting the last other stuff done and get M5 out the
door as the last milestone, and then close down on RC builds.

Then this meta-model can maybe be introduced in "steps" for 3.1, 3.2
etc. to give it more time to be more stable and maintainable.



On Wed, Jul 24, 2019 at 4:45 PM Guillaume Nodet <[hidden email]> wrote:

>
> Hey everyone !
>
> The last weeks, I've spend quite some time working on a camel metamodel.
> The idea is to invert the way things are built in camel so instead of
> generating the metamodel from the classes, the metamodel would be
> maintained manually and used to generate a bunch of things.
>
> This would bring the following benefits:
>   - the metamodel would necessarily be up to date
>   - generate the model classes (XyzDefinition classes) with an homogeneous
> fluent api (similar to the new endpoint DSL)
>   - get rid of some round tripping between java -> json -> java, copying
> json files everywhere , so great simplification of the build process
>   - the DSL allows type-safe + property placeholders / references everywhere
>   - generate xml readers / parsers without relying on JAXB
>   - brings extensibility of the DSL as it would be much easier to create
> DSLs based on other languages from the metamodel (yaml for example)
>
> I've pushed a branch that shows my experiments.  It's not fully working
> yet, but it gives a good idea. It's available at
>   https://github.com/gnodet/camel/tree/model
> I'm progressing a bit slowly as there are lots of step by step adjustements
> to do in order to keep compatibility of the DSL.
>
> Currently the model is generated from the json files, but the idea is to
> reverse this process and have the metamodel the real primary input for all
> generation.
> The model currently looks like:
>   https://gist.github.com/gnodet/75457febcca9a893c3c1a2b8499189b2
>
> The current JAXB model will need to be moved into a separate module so that
> it can be kept for compatibility without interfering with the new generated
> java DSL.
>
> So, still quite some work left, but I wanted to bring it to the community
> sooner rather than later, especially before I go on PTO for a few weeks
> where I'll be mostly offline.
> Happy to discuss anything or provide more infos.
>
> Cheers,
> Guillaume Nodet



--
Claus Ibsen
-----------------
http://davsclaus.com @davsclaus
Camel in Action 2: https://www.manning.com/ibsen2
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] Meta model

Zoran Regvart-2
In reply to this post by gnodet
Hi Guillaume & Cameleers,
I like that there's an effort to make the code more maintainable. I do
however feel that the source of truth is best kept in the Java code,
and perhaps this new model can be generated from that. I understand
the need for having a single strictly typed source of truth.

Could we have the flow somehow like Java -> (XML?) meta model -> code
generation of XML readers/writers and DSL?

I have an inkling that we could even generate the meta model in Java
and code generate from there without the model being in XML format, we
are kinda, while awkwardly, doing it right now.

I fear that unless we add more strict checking that if the XML meta
model is the source of truth that we'll easily diverge from the Java
code. For example, if we remove a property from the endpoint we need
tooling that'll fail the build if the meta model is not updated and
the property is removed there as well.

zoran

On Wed, Jul 24, 2019 at 4:45 PM Guillaume Nodet <[hidden email]> wrote:

>
> Hey everyone !
>
> The last weeks, I've spend quite some time working on a camel metamodel.
> The idea is to invert the way things are built in camel so instead of
> generating the metamodel from the classes, the metamodel would be
> maintained manually and used to generate a bunch of things.
>
> This would bring the following benefits:
>   - the metamodel would necessarily be up to date
>   - generate the model classes (XyzDefinition classes) with an homogeneous
> fluent api (similar to the new endpoint DSL)
>   - get rid of some round tripping between java -> json -> java, copying
> json files everywhere , so great simplification of the build process
>   - the DSL allows type-safe + property placeholders / references everywhere
>   - generate xml readers / parsers without relying on JAXB
>   - brings extensibility of the DSL as it would be much easier to create
> DSLs based on other languages from the metamodel (yaml for example)
>
> I've pushed a branch that shows my experiments.  It's not fully working
> yet, but it gives a good idea. It's available at
>   https://github.com/gnodet/camel/tree/model
> I'm progressing a bit slowly as there are lots of step by step adjustements
> to do in order to keep compatibility of the DSL.
>
> Currently the model is generated from the json files, but the idea is to
> reverse this process and have the metamodel the real primary input for all
> generation.
> The model currently looks like:
>   https://gist.github.com/gnodet/75457febcca9a893c3c1a2b8499189b2
>
> The current JAXB model will need to be moved into a separate module so that
> it can be kept for compatibility without interfering with the new generated
> java DSL.
>
> So, still quite some work left, but I wanted to bring it to the community
> sooner rather than later, especially before I go on PTO for a few weeks
> where I'll be mostly offline.
> Happy to discuss anything or provide more infos.
>
> Cheers,
> Guillaume Nodet



--
Zoran Regvart
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] Meta model

gnodet
In reply to this post by Claus Ibsen-2
Le jeu. 25 juil. 2019 à 10:47, Claus Ibsen <[hidden email]> a écrit :

> Hi
>
> Good to see more experiments, but as others have said in this mail
> thread, its too overwhelming to dive into and understand.
>

Yes, I definitely get that, especially as it's still work in progress.

On the java dsl side, the goal is to be 100% compatible, i.e. i'm trying to
generate the DSL without changing all the tests in camel-core.
However, this opens new possibilities, mainly about having a consistent way
to inject references and use property placeholders, especially in places
where this was not possible.



> So what I can see is that the model is now more "coded" in velocity vm
> and xml xslt files, than what we had before with the java model
> classes with JAXB annotations.



> Also I fail to see how it would understand if we add a new EIP today -
> how would you go about doing that?
>

So if the xml metamodel becomes the real input (see below), then the
developer would have to modify the metamodel to add an element to it (a
processor, language, dataformat, or maybe a property on an existing
element).
You'd build the camel-metamodel, then build camel-core which re-generates
all the data structures. At this point, the property will be added to the
java code, so the next step is to actually create or modify the reifier and
the runtime bits to actually use it.


>
> I do like that if we can have a lighter XML parser that is stax based than
> JAXB.
> And also if we can generate the XML XSD without JAXB at all.
>

Yes.


> And the meta-model that you link to is a XML file which seems like an
> aggregation of what we have today in camel-catalog in the various json
> files.
>

Yes, though this is temporary in my mind.   I think the main input should
be the xml metamodel which would be "manually" maintained.  From this file,
we should generate everything that we need, including the json files for
the catalog.


> As you are going on PTO for a while I think we should maybe keep this
> in mind that this work will not get completed or finished in the near
> future.
> We may consider getting the last other stuff done and get M5 out the
> door as the last milestone, and then close down on RC builds.
>
> Then this meta-model can maybe be introduced in "steps" for 3.1, 3.2
> etc. to give it more time to be more stable and maintainable.
>
>
>
> On Wed, Jul 24, 2019 at 4:45 PM Guillaume Nodet <[hidden email]> wrote:
> >
> > Hey everyone !
> >
> > The last weeks, I've spend quite some time working on a camel metamodel.
> > The idea is to invert the way things are built in camel so instead of
> > generating the metamodel from the classes, the metamodel would be
> > maintained manually and used to generate a bunch of things.
> >
> > This would bring the following benefits:
> >   - the metamodel would necessarily be up to date
> >   - generate the model classes (XyzDefinition classes) with an
> homogeneous
> > fluent api (similar to the new endpoint DSL)
> >   - get rid of some round tripping between java -> json -> java, copying
> > json files everywhere , so great simplification of the build process
> >   - the DSL allows type-safe + property placeholders / references
> everywhere
> >   - generate xml readers / parsers without relying on JAXB
> >   - brings extensibility of the DSL as it would be much easier to create
> > DSLs based on other languages from the metamodel (yaml for example)
> >
> > I've pushed a branch that shows my experiments.  It's not fully working
> > yet, but it gives a good idea. It's available at
> >   https://github.com/gnodet/camel/tree/model
> > I'm progressing a bit slowly as there are lots of step by step
> adjustements
> > to do in order to keep compatibility of the DSL.
> >
> > Currently the model is generated from the json files, but the idea is to
> > reverse this process and have the metamodel the real primary input for
> all
> > generation.
> > The model currently looks like:
> >   https://gist.github.com/gnodet/75457febcca9a893c3c1a2b8499189b2
> >
> > The current JAXB model will need to be moved into a separate module so
> that
> > it can be kept for compatibility without interfering with the new
> generated
> > java DSL.
> >
> > So, still quite some work left, but I wanted to bring it to the community
> > sooner rather than later, especially before I go on PTO for a few weeks
> > where I'll be mostly offline.
> > Happy to discuss anything or provide more infos.
> >
> > Cheers,
> > Guillaume Nodet
>
>
>
> --
> Claus Ibsen
> -----------------
> http://davsclaus.com @davsclaus
> Camel in Action 2: https://www.manning.com/ibsen2
>


--
------------------------
Guillaume Nodet
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] Meta model

gnodet
In reply to this post by Zoran Regvart-2
Your point is really interesting.

However, I think one of the main problem right now is that the metamodel
information is spread in the whole source tree (some bits in camel-core,
and lots in the various components).  This leads to a lot of hacks and
round trips in the build system (copying json files from various locations,
etc...).
Getting out of sync is already somehow a problem with dataformats and
languages, where the XyzDefinition are mapped to the actual dataformat /
language properties using introspection.  This is even more the case in
Camel 3 with the new endpoint DSL which basically does the same thing
(though it was already done using uri parameters).
I think if we have the metamodel from the beginning of the build, we should
be able to validate that the properties do exist in the final java classes
(components, endpoints, dataformats and languages).

The only way to have all the information from the java code and have a
clean build would be to generate the DSL (java + xml) after all components
/ langauges / dataformats would have been built, i.e. build camel-core at
the end, so that we could use the real java classes and use them.  However,
I'm not sure that this woud be a good idea, given the length of the build...


Le jeu. 25 juil. 2019 à 11:11, Zoran Regvart <[hidden email]> a écrit :

> Hi Guillaume & Cameleers,
> I like that there's an effort to make the code more maintainable. I do
> however feel that the source of truth is best kept in the Java code,
> and perhaps this new model can be generated from that. I understand
> the need for having a single strictly typed source of truth.
>
> Could we have the flow somehow like Java -> (XML?) meta model -> code
> generation of XML readers/writers and DSL?
>
> I have an inkling that we could even generate the meta model in Java
> and code generate from there without the model being in XML format, we
> are kinda, while awkwardly, doing it right now.
>
> I fear that unless we add more strict checking that if the XML meta
> model is the source of truth that we'll easily diverge from the Java
> code. For example, if we remove a property from the endpoint we need
> tooling that'll fail the build if the meta model is not updated and
> the property is removed there as well.
>



>
> zoran
>
> On Wed, Jul 24, 2019 at 4:45 PM Guillaume Nodet <[hidden email]> wrote:
> >
> > Hey everyone !
> >
> > The last weeks, I've spend quite some time working on a camel metamodel.
> > The idea is to invert the way things are built in camel so instead of
> > generating the metamodel from the classes, the metamodel would be
> > maintained manually and used to generate a bunch of things.
> >
> > This would bring the following benefits:
> >   - the metamodel would necessarily be up to date
> >   - generate the model classes (XyzDefinition classes) with an
> homogeneous
> > fluent api (similar to the new endpoint DSL)
> >   - get rid of some round tripping between java -> json -> java, copying
> > json files everywhere , so great simplification of the build process
> >   - the DSL allows type-safe + property placeholders / references
> everywhere
> >   - generate xml readers / parsers without relying on JAXB
> >   - brings extensibility of the DSL as it would be much easier to create
> > DSLs based on other languages from the metamodel (yaml for example)
> >
> > I've pushed a branch that shows my experiments.  It's not fully working
> > yet, but it gives a good idea. It's available at
> >   https://github.com/gnodet/camel/tree/model
> > I'm progressing a bit slowly as there are lots of step by step
> adjustements
> > to do in order to keep compatibility of the DSL.
> >
> > Currently the model is generated from the json files, but the idea is to
> > reverse this process and have the metamodel the real primary input for
> all
> > generation.
> > The model currently looks like:
> >   https://gist.github.com/gnodet/75457febcca9a893c3c1a2b8499189b2
> >
> > The current JAXB model will need to be moved into a separate module so
> that
> > it can be kept for compatibility without interfering with the new
> generated
> > java DSL.
> >
> > So, still quite some work left, but I wanted to bring it to the community
> > sooner rather than later, especially before I go on PTO for a few weeks
> > where I'll be mostly offline.
> > Happy to discuss anything or provide more infos.
> >
> > Cheers,
> > Guillaume Nodet
>
>
>
> --
> Zoran Regvart
>


--
------------------------
Guillaume Nodet
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] Meta model

Claus Ibsen-2
In reply to this post by gnodet
On Thu, Jul 25, 2019 at 11:28 AM Guillaume Nodet <[hidden email]> wrote:

>
> Le jeu. 25 juil. 2019 à 10:47, Claus Ibsen <[hidden email]> a écrit :
>
> > Hi
> >
> > Good to see more experiments, but as others have said in this mail
> > thread, its too overwhelming to dive into and understand.
> >
>
> Yes, I definitely get that, especially as it's still work in progress.
>
> On the java dsl side, the goal is to be 100% compatible, i.e. i'm trying to
> generate the DSL without changing all the tests in camel-core.
> However, this opens new possibilities, mainly about having a consistent way
> to inject references and use property placeholders, especially in places
> where this was not possible.
>

Ah okay so you are doing the same generating as endpoint dsl, where an
option that is eg a integer, you generate a builder for both
- integer
- string

Where the latter can be used for property placeholders.



>
>
> > So what I can see is that the model is now more "coded" in velocity vm
> > and xml xslt files, than what we had before with the java model
> > classes with JAXB annotations.
>

Okay so with these vm and xslt files then they seem a bit hardcore. I
am a little worried how we can maintain this.
But I can see they do the "hard" work of code generating from the metamodel.


>
>
> > Also I fail to see how it would understand if we add a new EIP today -
> > how would you go about doing that?
> >
>
> So if the xml metamodel becomes the real input (see below), then the
> developer would have to modify the metamodel to add an element to it (a
> processor, language, dataformat, or maybe a property on an existing
> element).
> You'd build the camel-metamodel, then build camel-core which re-generates
> all the data structures. At this point, the property will be added to the
> java code, so the next step is to actually create or modify the reifier and
> the runtime bits to actually use it.
>

Okay that is a fairly easy process.


>
> >
> > I do like that if we can have a lighter XML parser that is stax based than
> > JAXB.
> > And also if we can generate the XML XSD without JAXB at all.
> >
>
> Yes.
>
>
> > And the meta-model that you link to is a XML file which seems like an
> > aggregation of what we have today in camel-catalog in the various json
> > files.
> >
>
> Yes, though this is temporary in my mind.   I think the main input should
> be the xml metamodel which would be "manually" maintained.  From this file,
> we should generate everything that we need, including the json files for
> the catalog.
>

Ah okay so we have this single xml file.
Well frankly this file seems reasonable straightforward to understand and edit.

The only point is that for endpoints such jms etc, that has many
options and whatnot.
Then we have a build tool that generate and update the XML file with
all these endpoints?
And the same for components, eg we have component level options also.
And speaking of that, we may
also consider a fluent builder DSL for that too - like we have for endpoint DSL.




>
> > As you are going on PTO for a while I think we should maybe keep this
> > in mind that this work will not get completed or finished in the near
> > future.
> > We may consider getting the last other stuff done and get M5 out the
> > door as the last milestone, and then close down on RC builds.
> >
> > Then this meta-model can maybe be introduced in "steps" for 3.1, 3.2
> > etc. to give it more time to be more stable and maintainable.
> >
> >
> >
> > On Wed, Jul 24, 2019 at 4:45 PM Guillaume Nodet <[hidden email]> wrote:
> > >
> > > Hey everyone !
> > >
> > > The last weeks, I've spend quite some time working on a camel metamodel.
> > > The idea is to invert the way things are built in camel so instead of
> > > generating the metamodel from the classes, the metamodel would be
> > > maintained manually and used to generate a bunch of things.
> > >
> > > This would bring the following benefits:
> > >   - the metamodel would necessarily be up to date
> > >   - generate the model classes (XyzDefinition classes) with an
> > homogeneous
> > > fluent api (similar to the new endpoint DSL)
> > >   - get rid of some round tripping between java -> json -> java, copying
> > > json files everywhere , so great simplification of the build process
> > >   - the DSL allows type-safe + property placeholders / references
> > everywhere
> > >   - generate xml readers / parsers without relying on JAXB
> > >   - brings extensibility of the DSL as it would be much easier to create
> > > DSLs based on other languages from the metamodel (yaml for example)
> > >
> > > I've pushed a branch that shows my experiments.  It's not fully working
> > > yet, but it gives a good idea. It's available at
> > >   https://github.com/gnodet/camel/tree/model
> > > I'm progressing a bit slowly as there are lots of step by step
> > adjustements
> > > to do in order to keep compatibility of the DSL.
> > >
> > > Currently the model is generated from the json files, but the idea is to
> > > reverse this process and have the metamodel the real primary input for
> > all
> > > generation.
> > > The model currently looks like:
> > >   https://gist.github.com/gnodet/75457febcca9a893c3c1a2b8499189b2
> > >
> > > The current JAXB model will need to be moved into a separate module so
> > that
> > > it can be kept for compatibility without interfering with the new
> > generated
> > > java DSL.
> > >
> > > So, still quite some work left, but I wanted to bring it to the community
> > > sooner rather than later, especially before I go on PTO for a few weeks
> > > where I'll be mostly offline.
> > > Happy to discuss anything or provide more infos.
> > >
> > > Cheers,
> > > Guillaume Nodet
> >
> >
> >
> > --
> > Claus Ibsen
> > -----------------
> > http://davsclaus.com @davsclaus
> > Camel in Action 2: https://www.manning.com/ibsen2
> >
>
>
> --
> ------------------------
> Guillaume Nodet



--
Claus Ibsen
-----------------
http://davsclaus.com @davsclaus
Camel in Action 2: https://www.manning.com/ibsen2
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] Meta model

Claus Ibsen-2
In reply to this post by gnodet
Hi

Btw a follow up question.

Would we be able to get to a point where all the model classes
(xxxDefinition and helpers etc) can be dropped at runtime, eg for
camel-quarkus optimization?
And also the refiers as they are the builder from model -> processor.

Well what you experiment with is surely a great step in that direction too.



On Wed, Jul 24, 2019 at 4:45 PM Guillaume Nodet <[hidden email]> wrote:

>
> Hey everyone !
>
> The last weeks, I've spend quite some time working on a camel metamodel.
> The idea is to invert the way things are built in camel so instead of
> generating the metamodel from the classes, the metamodel would be
> maintained manually and used to generate a bunch of things.
>
> This would bring the following benefits:
>   - the metamodel would necessarily be up to date
>   - generate the model classes (XyzDefinition classes) with an homogeneous
> fluent api (similar to the new endpoint DSL)
>   - get rid of some round tripping between java -> json -> java, copying
> json files everywhere , so great simplification of the build process
>   - the DSL allows type-safe + property placeholders / references everywhere
>   - generate xml readers / parsers without relying on JAXB
>   - brings extensibility of the DSL as it would be much easier to create
> DSLs based on other languages from the metamodel (yaml for example)
>
> I've pushed a branch that shows my experiments.  It's not fully working
> yet, but it gives a good idea. It's available at
>   https://github.com/gnodet/camel/tree/model
> I'm progressing a bit slowly as there are lots of step by step adjustements
> to do in order to keep compatibility of the DSL.
>
> Currently the model is generated from the json files, but the idea is to
> reverse this process and have the metamodel the real primary input for all
> generation.
> The model currently looks like:
>   https://gist.github.com/gnodet/75457febcca9a893c3c1a2b8499189b2
>
> The current JAXB model will need to be moved into a separate module so that
> it can be kept for compatibility without interfering with the new generated
> java DSL.
>
> So, still quite some work left, but I wanted to bring it to the community
> sooner rather than later, especially before I go on PTO for a few weeks
> where I'll be mostly offline.
> Happy to discuss anything or provide more infos.
>
> Cheers,
> Guillaume Nodet



--
Claus Ibsen
-----------------
http://davsclaus.com @davsclaus
Camel in Action 2: https://www.manning.com/ibsen2
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] Meta model

lburgazzoli
Couple of questions:

- how does this play with external components ?
- what about annotations we have now i.e. for endpoints ? I found
particularity useful that types, enum values and so one are taken from the
java code

---
Luca Burgazzoli


On Thu, Jul 25, 2019 at 12:04 PM Claus Ibsen <[hidden email]> wrote:

> Hi
>
> Btw a follow up question.
>
> Would we be able to get to a point where all the model classes
> (xxxDefinition and helpers etc) can be dropped at runtime, eg for
> camel-quarkus optimization?
> And also the refiers as they are the builder from model -> processor.
>
> Well what you experiment with is surely a great step in that direction too.
>
>
>
> On Wed, Jul 24, 2019 at 4:45 PM Guillaume Nodet <[hidden email]> wrote:
> >
> > Hey everyone !
> >
> > The last weeks, I've spend quite some time working on a camel metamodel.
> > The idea is to invert the way things are built in camel so instead of
> > generating the metamodel from the classes, the metamodel would be
> > maintained manually and used to generate a bunch of things.
> >
> > This would bring the following benefits:
> >   - the metamodel would necessarily be up to date
> >   - generate the model classes (XyzDefinition classes) with an
> homogeneous
> > fluent api (similar to the new endpoint DSL)
> >   - get rid of some round tripping between java -> json -> java, copying
> > json files everywhere , so great simplification of the build process
> >   - the DSL allows type-safe + property placeholders / references
> everywhere
> >   - generate xml readers / parsers without relying on JAXB
> >   - brings extensibility of the DSL as it would be much easier to create
> > DSLs based on other languages from the metamodel (yaml for example)
> >
> > I've pushed a branch that shows my experiments.  It's not fully working
> > yet, but it gives a good idea. It's available at
> >   https://github.com/gnodet/camel/tree/model
> > I'm progressing a bit slowly as there are lots of step by step
> adjustements
> > to do in order to keep compatibility of the DSL.
> >
> > Currently the model is generated from the json files, but the idea is to
> > reverse this process and have the metamodel the real primary input for
> all
> > generation.
> > The model currently looks like:
> >   https://gist.github.com/gnodet/75457febcca9a893c3c1a2b8499189b2
> >
> > The current JAXB model will need to be moved into a separate module so
> that
> > it can be kept for compatibility without interfering with the new
> generated
> > java DSL.
> >
> > So, still quite some work left, but I wanted to bring it to the community
> > sooner rather than later, especially before I go on PTO for a few weeks
> > where I'll be mostly offline.
> > Happy to discuss anything or provide more infos.
> >
> > Cheers,
> > Guillaume Nodet
>
>
>
> --
> Claus Ibsen
> -----------------
> http://davsclaus.com @davsclaus
> Camel in Action 2: https://www.manning.com/ibsen2
>
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] Meta model

Zoran Regvart-2
In reply to this post by gnodet
Hi Guillaume,
since we're all piling up on you :) One more thing I'd like you to consider :)

I find Velocity templates a bit brittle and hard to maintain, would
you consider a Java code generator to create the Java source code?

I've found JavaPoet[1] pretty nifty in this regard. I've used it for
the OpenAPI to Rest DSL code generator we have[2].

Thanks :)

zoran

[1] https://github.com/square/javapoet
[2] https://github.com/apache/camel/tree/master/tooling/swagger-rest-dsl-generator/src/main/java/org/apache/camel/generator/swagger

On Wed, Jul 24, 2019 at 4:45 PM Guillaume Nodet <[hidden email]> wrote:

>
> Hey everyone !
>
> The last weeks, I've spend quite some time working on a camel metamodel.
> The idea is to invert the way things are built in camel so instead of
> generating the metamodel from the classes, the metamodel would be
> maintained manually and used to generate a bunch of things.
>
> This would bring the following benefits:
>   - the metamodel would necessarily be up to date
>   - generate the model classes (XyzDefinition classes) with an homogeneous
> fluent api (similar to the new endpoint DSL)
>   - get rid of some round tripping between java -> json -> java, copying
> json files everywhere , so great simplification of the build process
>   - the DSL allows type-safe + property placeholders / references everywhere
>   - generate xml readers / parsers without relying on JAXB
>   - brings extensibility of the DSL as it would be much easier to create
> DSLs based on other languages from the metamodel (yaml for example)
>
> I've pushed a branch that shows my experiments.  It's not fully working
> yet, but it gives a good idea. It's available at
>   https://github.com/gnodet/camel/tree/model
> I'm progressing a bit slowly as there are lots of step by step adjustements
> to do in order to keep compatibility of the DSL.
>
> Currently the model is generated from the json files, but the idea is to
> reverse this process and have the metamodel the real primary input for all
> generation.
> The model currently looks like:
>   https://gist.github.com/gnodet/75457febcca9a893c3c1a2b8499189b2
>
> The current JAXB model will need to be moved into a separate module so that
> it can be kept for compatibility without interfering with the new generated
> java DSL.
>
> So, still quite some work left, but I wanted to bring it to the community
> sooner rather than later, especially before I go on PTO for a few weeks
> where I'll be mostly offline.
> Happy to discuss anything or provide more infos.
>
> Cheers,
> Guillaume Nodet



--
Zoran Regvart
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] Meta model

jbonofre
Hi

Model driven makes sense.

I like velocity but I understand Zoran's comment.

Maybe we can imagine some pluggable: pojo describing model and binding.

Regards
JB

Le 25 juil. 2019 à 20:28, à 20:28, Zoran Regvart <[hidden email]> a écrit:

>Hi Guillaume,
>since we're all piling up on you :) One more thing I'd like you to
>consider :)
>
>I find Velocity templates a bit brittle and hard to maintain, would
>you consider a Java code generator to create the Java source code?
>
>I've found JavaPoet[1] pretty nifty in this regard. I've used it for
>the OpenAPI to Rest DSL code generator we have[2].
>
>Thanks :)
>
>zoran
>
>[1] https://github.com/square/javapoet
>[2]
>https://github.com/apache/camel/tree/master/tooling/swagger-rest-dsl-generator/src/main/java/org/apache/camel/generator/swagger
>
>On Wed, Jul 24, 2019 at 4:45 PM Guillaume Nodet <[hidden email]>
>wrote:
>>
>> Hey everyone !
>>
>> The last weeks, I've spend quite some time working on a camel
>metamodel.
>> The idea is to invert the way things are built in camel so instead of
>> generating the metamodel from the classes, the metamodel would be
>> maintained manually and used to generate a bunch of things.
>>
>> This would bring the following benefits:
>>   - the metamodel would necessarily be up to date
>>   - generate the model classes (XyzDefinition classes) with an
>homogeneous
>> fluent api (similar to the new endpoint DSL)
>>   - get rid of some round tripping between java -> json -> java,
>copying
>> json files everywhere , so great simplification of the build process
>>   - the DSL allows type-safe + property placeholders / references
>everywhere
>>   - generate xml readers / parsers without relying on JAXB
>>   - brings extensibility of the DSL as it would be much easier to
>create
>> DSLs based on other languages from the metamodel (yaml for example)
>>
>> I've pushed a branch that shows my experiments.  It's not fully
>working
>> yet, but it gives a good idea. It's available at
>>   https://github.com/gnodet/camel/tree/model
>> I'm progressing a bit slowly as there are lots of step by step
>adjustements
>> to do in order to keep compatibility of the DSL.
>>
>> Currently the model is generated from the json files, but the idea is
>to
>> reverse this process and have the metamodel the real primary input
>for all
>> generation.
>> The model currently looks like:
>>   https://gist.github.com/gnodet/75457febcca9a893c3c1a2b8499189b2
>>
>> The current JAXB model will need to be moved into a separate module
>so that
>> it can be kept for compatibility without interfering with the new
>generated
>> java DSL.
>>
>> So, still quite some work left, but I wanted to bring it to the
>community
>> sooner rather than later, especially before I go on PTO for a few
>weeks
>> where I'll be mostly offline.
>> Happy to discuss anything or provide more infos.
>>
>> Cheers,
>> Guillaume Nodet
>
>
>
>--
>Zoran Regvart
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] Meta model

gnodet
In reply to this post by Zoran Regvart-2
Le jeu. 25 juil. 2019 à 20:28, Zoran Regvart <[hidden email]> a écrit :

> Hi Guillaume,
> since we're all piling up on you :) One more thing I'd like you to
> consider :)
>
> I find Velocity templates a bit brittle and hard to maintain, would
> you consider a Java code generator to create the Java source code?
>

Lol ! I usually find those harder to use for the use cases we have and when
you want tight control over the written code.
I'd rather move all the smart things into a helper class and keep velocity
with almost only simple iterations on variables.

You can compare the code used to generate the endpoint DSL:

https://github.com/apache/camel/blob/master/tooling/maven/camel-package-maven-plugin/src/main/java/org/apache/camel/maven/packaging/EndpointDslMojo.java

https://github.com/gnodet/camel/blob/model/core/camel-core/src/generator/resources/endpoint.vm

If you really want the main executed code to be java, I'd rather use a
simple java writer rather than a java code generator.

That may be because of the way I'm working, but I usually work with the
generated source code when I need to change something it, and when I'm
happy, I'm changing the generator to reflect those changes, and this is
usually way easier when you're working on a template which actually looks
like the output.


>
> I've found JavaPoet[1] pretty nifty in this regard. I've used it for
> the OpenAPI to Rest DSL code generator we have[2].
>
> Thanks :)
>
> zoran
>
> [1] https://github.com/square/javapoet
> [2]
> https://github.com/apache/camel/tree/master/tooling/swagger-rest-dsl-generator/src/main/java/org/apache/camel/generator/swagger
>
> On Wed, Jul 24, 2019 at 4:45 PM Guillaume Nodet <[hidden email]> wrote:
> >
> > Hey everyone !
> >
> > The last weeks, I've spend quite some time working on a camel metamodel.
> > The idea is to invert the way things are built in camel so instead of
> > generating the metamodel from the classes, the metamodel would be
> > maintained manually and used to generate a bunch of things.
> >
> > This would bring the following benefits:
> >   - the metamodel would necessarily be up to date
> >   - generate the model classes (XyzDefinition classes) with an
> homogeneous
> > fluent api (similar to the new endpoint DSL)
> >   - get rid of some round tripping between java -> json -> java, copying
> > json files everywhere , so great simplification of the build process
> >   - the DSL allows type-safe + property placeholders / references
> everywhere
> >   - generate xml readers / parsers without relying on JAXB
> >   - brings extensibility of the DSL as it would be much easier to create
> > DSLs based on other languages from the metamodel (yaml for example)
> >
> > I've pushed a branch that shows my experiments.  It's not fully working
> > yet, but it gives a good idea. It's available at
> >   https://github.com/gnodet/camel/tree/model
> > I'm progressing a bit slowly as there are lots of step by step
> adjustements
> > to do in order to keep compatibility of the DSL.
> >
> > Currently the model is generated from the json files, but the idea is to
> > reverse this process and have the metamodel the real primary input for
> all
> > generation.
> > The model currently looks like:
> >   https://gist.github.com/gnodet/75457febcca9a893c3c1a2b8499189b2
> >
> > The current JAXB model will need to be moved into a separate module so
> that
> > it can be kept for compatibility without interfering with the new
> generated
> > java DSL.
> >
> > So, still quite some work left, but I wanted to bring it to the community
> > sooner rather than later, especially before I go on PTO for a few weeks
> > where I'll be mostly offline.
> > Happy to discuss anything or provide more infos.
> >
> > Cheers,
> > Guillaume Nodet
>
>
>
> --
> Zoran Regvart
>


--
------------------------
Guillaume Nodet