Incremental builds, can it be done?

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Incremental builds, can it be done?

Zoran Regvart-2
Hi,
There has been some talk about PR builds in another thread and I'm
just trying to gauge the idea of having incremental builds (or no
clean builds) for a project of this complexity.

Having a incremental build would allow us to have a build that
verifies PRs as, in theory, they would perform the build with just the
subset that is affected and would lead to faster build times.

There has been some prior work on getting incremental builds working
with Maven with some degree of success[1][2][3][4].

I wonder if anyone has tried to do this with Camel or similarly large
Maven project and has any stories to share?

zoran

[1] http://takari.io/book/40-lifecycle.html#the-takari-lifecycle
[2] https://github.com/vackosar/gitflow-incremental-builder
[3] https://svn.apache.org/repos/asf/maven/sandbox/trunk/shared/scm-changes-maven-extension/
[4] https://camel.apache.org/building.html#Building-ExecutingunittestsusingEkstazi
--
Zoran Regvart
Reply | Threaded
Open this post in threaded view
|

Re: Incremental builds, can it be done?

Peter Palaga
Hi Zoran,

I can offer some experience with Takari Lifecycle. I am using it in my
smallish srcdeps project [1]. I took it for its seamless integration
with Eclipse Sisu rather than for its incremental builds.

Takari is a replacement for the default Maven Lifecycle - i.e. Takari
replaces the default Maven plugins (compile, resources, jar, ...) with
its own variants. While their plugins serve the same purpose as the
default ones, the configuration may (an often is) not as rich as with
the default plugins. One prominent example is that they do not support
war packaging [2].

Hence when deciding whether Takari is a good choice one should in the
first place check carefully if their plugins support all necessary
configuration options. I am not sure if their incrementality claims
still hold when non-takari plugins are executed during the build.

As for their incremental compilation, they say [3] they can do it thanks
Eclipse Java Compiler (a.k.a. JDT Compiler). It is known that JDT
Compiler and OpenJDK's javac sometimes differ in what Java source they
accept [4]. That might be seen as an argument against using JDT Compiler
and Takari inside CI.

Thanks,

-- Peter

[1] https://github.com/srcdeps/srcdeps-core and
https://github.com/srcdeps/srcdeps-maven
[2] https://github.com/takari/takari-lifecycle/issues/8
[3] http://takari.io/2014/10/16/incremental-compilation.html
[4] https://bugs.eclipse.org/bugs/show_bug.cgi?id=520916 
https://bugs.openjdk.java.net/browse/JDK-8168644

On 15/10/17 21:34, Zoran Regvart wrote:

> Hi,
> There has been some talk about PR builds in another thread and I'm
> just trying to gauge the idea of having incremental builds (or no
> clean builds) for a project of this complexity.
>
> Having a incremental build would allow us to have a build that
> verifies PRs as, in theory, they would perform the build with just the
> subset that is affected and would lead to faster build times.
>
> There has been some prior work on getting incremental builds working
> with Maven with some degree of success[1][2][3][4].
>
> I wonder if anyone has tried to do this with Camel or similarly large
> Maven project and has any stories to share?
>
> zoran
>
> [1] http://takari.io/book/40-lifecycle.html#the-takari-lifecycle
> [2] https://github.com/vackosar/gitflow-incremental-builder
> [3] https://svn.apache.org/repos/asf/maven/sandbox/trunk/shared/scm-changes-maven-extension/
> [4] https://camel.apache.org/building.html#Building-ExecutingunittestsusingEkstazi
>

Reply | Threaded
Open this post in threaded view
|

Re: Incremental builds, can it be done?

Kevin Earls
Hi Zoran,

It's been a while, so I can't offer any precise numbers on this, but for a
while I tried doing incremental builds of Camel on the Fuse Jenkins.
Sometimes this would result in a relatively quick (45min to 1 hour) build.
However, that was more of an exception.

Generally, I think because so many things depend on core, Jenkins would do
a full or nearly full build, so we never ended up saving that much time.

Sorry my memory is not better and I can't offer more detail on this.

Kevin


On Sun, Oct 15, 2017 at 10:49 PM, Peter Palaga <[hidden email]> wrote:

> Hi Zoran,
>
> I can offer some experience with Takari Lifecycle. I am using it in my
> smallish srcdeps project [1]. I took it for its seamless integration with
> Eclipse Sisu rather than for its incremental builds.
>
> Takari is a replacement for the default Maven Lifecycle - i.e. Takari
> replaces the default Maven plugins (compile, resources, jar, ...) with its
> own variants. While their plugins serve the same purpose as the default
> ones, the configuration may (an often is) not as rich as with the default
> plugins. One prominent example is that they do not support war packaging
> [2].
>
> Hence when deciding whether Takari is a good choice one should in the
> first place check carefully if their plugins support all necessary
> configuration options. I am not sure if their incrementality claims still
> hold when non-takari plugins are executed during the build.
>
> As for their incremental compilation, they say [3] they can do it thanks
> Eclipse Java Compiler (a.k.a. JDT Compiler). It is known that JDT Compiler
> and OpenJDK's javac sometimes differ in what Java source they accept [4].
> That might be seen as an argument against using JDT Compiler and Takari
> inside CI.
>
> Thanks,
>
> -- Peter
>
> [1] https://github.com/srcdeps/srcdeps-core and
> https://github.com/srcdeps/srcdeps-maven
> [2] https://github.com/takari/takari-lifecycle/issues/8
> [3] http://takari.io/2014/10/16/incremental-compilation.html
> [4] https://bugs.eclipse.org/bugs/show_bug.cgi?id=520916
> https://bugs.openjdk.java.net/browse/JDK-8168644
>
>
> On 15/10/17 21:34, Zoran Regvart wrote:
>
>> Hi,
>> There has been some talk about PR builds in another thread and I'm
>> just trying to gauge the idea of having incremental builds (or no
>> clean builds) for a project of this complexity.
>>
>> Having a incremental build would allow us to have a build that
>> verifies PRs as, in theory, they would perform the build with just the
>> subset that is affected and would lead to faster build times.
>>
>> There has been some prior work on getting incremental builds working
>> with Maven with some degree of success[1][2][3][4].
>>
>> I wonder if anyone has tried to do this with Camel or similarly large
>> Maven project and has any stories to share?
>>
>> zoran
>>
>> [1] http://takari.io/book/40-lifecycle.html#the-takari-lifecycle
>> [2] https://github.com/vackosar/gitflow-incremental-builder
>> [3] https://svn.apache.org/repos/asf/maven/sandbox/trunk/shared/
>> scm-changes-maven-extension/
>> [4] https://camel.apache.org/building.html#Building-Executinguni
>> ttestsusingEkstazi
>>
>>
>