Hudson and Unit Tests

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Hudson and Unit Tests

chrislovecnm
A quick question

Why are the tests skipped during the CI Hudson builds? I have filed three bugs today that deal directly with unit tests not building or failing. IMHO these may have been caught if the test where not skipped.  Or a hudson build could be setup that nightly runs the unit tests?

I am sure that there is a reason why the tests are off, but I was just curious.  Plus a bit frustrated that the build is broken :)

Chris
Reply | Threaded
Open this post in threaded view
|

Re: Hudson and Unit Tests

Willem.Jiang
Administrator
I found this issue weeks ago, it looks like we just use the Apache
Hundson to deploy the camel kits now.

And Gert said we can create two Hudson jobs (one for deployment and
other for running test) for Camel.

It's time to get them started :)

Willem

chrislovecnm wrote:

> A quick question
>
> Why are the tests skipped during the CI Hudson builds? I have filed three
> bugs today that deal directly with unit tests not building or failing. IMHO
> these may have been caught if the test where not skipped.  Or a hudson build
> could be setup that nightly runs the unit tests?
>
> I am sure that there is a reason why the tests are off, but I was just
> curious.  Plus a bit frustrated that the build is broken :)
>
> Chris

Reply | Threaded
Open this post in threaded view
|

Re: Hudson and Unit Tests

Christian Schneider
Hi Willem,

does it make sense to deploy if not all test succeed? This causes that
sometimes snapshots will have low quality.
I would prefer to only deploy when all tests work. Skipping some tests
is sure ok for some time if an issue can not be worked
out quickly. What do you think?

Greetings

Christian


Am 19.01.2010 03:10, schrieb Willem Jiang:

> I found this issue weeks ago, it looks like we just use the Apache
> Hundson to deploy the camel kits now.
>
> And Gert said we can create two Hudson jobs (one for deployment and
> other for running test) for Camel.
>
> It's time to get them started :)
>
> Willem
>
> chrislovecnm wrote:
>> A quick question
>>
>> Why are the tests skipped during the CI Hudson builds? I have filed
>> three
>> bugs today that deal directly with unit tests not building or
>> failing. IMHO
>> these may have been caught if the test where not skipped.  Or a
>> hudson build
>> could be setup that nightly runs the unit tests?
>>
>> I am sure that there is a reason why the tests are off, but I was just
>> curious.  Plus a bit frustrated that the build is broken :)
>>
>> Chris
>
>

Reply | Threaded
Open this post in threaded view
|

Re: Hudson and Unit Tests

Claus Ibsen-2
On Tue, Jan 19, 2010 at 8:31 AM, Christian Schneider
<[hidden email]> wrote:
> Hi Willem,
>
> does it make sense to deploy if not all test succeed? This causes that
> sometimes snapshots will have low quality.
> I would prefer to only deploy when all tests work. Skipping some tests is
> sure ok for some time if an issue can not be worked
> out quickly. What do you think?
>

No I think its great as it is.

Camel have 4600+ unit test and still counting. It takes like 3-4h on
the hudson server to complete a test and it can break somewhere in for
some strange reasons. We see this on the Fuse CI platforms where we
got 10+ platforms to run Camel every single day.

At Apache people want to latest code in SNAPSHOT which also can help
when they want to verify a fix committed fixed their issue.
If we only deploy if 100% green then SNAPSHOT could easily turn weeks
old or longer.

The current setup is great as it is IMHO.

If you want SNAPSHOT which have 100% green they you can use the
SNAPSHOTs from the fuse repos.


> Greetings
>
> Christian
>
>
> Am 19.01.2010 03:10, schrieb Willem Jiang:
>>
>> I found this issue weeks ago, it looks like we just use the Apache Hundson
>> to deploy the camel kits now.
>>
>> And Gert said we can create two Hudson jobs (one for deployment and other
>> for running test) for Camel.
>>
>> It's time to get them started :)
>>
>> Willem
>>
>> chrislovecnm wrote:
>>>
>>> A quick question
>>>
>>> Why are the tests skipped during the CI Hudson builds? I have filed three
>>> bugs today that deal directly with unit tests not building or failing.
>>> IMHO
>>> these may have been caught if the test where not skipped.  Or a hudson
>>> build
>>> could be setup that nightly runs the unit tests?
>>>
>>> I am sure that there is a reason why the tests are off, but I was just
>>> curious.  Plus a bit frustrated that the build is broken :)
>>>
>>> Chris
>>
>>
>
>



--
Claus Ibsen
Apache Camel Committer

Author of Camel in Action: http://www.manning.com/ibsen/
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus
Reply | Threaded
Open this post in threaded view
|

Re: Hudson and Unit Tests

Willem.Jiang
Administrator
The SNAPSHOT release means a unstable release, I'm +1 for the publish
the SNAPSHOT everyday.
In most case we just don't recommend user to use the SNAPSHOT release,
only for new feature or some other integration test we use the SNAPSHOT.

Back to the Camel unit tests, as most unit test need to start the whole
camel context, it's a time consuming work, so most unit tests take more
than 1 second to run. Maybe we need find a way to speed up the
camel-unit test by build it separately.

Willem

Claus Ibsen wrote:

> On Tue, Jan 19, 2010 at 8:31 AM, Christian Schneider
> <[hidden email]> wrote:
>> Hi Willem,
>>
>> does it make sense to deploy if not all test succeed? This causes that
>> sometimes snapshots will have low quality.
>> I would prefer to only deploy when all tests work. Skipping some tests is
>> sure ok for some time if an issue can not be worked
>> out quickly. What do you think?
>>
>
> No I think its great as it is.
>
> Camel have 4600+ unit test and still counting. It takes like 3-4h on
> the hudson server to complete a test and it can break somewhere in for
> some strange reasons. We see this on the Fuse CI platforms where we
> got 10+ platforms to run Camel every single day.
>
> At Apache people want to latest code in SNAPSHOT which also can help
> when they want to verify a fix committed fixed their issue.
> If we only deploy if 100% green then SNAPSHOT could easily turn weeks
> old or longer.
>
> The current setup is great as it is IMHO.
>
> If you want SNAPSHOT which have 100% green they you can use the
> SNAPSHOTs from the fuse repos.
>
>
>> Greetings
>>
>> Christian
>>
>>
>> Am 19.01.2010 03:10, schrieb Willem Jiang:
>>> I found this issue weeks ago, it looks like we just use the Apache Hundson
>>> to deploy the camel kits now.
>>>
>>> And Gert said we can create two Hudson jobs (one for deployment and other
>>> for running test) for Camel.
>>>
>>> It's time to get them started :)
>>>
>>> Willem
>>>
>>> chrislovecnm wrote:
>>>> A quick question
>>>>
>>>> Why are the tests skipped during the CI Hudson builds? I have filed three
>>>> bugs today that deal directly with unit tests not building or failing.
>>>> IMHO
>>>> these may have been caught if the test where not skipped.  Or a hudson
>>>> build
>>>> could be setup that nightly runs the unit tests?
>>>>
>>>> I am sure that there is a reason why the tests are off, but I was just
>>>> curious.  Plus a bit frustrated that the build is broken :)
>>>>
>>>> Chris
>>>
>>
>
>
>

Reply | Threaded
Open this post in threaded view
|

Re: Hudson and Unit Tests

gertv
L.S.,

I also think it makes sense to deploy snapshots, even for unstable
builds, but currently this requires us to disable the unit tests on
one build and create a second CI build to run the unit tests.  FWIW, I
created a patch for HUDSON-3773 [1] which would allow Hudson to deploy
a SNAPSHOT even for unstable builds.  This will allow us to have a
single build definition to run tests and deploy regular snapshots.

Regards,

Gert Vanthienen
------------------------
Open Source SOA: http://fusesource.com
Blog: http://gertvanthienen.blogspot.com/

[1] http://issues.hudson-ci.org/browse/HUDSON-3773



2010/1/19 Willem Jiang <[hidden email]>:

> The SNAPSHOT release means a unstable release, I'm +1 for the publish the
> SNAPSHOT everyday.
> In most case we just don't recommend user to use the SNAPSHOT release,
> only for new feature or some other integration test we use the SNAPSHOT.
>
> Back to the Camel unit tests, as most unit test need to start the whole
> camel context, it's a time consuming work, so most unit tests take more than
> 1 second to run. Maybe we need find a way to speed up the camel-unit test by
> build it separately.
>
> Willem
>
> Claus Ibsen wrote:
>>
>> On Tue, Jan 19, 2010 at 8:31 AM, Christian Schneider
>> <[hidden email]> wrote:
>>>
>>> Hi Willem,
>>>
>>> does it make sense to deploy if not all test succeed? This causes that
>>> sometimes snapshots will have low quality.
>>> I would prefer to only deploy when all tests work. Skipping some tests is
>>> sure ok for some time if an issue can not be worked
>>> out quickly. What do you think?
>>>
>>
>> No I think its great as it is.
>>
>> Camel have 4600+ unit test and still counting. It takes like 3-4h on
>> the hudson server to complete a test and it can break somewhere in for
>> some strange reasons. We see this on the Fuse CI platforms where we
>> got 10+ platforms to run Camel every single day.
>>
>> At Apache people want to latest code in SNAPSHOT which also can help
>> when they want to verify a fix committed fixed their issue.
>> If we only deploy if 100% green then SNAPSHOT could easily turn weeks
>> old or longer.
>>
>> The current setup is great as it is IMHO.
>>
>> If you want SNAPSHOT which have 100% green they you can use the
>> SNAPSHOTs from the fuse repos.
>>
>>
>>> Greetings
>>>
>>> Christian
>>>
>>>
>>> Am 19.01.2010 03:10, schrieb Willem Jiang:
>>>>
>>>> I found this issue weeks ago, it looks like we just use the Apache
>>>> Hundson
>>>> to deploy the camel kits now.
>>>>
>>>> And Gert said we can create two Hudson jobs (one for deployment and
>>>> other
>>>> for running test) for Camel.
>>>>
>>>> It's time to get them started :)
>>>>
>>>> Willem
>>>>
>>>> chrislovecnm wrote:
>>>>>
>>>>> A quick question
>>>>>
>>>>> Why are the tests skipped during the CI Hudson builds? I have filed
>>>>> three
>>>>> bugs today that deal directly with unit tests not building or failing.
>>>>> IMHO
>>>>> these may have been caught if the test where not skipped.  Or a hudson
>>>>> build
>>>>> could be setup that nightly runs the unit tests?
>>>>>
>>>>> I am sure that there is a reason why the tests are off, but I was just
>>>>> curious.  Plus a bit frustrated that the build is broken :)
>>>>>
>>>>> Chris
>>>>
>>>
>>
>>
>>
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Hudson and Unit Tests

Christian Schneider
In reply to this post by Claus Ibsen-2
Hi Claus,

I understand that it is not possible to run the unit tests for a
snapshot build at the moment. Of course they can only be switched on if
they run reliably.

I normally run the tests before a checkin (or before I submit a patch in
case of camel) to make sure that I do not break anything. If some tests
fail or worse fail only sometimes it is extremly difficult to decide if
the failure was because of my change. Any idea how to handle this?

Greetings

Christian


Am 19.01.2010 09:13, schrieb Claus Ibsen:

>
> No I think its great as it is.
> Camel have 4600+ unit test and still counting. It takes like 3-4h on
> the hudson server to complete a test and it can break somewhere in for
> some strange reasons. We see this on the Fuse CI platforms where we
> got 10+ platforms to run Camel every single day.
>
> At Apache people want to latest code in SNAPSHOT which also can help
> when they want to verify a fix committed fixed their issue.
> If we only deploy if 100% green then SNAPSHOT could easily turn weeks
> old or longer.
>
> The current setup is great as it is IMHO.
>
> If you want SNAPSHOT which have 100% green they you can use the
> SNAPSHOTs from the fuse repos.
>
>
>    
>> Greetings
>>
>> Christian
>>
>>
>> Am 19.01.2010 03:10, schrieb Willem Jiang:
>>      
>>> I found this issue weeks ago, it looks like we just use the Apache Hundson
>>> to deploy the camel kits now.
>>>
>>> And Gert said we can create two Hudson jobs (one for deployment and other
>>> for running test) for Camel.
>>>
>>> It's time to get them started :)
>>>
>>> Willem
>>>
>>> chrislovecnm wrote:
>>>        
>>>> A quick question
>>>>
>>>> Why are the tests skipped during the CI Hudson builds? I have filed three
>>>> bugs today that deal directly with unit tests not building or failing.
>>>> IMHO
>>>> these may have been caught if the test where not skipped.  Or a hudson
>>>> build
>>>> could be setup that nightly runs the unit tests?
>>>>
>>>> I am sure that there is a reason why the tests are off, but I was just
>>>> curious.  Plus a bit frustrated that the build is broken :)
>>>>
>>>> Chris
>>>>          
>>>
>>>        
>>
>>      
>
>
>    

Reply | Threaded
Open this post in threaded view
|

Re: Hudson and Unit Tests

chrislovecnm
In reply to this post by gertv
Since we are on the topic. Do you guys use or plan to use Hudson along with the maven release plug-in to cut releases?

Chris

P.S. Thanks for the active response!
Gert Vanthienen wrote
L.S.,

I also think it makes sense to deploy snapshots, even for unstable
builds, but currently this requires us to disable the unit tests on
one build and create a second CI build to run the unit tests.  FWIW, I
created a patch for HUDSON-3773 [1] which would allow Hudson to deploy
a SNAPSHOT even for unstable builds.  This will allow us to have a
single build definition to run tests and deploy regular snapshots.

Regards,

Gert Vanthienen
------------------------
Open Source SOA: http://fusesource.com
Blog: http://gertvanthienen.blogspot.com/

[1] http://issues.hudson-ci.org/browse/HUDSON-3773



2010/1/19 Willem Jiang <willem.jiang@gmail.com>:
> The SNAPSHOT release means a unstable release, I'm +1 for the publish the
> SNAPSHOT everyday.
> In most case we just don't recommend user to use the SNAPSHOT release,
> only for new feature or some other integration test we use the SNAPSHOT.
>
> Back to the Camel unit tests, as most unit test need to start the whole
> camel context, it's a time consuming work, so most unit tests take more than
> 1 second to run. Maybe we need find a way to speed up the camel-unit test by
> build it separately.
>
> Willem
>
> Claus Ibsen wrote:
>>
>> On Tue, Jan 19, 2010 at 8:31 AM, Christian Schneider
>> <chris@die-schneider.net> wrote:
>>>
>>> Hi Willem,
>>>
>>> does it make sense to deploy if not all test succeed? This causes that
>>> sometimes snapshots will have low quality.
>>> I would prefer to only deploy when all tests work. Skipping some tests is
>>> sure ok for some time if an issue can not be worked
>>> out quickly. What do you think?
>>>
>>
>> No I think its great as it is.
>>
>> Camel have 4600+ unit test and still counting. It takes like 3-4h on
>> the hudson server to complete a test and it can break somewhere in for
>> some strange reasons. We see this on the Fuse CI platforms where we
>> got 10+ platforms to run Camel every single day.
>>
>> At Apache people want to latest code in SNAPSHOT which also can help
>> when they want to verify a fix committed fixed their issue.
>> If we only deploy if 100% green then SNAPSHOT could easily turn weeks
>> old or longer.
>>
>> The current setup is great as it is IMHO.
>>
>> If you want SNAPSHOT which have 100% green they you can use the
>> SNAPSHOTs from the fuse repos.
>>
>>
>>> Greetings
>>>
>>> Christian
>>>
>>>
>>> Am 19.01.2010 03:10, schrieb Willem Jiang:
>>>>
>>>> I found this issue weeks ago, it looks like we just use the Apache
>>>> Hundson
>>>> to deploy the camel kits now.
>>>>
>>>> And Gert said we can create two Hudson jobs (one for deployment and
>>>> other
>>>> for running test) for Camel.
>>>>
>>>> It's time to get them started :)
>>>>
>>>> Willem
>>>>
>>>> chrislovecnm wrote:
>>>>>
>>>>> A quick question
>>>>>
>>>>> Why are the tests skipped during the CI Hudson builds? I have filed
>>>>> three
>>>>> bugs today that deal directly with unit tests not building or failing.
>>>>> IMHO
>>>>> these may have been caught if the test where not skipped.  Or a hudson
>>>>> build
>>>>> could be setup that nightly runs the unit tests?
>>>>>
>>>>> I am sure that there is a reason why the tests are off, but I was just
>>>>> curious.  Plus a bit frustrated that the build is broken :)
>>>>>
>>>>> Chris
>>>>
>>>
>>
>>
>>
>
>


-----
---
Gert Vanthienen
http://gertvanthienen.blogspot.com
Reply | Threaded
Open this post in threaded view
|

Re: Hudson and Unit Tests

hadrian
In reply to this post by Christian Schneider
Well, this is a bit of a gray area inherent to distributed teams. What you do Christian is great, don't change that :). However expect to run into frustrating situations now and then.

In an ideal world all tests would pass, we would not depend on an internet connection and a build would not take more than 5 minutes. And yes, we would not have snapshot dependencies! I ran into both compile and test issues countless times because a dependent snapshot changed under me, and spent hours investigating stuff like that. Well, that's much better now. Sometimes after changing in core I run all the tests, making sure there are no adverse effects on other components. By the time my tests are done, there may be 2-3 newer commits on the trunk and have to start over. Doing things by the book, doesn't always scale and all in all things are not bad at all.

That said, I think we could do a better job with snapshots. What about taking turns on monitoring Hudson and giving a higher prio to fixing issues there? Another solution I toyed with a while ago is moving to a distributed build using GrigGain or something like that to hopefully cut the build time to a fraction. Anybody with some spare cycles?

Hadrian


On Jan 19, 2010, at 4:31 PM, Christian Schneider wrote:

> Hi Claus,
>
> I understand that it is not possible to run the unit tests for a snapshot build at the moment. Of course they can only be switched on if they run reliably.
>
> I normally run the tests before a checkin (or before I submit a patch in case of camel) to make sure that I do not break anything. If some tests fail or worse fail only sometimes it is extremly difficult to decide if the failure was because of my change. Any idea how to handle this?
>
> Greetings
>
> Christian
>
>
> Am 19.01.2010 09:13, schrieb Claus Ibsen:
>>
>> No I think its great as it is.
>> Camel have 4600+ unit test and still counting. It takes like 3-4h on
>> the hudson server to complete a test and it can break somewhere in for
>> some strange reasons. We see this on the Fuse CI platforms where we
>> got 10+ platforms to run Camel every single day.
>>
>> At Apache people want to latest code in SNAPSHOT which also can help
>> when they want to verify a fix committed fixed their issue.
>> If we only deploy if 100% green then SNAPSHOT could easily turn weeks
>> old or longer.
>>
>> The current setup is great as it is IMHO.
>>
>> If you want SNAPSHOT which have 100% green they you can use the
>> SNAPSHOTs from the fuse repos.
>>
>>
>>  
>>> Greetings
>>>
>>> Christian
>>>
>>>
>>> Am 19.01.2010 03:10, schrieb Willem Jiang:
>>>    
>>>> I found this issue weeks ago, it looks like we just use the Apache Hundson
>>>> to deploy the camel kits now.
>>>>
>>>> And Gert said we can create two Hudson jobs (one for deployment and other
>>>> for running test) for Camel.
>>>>
>>>> It's time to get them started :)
>>>>
>>>> Willem
>>>>
>>>> chrislovecnm wrote:
>>>>      
>>>>> A quick question
>>>>>
>>>>> Why are the tests skipped during the CI Hudson builds? I have filed three
>>>>> bugs today that deal directly with unit tests not building or failing.
>>>>> IMHO
>>>>> these may have been caught if the test where not skipped.  Or a hudson
>>>>> build
>>>>> could be setup that nightly runs the unit tests?
>>>>>
>>>>> I am sure that there is a reason why the tests are off, but I was just
>>>>> curious.  Plus a bit frustrated that the build is broken :)
>>>>>
>>>>> Chris
>>>>>        
>>>>
>>>>      
>>>
>>>    
>>
>>
>>  
>