We've run into an issue with travis where adding new make files causes the tests to fail with an "write error" when running distclean.
For example: https://travis-ci.org/cockpit-project/cockpit/jobs/76408030.
We haven't been able to reproduce these failures anywhere else. In addition their new container based infrastructure that they are moving towards doesn't support sudo which we need for our reauthorize tests. So we are looking at replacing travis with a different CI service.
I looked at semaphoreci and circleci. I was eventually able to get our test suite to pass on both systems. Both seems to have most of the same github integration / badges but Circleci is based on Ubuntu 12 and we'd really like to be able to use something newer.
You can see my cockpit fork on semaphoreci here: https://semaphoreci.com/petervo/cockpit
PROS:
- It works and uses a newer OS (Ubuntu 14) - Allows you to ssh into instances to run / debug tests.
CONS:
- There is only one configuration per project that is configured via their website. So you can't change the commands on a per branch / PR basis. Additionally the webui has some bugs and usability issues that end up requiring a few tries before it gets the commands in the right order and in the right threads. - We have to run on their container based beta platform as their normal platform doesn't fully support inotify and causes tests to fail. - Doesn't do IRC notifications out of the box. Though we could probably implement something using webhooks if we wanted to.
What does everyone think about switching? Worth doing or should we keep looking at other options?
Peter petervo@redhat.com writes:
PROS:
- It works and uses a newer OS (Ubuntu 14)
- Allows you to ssh into instances to run / debug tests.
This is very, very nice.
CONS:
- There is only one configuration per project that is configured via
their website. So you can't change the commands on a per branch / PR basis.
We could have a script in our sources that does all the work and configure semaphoreci to just run that script. Would that work?
- We have to run on their container based beta platform as their normal
platform doesn't fully support inotify and causes tests to fail.
This is weird. How do they manage to not fully support inotify? Is it a weird/broken filesystem? Some absurdly low limits somewhere?
- Doesn't do IRC notifications out of the box. Though we could probably
implement something using webhooks if we wanted to.
Personally, I didn't find the notifications very useful.
What does everyone think about switching? Worth doing or should we keep looking at other options?
Right now the options are containers in either Semaphore or Travis, right? (Which means we need to adjust the tests that need sudo, right?)
And switching to containers in Travis might just make the weird "write error" bug disappear, I guess.
But since you have done the work, and Semaphore has the "ssh" killer feature, I vote to switch to Semaphore.
On 24.08.2015 08:54, Marius Vollmer wrote:
Peter petervo@redhat.com writes:
PROS:
- It works and uses a newer OS (Ubuntu 14)
- Allows you to ssh into instances to run / debug tests.
This is very, very nice.
CONS:
Obviously another big con is that this stuff is not Open Source. These are proprietary services. But so is GitHub. But until we have more resources, and/or an open source hosted CI service ... we'll probably have to ignore this con.
- There is only one configuration per project that is configured via
their website. So you can't change the commands on a per branch / PR basis.
We could have a script in our sources that does all the work and configure semaphoreci to just run that script. Would that work?
- We have to run on their container based beta platform as their normal
platform doesn't fully support inotify and causes tests to fail.
This is weird. How do they manage to not fully support inotify? Is it a weird/broken filesystem? Some absurdly low limits somewhere?
NFS.
- Doesn't do IRC notifications out of the box. Though we could probably
implement something using webhooks if we wanted to.
Personally, I didn't find the notifications very useful.
What does everyone think about switching? Worth doing or should we keep looking at other options?
Right now the options are containers in either Semaphore or Travis, right? (Which means we need to adjust the tests that need sudo, right?)
sudo can run in a container. It's just the Travis containers that don't allow this. Presumably because it's easier to break out of a container if you have root.
And switching to containers in Travis might just make the weird "write error" bug disappear, I guess.
But more importantly the Travis containers have an ancient operating system (from 2012) and can't be updated (due to no root access).
But since you have done the work, and Semaphore has the "ssh" killer feature, I vote to switch to Semaphore.
Me too.
Stef
This is weird. How do they manage to not fully support inotify? Is it a weird/broken filesystem? Some absurdly low limits somewhere?
NFS.
Oh, I see.
sudo can run in a container. It's just the Travis containers that don't allow this. [...]
Ok, so Semaphore does allow it and we don't need to change the tests? That is another big pro.
And switching to containers in Travis might just make the weird "write error" bug disappear, I guess.
But more importantly the Travis containers have an ancient operating system (from 2012) and can't be updated (due to no root access).
Yes, the pros are very strong.
On 24.08.2015 09:03, Stef Walter wrote:
On 24.08.2015 08:54, Marius Vollmer wrote:
Peter petervo@redhat.com writes:
PROS:
- It works and uses a newer OS (Ubuntu 14)
- Allows you to ssh into instances to run / debug tests.
This is very, very nice.
CONS:
Obviously another big con is that this stuff is not Open Source. These are proprietary services. But so is GitHub. But until we have more resources, and/or an open source hosted CI service ... we'll probably have to ignore this con.
I was wrong about Travis. Travis is Open Source. Yay. Good for them. That changes how much effort I feel we should dedicate to helping make it work, filing bugs, etc.
A little bird also whispered in my ear that Marius has found a work-around for the Travis bug that has been bothering us.
Stef
On Mon, Aug 24, 2015 at 8:45 AM, Stef Walter stefw@redhat.com wrote:
On 24.08.2015 09:03, Stef Walter wrote:
On 24.08.2015 08:54, Marius Vollmer wrote:
Peter petervo@redhat.com writes:
PROS:
- It works and uses a newer OS (Ubuntu 14)
- Allows you to ssh into instances to run / debug tests.
This is very, very nice.
CONS:
Obviously another big con is that this stuff is not Open Source. These are proprietary services. But so is GitHub. But until we have more resources, and/or an open source hosted CI service ... we'll probably have to ignore this con.
I was wrong about Travis. Travis is Open Source. Yay. Good for them. That changes how much effort I feel we should dedicate to helping make it work, filing bugs, etc.
A little bird also whispered in my ear that Marius has found a work-around for the Travis bug that has been bothering us.
Not sure if this would help, but the AOS team has a Jenkins server where we could do some more sophisticated CI testing:
https://ci.openshift.redhat.com/jenkins/
I am one of the maintainers so I'd be happy to help set things up if this was of interest.
Stef
cockpit-devel mailing list cockpit-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/cockpit-devel
On 24.08.2015 15:15, Harrison Ripps wrote:
On Mon, Aug 24, 2015 at 8:45 AM, Stef Walter <stefw@redhat.com mailto:stefw@redhat.com> wrote:
On 24.08.2015 09 <tel:24.08.2015%2009>:03, Stef Walter wrote: > On 24.08.2015 08 <tel:24.08.2015%2008>:54, Marius Vollmer wrote: >> Peter <petervo@redhat.com <mailto:petervo@redhat.com>> writes: >> >>> PROS: >>> >>> - It works and uses a newer OS (Ubuntu 14) >>> - Allows you to ssh into instances to run / debug tests. >> >> This is very, very nice. >> >>> CONS: > > Obviously another big con is that this stuff is not Open Source. These > are proprietary services. But so is GitHub. But until we have more > resources, and/or an open source hosted CI service ... we'll probably > have to ignore this con. I was wrong about Travis. Travis is Open Source. Yay. Good for them. That changes how much effort I feel we should dedicate to helping make it work, filing bugs, etc. A little bird also whispered in my ear that Marius has found a work-around for the Travis bug that has been bothering us.Not sure if this would help, but the AOS team has a Jenkins server where we could do some more sophisticated CI testing:
https://ci.openshift.redhat.com/jenkins/
I am one of the maintainers so I'd be happy to help set things up if this was of interest.
Interesting. Have you migrated anything from Travis to that Jenkins instance? What did the process end up looking like?
But this is actually more interesting for the workloads we don't run on Travis ... our integration tests ... which launch tons of VMs within the space of a few minutes. Is that the sort of thing your Jenkins instance can run? ie: nested virtualization, and/or root access etc?
Cheers,
Stef
On Mon, Aug 24, 2015 at 9:27 AM, Stef Walter stefw@redhat.com wrote:
On 24.08.2015 15:15, Harrison Ripps wrote:
On Mon, Aug 24, 2015 at 8:45 AM, Stef Walter <stefw@redhat.com mailto:stefw@redhat.com> wrote:
On 24.08.2015 09 <tel:24.08.2015%2009>:03, Stef Walter wrote: > On 24.08.2015 08 <tel:24.08.2015%2008>:54, Marius Vollmer wrote: >> Peter <petervo@redhat.com <mailto:petervo@redhat.com>> writes: >> >>> PROS: >>> >>> - It works and uses a newer OS (Ubuntu 14) >>> - Allows you to ssh into instances to run / debug tests. >> >> This is very, very nice. >> >>> CONS: > > Obviously another big con is that this stuff is not Open Source.These
> are proprietary services. But so is GitHub. But until we have more > resources, and/or an open source hosted CI service ... we'llprobably
> have to ignore this con. I was wrong about Travis. Travis is Open Source. Yay. Good for them. That changes how much effort I feel we should dedicate to helpingmake
it work, filing bugs, etc. A little bird also whispered in my ear that Marius has found a work-around for the Travis bug that has been bothering us.Not sure if this would help, but the AOS team has a Jenkins server where we could do some more sophisticated CI testing:
https://ci.openshift.redhat.com/jenkins/
I am one of the maintainers so I'd be happy to help set things up if this was of interest.
Interesting. Have you migrated anything from Travis to that Jenkins instance? What did the process end up looking like?
I have not done a Travis to Jenkins migration. However, based on looking at a reference Travis config[1], I'd say the porting is relatively painless. The Jenkins host needs to have the right tools installed on it and then you write a shell script for Jenkins to run.
[1]: https://github.com/openshift/origin/blob/master/.travis.yml
But this is actually more interesting for the workloads we don't run on Travis ... our integration tests ... which launch tons of VMs within the space of a few minutes. Is that the sort of thing your Jenkins instance can run? ie: nested virtualization, and/or root access etc?
The public AOS Jenkins system is running on an EC2 instance. With its current workload, I don't think it has enough power to stand up a bunch of locally hosted VMs for testing. However, if the tests could be modified to spin up EC2 instances, or instances on the Red Hat public OpenStack cluster, then the AOS Jenkins host could certainly handle the workload of firing them up, running tests, and reporting on the test results.
Cheers,
Stef
cockpit-devel mailing list cockpit-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/cockpit-devel
On 24.08.2015 15:44, Harrison Ripps wrote:
On Mon, Aug 24, 2015 at 9:27 AM, Stef Walter <stefw@redhat.com mailto:stefw@redhat.com> wrote:
On 24.08.2015 15 <tel:24.08.2015%2015>:15, Harrison Ripps wrote: > > On Mon, Aug 24, 2015 at 8:45 AM, Stef Walter <stefw@redhat.com <mailto:stefw@redhat.com> > <mailto:stefw@redhat.com <mailto:stefw@redhat.com>>> wrote: > > On 24.08.2015 09 <tel:24.08.2015%2009> <tel:24.08.2015%2009>:03, Stef Walter wrote: > > On 24.08.2015 08 <tel:24.08.2015%2008> <tel:24.08.2015%2008>:54, Marius Vollmer wrote: > >> Peter <petervo@redhat.com <mailto:petervo@redhat.com> <mailto:petervo@redhat.com <mailto:petervo@redhat.com>>> writes: > >> > >>> PROS: > >>> > >>> - It works and uses a newer OS (Ubuntu 14) > >>> - Allows you to ssh into instances to run / debug tests. > >> > >> This is very, very nice. > >> > >>> CONS: > > > > Obviously another big con is that this stuff is not Open Source. These > > are proprietary services. But so is GitHub. But until we have more > > resources, and/or an open source hosted CI service ... we'll probably > > have to ignore this con. > > I was wrong about Travis. Travis is Open Source. Yay. Good for them. > That changes how much effort I feel we should dedicate to helping make > it work, filing bugs, etc. > > A little bird also whispered in my ear that Marius has found a > work-around for the Travis bug that has been bothering us. > > > Not sure if this would help, but the AOS team has a Jenkins server where > we could do some more sophisticated CI testing: > > https://ci.openshift.redhat.com/jenkins/ > > I am one of the maintainers so I'd be happy to help set things up if > this was of interest. Interesting. Have you migrated anything from Travis to that Jenkins instance? What did the process end up looking like?I have not done a Travis to Jenkins migration. However, based on looking at a reference Travis config[1], I'd say the porting is relatively painless. The Jenkins host needs to have the right tools installed on it and then you write a shell script for Jenkins to run.
But this is actually more interesting for the workloads we don't run on Travis ... our integration tests ... which launch tons of VMs within the space of a few minutes. Is that the sort of thing your Jenkins instance can run? ie: nested virtualization, and/or root access etc?The public AOS Jenkins system is running on an EC2 instance. With its current workload, I don't think it has enough power to stand up a bunch of locally hosted VMs for testing. However, if the tests could be modified to spin up EC2 instances, or instances on the Red Hat public OpenStack cluster, then the AOS Jenkins host could certainly handle the workload of firing them up, running tests, and reporting on the test results.
True, but we snapshot images, and then spin up VMs, and do a test and shutdown each within 10 seconds or so. For day to day use (vetting pull requests) it makes sense to stay on some performant hardware.
Stef
On Mon, Aug 24, 2015 at 9:54 AM, Stef Walter stefw@redhat.com wrote:
On 24.08.2015 15:44, Harrison Ripps wrote:
On Mon, Aug 24, 2015 at 9:27 AM, Stef Walter <stefw@redhat.com mailto:stefw@redhat.com> wrote:
On 24.08.2015 15 <tel:24.08.2015%2015>:15, Harrison Ripps wrote: > > On Mon, Aug 24, 2015 at 8:45 AM, Stef Walter <stefw@redhat.com> <mailto:stefw@redhat.com <mailto:stefw@redhat.com>>> wrote: > > On 24.08.2015 09 <tel:24.08.2015%2009> <tel:24.08.2015%2009>:03, Stef Walter wrote: > > On 24.08.2015 08 <tel:24.08.2015%2008> <tel:24.08.2015%2008>:54, Marius Vollmer wrote: > >> Peter <petervo@redhat.com <mailto:petervo@redhat.com> <mailto:petervo@redhat.com <mailto:petervo@redhat.com>>> writes: > >> > >>> PROS: > >>> > >>> - It works and uses a newer OS (Ubuntu 14) > >>> - Allows you to ssh into instances to run / debug tests. > >> > >> This is very, very nice. > >> > >>> CONS: > > > > Obviously another big con is that this stuff is not OpenSource. These
> > are proprietary services. But so is GitHub. But until wehave more
> > resources, and/or an open source hosted CI service ... we'llprobably
> > have to ignore this con. > > I was wrong about Travis. Travis is Open Source. Yay. Good forthem.
> That changes how much effort I feel we should dedicate tohelping make
> it work, filing bugs, etc. > > A little bird also whispered in my ear that Marius has found a > work-around for the Travis bug that has been bothering us. > > > Not sure if this would help, but the AOS team has a Jenkins serverwhere
> we could do some more sophisticated CI testing: > > https://ci.openshift.redhat.com/jenkins/ > > I am one of the maintainers so I'd be happy to help set things upif
> this was of interest. Interesting. Have you migrated anything from Travis to that Jenkins instance? What did the process end up looking like?I have not done a Travis to Jenkins migration. However, based on looking at a reference Travis config[1], I'd say the porting is relatively painless. The Jenkins host needs to have the right tools installed on it and then you write a shell script for Jenkins to run.
But this is actually more interesting for the workloads we don't runon
Travis ... our integration tests ... which launch tons of VMs withinthe
space of a few minutes. Is that the sort of thing your Jenkinsinstance
can run? ie: nested virtualization, and/or root access etc?The public AOS Jenkins system is running on an EC2 instance. With its current workload, I don't think it has enough power to stand up a bunch of locally hosted VMs for testing. However, if the tests could be modified to spin up EC2 instances, or instances on the Red Hat public OpenStack cluster, then the AOS Jenkins host could certainly handle the workload of firing them up, running tests, and reporting on the test results.
True, but we snapshot images, and then spin up VMs, and do a test and shutdown each within 10 seconds or so. For day to day use (vetting pull requests) it makes sense to stay on some performant hardware.
Understood. I may still be able to help with this one. I'll follow up after I've had a chance to talk to our ops team.
Stef
cockpit-devel mailing list cockpit-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/cockpit-devel
Stef Walter stefw@redhat.com writes:
A little bird also whispered in my ear that Marius has found a work-around for the Travis bug that has been bothering us.
Yes, see here https://github.com/travis-ci/travis-ci/issues/4704
Since I can't seem to let go of this, I'll probably try to make a simple reproducer and maybe even dig into the code a bit. (But I have to say I don't know enough about ptys to be of immediate help here.)
On 25.08.2015 08:10, Marius Vollmer wrote:
Stef Walter stefw@redhat.com writes:
A little bird also whispered in my ear that Marius has found a work-around for the Travis bug that has been bothering us.
Yes, see here https://github.com/travis-ci/travis-ci/issues/4704
Since I can't seem to let go of this, I'll probably try to make a simple reproducer and maybe even dig into the code a bit. (But I have to say I don't know enough about ptys to be of immediate help here.)
For this case here, a PTY acts a lot like a socketpair(). When you create it you get two file descriptors both readable/writable. The process is connected to one side, and the other side is read/written by the terminal (Travis logging system in this case). As you may expect, there's no concept of stderr in a PTY ... it's all one output stream (eg: it's all mixed in your terminal).
Stef
cockpit-devel@lists.fedorahosted.org