Resending to keep things public...
tl;dr; of the tl;dr;
Tim and I/lbrabec shared the same worry, that by being overly-facilitating
to docker, we might go into the spiral of doom, where in the end we'd have
to have and support specific tooling for all of the projects.
Using pytest for docker testing seemed silly to me, since I thought that we
should be adding a layer of docker-specific convenience code, thus going
into the spiral of doom, instead of using some pre-existing docker
Tim was worried that using Tutum (docker specific tool) sends us down the
In the end, we agreed that Taskotron should be first and foremost an
universal runner. Tim mentioned pytest (AFAIK) because we will need to be
able to consume some more "standard" output format than result yaml (and
return code), and pytest might be a good source of this, while at the same
time providing kind of OK testsuite-like behavior.
For the problem at hand (but this is universal for the future problems
too), we say:
1) we understand these output formats (result yaml, return code, in the
furture probably maybe junit)
2) write the test any way you want (bash to compiled C), and as long as
you provide one of the options from #1 as output, we're fine
3) if we did the tests, we'd do it this way: [insert foobar test using
docker containers], but #1 and #2 are still valid
Where #3 is the "reference implementation" aka "simple piece of code, that
shows how we'd do it, but is by no means binding".
We do not know, nor say, that pytest is _the_ tool for docker testing, and
we realistically expect that most of the tests will be just a bash script,
that will do what's necessary, and we (lbrabec) will just try and provide
_some_ reference implementation for a docker test in taskotron.
Tim, If I left something out, or messed up, please correct me. I'm going to
have a beer now...
(04:51:29 PM) lbrabec: welcome guys :)
(04:51:45 PM) tflink: is this the place where all the docker things are
(04:52:04 PM) jskladan: and where we burn the witches tooo
(04:52:16 PM) tflink: this is an acceptable solution to docker
(04:52:38 PM) tflink: how do we test docker? we burn witches!
(04:54:46 PM) jskladan: so ad pytest & docker - I don't really see the
profit of making the image maintainers write tests in pytest - although I
get the "it is a testsuite" argument, docker is mostly interfaced with via
command line, and that is one of the worst things to do in Python.
(04:54:46 PM) jskladan: On top of that - why force a choice on the
maintainers, instead of just allowing them to "write a test(suite)" any way
they want (heck, even using pytest, if that's their choice), and just
running it as a regular task via taskotron?
(04:55:09 PM) tflink: who's forcing anyone to do anything?
(04:55:50 PM) jskladan: then I'm misunderstanding, and still don't
understand why should _we_ use pytest in any way
(04:55:52 PM) tflink: eh, it can be wrapped to be much less painful
(04:56:22 PM) tflink: because it offers a default option so there's not so
much overwhelming choice
(04:57:30 PM) tflink: if there's an easy(ish) default, that's what many
people will end up using
(04:57:45 PM) jskladan: from my POW, I don't see why we should treat Docker
testing any different than, ie. package-specific tests
(04:58:13 PM) tflink: which is why i was suggesting that we look into
something more generic
(04:58:34 PM) jskladan: more generic than what?
(04:58:45 PM) tflink: not something specific to docker
(04:59:11 PM) tflink: something that allows grouping of commands/actions
into test cases and makes reporting results easy for users
(05:00:32 PM) tflink: why would having everyone come up with their own
solution for that use case be better?
(05:00:42 PM) lbrabec: i always thought that the generic thing is
taskotron, and we are going to provide docker directive, that runs the
(05:00:55 PM) tflink: we can
(05:00:59 PM) jskladan: ok, let me ask it in a different way - are we going
to "remove the overwhelming choice" for the package-specific tests too?
(05:01:31 PM) tflink: but there's a limit to what we're going to be able to
support if we do "this is for docker, this is for kubernetes, this is for
(05:01:38 PM) tflink: jskladan: I'd like to, yes
(05:02:05 PM) jskladan: so who's going to tell all the devs "well, what you
have now is nice, but you should really rewrite it to pytest"
(05:02:05 PM) jskladan: ?
(05:02:31 PM) tflink: but note that the base thing in my mind is "this is a
default which will likely make your lives easier. if there's a better tool,
please use it - if it returns something we can understand, it doesn't
matter but we won't be able to help as much if you run into issues"
(05:02:51 PM) tflink: nobody?
(05:03:33 PM) tflink: the target folks here are people writing tests for
things beyond what's already upstream
(05:04:03 PM) tflink: there is no way that we're going to be able to
mandate a single framework/runner/etc.
(05:04:08 PM) tflink: well, we could
(05:04:23 PM) tflink: but i don't think it would work and I'd consider that
a form of failure
(05:05:55 PM) tflink: my emphasis is that people can use pretty much
whatever they want. so long as it can be run on the command line and
returns some form of input we can parse (xunit xml, resultsyaml, tap ...),
I don't much care
(05:06:26 PM) jskladan: so, I think that on some fundamental level, we do
agree :) I'm just not sure where (if anywhere) we diverge
(05:07:58 PM) jskladan: since, what I though is the solution, was to just
say "write a script, we'll run it", and that's it. What we wanted the
for was to remove the burden of us making the tooling to interface with
(05:08:27 PM) jskladan: like, even though it does not really make much
sense to me, you might want to run a command inside the docker container
(05:08:47 PM) tflink: if it works really well, using tutum for that is an
(05:08:58 PM) jskladan: so instead of us trying to put together the code
for it, we'd just use that code that exists
(05:09:15 PM) jskladan: whether we should make the tooling around it is
(05:09:16 PM) tflink: which is always a balancing act, in my opinion
(05:09:38 PM) jskladan: but on the fundamental level, my idea of what
"Testing docker images" means for us is functional testing, basically
(05:09:50 PM) jskladan: you'd run the container, and then try to connect to
(05:09:54 PM) tflink: yep
(05:10:09 PM) tflink: maybe even getting into "here are 3 containers, make
sure they work together"
(05:10:58 PM) tflink: the docker ecosystem worries me more than most others
due to it's tendency to change drastically
(05:11:39 PM) tflink: which tempts me to say that it might be wise to write
our own interfaces and keep them as simple as possible
(05:12:08 PM) jskladan: yup - it's like - I would not want to write a code
for them to set up the three containers, and connect them, and so on - that
would mean that we are in need of a declaratory language, to say what's to
(05:12:15 PM) jskladan: and the fact is - docker already has that
(05:12:17 PM) jskladan: the compose files
(05:12:58 PM) tflink: ok
(05:13:31 PM) jskladan: tflink: ad "write our own interfaces and keep them
as simple as possible" - that's what we already do, IMHO, via the formulae
(05:13:40 PM) jskladan: and result yaml
(05:13:59 PM) jskladan: input: any executable, output: result yaml
(05:14:00 PM) jskladan: done :)
(05:14:23 PM) tflink: and I'm not arguing against that
(05:14:30 PM) jskladan: and I understand that adding "something else,
hopefully standard" to output is a good idea
(05:14:44 PM) tflink: to output?
(05:14:53 PM) jskladan: to the output standard
(05:15:12 PM) tflink: yeah, we're going to have to do that eventually
(05:15:14 PM) jskladan: so it's not result yaml, but also ie. pytest's
(05:15:31 PM) jskladan: where's what I see is probably the gain you feel
(05:16:10 PM) tflink: but you don't think that the idea of having a
"default choice" for writing tests is a good idea?
(05:16:13 PM) jskladan: and by pytest's output, I really do mean junit, or
however is the "defacto standard" called
(05:16:40 PM) tflink: i figure we'll find a few options and specify them as
"stuff we understand"
(05:17:12 PM) *garretraziel left the room (quit: Quit: garretraziel).*
(05:18:13 PM) jskladan: well, I think that we should really just define the
interface, not provide the "default choice", but the border between the two
is probably blurred
(05:18:20 PM) ***tflink figured that having a "default" with a bunch of
examples would be helpful overall so long as there's no "first class
citizen" stuff involved
(05:18:51 PM) tflink: why are those two things mutually exclusive?
(05:19:15 PM) jskladan: I'm not saying they are
(05:19:56 PM) tflink: hrm, let me try rephrasing things
(05:20:10 PM) tflink: would you have the same objection if the "default
option" wasn't part of libtaskotron?
(05:21:35 PM) jskladan: *shrugs* I probably don't understand the question
(05:23:16 PM) tflink: is your concern with libtaskotron growing too many
arms? ie, trying to be too many things and that the paradigm of "we set up,
you execute" could be hurt by having too many different things?
(05:23:25 PM) jskladan: but we have diverged a bit, I guess so when you
said "usiepytest to test docker" - as I understand it now, you did not mean
that we should write a codebase/library to make testing docker stuff using
pytest easier (like pre-coding the fixtures _we_ think are usefull, or
something like that) - beacause that would go against 'but there's a limit
to what we're going to be able to support if we do "this is for docker,
this is for kubernetes, this is for modules ..."'
(05:23:26 PM) tflink: that's an odd question, i'll rephrase
(05:24:01 PM) jskladan: tflink: yup, I guess it could be said like that
(05:24:51 PM) jskladan: I think that we basically share the same worries -
I don't want to get to the point where we over-facilitate for all of the
(05:25:11 PM) tflink: which makes sense. would you have the same concern if
the pytest thingy was separate from libtaskotron?
(05:25:37 PM) tflink: so it was a "here's an example of something that
works with libtaskotron, you can use it or just make sure your tool behaves
close enough to it"
(05:25:49 PM) jskladan: that does not mean, though, that I don't see value
in package-specific testing, or what you discussed with kamil lately (the
CI for our repos thing),
(05:26:08 PM) tflink: yeah, I see these as somewhat separate concerns
(05:26:18 PM) jskladan: exactly
(05:27:50 PM) jskladan: ad "here's example of something..." - I really
don't have a problem with that. My concern with "pytest for docker" was
that we are to bend pytest fo make docker testing easier (than something
else, undefined) to be honest
(05:28:09 PM) jskladan: and if I understand your problem with tutum
correctly now, we basically share the same worries
(05:28:31 PM) jskladan: that by using something too specific, we are going
into the spiral of doom
(05:28:36 PM) jskladan: of specific things
(05:28:57 PM) tflink: yeah, it was less bending pytest to work better for
docker than wondering if the docker-specific things are worth dealing with
or if we should just have a default that works well enough
(05:29:00 PM) tflink: yep
(05:29:26 PM) jskladan: it was just that I thought that you want to add
(docker specific) convenience layer over pytest, and I wanted to use
something that's there (tutum) isntead of us being responsible for the
(05:29:32 PM) jskladan: especially since docker changes day to day
(05:29:59 PM) tflink: maybe in starting up images/instances
(05:30:06 PM) jskladan: and this was the concern lbrabec voiced quite a lot
- that we should not have our own convenience code, since docker changes
all the time
(05:30:34 PM) tflink: ie, make it reasonable for libtaskotron to start
images and hand off stuff to whichever task is running
(05:30:52 PM) tflink: the more I'm thinking about it, the more I like the
idea of having a reference implementation of a testing tool
(05:31:46 PM) tflink: "here's the tool, here's how it interacts with
libtaskotron. you're welcome to use it, extend it or use something
different that behaves close enough to it"
(05:32:33 PM) tflink: but to be a bit more clear (even if I am topic
jumping a bit) - I agree with trying to avoid the "spiral of death" that
you mentioned earlier
(05:32:43 PM) tflink: trying to be all things to all people is a recipe for
disaster and failure
(05:33:18 PM) jskladan: agreed
(05:35:15 PM) tflink: what are your thoughts on the reference
(05:36:11 PM) jskladan: so to sum it up, for docker (not only, but it's the
problem at hand), we could say:
(05:36:11 PM) jskladan: 1) we understand these output formats (result yaml,
return code, in the furture probably maybe junit)
(05:36:11 PM) jskladan: 2) write the test any way you want (bash to
compiled C), and as long as you provide one of the options from #1 as
output, we're fine
(05:36:11 PM) jskladan: 3) if we did the tests, we'd do it this way:
[insert foobar test using docker containers], but #1 and #2 are still valid
(05:37:14 PM) jskladan: where #3 is the "reference implementation" aka
"proof of concept code" aka "magic land full of unicorns"
(05:37:26 PM) jskladan: for the problem at hand
(05:38:08 PM) tflink: yeah, that sounds good
(05:38:30 PM) tflink: i suspect that it would be wise to keep 3 as dumb as
(05:38:30 PM) jskladan: I'm still not convinced that pytest is the way to
go for docker specifically, but hey, that's an implementation issue now
(05:38:49 PM) tflink: pytest was just an example, I have no idea if it
would be a good choice for docker
(05:38:52 PM) jskladan: yup
(05:38:55 PM) jskladan: superb!
(05:39:08 PM) tflink: I really don't know that much about docker
(05:39:20 PM) tflink: so if i'm suggesting silly things, I assume you all
will let me know :)
(05:39:36 PM) jskladan: yeah, and I get what you meant by that now, and I
understand the concerns you have/had
(05:40:11 PM) tflink: it was really a "i don't think anyone knows what
'docker testing' means right now, what can we use for a generic,
cover-our-asses solution so we're not caught unprepared?"
(05:40:13 PM) jskladan: I was just having an aggro on "using pytest for
docker testing is silly", and did not understand where you're coming from
(05:40:21 PM) tflink: no worries
(05:40:30 PM) jskladan: sure thing
(05:40:57 PM) tflink: lbrabec: does this make more sense for you as well?
(05:41:17 PM) jskladan: and I belive we all shared the same idea - that was
the reason for Lukas going the tutum way - since Docker "bought" it, and it
removed the "Docker is changing all the time, and we are responsible" issue
(05:41:56 PM) lbrabec: lbrabec: yep x)
(05:42:17 PM) jskladan: lbrabec: compass making you talk to yourself?
(05:42:18 PM) jskladan: :D
(05:42:28 PM) lbrabec: kind of :D
(05:42:33 PM) tflink: that sounds about right
(05:42:52 PM) tflink: compass causing folks to talk to themselves
(05:43:34 PM) tflink: but docker or arm could do that, too :)
(05:44:29 PM) tflink: in case it's not clear - there's a decent number of
things going on which are pretty hazy
(05:44:41 PM) tflink: if something doesn't make sense, speak up, please
(05:45:11 PM) tflink: too many meetings can cause questionable ideas to
(05:49:07 PM) jskladan: :)
(05:50:09 PM) jskladan: well I'm glad this is sorted out, and to be honest
I/(us in the office) needed an hour of so of a heated discussion just to
wrap our heads around the "docker testing" as a whole
(05:50:26 PM) jskladan: and I'
(05:50:59 PM) tflink: if you've wrapped your head around docker testing,
you may be ahead of me :)
(05:51:27 PM) jskladan: hehe, I would not go that far, but we at least
tried to define what we expect of it, and what we thing the use-cases might
(05:51:51 PM) jskladan: like, that running some testsuite _inside_ the
container, is probably not the way to go
(05:52:04 PM) jskladan: and that it should be more of an
(05:52:15 PM) tflink: yeah, i can see use cases for that but I'm not very
interested in those
(05:52:24 PM) jskladan: in the "run the container, check that the service
[http, postgres] works fine"
(05:52:50 PM) jskladan: or maybe "run these two containers, connect them,
and make sure it works ok"
(05:52:58 PM) tflink: yep
(05:53:08 PM) jskladan: but that we should keep this on the test
developers, since they know what to do
(05:53:24 PM) jskladan: and I think most of the "run and connect stuff"
will just be bash scripts
(05:53:29 PM) jskladan: based on docker behaves
(05:53:30 PM) tflink: exactly
(05:53:43 PM) jskladan: *how docker behaves
(05:53:43 PM) tflink: I don't pretend to be an expert in docker testing,
kernel testing or much else
(05:53:47 PM) jskladan: yeah
(05:54:03 PM) jskladan: ok, superb, I'm really happy we got to the same page
(05:54:07 PM) tflink: I'm more interested in providing a way to make
running automated tests and getting results easy
(05:54:16 PM) tflink: yeah, same here
(05:54:22 PM) tflink: sorry for not explaining things better
(05:54:42 PM) jskladan: no problem, really, it's the problem with written
(05:55:32 PM) tflink: and not being in the same office
(05:55:43 PM) tflink: which is pretty much the same problem :)
(05:56:03 PM) jskladan: do you have anything other we should discuss,
related to this? If not, I'll be leaving the office soon to get some
dinner, and beer. A lot of beer :D
(05:56:17 PM) tflink: nothing for today
(05:56:23 PM) tflink: beer sounds like a good plan to me
(05:56:34 PM) tflink: a natural consequence of working with docker :)
(05:56:46 PM) tflink: or test automation, come to think of it
(05:58:59 PM) jskladan: :)