On Wed, Feb 8, 2017 at 2:26 PM, Kamil Paral <kparal@redhat.com> wrote:
This is what I meant - keeping item as is, but being able to pass another structure to the formula, which can then be used from it. I'd still like to keep the item to a single string, so it can be queried easily in the resultsdb. The item should still represent what was tested. It's just that I want to be able to pass arbitrary data to the formulae, without the need for ugly hacks like we have seen with the git commits lately.

So, the question is now how much we want the `item` to uniquely identify the item under test. Currently we mostly do (rpmlint, rpmgrill) and sometimes don't (depcheck, because item is NVR, but the full ID is NEVRA, and we store arch in the results extradata section).


I still kind of believe that the `item` should be chosen with great respect to what actually is the item under test, but it also really depends on what you want to do with it later on. Note that the `item` is actually a convention (yay, more water to adamw's "if we only had some awesome new project" mill), and is not enforced in any way. I believe that there should be firm rules (once again - conventions) on what the item is for each "well known" item type, so you can kind-of assume that if you query for `item=foo&type=koji_build` you are getting the results related to that build.
As we were discussing privately with the item types (I'm not going to go into much detail here, but for the rest of you guys - I'm contemplating making the types more general, and using more of the 'metadata' to store additional spefics - like replacing `type=koji_build` with `type=build, source=koji`, or `type=build, source=brew` - on the high level, you know that a package/build was tested, and you don't really care where it came from, but you sometimes might care, and so there is the additional metadata stored. We could even have more types stored for one results, or I don't know... It's complicated), the idea behind item is that it should be a reasonable value, that carries the "what was tested" information, and you will use the other "extra-data" fields to provide more details (like we kind-of want to do with arch, but we don't really..). The reason for it to be 'reasonable value" and not "superset of all values that we have" is to make the general querying a bit more straightforward.
 
If we have structured input data, what happens to `item` for check_modulemd? Currently it is "namespace/module#commithash". Will it stay the same, and they'll just avoid parsing it because we'll also provide ${data.namespace}, ${data.module} and ${data.hash}? Or will the `item` be perhaps just "module" (and the rest will be stored as extradata)? What happens when we have a generic git_commit type, and the source can be an arbitrary service? Will have some convention to use item as "giturl#commithash"?


Once again - whatever makes sense as the item. For me that would be the Repo/SHA combo, with server, repo, branch, and commit in extradata.
And it comes to "storing as much relevant metadata as possible" once again. The thing is, that as long as stuff is predictable, it almost does not matter what it is, and it once again points out how good of an idea is the conventions stuff. I believe that we are now storing much less metadata in resultsdb than we should, and it is caused mostly by the fact that
 - we did not really need to use the results much so far
 - it is pretty hard to pass data into libtaskotron, and querying all the services all the time, to get the metadata, is/was deemed a bad idea - why do it ourselves, if the consumer can get it himself. They know that it is koji_build, so they can query koji.

There is a fine balance to be struck, IMO, so we don't end up storing "all the data" in resultsdb. But I believe that the stuff relevant for the result consumption should be there.


Because the ideal way would be to store the whole item+data structure as item in resultsdb. But that's hard to query for humans, so we want a simple string as an identifier.

This, for me, is once again about being predictable. As I said above, I still think that `item` should be a reasonable identifier, but not necessary a superset of all the info. That is what the extra data is for. Talking about...
 
But sometimes there can be a lot of data points which uniquely identify the thing under test only when you specify it all (for example what Dan wrote, sometimes the ID is the old NVR *plus* the new NVR). Will we want to somehow combine them into a single item value? We should give some directions how people should construct their items.


My gut feeling here would be storing the "new NVR" (the thing that actually caused the test to be executed) as item, and adding 'old nvr' to extra data. But I'm not that familiar with the specific usecase. To me, this would make sense, because when you query for "this NVR related results" you'd get the results too. If you wanted to have the specifics, then you could narrow that search by using the relevant extra-data key-vals.


Once, for each item :)  If a task developer wants to execute his task on a NVR, he'll need to prepare the structured input data for each NVR he wants to test. That might not be difficult, but it's more work than it is currently, so we should know whether we're fine with that. I guess we are, but there can be some gotchas, e.g. sometimes it might not be obvious how to get some extra data. For example:

nvr = htop-1.0-2.fc25
inputdata = {name: htop; epoch: 0; build_id: 482735}


I think that you are, once again, being overly defensive. The task does not _need_ to use/access all the data that the 'item'(type) can possible have associated with it. And the values the task needs, the task-developer should be able to provide. I mean - the developer knows, what he wants to test, right? And if not, is it a good idea for him to put the test together in the first place?

 
We'll probably end up having a mix of necessary and convenience values in the inputdata. "name" is probably a convenience value here, so that tasks don't have to parse if they need to use it in a certain directive. "epoch" might be an important value for some test cases, and let's say we learn the value in trigger during scheduling investigation, so we decide to pass it down. But that information is not that easy to get manually. If you know what to do, you'll open up a particular koji page and see it. But you can also be clueless about how to figure it out. The same goes for build_id, again can be important, but also can be retrieved later, so more of a convenience data (saving you from writing a koji query). This is just an example for illustration, might not match real-world use cases.


If you are clueless about getting information necessary for your test to run, you (IMO) should not be writing the test. Because that most probably mean, that you do not have the necessary expertise to design the test in the first place.

Makes sense?

I'm a bit torn between providing as much useful data as we can when scheduling (because a) yaml formulas are very limited and you can't do stuff like string parsing/splitting b) might save you a lot of work/code to have this data presented to you right from the start), and the easy manual execution (when you need to gather and provide all that data manually). It's probably about finding the right balance. We can't avoid having structured multi-data input, I don't think.


As I said already, I firmly believe that we should be providing as much reasonable, useful data as possible. Most of the tests will not need it all, and if they do, the maintainers/developers should know what the possible values are anyway.
It sure can be a minor pita to put together the first "testing" datafiles, but you only need to do it once.

Joza