#244: Find the best way how to use fas.conf --------------------+------------------------------------------------------- Reporter: kparal | Owner: Type: task | Status: new Priority: major | Milestone: 0.4.4 Component: core | Keywords: --------------------+------------------------------------------------------- For the purpose of sending comments to Bodhi, we need a configuration file fas.conf containing username and password of the autoqa user in the Fedora Account System. The problem is that it's not sufficient to have this configuration file defined on AutoQA server, because we need to send these comments from the clients. Find a best way how to achieve that. Suggestions:
1. fas.conf on every client - tedious and hard to maintain 1. fas.conf on some clients - we can use autotest labels to mark only a few clients as containing fas.conf; tests using fas.conf would then require these clients; fas.conf checksum may be compared to decide whether client's fas.conf is up-to-date 1. copy fas.conf from the server, but don't store it in the results dir - ideal, if autotest server supports this, need some investigation 1. ask some service to send a comment - it may run this on the server, but it's another layer, need to solve authentication and so on
This is probably a temporary hack until we have better solution based on ResultDB, so some simple solution is preferred.
#244: Find the best way how to use fas.conf --------------------+------------------------------------------------------- Reporter: kparal | Owner: Type: task | Status: new Priority: major | Milestone: 0.4.4 Component: core | Resolution: Keywords: | --------------------+------------------------------------------------------- Comment (by jlaska):
Replying to [ticket:244 kparal]:
- fas.conf on every client - tedious and hard to maintain
Not terrible really, and we are doing this already with irb.conf. So it's not perfect, but not that bad. And given the frequency that we provision test clients, it's not so tedious. I think we just need to decide whether the configuration information being recorded is test-specific ... or generally useful for all/other tests. I think it's the later, which means a per-test fas.conf is kind of silly. Since we now have /etc/autoqa/ I think we can move this from the test into that system-wide configuration directory.
- fas.conf on some clients - we can use autotest labels to mark only a
few clients as containing fas.conf; tests using fas.conf would then require these clients; fas.conf checksum may be compared to decide whether client's fas.conf is up-to-date
I don't know the benefits of this are over the #1. My sense is this adds extra complication to #1. But I might be missing some important details.
- copy fas.conf from the server, but don't store it in the results dir
- ideal, if autotest server supports this, need some investigation
Hmm, I don't know.
- ask some service to send a comment - it may run this on the server,
but it's another layer, need to solve authentication and so on
This is probably a temporary hack until we have better solution based on
ResultDB, so some simple solution is preferred.
I'd vote for #1 for simplicity. I have to play the role of "puppet" already, so I don't mind including deploying an updated /etc/autoqa/fas.conf configuration to all test clients.
#244: Find the best way how to use fas.conf --------------------+------------------------------------------------------- Reporter: kparal | Owner: Type: task | Status: new Priority: major | Milestone: 0.4.4 Component: core | Resolution: Keywords: | --------------------+------------------------------------------------------- Comment (by kparal):
Replying to [comment:1 jlaska]:
ad 2) I don't know the benefits of this are over the #1. My sense is this
adds extra complication to #1. But I might be missing some important details.
Well, the idea is that we don't have to maintain fas.conf on so many clients. Also if we add a new autotest client and forget to fill-in its fas.conf, some test requiring fas.conf may be scheduled on that client and then silently do nothing (not report anything). If we use autotest labels, this is more unlikely to happen.
- copy fas.conf from the server, but don't store it in the results
dir - ideal, if autotest server supports this, need some investigation
Hmm, I don't know.
Autotest server tarballs the whole test and then transfers it onto the client. So if we have a symlink in the test directory pointing at server's fas.conf, and if we manage autotest to archive the target file and not the symlink itself, it would work quite fine. But still, I looked at it very briefly.
I'd vote for #1 for simplicity. I have to play the role of "puppet"
already, so I don't mind including deploying an updated /etc/autoqa/fas.conf configuration to all test clients.
Originally I supposed we should do more complex solution, but as you say -- it's not really so much work after all, and it's just a temporary hack. If we don't find a better solution easily, I'd vote for #1 also.
#244: Find the best way how to use fas.conf --------------------+------------------------------------------------------- Reporter: kparal | Owner: kparal Type: task | Status: assigned Priority: major | Milestone: 0.4.4 Component: core | Resolution: Keywords: | --------------------+------------------------------------------------------- Changes (by kparal):
* owner: => kparal * status: new => assigned
Comment:
My findings:
* Autotest server transfers the whole test directory to the client. If we place any additional file into testdir on the server, it will also be available on the client. * Autotest uses tar to archive the whole test directory. I tried to place symlink to fas.conf into it, but tar by default archives the symlink, not the symlink target. * The code responsible for calling tar is in autotest/client/common_lib/base_packages.py (tar_package() method). * I have managed to add autotest/client/common_lib/site_packages.py which overrides default tar_package() method.
Two possible solutions: 1. In our site-specific tar_package() method let's add '-h' option to tar (de-referencing symlinks). The whole method code must be duplicated. 2. In our site-specific tar_package() let's forcibly copy /etc/autoqa/fas.conf to test dir and then call the parent tar_package() method. No code is duplicated.
Whichever solution we choose I think it's better than maintaining fas.conf on all autotest clients. The first solution seems cleaner to me although it duplicates the original code. Example code attached.
Solves ticket #244. Enables us to attach configuration files (like fas.conf) to individual tests just by placing a symlink there. --- Makefile | 1 + autoqa.spec | 1 + lib/autotest/site_packages.py | 32 ++++++++++++++++++++++++++++++++ 3 files changed, 34 insertions(+), 0 deletions(-) create mode 100644 lib/autotest/site_packages.py
diff --git a/Makefile b/Makefile index adbead8..ac4ddeb 100644 --- a/Makefile +++ b/Makefile @@ -30,6 +30,7 @@ install: build for t in tests/*; do cp -a $$t $(PREFIX)$(TEST_DIR); done install -d $(PREFIX)$(AUTOTEST_DIR)/client/bin install -m 0644 lib/autotest/site_utils.py $(PREFIX)$(AUTOTEST_DIR)/client/bin + install -m 0644 lib/autotest/site_packages.py $(PREFIX)$(AUTOTEST_DIR)/client/common_lib ( cd lib/python; $(PYTHON) setup.py install --skip-build --root $(PREFIX)/ ) ## front-ends/israwhidebroken install -d $(PREFIX)/usr/sbin diff --git a/autoqa.spec b/autoqa.spec index d30251a..261b20a 100644 --- a/autoqa.spec +++ b/autoqa.spec @@ -85,6 +85,7 @@ rm -rf $RPM_BUILD_ROOT %{testdir} %{hookdir} %{_datadir}/autotest/client/bin/site_utils.py* +%{_datadir}/autotest/client/common_lib/site_packages.py* %{python_sitelib}/autoqa*
diff --git a/lib/autotest/site_packages.py b/lib/autotest/site_packages.py new file mode 100644 index 0000000..445e916 --- /dev/null +++ b/lib/autotest/site_packages.py @@ -0,0 +1,32 @@ +# site_packages.py - site-specific autotest packages methods for use in autoqa +# +# this must be copied to autotest/client/common_lib, because it is imported by +# autotest/client/common_lib/packages.py +# +# Author: Kamil Paral kparal@redhat.com + +import os +from autotest_lib.client.common_lib import utils, base_packages + +class SitePackageManager(base_packages.BasePackageManager): + # same as common_lib.base_packages.tar_package, but with de-referencing of symlinks enabled + def tar_package(self, pkg_name, src_dir, dest_dir, exclude_string=None): + ''' + Create a tar.bz2 file with the name 'pkg_name' say test-blah.tar.bz2. + Excludes the directories specified in exclude_string while tarring + the source. Returns the tarball path. + ''' + tarball_path = os.path.join(dest_dir, pkg_name) + temp_path = tarball_path + '.tmp' + # AutoQA change -> add '-h' option to dereference symlinks (to be used with fas.conf, etc) + cmd = "tar -h -cvjf %s -C %s %s " % (temp_path, src_dir, exclude_string) + + try: + utils.system(cmd) + except: + os.unlink(temp_path) + raise + + os.rename(temp_path, tarball_path) + return tarball_path +
#244: Find the best way how to use fas.conf --------------------+------------------------------------------------------- Reporter: kparal | Owner: kparal Type: task | Status: assigned Priority: major | Milestone: 0.4.4 Component: core | Resolution: Keywords: | --------------------+------------------------------------------------------- Comment (by jlaska):
Replying to [comment:3 kparal]:
Whichever solution we choose I think it's better than maintaining
fas.conf on all autotest clients. The first solution seems cleaner to me although it duplicates the original code. Example code attached.
Nice detective work! I can't think of any reasons why we might need different config files on different client systems, can you? Meaning, maybe our F13 client systems have a slightly different config from our F14 systems etc...
With either solution, is the fas.conf available when viewing the test results? Or is the fas.conf only copied into the base directory for the test ... not the results?
Assuming it's not visible when viewing test results from the web-ui, I like your suggestion #2. Should we extend that support further to add an additional method that copies all autoqa config files from the autotest server to the test client before running a test? That would be a real nice improvement for the AutoQA sysadmin.
#244: Find the best way how to use fas.conf --------------------+------------------------------------------------------- Reporter: kparal | Owner: kparal Type: task | Status: assigned Priority: major | Milestone: 0.4.4 Component: core | Resolution: Keywords: | --------------------+------------------------------------------------------- Comment (by kparal):
Replying to [comment:4 jlaska]:
Nice detective work! I can't think of any reasons why we might need
different config files on different client systems, can you?
Certainly not for fas.conf.
Meaning, maybe our F13 client systems have a slightly different config
from our F14 systems etc...
autoqa.conf is currently only configured on the server and transferred to client hosts by writing it out into test's control file. That means we don't really have any configuration changes for F13/F14/etc.
However, this approach is not really usable for autoqa.conf, because we modify autoqa.conf on the fly before writing it out into the control file. It is really meant as a quick workaround for transferring fas.conf (static and shared config file) into the test dir on the client. Because fas.conf should be temporary, this workaround should also be temporary.
With either solution, is the fas.conf available when viewing the test
results?
No, at least I couldn't find it in the results.
Or is the fas.conf only copied into the base directory for the test ...
not the results?
That should be exactly the case.
Assuming it's not visible when viewing test results from the web-ui, I
like your suggestion #2.
Do you mean the second solution in comment 3? I personally think the first solution from comment 3 is better, because it allows us to add/remove/rename symlinks to config files without deploying patched autoqa version on the production machine.
Should we extend that support further to add an additional method that
copies all autoqa config files from the autotest server to the test client before running a test? That would be a real nice improvement for the AutoQA sysadmin.
We don't have any more config files apart from fas.conf and autoqa.conf (that is transferred other way), or do we (necessary for the client)? I don't know about them :)
If we need to copy more such files and do it non-temporarily, I would rather ask autotest developers whether they could add such feature for us.
Btw, the patch for solution 1 from comment 3 is in autoqa-devel now:[[BR]] https://fedorahosted.org/pipermail/autoqa-devel/2010-November/001368.html
#244: Find the best way how to use fas.conf --------------------+------------------------------------------------------- Reporter: kparal | Owner: kparal Type: task | Status: assigned Priority: major | Milestone: 0.4.4 Component: core | Resolution: Keywords: | --------------------+------------------------------------------------------- Comment (by jlaska):
Replying to [comment:5 kparal]:
However, this approach is not really usable for autoqa.conf, because we
modify autoqa.conf on the fly before writing it out into the control file. It is really meant as a quick workaround for transferring fas.conf (static and shared config file) into the test dir on the client. Because fas.conf should be temporary, this workaround should also be temporary.
Oh that's right we do modify autoqa.conf and send it to the client. Thanks.
Assuming it's not visible when viewing test results from the web-ui, I
like your suggestion #2.
Do you mean the second solution in comment 3? I personally think the
first solution from comment 3 is better, because it allows us to add/remove/rename symlinks to config files without deploying patched autoqa version on the production machine.
I'm just not sure whether the '-h' tar option was omitted intentionally by upstream autotest, that'd be my only hesitation with changing that core autotest behavior.
Should we extend that support further to add an additional method that
copies all autoqa config files from the autotest server to the test client before running a test? That would be a real nice improvement for the AutoQA sysadmin.
We don't have any more config files apart from fas.conf and autoqa.conf
(that is transferred other way), or do we (necessary for the client)? I don't know about them :)
repoinfo.conf, and irb.conf could benefit from this (although irb is kind of an unmaintained feature).
From an rpm packaging standpoint, I (and rpmlint) hate having config files live outside /etc. Additionally, I wonder if your second solution might be more future-proof since you're not replacing/duplicating the existing autotest tar() method, but simply calling it. Of course, either approach will do nicely ... and we'll adjust as needed.
#244: Find the best way how to use fas.conf --------------------+------------------------------------------------------- Reporter: kparal | Owner: kparal Type: task | Status: assigned Priority: major | Milestone: 0.4.4 Component: core | Resolution: Keywords: | --------------------+------------------------------------------------------- Comment (by kparal):
Replying to [comment:6 jlaska]:
Additionally, I wonder if your second solution might be more future-
proof since you're not replacing/duplicating the existing autotest tar() method, but simply calling it. Of course, either approach will do nicely ... and we'll adjust as needed.
Ok, I have attached a code that will copy all AutoQA config files into the source directory of our test before it is transferred to client, then it will standard Autotest methods for tarring and transferring these files and finally all those config files are removed from that test's directory. Any concerns?
There is one change needed, though. Currently /usr/share/autotest/client/site_tests/* are owned by root. It needs to be owned by autotest (this requires change in autoqa packaging). What do you think, James, is there some problem in that?
#244: Find the best way how to use fas.conf --------------------+------------------------------------------------------- Reporter: kparal | Owner: kparal Type: task | Status: assigned Priority: major | Milestone: 0.4.4 Component: core | Resolution: Keywords: | --------------------+------------------------------------------------------- Comment (by jlaska):
Replying to [comment:7 kparal]:
There is one change needed, though. Currently
/usr/share/autotest/client/site_tests/* are owned by root. It needs to be owned by autotest (this requires change in autoqa packaging). What do you think, James, is there some problem in that?
I can't think of anything off-hand. I've created ticket#250 to track the packaging change.
#244: Find the best way how to use fas.conf --------------------+------------------------------------------------------- Reporter: kparal | Owner: kparal Type: task | Status: closed Priority: major | Milestone: 0.4.4 Component: core | Resolution: fixed Keywords: | --------------------+------------------------------------------------------- Changes (by kparal):
* status: assigned => closed * resolution: => fixed
Comment:
Pushed to master as commit 1b1d73030401a5859751b871043c15a2f8877064. Thanks James, closing.
autoqa-devel@lists.fedorahosted.org