#383: Depcheck and Upgradepath crash if yourls is not configured ----------------------+----------------------------------------------------- Reporter: tflink | Owner: Type: defect | Status: new Priority: critical | Milestone: 0.7.0 Component: core | Keywords: ----------------------+----------------------------------------------------- If yourls is not configured (empty configuration file), depcheck and upgrade path crash at the end of the test when URL shortening is attempted.
There is an example of this in the staging environment: http://autoqa- stg.fedoraproject.org/results/4194-autotest/virt24.qa/debug/client.0.DEBUG
{{{ 11/11 15:28:23 INFO | test:0358| Log created: /usr/share/autotest/results/default/depcheck/results/wicd- kde-0.2.3-3.fc1.html 11/11 15:28:23 ERROR| test:0473| -------------------------------------------------------------------------------- 11/11 15:28:23 ERROR| test:0473| Traceback (most recent call last): 11/11 15:28:23 ERROR| test:0473| File "/usr/lib/python2.7/site- packages/autoqa/decorators.py", line 72, in newf 11/11 15:28:23 ERROR| test:0473| f_result = f(*args, **kwargs) #call the decorated function 11/11 15:28:23 ERROR| test:0473| File "/usr/share/autotest/tests/depcheck/depcheck.py", line 362, in run_once 11/11 15:28:23 ERROR| test:0473| self.post_results(td, bodhi = {'title': title}) 11/11 15:28:23 ERROR| test:0473| File "/usr/lib/python2.7/site- packages/autoqa/test.py", line 231, in post_results 11/11 15:28:23 ERROR| test:0473| self._send_email(test_detail, optin=optin, logname=logname) 11/11 15:28:23 ERROR| test:0473| File "/usr/lib/python2.7/site- packages/autoqa/test.py", line 292, in _send_email 11/11 15:28:23 ERROR| test:0473| logname=logname) 11/11 15:28:23 ERROR| test:0473| File "/usr/lib/python2.7/site- packages/autoqa/test.py", line 334, in _create_email_body 11/11 15:28:23 ERROR| test:0473| url = self._shorten_url(self.url['result_dir'] + urllib.quote(logname), test_detail) 11/11 15:28:23 ERROR| test:0473| File "/usr/lib/python2.7/site- packages/autoqa/test.py", line 179, in _shorten_url 11/11 15:28:23 ERROR| test:0473| url = self._yourls_client.shorten(url) 11/11 15:28:23 ERROR| test:0473| File "/usr/lib/python2.7/site- packages/yourls/client.py", line 104, in shorten 11/11 15:28:23 ERROR| test:0473| raw_data = json.loads(self._send_request(args)) 11/11 15:28:23 ERROR| test:0473| File "/usr/lib/python2.7/site- packages/yourls/client.py", line 81, in _send_request 11/11 15:28:23 ERROR| test:0473| r = urllib2.urlopen(req) 11/11 15:28:23 ERROR| test:0473| File "/usr/lib64/python2.7/urllib2.py", line 126, in urlopen 11/11 15:28:23 ERROR| test:0473| return _opener.open(url, data, timeout) 11/11 15:28:23 ERROR| test:0473| File "/usr/lib64/python2.7/urllib2.py", line 384, in open 11/11 15:28:23 ERROR| test:0473| protocol = req.get_type() 11/11 15:28:23 ERROR| test:0473| File "/usr/lib64/python2.7/urllib2.py", line 245, in get_type 11/11 15:28:23 ERROR| test:0473| raise ValueError, "unknown url type: %s" % self.__original 11/11 15:28:23 ERROR| test:0473| ValueError: unknown url type: }}}
This may end up being a bug in python-yourls but that's not clear yet.
#383: Depcheck and Upgradepath crash if yourls is not configured ----------------------+----------------------------------------------------- Reporter: tflink | Owner: Type: defect | Status: new Priority: critical | Milestone: 0.7.0 Component: core | Resolution: Keywords: | ----------------------+----------------------------------------------------- Comment (by tflink):
Well, my dev setup of AutoQA is completely borked right now and I can't figure out how to fix it without a complete re-setup of autotest. If someone else wants to look into this, feel free. Otherwise, I'll look at it next week.
#383: Depcheck and Upgradepath crash if yourls is not configured ----------------------+----------------------------------------------------- Reporter: tflink | Owner: Type: defect | Status: new Priority: critical | Milestone: 0.7.0 Component: core | Resolution: Keywords: | ----------------------+----------------------------------------------------- Comment (by mkrizek):
Quick patch: http://fpaste.org/TS8e/ (also I pushed patch fixing missing import)
The problem/feature seems to be that python-yourls checks args whether they are None and therefore accepts empty strings (those are in theory valid). So options are the patch or, as Kamil suggested, checking this in python-yourls.
#383: Depcheck and Upgradepath crash if yourls is not configured ----------------------+----------------------------------------------------- Reporter: tflink | Owner: Type: defect | Status: new Priority: critical | Milestone: 0.7.0 Component: core | Resolution: Keywords: | ----------------------+----------------------------------------------------- Comment (by tflink):
Yeah, that needs to be fixed in python-yourls - empty urls are not valid. I'll push a fix and re-build.
#383: Depcheck and Upgradepath crash if yourls is not configured ----------------------+----------------------------------------------------- Reporter: tflink | Owner: Type: defect | Status: new Priority: critical | Milestone: 0.7.0 Component: core | Resolution: Keywords: | ----------------------+----------------------------------------------------- Comment (by tflink):
for the sake of documentation, filed "upstream"
https://github.com/tflink/python-yourls/issues/2
#383: Depcheck and Upgradepath crash if yourls is not configured ----------------------+----------------------------------------------------- Reporter: tflink | Owner: tflink Type: defect | Status: assigned Priority: critical | Milestone: 0.7.0 Component: core | Resolution: Keywords: | ----------------------+----------------------------------------------------- Changes (by tflink):
* owner: => tflink * status: new => assigned
Comment:
python-yourls has been fixed and updated: * [https://admin.fedoraproject.org/updates/python-yourls-0.1.2-1.fc16 F16 Update] * [https://admin.fedoraproject.org/updates/python-yourls-0.1.2-1.fc15 F15 Update] * [https://admin.fedoraproject.org/updates/python-yourls-0.1.2-1.el6 EL6 Update]
#383: Depcheck and Upgradepath crash if yourls is not configured ----------------------+----------------------------------------------------- Reporter: tflink | Owner: tflink Type: defect | Status: assigned Priority: critical | Milestone: 0.7.0 Component: core | Resolution: Keywords: | ----------------------+----------------------------------------------------- Comment (by kparal):
I still receive a traceback when I fill in everything in yourls.conf except username.
{{{
12:00:55 ERROR| Traceback (most recent call last): 12:00:55 ERROR| File "/usr/lib/python2.7/site- packages/autoqa/decorators.py", line 72, in newf 12:00:55 ERROR| f_result = f(*args, **kwargs) #call the decorated function 12:00:55 ERROR| File "/usr/share/autotest/client/site_tests/rpmlint/rpmlint.py", line 85, in run_once 12:00:55 ERROR| self.post_results(optin={'pkg': name, 'kojitag': kojitag}) 12:00:55 ERROR| File "/usr/lib/python2.7/site-packages/autoqa/test.py", line 232, in post_results 12:00:55 ERROR| self._send_email(test_detail, optin=optin, logname=logname) 12:00:55 ERROR| File "/usr/lib/python2.7/site-packages/autoqa/test.py", line 293, in _send_email 12:00:55 ERROR| logname=logname) 12:00:55 ERROR| File "/usr/lib/python2.7/site-packages/autoqa/test.py", line 335, in _create_email_body 12:00:55 ERROR| url = self._shorten_url(self.url['result_dir'] + urllib.quote(logname), test_detail) 12:00:55 ERROR| File "/usr/lib/python2.7/site-packages/autoqa/test.py", line 180, in _shorten_url 12:00:55 ERROR| url = self._yourls_client.shorten(url) 12:00:55 ERROR| File "/usr/lib/python2.7/site- packages/yourls/client.py", line 109, in shorten 12:00:55 ERROR| if raw_data['status'] == 'fail': 12:00:55 ERROR| KeyError: 'status' }}}
Tim, please make following changes in yourls/client.py: * Test all arguments in __init__() for emptiness, not for None. Use "if not username" instead of "if username is None". The same for password, apiurl, etc. * Don't use sys.exit(), throw an exception instead. * Define a base exception YourlsError(StandardError) and inherit all other exceptions from it. We can then catch just this base exception if needed. * Make sure raw_data in shorten() contains required keys before accessing it, or guard the whole block with some YourlsError descendant exception.
After new release of python-yourls is released, we will adjust _get_yourls_client() and _shorten_url() in test.py to catch YourlsError exceptions and our test won't crash if something in python-yourls goes wrong.
#383: Depcheck and Upgradepath crash if yourls is not configured ----------------------+----------------------------------------------------- Reporter: tflink | Owner: tflink Type: defect | Status: assigned Priority: critical | Milestone: 0.7.0 Component: core | Resolution: Keywords: | ----------------------+----------------------------------------------------- Comment (by kparal):
Preliminary patch attached. It seems to work, but I don't have much insight into communication with yourls.
#383: Depcheck and Upgradepath crash if yourls is not configured ----------------------+----------------------------------------------------- Reporter: tflink | Owner: tflink Type: defect | Status: assigned Priority: critical | Milestone: 0.7.0 Component: core | Resolution: Keywords: | ----------------------+----------------------------------------------------- Comment (by tflink):
Error handling? What is this error handling nonsense of which you speak? ... I kid, I kid.
Anyhow, I've added or improved the error handling thoughout python-yourls and pushed updates for F15, F16 and EL6. They should hit updates-testing @ the next push. * https://admin.fedoraproject.org/updates/python-yourls-0.2.0-1.fc15 * https://admin.fedoraproject.org/updates/python-yourls-0.2.0-1.fc16 * https://admin.fedoraproject.org/updates/python-yourls-0.2.0-1.el6
I also submitted a patch to reviewboard to update AutoQA to use the slightly changed exception mechanisms in python-yourls 0.2.0. The patch also simplifies the yourls.conf a bit by combining the baseurl and apiurl configuration options in addition to removing the urltitle option (which wasn't used). * https://fedorahosted.org/reviewboard/r/242
Once other have had the chance to look at the code, we can push to master and do another build (either another pre-release or go for 0.7.0).
If you try this patch, make sure you have python-yourls 0.2.0. Crashing and burning at initialization will result otherwise - the same is true about updating python-yourls without using my patch or something similar.
Once other have had the chance to look at the code, we can push to master and do another build (either another pre-release or go for 0.7.0).
Tim,
are we able to deploy arbitrary git commit to staging, or do we have to tag it as "pre-release" every time we want to update staging?
On Wed, 23 Nov 2011 04:11:28 -0500 (EST) Kamil Paral kparal@redhat.com wrote:
Once other have had the chance to look at the code, we can push to master and do another build (either another pre-release or go for 0.7.0).
Tim,
are we able to deploy arbitrary git commit to staging, or do we have to tag it as "pre-release" every time we want to update staging?
I've not been tagging anything in git, just updating the spec and pushing to master.
Am I missing part of the process somewhere?
Tim
I don't think that you're missing something. This is probably just an communication issue - IMHO Kamil thought that you have some "automated deployment" tool in your hands, which will always checkout and update staging to the latest 'pre-release' tag.
j.
----- Original Message -----
From: "Tim Flink" tflink@redhat.com To: "AutoQA development" autoqa-devel@lists.fedorahosted.org Sent: Wednesday, November 23, 2011 3:44:46 PM Subject: Re: Deploying to staging
On Wed, 23 Nov 2011 04:11:28 -0500 (EST) Kamil Paral kparal@redhat.com wrote:
Once other have had the chance to look at the code, we can push to master and do another build (either another pre-release or go for 0.7.0).
Tim,
are we able to deploy arbitrary git commit to staging, or do we have to tag it as "pre-release" every time we want to update staging?
I've not been tagging anything in git, just updating the spec and pushing to master.
Am I missing part of the process somewhere?
Tim
autoqa-devel mailing list autoqa-devel@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/autoqa-devel
On Wed, 23 Nov 2011 09:53:02 -0500 (EST) Josef Skladanka jskladan@redhat.com wrote:
I don't think that you're missing something. This is probably just an communication issue - IMHO Kamil thought that you have some "automated deployment" tool in your hands, which will always checkout and update staging to the latest 'pre-release' tag.
j.
I wish I had an automated tool for that but haven't had the time to write anything.
BTW, I'm planning to update staging to 0.7.0-0.3.pre as soon as python-yourls-0.2.0-1.el6 hits updates-testing.
Tim
Sorry for not being clear for the first time. My question is:
Do we have to tag the commit we want to deploy to staging (like 0.7.0-0.pre3), or are we able to do that without using tags (just building RPM from arbitrary commit ABC123 and installing)?
Because I don't really see the point in tagging pre-releases (it just clutters the view), if it is not required for the deployment process.
Sorry for not being clear for the first time. My question is:
Do we have to tag the commit we want to deploy to staging (like 0.7.0-0.pre3), or are we able to do that without using tags (just building RPM from arbitrary commit ABC123 and installing)?
Because I don't really see the point in tagging pre-releases (it just clutters the view), if it is not required for the deployment process.
Oh I got confused. Git tagging is not needed for deployment to staging, but autoqa.spec bumping seems to be. Can we do it without bumping .spec version?
On Wed, 23 Nov 2011 11:31:32 -0500 (EST) Kamil Paral kparal@redhat.com wrote:
Sorry for not being clear for the first time. My question is:
Do we have to tag the commit we want to deploy to staging (like 0.7.0-0.pre3), or are we able to do that without using tags (just building RPM from arbitrary commit ABC123 and installing)?
Because I don't really see the point in tagging pre-releases (it just clutters the view), if it is not required for the deployment process.
Oh I got confused. Git tagging is not needed for deployment to staging, but autoqa.spec bumping seems to be. Can we do it without bumping .spec version?
If I did a 'yum reinstall' instead of a 'yum update', sure. Another option would be to change the spec locally instead of pushing changes to the git repo.
I'm fine with either, though. The pre-release changes to the spec do seem rather noisy, though.
Tim
#383: Depcheck and Upgradepath crash if yourls is not configured ----------------------+----------------------------------------------------- Reporter: tflink | Owner: tflink Type: defect | Status: assigned Priority: critical | Milestone: 0.7.0 Component: core | Resolution: Keywords: | ----------------------+----------------------------------------------------- Comment (by kparal):
Reviewed.
#383: Depcheck and Upgradepath crash if yourls is not configured ----------------------+----------------------------------------------------- Reporter: tflink | Owner: tflink Type: defect | Status: assigned Priority: critical | Milestone: 0.7.0 Component: core | Resolution: Keywords: | ----------------------+----------------------------------------------------- Comment (by tflink):
Replying to [comment:9 kparal]:
Reviewed.
I added KeyError back in to the caught exceptions, applied the patch and updated the spec to 0.7.0-0.3.pre.
I'll update the documentation to reflect the changes in configuration and will update staging when python-yourls-0.2.0-1 hits updates-testing at the next push.
#383: Depcheck and Upgradepath crash if yourls is not configured ----------------------+----------------------------------------------------- Reporter: tflink | Owner: tflink Type: defect | Status: assigned Priority: critical | Milestone: 0.7.0 Component: core | Resolution: Keywords: | ----------------------+----------------------------------------------------- Comment (by kparal):
It doesn't really matter whether python-yourls is in updates-testing or not. Unless it is in stable updates, you'll have to manually update all the clients anyway.
(That raises an interesting question whether we should enable updates- testing on staging clients by default, at least for autoqa dependencies).
#383: Depcheck and Upgradepath crash if yourls is not configured ----------------------+----------------------------------------------------- Reporter: tflink | Owner: tflink Type: defect | Status: closed Priority: critical | Milestone: 0.7.0 Component: core | Resolution: fixed Keywords: | ----------------------+----------------------------------------------------- Changes (by kparal):
* status: assigned => closed * resolution: => fixed
Comment:
Closing, problem solved.
autoqa-devel@lists.fedorahosted.org