[copr] skvidal-backend: more todo's highlighting a bug (b336c6d)
by Seth Vidal
Repository : http://git.fedorahosted.org/cgit/copr.git
On branch : skvidal-backend
>---------------------------------------------------------------
commit b336c6d22bc863b44410089d936504d2b2065c2b
Author: Seth Vidal <skvidal(a)fedoraproject.org>
Date: Sat Dec 8 13:04:44 2012 -0500
more todo's highlighting a bug
>---------------------------------------------------------------
TODO-backend | 2 ++
1 files changed, 2 insertions(+), 0 deletions(-)
diff --git a/TODO-backend b/TODO-backend
index 058330e..4cc5ca1 100644
--- a/TODO-backend
+++ b/TODO-backend
@@ -15,3 +15,5 @@
- work on a way to find and cancel a specific build that's happening other than just killing the instance
- determine if it is properly checking the timeout from a dead instance
- maybe dump out the PID of the worker that is running so we know which one to kill?
+- failure/success not being returned correctly. Should check for 'fail' in the directories and return based on
+ that. also anything lacking success is a failure.
11 years, 4 months
[copr] skvidal-backend: add another TODO note (04b2f89)
by skvidal@fedorahosted.org
Repository : http://git.fedorahosted.org/cgit/copr.git
On branch : skvidal-backend
>---------------------------------------------------------------
commit 04b2f891e06aaa4ea79188d641594af34d24fba7
Author: Seth Vidal <skvidal(a)fedoraproject.org>
Date: Fri Dec 7 17:05:54 2012 -0500
add another TODO note
>---------------------------------------------------------------
TODO-backend | 1 +
1 files changed, 1 insertions(+), 0 deletions(-)
diff --git a/TODO-backend b/TODO-backend
index a978e13..058330e 100644
--- a/TODO-backend
+++ b/TODO-backend
@@ -14,3 +14,4 @@
- refactor mockremote/dispatcher.worker together?
- work on a way to find and cancel a specific build that's happening other than just killing the instance
- determine if it is properly checking the timeout from a dead instance
+- maybe dump out the PID of the worker that is running so we know which one to kill?
11 years, 4 months
[copr] skvidal-backend: remove/add todo's (25b9a4f)
by Seth Vidal
Repository : http://git.fedorahosted.org/cgit/copr.git
On branch : skvidal-backend
>---------------------------------------------------------------
commit 25b9a4f18ee342a0bbc797c796f27ec8756ee705
Author: Seth Vidal <skvidal(a)fedoraproject.org>
Date: Fri Dec 7 16:47:52 2012 -0500
remove/add todo's
>---------------------------------------------------------------
TODO-backend | 3 ++-
1 files changed, 2 insertions(+), 1 deletions(-)
diff --git a/TODO-backend b/TODO-backend
index e532fc9..a978e13 100644
--- a/TODO-backend
+++ b/TODO-backend
@@ -7,9 +7,10 @@
- LOADS of fixme and catching weird conditions
- make logging from mockremote more sane and coinsistent
- mock configs should be pushed to instances at creation time
- - should include enabled package_state pluging
- single url to repos, not mirrorlists
- consider making each worker return job to a completed queue so the primary
process can do other kinds of notification
- email notifications from backend?
- refactor mockremote/dispatcher.worker together?
+- work on a way to find and cancel a specific build that's happening other than just killing the instance
+- determine if it is properly checking the timeout from a dead instance
11 years, 4 months
[copr] skvidal-backend: log how long the instance spawn takes (24fbf8d)
by Seth Vidal
Repository : http://git.fedorahosted.org/cgit/copr.git
On branch : skvidal-backend
>---------------------------------------------------------------
commit 24fbf8d00b341e931ab436d21836e025f91343f0
Author: Seth Vidal <skvidal(a)fedoraproject.org>
Date: Fri Dec 7 10:48:52 2012 -0500
log how long the instance spawn takes
>---------------------------------------------------------------
backend/dispatcher.py | 3 +++
1 files changed, 3 insertions(+), 0 deletions(-)
diff --git a/backend/dispatcher.py b/backend/dispatcher.py
index a62f5f1..9b0399d 100644
--- a/backend/dispatcher.py
+++ b/backend/dispatcher.py
@@ -105,6 +105,7 @@ class Worker(multiprocessing.Process):
def spawn_instance(self):
"""call the spawn playbook to startup/provision a building instance"""
self.callback.log('spawning instance begin')
+ start = time.time()
stats = callbacks.AggregateStats()
playbook_cb = SilentPlaybookCallbacks(verbose=False)
@@ -117,6 +118,8 @@ class Worker(multiprocessing.Process):
play.run()
self.callback.log('spawning instance end')
+ self.callback.log('Instance spawn/provision took %s sec' % (time.time() - start))
+
if self.ip:
return self.ip
11 years, 4 months
[copr] skvidal-backend: add TODO list from my paper lists (cc1f0ba)
by skvidal@fedorahosted.org
Repository : http://git.fedorahosted.org/cgit/copr.git
On branch : skvidal-backend
>---------------------------------------------------------------
commit cc1f0ba6e55734f7f453a47321cbd778b6453de3
Author: Seth Vidal <skvidal(a)fedoraproject.org>
Date: Thu Dec 6 17:25:35 2012 -0500
add TODO list from my paper lists
>---------------------------------------------------------------
TODO-backend | 15 +++++++++++++++
1 files changed, 15 insertions(+), 0 deletions(-)
diff --git a/TODO-backend b/TODO-backend
new file mode 100644
index 0000000..e532fc9
--- /dev/null
+++ b/TODO-backend
@@ -0,0 +1,15 @@
+
+- change instance type by build request for more mem/procs/extend timeouts
+ - use extra-vars?
+ - need ansible 0.9?
+- auto-timer/cleanup script for old instances that may have been orphaned
+- prune out builders when we drop the number of them active
+- LOADS of fixme and catching weird conditions
+- make logging from mockremote more sane and coinsistent
+- mock configs should be pushed to instances at creation time
+ - should include enabled package_state pluging
+ - single url to repos, not mirrorlists
+- consider making each worker return job to a completed queue so the primary
+ process can do other kinds of notification
+- email notifications from backend?
+- refactor mockremote/dispatcher.worker together?
11 years, 4 months
[copr] skvidal-backend: example config file (85be22f)
by skvidal@fedorahosted.org
Repository : http://git.fedorahosted.org/cgit/copr.git
On branch : skvidal-backend
>---------------------------------------------------------------
commit 85be22f2287f53a38538194fb7bdc6000c09707d
Author: Seth Vidal <skvidal(a)fedoraproject.org>
Date: Thu Dec 6 16:49:48 2012 -0500
example config file
>---------------------------------------------------------------
copr-be.conf.example | 16 ++++++++++++++++
1 files changed, 16 insertions(+), 0 deletions(-)
diff --git a/copr-be.conf.example b/copr-be.conf.example
new file mode 100644
index 0000000..ffeb235
--- /dev/null
+++ b/copr-be.conf.example
@@ -0,0 +1,16 @@
+[backend]
+results_baseurl=http://copr-be.cloud.fedoraproject.org/results
+frontend_url=http://copr-fe.cloud.fedoraproject.org/backend
+frontend_auth=backend_password_from_fe_config
+spawn_playbook=/srv/copr-work/provision/builderpb.yml
+terminate_playbook=/srv/copr-work/provision/terminatepb.yml
+jobsdir=/srv/copr-work/jobs
+destdir=/srv/copr-repo/results
+sleeptime=30
+num_workers=5
+logfile=/srv/copr-work/logs/copr.log
+worker_logdir=/srv/copr-work/logs/workers/
+
+
+[builder]
+timeout=3600
11 years, 4 months
[copr] skvidal-backend: output what pkgs we're building to the logs (3094750)
by Seth Vidal
Repository : http://git.fedorahosted.org/cgit/copr.git
On branch : skvidal-backend
>---------------------------------------------------------------
commit 3094750f307303f0f1058240f4787a4e7405505a
Author: Seth Vidal <skvidal(a)fedoraproject.org>
Date: Thu Dec 6 16:41:10 2012 -0500
output what pkgs we're building to the logs
>---------------------------------------------------------------
backend/dispatcher.py | 3 ++-
1 files changed, 2 insertions(+), 1 deletions(-)
diff --git a/backend/dispatcher.py b/backend/dispatcher.py
index c113e8c..a62f5f1 100644
--- a/backend/dispatcher.py
+++ b/backend/dispatcher.py
@@ -272,7 +272,8 @@ class Worker(multiprocessing.Process):
# start the build - most importantly license checks.
- self.callback.log('Starting build: builder=%r timeout=%r destdir=%r chroot=%r repos=%r' % (ip, job.timeout, job.destdir, chroot, str(job.repos)))
+ self.callback.log('Starting build: id=%r builder=%r timeout=%r destdir=%r chroot=%r repos=%r' % (job.build_id,ip, job.timeout, job.destdir, chroot, str(job.repos)))
+ self.callback.log('building pkgs: %s' % ' '.join(job.pkgs))
try:
chrootlogfile = chroot_destdir + '/mockremote.log'
mr = mockremote.MockRemote(builder=ip, timeout=job.timeout,
11 years, 4 months
[copr] skvidal-backend: post status to frontend using python-requests (707d001)
by Seth Vidal
Repository : http://git.fedorahosted.org/cgit/copr.git
On branch : skvidal-backend
>---------------------------------------------------------------
commit 707d00179d06651528caa41cae30ca573afcd83d
Author: Seth Vidal <skvidal(a)fedoraproject.org>
Date: Thu Dec 6 15:16:16 2012 -0500
post status to frontend using python-requests
>---------------------------------------------------------------
backend/dispatcher.py | 61 +++++++++++++++++++++++++++++++++++++++---------
1 files changed, 49 insertions(+), 12 deletions(-)
diff --git a/backend/dispatcher.py b/backend/dispatcher.py
index db800cc..c113e8c 100644
--- a/backend/dispatcher.py
+++ b/backend/dispatcher.py
@@ -3,7 +3,6 @@
import os
import sys
-import shutil
import multiprocessing
import time
import Queue
@@ -15,13 +14,11 @@ import ansible
import ansible.playbook
import ansible.errors
from ansible import callbacks
+import requests
-
-#FIXME - this should be emitting to the per-worker log file
-# so we can know what is going on and where
class SilentPlaybookCallbacks(callbacks.PlaybookCallbacks):
''' playbook callbacks - quietly! '''
@@ -162,22 +159,60 @@ class Worker(multiprocessing.Process):
jobdata.timeout = build['timeout']
jobdata.destdir = self.opts.destdir + '/' + build['copr']['owner']['name'] + '/' + build['copr']['name'] + '/'
jobdata.build_id = build['id']
+ jobdata.results = self.opts.results_baseurl + '/' + build['copr']['owner']['name'] + '/' + build['copr']['name'] + '/'
jobdata.copr_id = build['copr']['id']
jobdata.user_id = build['user_id']
return jobdata
+ # maybe we move this to the callback?
+ def post_to_frontend(self, data):
+ """send data to frontend"""
+
+ headers = {'content-type': 'application/json'}
+ url='%s/update_builds/' % self.opts.frontend_url
+ auth=('user', self.opts.frontend_auth)
+
+ msg = None
+ try:
+ r = requests.post(url, data=json.dumps(data), auth=auth,
+ headers=headers)
+ if r.status_code != 200:
+ msg = 'Failed to submit to frontend: %s: %s' % (r.status_code, r.text)
+ except requests.RequestException, e:
+ msg = 'Post request failed: %s' % e
+
+ if msg:
+ self.callback.log(msg)
+ return False
+
+ return True
+
+ # maybe we move this to the callback?
+ def mark_started(self, job):
+
+ build = {'id':job.build_id,
+ 'started_on': job.started_on,
+ 'results': job.results,
+ }
+ data = {'builds':[build]}
+
+ if not self.post_to_frontend(data):
+ raise errors.CoprWorkerError, "Could not communicate to front end to submit status info"
+
+ # maybe we move this to the callback?
def return_results(self, job):
- """write out a completed json file to the results dir and submit the results to the frontend"""
self.callback.log('%s status %s. Took %s seconds' % (job.build_id, job.status, job.ended_on - job.started_on))
- jobfilename = os.path.basename(job.jobfile)
- shutil.move(job.jobfile, job.destdir + '/' + jobfilename)
- # reform json file with completed info (completed time, url, status, etc)
- # post it via requests to url with passcode
- # profit
+ build = {'id':job.build_id,
+ 'ended_on': job.ended_on,
+ 'status': job.status,
+ }
+ data = {'builds':[build]}
+
+ if not self.post_to_frontend(data):
+ raise errors.CoprWorkerError, "Could not communicate to front end to submit results"
- #FIXME - this should either return job status/results
- # into a queue or it should submit results directly to the frontend
+ os.unlink(job.jobfile)
def run(self):
# worker should startup and check if it can function
@@ -214,6 +249,8 @@ class Worker(multiprocessing.Process):
status = 1
job.started_on = time.time()
+ self.mark_started(job)
+
for chroot in job.chroots:
chroot_destdir = job.destdir + '/' + chroot
11 years, 4 months
[copr] skvidal-backend: - touch up how mockremote logs downloads to make it less confusing - note for tomorrow - (9515168)
by Seth Vidal
Repository : http://git.fedorahosted.org/cgit/copr.git
On branch : skvidal-backend
>---------------------------------------------------------------
commit 9515168b490f853813aaed3b0605160e2245a3d3
Author: Seth Vidal <skvidal(a)fedoraproject.org>
Date: Thu Dec 6 02:51:28 2012 -0500
- touch up how mockremote logs downloads to make it less confusing
- note for tomorrow -
>---------------------------------------------------------------
backend/dispatcher.py | 5 +++++
backend/mockremote.py | 4 ++--
2 files changed, 7 insertions(+), 2 deletions(-)
diff --git a/backend/dispatcher.py b/backend/dispatcher.py
index 69aafd6..db800cc 100644
--- a/backend/dispatcher.py
+++ b/backend/dispatcher.py
@@ -171,6 +171,11 @@ class Worker(multiprocessing.Process):
self.callback.log('%s status %s. Took %s seconds' % (job.build_id, job.status, job.ended_on - job.started_on))
jobfilename = os.path.basename(job.jobfile)
shutil.move(job.jobfile, job.destdir + '/' + jobfilename)
+
+ # reform json file with completed info (completed time, url, status, etc)
+ # post it via requests to url with passcode
+ # profit
+
#FIXME - this should either return job status/results
# into a queue or it should submit results directly to the frontend
diff --git a/backend/mockremote.py b/backend/mockremote.py
index 3d39f3d..2c641c8 100755
--- a/backend/mockremote.py
+++ b/backend/mockremote.py
@@ -187,11 +187,11 @@ class CliLogCallBack(DefaultCallBack):
self.log(msg)
def start_download(self, pkg):
- msg = "Start Download: %s" % pkg
+ msg = "Start retrieve results for: %s" % pkg
self.log(msg)
def end_download(self, pkg):
- msg = "End Download: %s" % pkg
+ msg = "End retrieve results for: %s" % pkg
self.log(msg)
def error(self, msg):
11 years, 4 months
[copr] skvidal-backend: log output a little simpler to read (c4f21e6)
by skvidal@fedorahosted.org
Repository : http://git.fedorahosted.org/cgit/copr.git
On branch : skvidal-backend
>---------------------------------------------------------------
commit c4f21e6c2b94af0a400d70730d21f0b6f2b437fc
Author: Seth Vidal <skvidal(a)fedoraproject.org>
Date: Thu Dec 6 02:34:48 2012 -0500
log output a little simpler to read
>---------------------------------------------------------------
backend/dispatcher.py | 2 +-
copr-be.py | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/backend/dispatcher.py b/backend/dispatcher.py
index fc9a958..69aafd6 100644
--- a/backend/dispatcher.py
+++ b/backend/dispatcher.py
@@ -76,7 +76,7 @@ class WorkerCallback(object):
now = time.strftime('%F %T')
try:
- open(self.logfile, 'a').write(str(now) + ':' + msg + '\n')
+ open(self.logfile, 'a').write(str(now) + ': ' + msg + '\n')
except (IOError, OSError), e:
print >>sys.stderr, 'Could not write to logfile %s - %s' % (self.logfile, str(e))
diff --git a/copr-be.py b/copr-be.py
index 6564134..6631ac4 100644
--- a/copr-be.py
+++ b/copr-be.py
@@ -92,7 +92,7 @@ class CoprBackend(object):
def log(self, msg):
now = time.strftime('%F %T')
- output = str(now) + ':' + msg
+ output = str(now) + ': ' + msg
if not self.opts.daemonize:
print output
11 years, 4 months