r5290 - in branches/play_plumage: cumin/bin cumin/etc cumin/model/plumage cumin/python/cumin mint/python/mint mint/python/mint/plumage parsley/python/parsley rosemary/python/rosemary
by tmckay@fedoraproject.org
Author: tmckay
Date: 2012-04-11 14:17:47 +0000 (Wed, 11 Apr 2012)
New Revision: 5290
Modified:
branches/play_plumage/cumin/bin/cumin-report
branches/play_plumage/cumin/etc/cumin.conf
branches/play_plumage/cumin/model/plumage/plumage.xml
branches/play_plumage/cumin/model/plumage/rosemary.xml
branches/play_plumage/cumin/python/cumin/config.py
branches/play_plumage/cumin/python/cumin/model.py
branches/play_plumage/mint/python/mint/expire.py
branches/play_plumage/mint/python/mint/plumage/main.py
branches/play_plumage/mint/python/mint/plumage/update.py
branches/play_plumage/mint/python/mint/update.py
branches/play_plumage/parsley/python/parsley/config.py
branches/play_plumage/rosemary/python/rosemary/model.py
branches/play_plumage/rosemary/python/rosemary/sqloperation.py
Log:
Many changes to support generic data in rosemary.
* Add new 'timestamp' legal value for the 'persistent' attribute in *.xml.
Classes marked with persistent="timestamp" will have object and sample data
expired based on interval and threshold values in cumin.conf.
* Add new 'timestamp' attribute for properties and statistics. A field marked
with "timestamp='y'" will be used by Cumin as the date field that governs
expiration, etc. For qmf data, _qmf_update_time has always been considered
the timestamp field.
* RosemaryClass and RosemaryObject have methods to get at the timestamp field
name and value.
* Add new 'source' tag in rosemary.xml for generic data classes. This tag will
hold information for retrieving the data, like a query string for example.
* Fixup expire thread to delete objects as well as samples if object deletion
is defined, and optionally loop through designated classes and packages rather
than the whole data model.
* Add --delete-on-start flag to cumin-report that will delete all of the
timestamp persistent data handled by the instance at startup. Combined with
--init-only, this gives a quick command line method of dumping reporing data.
* Add ConfigIntervalParameter type in config.py. This type allows numeric
values followed by s, m, h, d, or y as a unit designator. This allows large
intervals be be expressed naturally, such 3y for 3 years. All intervals in
cumin.conf are now this type.
* Set defaults for reporting expire/vacuum intervals in standard cumin.conf
expire-interval: 1d
expire-threshold: 90d
vacuum-interval: 1h
Modified: branches/play_plumage/cumin/bin/cumin-report
===================================================================
--- branches/play_plumage/cumin/bin/cumin-report 2012-04-09 21:34:22 UTC (rev 5289)
+++ branches/play_plumage/cumin/bin/cumin-report 2012-04-11 14:17:47 UTC (rev 5290)
@@ -116,6 +116,7 @@
parser.add_option("--daemon", action="store_true", default=False)
parser.add_option("--no-vacuum", dest="vacuum_enabled", action="store_false")
parser.add_option("--no-expire", dest="expire_enabled", action="store_false")
+ parser.add_option("--delete-on-start", action="store_true", default=False)
# Get options
opts, args = parser.parse_args()
@@ -155,10 +156,11 @@
mint.vacuum_enabled = opts.vacuum_enabled
mint.vacuum_thread.interval = values.vacuum_interval
+ # Data model init has been moved here in the
+ # bootstrap process so we can process the binding options next...
mint.check()
- mint.init()
- # Handle QMF class binding options
+ # Handle class binding options
if values.include_classes or values.exclude_classes:
return_code, includes = process_classes(mint,
values.include_classes,
@@ -170,7 +172,19 @@
mint.classes = includes.difference(excludes)
+ # Finish initialization
+ mint.init()
+ # If we've been instructed to delete timestamp data at
+ # startup, set the flag in the update thread where the
+ # delete logic is located...
+ mint.update_thread.del_on_start = opts.delete_on_start
+
+ # ... and if furthermore the init_only flag is set,
+ # cause the delete to happen before we return
+ if opts.init_only and opts.delete_on_start:
+ mint.update_thread._delete_all_objects()
+
# If init_only was set or we failed init, don't proceed...
if not opts.init_only and not return_code:
passed_init = 0
Modified: branches/play_plumage/cumin/etc/cumin.conf
===================================================================
--- branches/play_plumage/cumin/etc/cumin.conf 2012-04-09 21:34:22 UTC (rev 5289)
+++ branches/play_plumage/cumin/etc/cumin.conf 2012-04-11 14:17:47 UTC (rev 5290)
@@ -86,6 +86,8 @@
# section based on 'persona'. See additional comments in [data] section.
#datas: data
+# Reporting is off by default.
+# To enable reporting features, uncomment the following line.
# reports: report
# ****************************************************
@@ -181,7 +183,14 @@
[report]
# include-classes: [all]
# exclude-classes: [none]
+# log-file: $CUMIN_HOME/log/report.log
+# expire-enabled: True
+expire-interval: 1d
+expire-threshold: 90d
+# vacuum-enabled: True
+vacuum-interval: 1h
+
# *************** Some parameter explanations by section ***************
## [common]
Modified: branches/play_plumage/cumin/model/plumage/plumage.xml
===================================================================
--- branches/play_plumage/cumin/model/plumage/plumage.xml 2012-04-09 21:34:22 UTC (rev 5289)
+++ branches/play_plumage/cumin/model/plumage/plumage.xml 2012-04-11 14:17:47 UTC (rev 5290)
@@ -1,5 +1,5 @@
<?xml version="1.0" encoding="utf-8"?>
-<schema package="com.redhat.grid.plumage" package_type="plumage">
+<schema package="com.redhat.grid.plumage" package_type="generic">
<class name="OSUtil">
<property name="uuid" unique="y"
type="uuid" access="RC" desc="UUID of System Image"/>
Modified: branches/play_plumage/cumin/model/plumage/rosemary.xml
===================================================================
--- branches/play_plumage/cumin/model/plumage/rosemary.xml 2012-04-09 21:34:22 UTC (rev 5289)
+++ branches/play_plumage/cumin/model/plumage/rosemary.xml 2012-04-11 14:17:47 UTC (rev 5290)
@@ -1,6 +1,7 @@
<model>
- <package name="com.redhat.grid.plumage">
+ <package name="com.redhat.grid.plumage" persistent="time">
<class name="OSUtil">
+ <source>some stuff goes here about a query</source>
<property name="osName">
<title>OS</title>
</property>
@@ -12,6 +13,10 @@
<statistic name="avail">
<title>Available</title>
</statistic>
+
+ <statistic name="update_time" timestamp="y">
+ </statistic>
+
</class>
</package>
Modified: branches/play_plumage/cumin/python/cumin/config.py
===================================================================
--- branches/play_plumage/cumin/python/cumin/config.py 2012-04-09 21:34:22 UTC (rev 5289)
+++ branches/play_plumage/cumin/python/cumin/config.py 2012-04-11 14:17:47 UTC (rev 5290)
@@ -60,16 +60,16 @@
param = ConfigParameter(data, "expire-enabled", bool)
param.default = True
- param = ConfigParameter(data, "expire-interval", int)
+ param = ConfigIntervalParameter(data, "expire-interval")
param.default = 60 * 60 # 1 hour
- param = ConfigParameter(data, "expire-threshold", int)
+ param = ConfigIntervalParameter(data, "expire-threshold")
param.default = 24 * 60 * 60 # 1 day
param = ConfigParameter(data, "vacuum-enabled", bool)
param.default = True
- param = ConfigParameter(data, "vacuum-interval", int)
+ param = ConfigIntervalParameter(data, "vacuum-interval")
param.default = 60 * 60 # 1 hour
class CuminDataConfig(CuminConfig):
@@ -98,7 +98,7 @@
param.default = "web"
param = ConfigParameter(master, "reports", str)
- param.default = "report"
+ param.default = ""
class CuminConfigSection(ConfigSection):
def __init__(self, config, name, strict_section=False):
@@ -172,7 +172,7 @@
param = ConfigParameter(self, "aviary-suds-logs", bool)
param.default = False
- param = ConfigParameter(self, "update-interval", int)
+ param = ConfigIntervalParameter(self, "update-interval")
param.default = 10
param = ConfigParameter(self, "max-qmf-table-sort", int)
@@ -201,7 +201,8 @@
param.default = "grid"
param = ConfigParameter(self, "fast-view-attributes", str)
- param.default = "JobStatus,Cmd,Args,ExitStatus,JobStartDate,LastRemoteHost,LastJobStatus,Owner"
+ param.default = "JobStatus,Cmd,Args,ExitStatus,JobStartDate,"\
+ "LastRemoteHost,LastJobStatus,Owner"
param = ConfigParameter(self, "notification-timeout", int)
param.default = 180
Modified: branches/play_plumage/cumin/python/cumin/model.py
===================================================================
--- branches/play_plumage/cumin/python/cumin/model.py 2012-04-09 21:34:22 UTC (rev 5289)
+++ branches/play_plumage/cumin/python/cumin/model.py 2012-04-11 14:17:47 UTC (rev 5290)
@@ -318,8 +318,6 @@
return retval
class SamplesSqlAdapter(SqlAdapter):
- qmf_update_col = '_qmf_update_time'
-
def __init__(self, app, cls, sig):
super(SamplesSqlAdapter, self).__init__(app, cls.sql_samples_table)
@@ -327,12 +325,14 @@
for f in filters:
self.query.add_filter(f)
+ self.update_col = cls.get_timestamp_col()
+
def get_sql_options(self, options):
return options
def avg_samples(self, stat, secs, interval, secs2):
stat_col = self.table._columns_by_name[stat.name]
- updated_col = self.table._columns_by_name[self.qmf_update_col]
+ updated_col = self.table._columns_by_name[self.update_col]
org_filters = list(self.query.filters)
max_col = "max(%s) as interval_end" % updated_col.identifier
@@ -374,7 +374,7 @@
org_filters = list(self.query.filters)
stat_col = self.table._columns_by_name[stat.name]
- updated_col = self.table._columns_by_name[self.qmf_update_col]
+ updated_col = self.table._columns_by_name[self.update_col]
columns = list()
columns.append(updated_col.identifier)
@@ -407,7 +407,7 @@
return samples
def recent(self):
- updated_col = self.table._columns_by_name[self.qmf_update_col]
+ updated_col = self.table._columns_by_name[self.update_col]
columns = list()
columns.append(updated_col.identifier)
Modified: branches/play_plumage/mint/python/mint/expire.py
===================================================================
--- branches/play_plumage/mint/python/mint/expire.py 2012-04-09 21:34:22 UTC (rev 5289)
+++ branches/play_plumage/mint/python/mint/expire.py 2012-04-11 14:17:47 UTC (rev 5290)
@@ -4,11 +4,18 @@
log = logging.getLogger("mint.expire")
class ExpireThread(MintPeriodicProcessThread):
- def __init__(self, app):
- super(ExpireThread, self).__init__(app, 60 * 60)
+ def __init__(self, app, interval=3600, threshold=24*3600):
+ super(ExpireThread, self).__init__(app, interval)
- self.threshold = 24 * 3600
+ self.threshold = threshold
+ # Default for the package list is all the packages in the
+ # model, but this may be set by the app
+ self.packages = self.app.model._packages
+ # If a specific class list is given, then self.packages
+ # will be ignored and we will iterate over the classes
+ self.classes = []
+
def init(self):
super(ExpireThread, self).init()
@@ -30,23 +37,37 @@
conn = self.app.database.get_connection()
- try:
- cursor = conn.cursor()
+ def loop_body(cls, cursor):
+ count = 0
+ if cls._storage == "none":
+ return count
- for pkg in self.app.model._packages:
- for cls in pkg._classes:
- if cls._storage == "none":
- continue
+ values = {"seconds": threshold}
+ if cls.sql_samples_delete is not None:
+ cls.sql_samples_delete.execute(cursor, values)
+ log.debug("Deleted %i samples %s", cursor.rowcount, cls)
+ count = cursor.rowcount
- values = {"seconds": threshold}
- cls.sql_samples_delete.execute(cursor, values)
+ # For timestamp persistent classes, we get rid of the
+ # objects in additoin to the samples
+ if hasattr(cls, "sql_timestamp_delete") and \
+ cls.sql_timestamp.delete is not None:
+ cls.sql_timestamp_delete.execute(cursor, values)
+ log.debug("Deleted %i objects %s", cursor.rowcount, cls)
+ count += cursor.rowcount
- conn.commit()
+ conn.commit()
+ return count
- log.debug("Deleted %i %s", cursor.rowcount, cls)
-
- count += cursor.rowcount
+ try:
+ cursor = conn.cursor()
+ if len(self.classes) > 0:
+ for cls in self.classes:
+ count += loop_body(cls, cursor)
+ else:
+ for pkg in self.packages:
+ for cls in pkg._classes:
+ count += loop_body(cls, cursor)
finally:
conn.close()
-
- log.info("Expired %i samples", count)
+ log.info("Expired %i rows", count)
Modified: branches/play_plumage/mint/python/mint/plumage/main.py
===================================================================
--- branches/play_plumage/mint/python/mint/plumage/main.py 2012-04-09 21:34:22 UTC (rev 5289)
+++ branches/play_plumage/mint/python/mint/plumage/main.py 2012-04-11 14:17:47 UTC (rev 5290)
@@ -24,10 +24,10 @@
self.update_thread = UpdateThread(self)
self.session = PlumageSession(self, server_url)
- self.expire_enabled = False
+ self.expire_enabled = True
self.expire_thread = ExpireThread(self)
- self.vacuum_enabled = False
+ self.vacuum_enabled = True
self.vacuum_thread = VacuumThread(self)
self.print_event_level = 0
@@ -43,6 +43,7 @@
self.model.check()
self.database.check()
+ self.model.init()
def init(self):
log.info("Initializing %s", self)
@@ -53,26 +54,38 @@
log.info("Expiration is %s", state(self.expire_enabled))
log.info("Vacuum is %s", state(self.vacuum_enabled))
- self.model.init()
self.database.init()
self.update_thread.init()
- self.session.init()
- #self.expire_thread.init()
- #self.vacuum_thread.init()
+ self.session.init()
+ # The package and class lists will be
+ # processed here
+ self.session.init_classes()
+ if self.expire_enabled:
+ self.expire_thread.init()
+
+ if self.vacuum_enabled:
+ self.vacuum_thread.init()
+
def start(self):
log.info("Starting %s", self)
-
+
self.update_thread.start()
- self.session.init_classes()
+
self.session.start()
- #if self.expire_enabled:
- # self.expire_thread.start()
+ if self.expire_enabled:
+ # If we set these values here, we can have each cumin-report
+ # instance run expiration for its own classes...
+ # Default is like cumin-data, where a single instance runs
+ # expiration across all classes in the model
+ #self.expire_thread.packages = self.packages
+ #self.expire_thread.classes = self.classes
+ self.expire_thread.start()
- #if self.vacuum_enabled:
- # self.vacuum_thread.start()
+ if self.vacuum_enabled:
+ self.vacuum_thread.start()
def stop(self):
log.info("Stopping %s", self)
@@ -80,11 +93,11 @@
self.update_thread.stop()
self.session.stop()
- #if self.expire_enabled:
- # self.expire_thread.stop()
+ if self.expire_enabled:
+ self.expire_thread.stop()
- #if self.vacuum_enabled:
- # self.vacuum_thread.stop()
+ if self.vacuum_enabled:
+ self.vacuum_thread.stop()
def __repr__(self):
return self.__class__.__name__
Modified: branches/play_plumage/mint/python/mint/plumage/update.py
===================================================================
--- branches/play_plumage/mint/python/mint/plumage/update.py 2012-04-09 21:34:22 UTC (rev 5289)
+++ branches/play_plumage/mint/python/mint/plumage/update.py 2012-04-11 14:17:47 UTC (rev 5290)
@@ -33,19 +33,36 @@
self.halt_on_error = False
+ # For timestamp persistent data, if this flag is True,
+ # we will delete the objects and samples on startup.
+ # This is to allow purging of large data sets.
+ self.del_on_start = False
+
def _delete_all_objects(self):
def loop_body(cls):
if cls._storage != "none":
- if cls.check_persistent():
- log.debug("Skipping persistent class " + str(cls))
- else:
+ persistent = cls.check_persistent()
+ if (persistent == "session") or \
+ (persistent == "timestamp" and self.del_on_start):
cls.delete_selection(self.cursor)
+ if persistent == "timestamp":
+ # del_on_start must be true...
+ cls.delete_sample_selection(self.cursor)
self.cursor.connection.commit()
+ else:
+ log.debug("Skipping persistent class " + str(cls))
- log.debug("Delete all objects by bound packages " + str(self.app.packages))
- for pkg in self.app.packages:
- for cls in pkg._classes:
+ if len(self.app.classes):
+ log.debug("Delete all objects by bound classes " +\
+ str(self.app.classes))
+ for cls in self.app.classes:
loop_body(cls)
+ else:
+ log.debug("Delete all objects by bound packages " +\
+ str(self.app.packages))
+ for pkg in self.app.packages:
+ for cls in pkg._classes:
+ loop_body(cls)
def init(self):
self.conn = self.app.database.get_connection()
@@ -59,6 +76,7 @@
self.stats.enqueued += 1
def run(self):
+ self._delete_all_objects()
while True:
if self.stop_requested:
break
Modified: branches/play_plumage/mint/python/mint/update.py
===================================================================
--- branches/play_plumage/mint/python/mint/update.py 2012-04-09 21:34:22 UTC (rev 5289)
+++ branches/play_plumage/mint/python/mint/update.py 2012-04-11 14:17:47 UTC (rev 5290)
@@ -26,22 +26,34 @@
self.halt_on_error = False
+ # For timestamp persistent data, if this flag is True,
+ # we will delete the objects and samples on startup.
+ # This is to allow purging of large data sets.
+ self.del_on_start = False
+
def _delete_all_objects(self):
def loop_body(cls):
if cls._storage != "none":
- if cls.check_persistent():
- log.debug("Skipping persistent class " + str(cls))
- else:
+ persistent = cls.check_persistent()
+ if (persistent == "session") or \
+ (persistent == "timestamp" and self.del_on_start):
cls.delete_selection(self.cursor)
+ if persistent == "timestamp":
+ # del_on_start must be true...
+ cls.delete_sample_selection(self,cursor)
self.cursor.connection.commit()
+ else:
+ log.debug("Skipping persistent class " + str(cls))
if len(self.app.qmf_classes):
- log.debug("Delete all objects by bound classes " + str(self.app.qmf_classes))
+ log.debug("Delete all objects by bound classes " +\
+ str(self.app.qmf_classes))
for cls in self.app.qmf_classes:
loop_body(cls)
else:
# We bound all classes. Loop over model
- log.debug("Delete all objects by bound packages " + str(self.app.qmf_packages))
+ log.debug("Delete all objects by bound packages " +\
+ str(self.app.qmf_packages))
for pkg in self.app.qmf_packages:
for cls in pkg._classes:
loop_body(cls)
@@ -621,22 +633,25 @@
stats.agents_updated += 1
- def delete_agent_objects(self, cursor, stats, agent, bound_classes, bound_packages):
+ def delete_agent_objects(self, cursor, stats, agent,
+ bound_classes, bound_packages):
def loop_body(cls):
- if cls._storage != "none" and not cls.check_persistent():
+ if cls._storage != "none" and cls.check_persistent() == "session":
count = cls.delete_selection(cursor, _qmf_agent_id=agent.id)
stats.objects_deleted += count
#stats.objects_deleted_by_class[cls] += count
cursor.connection.commit()
if len(bound_classes):
- log.debug("Delete agent objects by bound classes " + str(bound_classes))
+ log.debug("Delete agent objects by bound classes " +\
+ str(bound_classes))
for cls in bound_classes:
loop_body(cls)
else:
# We bound all classes. Loop over model
- log.debug("Delete agent objects by bound packages " + str(bound_packages))
+ log.debug("Delete agent objects by bound packages " +\
+ str(bound_packages))
for pkg in bound_packages:
for cls in pkg._classes:
loop_body(cls)
Modified: branches/play_plumage/parsley/python/parsley/config.py
===================================================================
--- branches/play_plumage/parsley/python/parsley/config.py 2012-04-09 21:34:22 UTC (rev 5289)
+++ branches/play_plumage/parsley/python/parsley/config.py 2012-04-11 14:17:47 UTC (rev 5290)
@@ -63,23 +63,43 @@
return out_values
-class ConfigParameter(object):
- def __init__(self, section, name, type):
+class BaseConfigParameter(object):
+ def __init__(self, section, name):
assert isinstance(section, ConfigSection)
self.section = section
self.section.parameters.append(self)
self.name = name
- self.type = type
-
self.default = None
def unmarshal(self, string):
+ raise Exception("Not implemented")
+
+class ConfigParameter(BaseConfigParameter):
+ def __init__(self, section, name, type):
+ super(ConfigParameter, self).__init__(section, name)
+
+ self.type = type
+
+ def unmarshal(self, string):
if self.type == bool:
return string.lower() not in ("f", "false", "no", "0")
return self.type(string)
+class ConfigIntervalParameter(BaseConfigParameter):
+ def unmarshal(self, value):
+ mult = {'s': 1, 'm': 60, 'h': 3600, 'd': 86400, 'y': 31557600}
+ unit = 's'
+
+ if value[-1:].isalpha():
+ unit = value[-1:]
+ value = value[:-1]
+
+ if not unit in mult:
+ raise ValueError
+ return float(value) * mult[unit]
+
class ConfigValues(dict):
def __getattr__(self, name):
return self[name]
Modified: branches/play_plumage/rosemary/python/rosemary/model.py
===================================================================
--- branches/play_plumage/rosemary/python/rosemary/model.py 2012-04-09 21:34:22 UTC (rev 5289)
+++ branches/play_plumage/rosemary/python/rosemary/model.py 2012-04-11 14:17:47 UTC (rev 5290)
@@ -109,6 +109,16 @@
pass
return youngest
+ @classmethod
+ def handle_persistent(cls, persistent):
+ if persistent in ("y", "database"):
+ return "database"
+ elif persistent in ("n", "session"):
+ return "session"
+ elif persistent == "timestamp":
+ return persistent
+ return None
+
class RosemaryPackage(object):
def __init__(self, model, name, pkg_type):
self._model = model
@@ -118,17 +128,24 @@
self.pkg_type = "qmf"
self.xml_class = self.get_xml_class()
- # Set in the extension xml file. This marks a package as
- # containing persistent data which should not be deleted
- # by cumin-data on startup or in association with
- # the comings and goings of agents. Rather, it
- # should be manipulated by specific interfaces.
- # com.redhat.cumin is an example.
+ # Set in the extension xml file. This specifies how
+ # object data deletion is handled for classes in the
+ # package.
+ # There are 3 possibilities for 'persistent' in the xml:
+ # "y" or "database" -- data is only deleted manually
+ # "n" or "session" -- data is deleted on startup (default)
+ # "timestamp" -- data is deleted by the expire thread
+ # along with samples, governed by the same threshold
+
+ # In all 3 cases, sample data is deleted by the expire thread
+ # if it is enabled, governed by the threshold. Persistent
+ # only affects object data.
+
# Note, "persistent" may also be set at the class
- # level in cases where packages contain both persistent
- # and non-persistent data.
- self.persistent = False
+ # level in cases where packages contain classes with
+ # different semantics.
+ self.persistent = "session"
self._model._packages.append(self)
self._model._packages_by_name[self._name] = self
@@ -177,7 +194,13 @@
def extend(self, elem):
persistent = elem.get("persistent")
if persistent is not None:
- self.persistent = persistent == "y"
+ p = RosemaryModel.handle_persistent(persistent)
+ if p is None:
+ log.debug("Persistent value '%s' invalid for '%s',"\
+ "using default of '%s'" % \
+ (persistent, self._name, self.persistent))
+ else:
+ self.persistent = p
for child in elem.findall("class"):
cls = self._classes_by_name[child.get("name")]
@@ -251,8 +274,14 @@
self._sample_index_col_groups = defaultdict(list)
self._id = RosemaryAttribute(self, "_id")
- self.qmf_headers = False
+ # Designates which field we will use as a timestamp
+ # in Objects of this type.
+ self.timestamp = ""
+
+ def get_timestamp_col(self):
+ return self.timestamp
+
def load(self, elem, groups_by_name):
log.debug("Loading %s", self)
@@ -297,16 +326,36 @@
persistent = elem.get("persistent")
if persistent is not None:
- self._persistent = persistent == "y"
+ p = RosemaryModel.handle_persistent(persistent)
+ if p is None:
+ log.debug("Persistent value '%s' invalid for '%s',"\
+ "using default of '%s'" % \
+ (persistent, self._name, self._persistent))
+ else:
+ self._persistent = p
for child in elem.findall("property"):
prop = self._properties_by_name[child.get("name")]
prop.extend(child)
+ # If the attribute is flagged as the timestamp field,
+ # then remember the name of the field so that it can
+ # be set as the timestamp field when a new object is
+ # created
+ if prop.timestamp:
+ self.timestamp = prop.name
+
for child in elem.findall("statistic"):
stat = self._statistics_by_name[child.get("name")]
stat.extend(child)
+ # If the attribute is flagged as the timestamp field,
+ # then remember the name of the field so that it can
+ # be set as the timestamp field when a new object is
+ # created
+ if stat.timestamp:
+ self.timestamp = stat.name
+
# for child in elem.findall("method"):
# meth = self._methods_by_name[child.get("name")]
# meth.extend(child)
@@ -353,6 +402,7 @@
self.add_sql_constraints()
self.add_sql_operations()
+ self.add_sql_sample_operations()
self.add_sql_sample_indices()
def init_references(self):
@@ -375,10 +425,6 @@
for k,v in self._unique_col_groups.iteritems():
SqlUniqueConstraint(self.sql_table, k, v)
- def add_sql_sample_indices(self):
- for k,v in self._sample_index_col_groups.iteritems():
- SqlIndex(self._package.sql_schema, k, v)
-
def add_sql_operations(self):
self.sql_get_new_id = SqlGetNewId(self.sql_table, self.sql_sequence)
self.sql_select_by_id = SqlSelectObject(self.sql_table)
@@ -386,13 +432,44 @@
self.sql_update_object = SqlUpdateObject(self.sql_table)
self.sql_delete_object = SqlDeleteObject(self.sql_table)
+ # If we've been marked as timestamp persistent, then
+ # add an operation to delete objects by timestamp just
+ # like samples
+ if self.check_persistent() == "timestamp":
+ try:
+ timestamp = getattr(self.sql_table, self.timestamp)
+ self.sql_timestamp_delete = SqlDeleteObjectSamples(
+ self.sql_table,
+ timestamp)
+ except:
+ pass
+
def add_samples_sql(self):
name = "%s_samples" % self._name
table = SqlTable(self._package.sql_schema, name)
-
self.sql_samples_table = table
- self.sql_samples_insert = SqlInsertObjectSamples(table, self.qmf_headers)
+ def add_sql_sample_indices(self):
+ # Add indices based on the unique column groups
+ # for the sample table
+ for k,v in self._sample_index_col_groups.iteritems():
+ SqlIndex(self._package.sql_schema, k, v)
+
+ def add_sql_sample_operations(self):
+ self.sql_samples_insert = SqlInsertObjectSamples(
+ self.sql_samples_table)
+
+ # Add delete by timestamp if the timestamp field is defined
+ try:
+ timestamp = getattr(self.sql_samples_table, self.timestamp)
+ self.sql_samples_delete = SqlDeleteObjectSamples(
+ self.sql_samples_table,
+ timestamp)
+ except:
+ self.sql_samples_delete = None
+ log.debug("Missing or invalid timestamp designation for class " \
+ "'%s', check xml" % self._name)
+
def get_object(self, cursor, **criteria):
sql = SqlSelect(self.sql_table)
@@ -536,17 +613,23 @@
self.sql_delete_object.execute(cursor, obj.__dict__)
- def delete_selection(self, cursor, **criteria):
- sql = SqlDelete(self.sql_table)
+ def _delete_selection(self, cursor, table, **criteria):
+ sql = SqlDelete(table)
for name in criteria:
- column = self.sql_table._columns_by_name[name]
+ column = table._columns_by_name[name]
sql.add_filter(SqlValueFilter(column))
sql.execute(cursor, criteria)
-
return cursor.rowcount
+ def delete_selection(self, cursor, **criteria):
+ return self._delete_selection(cursor, self.sql_table, **criteria)
+
+ def delete_sample_selection(self, cursor, **criteria):
+ return self._delete_selection(cursor,
+ self.sql_samples_table, **criteria)
+
def __repr__(self):
args = (self.__class__.__name__, self._package._name, self._name)
return "%s(%s,%s)" % args
@@ -555,7 +638,6 @@
def __init__(self, package, name):
super(Qmf_class, self).__init__(package, name)
- self.qmf_headers = True
name = "_qmf_agent_id"
self._qmf_agent_id = RosemaryHeader(self, name, sql_text)
self._qmf_agent_id.title = "Agent ID"
@@ -577,10 +659,21 @@
self._qmf_delete_time.title = "Delete Time"
self._qmf_delete_time.optional = True
+ # Default timestamp field for Qmf objects will be _qmf_update_time
+ self.timestamp = "_qmf_update_time"
+
def add_sql_operations(self):
super(Qmf_class, self).add_sql_operations()
self.sql_select_by_qmf_id = SqlSelectObjectByQmfId(self.sql_table)
+ def add_sql_sample_operations(self):
+ super(Qmf_class, self).add_sql_sample_operations()
+
+ # Okay, our insert operation needs some extra columns
+ # Add them here to the operation created by our parent.
+ self.sql_samples_insert.extra_column_names = ("_qmf_agent_id",
+ "_qmf_object_id")
+
def add_sql_constraints(self):
# Add agent and object columns as a unique column set
name = "%s_qmf_id_uq" % self._name
@@ -590,20 +683,13 @@
def add_samples_sql(self):
super(Qmf_class, self).add_samples_sql()
- agent_col = SqlColumn(self.sql_samples_table,
- "_qmf_agent_id", sql_text)
+ a = SqlColumn(self.sql_samples_table, "_qmf_agent_id", sql_text)
+ o = SqlColumn(self.sql_samples_table, "_qmf_object_id", sql_text)
+ SqlColumn(self.sql_samples_table, "_qmf_update_time", sql_timestamp)
- object_col = SqlColumn(self.sql_samples_table,
- "_qmf_object_id", sql_text)
-
name = "%s_qmf_id_idx" % self.sql_samples_table._name
- SqlIndex(self._package.sql_schema, name, (agent_col, object_col))
+ self._sample_index_col_groups[name] = (a, o)
- SqlColumn(self.sql_samples_table, "_qmf_update_time", sql_timestamp)
-
- # We will need another way to delete non-qmf samples
- self.sql_samples_delete = SqlDeleteObjectSamples(self.sql_samples_table)
-
def get_object_by_qmf_id(self, cursor, agent_id, object_id):
obj = self._get_rosemary_object(None)
obj._qmf_agent_id = agent_id
@@ -612,8 +698,7 @@
return obj
def load_object_by_qmf_id(self, cursor, obj):
- assert isinstance(obj, RosemaryObject)
- assert self.qmf_headers
+ assert isinstance(obj, RosemaryQmfObject)
assert obj._qmf_agent_id, obj
assert obj._qmf_object_id, obj
@@ -660,16 +745,17 @@
def _get_rosemary_object(self, id):
return RosemaryQmfObject(self, id)
-class Plumage_class(RosemaryClass):
+class Generic_class(RosemaryClass):
def __init__(self, package, name):
- super(Plumage_class, self).__init__(package, name)
+ super(Generic_class, self).__init__(package, name)
# Columns marked unique will be added to the
# sample table
self.add_unique_to_samples = True
+ self.source = ""
def init(self):
- super(Plumage_class, self).init()
+ super(Generic_class, self).init()
# Store a sorted list of the names of unique
# column groups
@@ -686,6 +772,11 @@
for c in self.signature_cols:
self.sql_select_signature.add_filter(SqlValueFilter(c))
+ def extend(self, elem):
+ super(Generic_class, self).extend(elem)
+
+ self.source = elem.findtext("source")
+
def load_class_components(self, elem):
for child in elem.findall("property"):
name = child.get("name")
@@ -780,6 +871,7 @@
self.title = elem.findtext("title")
self.short_title = elem.findtext("short")
formatter = elem.findtext("formatter")
+ self.timestamp = elem.get("timestamp", "n") == "y"
# Allow overloading of optional in the extensions
# This is nice, because sometimes we have to mark
@@ -952,6 +1044,8 @@
arg.load(child)
def extend(self, elem):
+
+
pass
def init(self):
@@ -1024,6 +1118,14 @@
self._save_time = None
self._sample_time = None
+ # This will be a generic alias that refers to a timestamp
+ # field. The actual field may be different for different classes,
+ # but this name will work as an alias
+ try:
+ self._timestamp = getattr(self, cls.timestamp)
+ except:
+ self._timestamp = None
+
# XXX prefix these with _
def load(self, cursor):
self._class.load_object_by_id(cursor, self)
@@ -1067,8 +1169,14 @@
return value
def get_timestamp(self):
- raise Exception("Not implemented")
+ return self._timestamp
+ def get_signature(self):
+ # return an ordered list of values that can uniquely
+ # identify this object. Let the class determine
+ # the list.
+ return self._class.get_signature(self)
+
def __repr__(self):
name = self.__class__.__name__
args = (name, self._class, self._id, self._save_time)
@@ -1081,15 +1189,6 @@
self._qmf_create_time = datetime.now()
self._qmf_update_time = datetime.now()
- def get_timestamp(self):
- return self._qmf_update_time
-
- def get_signature(self):
- # return an ordered list of values that can uniquely
- # identify this object. Let the class determine
- # the list.
- return self._class.get_signature(self)
-
class RosemaryNotFound(Exception):
pass
Modified: branches/play_plumage/rosemary/python/rosemary/sqloperation.py
===================================================================
--- branches/play_plumage/rosemary/python/rosemary/sqloperation.py 2012-04-09 21:34:22 UTC (rev 5289)
+++ branches/play_plumage/rosemary/python/rosemary/sqloperation.py 2012-04-11 14:17:47 UTC (rev 5290)
@@ -96,10 +96,10 @@
return "insert into %s (%s) values (%s)" % (table, cols, vals)
class SqlInsertObjectSamples(SqlOperation):
- def __init__(self, table, qmf_headers):
+ def __init__(self, table):
super(SqlInsertObjectSamples, self).__init__(table)
- self.qmf_headers = qmf_headers
+ self.extra_column_names = ()
def emit(self, columns, options=None):
table = getattr(self.table, "identifier", self.table)
@@ -108,12 +108,9 @@
cols = ["\"%s\"" % x for x in names]
vals = ["%%(%s)s" % x for x in names]
- if self.qmf_headers:
- cols.append("\"_qmf_agent_id\"")
- cols.append("\"_qmf_object_id\"")
-
- vals.append("%(_qmf_agent_id)s")
- vals.append("%(_qmf_object_id)s")
+ for c in self.extra_column_names:
+ cols.append("\"%s\"" % c)
+ vals.append("%%(%s)s" % c)
cols = ", ".join(cols)
vals = ", ".join(vals)
@@ -163,10 +160,9 @@
self.add_filter(SqlValueFilter(self.table._id))
class SqlDeleteObjectSamples(SqlDelete):
- def __init__(self, table):
+ def __init__(self, table, time_col):
super(SqlDeleteObjectSamples, self).__init__(table)
- this = self.table._qmf_update_time
that = "now() - interval '%(seconds)s seconds'"
- self.add_filter(SqlComparisonFilter(this, that, "<"))
+ self.add_filter(SqlComparisonFilter(time_col, that, "<"))
12 years
r5289 - in branches/play_plumage: cumin/bin mint/python/mint/plumage
by tmckay@fedoraproject.org
Author: tmckay
Date: 2012-04-09 21:34:22 +0000 (Mon, 09 Apr 2012)
New Revision: 5289
Modified:
branches/play_plumage/cumin/bin/cumin-report
branches/play_plumage/mint/python/mint/plumage/main.py
branches/play_plumage/mint/python/mint/plumage/session.py
Log:
Fix up classes init
Modified: branches/play_plumage/cumin/bin/cumin-report
===================================================================
--- branches/play_plumage/cumin/bin/cumin-report 2012-04-09 21:21:50 UTC (rev 5288)
+++ branches/play_plumage/cumin/bin/cumin-report 2012-04-09 21:34:22 UTC (rev 5289)
@@ -168,7 +168,7 @@
values.exclude_classes,
opts.section)
- mint.qmf_classes = includes.difference(excludes)
+ mint.classes = includes.difference(excludes)
# If init_only was set or we failed init, don't proceed...
Modified: branches/play_plumage/mint/python/mint/plumage/main.py
===================================================================
--- branches/play_plumage/mint/python/mint/plumage/main.py 2012-04-09 21:21:50 UTC (rev 5288)
+++ branches/play_plumage/mint/python/mint/plumage/main.py 2012-04-09 21:34:22 UTC (rev 5289)
@@ -65,6 +65,7 @@
log.info("Starting %s", self)
self.update_thread.start()
+ self.session.init_classes()
self.session.start()
#if self.expire_enabled:
Modified: branches/play_plumage/mint/python/mint/plumage/session.py
===================================================================
--- branches/play_plumage/mint/python/mint/plumage/session.py 2012-04-09 21:21:50 UTC (rev 5288)
+++ branches/play_plumage/mint/python/mint/plumage/session.py 2012-04-09 21:34:22 UTC (rev 5289)
@@ -20,6 +20,7 @@
def init(self):
log.info("Initializing %s", self)
+ def init_classes(self):
# Apply the package filter to the class list
if len(self.app.classes):
black_list = set()
@@ -47,7 +48,6 @@
self.app,
self.server_url,
cls))
-
def start(self):
log.info("Starting %s", self)
for t in self.threads:
12 years
r5288 - in branches/play_plumage: cumin/etc cumin/model/plumage cumin/python/cumin mint/python/mint/plumage
by tmckay@fedoraproject.org
Author: tmckay
Date: 2012-04-09 21:21:50 +0000 (Mon, 09 Apr 2012)
New Revision: 5288
Modified:
branches/play_plumage/cumin/etc/cumin.conf
branches/play_plumage/cumin/model/plumage/plumage.xml
branches/play_plumage/cumin/python/cumin/config.py
branches/play_plumage/mint/python/mint/plumage/main.py
branches/play_plumage/mint/python/mint/plumage/session.py
Log:
Allow classes for a report instance to be specified from cumin.conf
Add update_time to pretend data.
Modified: branches/play_plumage/cumin/etc/cumin.conf
===================================================================
--- branches/play_plumage/cumin/etc/cumin.conf 2012-04-09 19:48:49 UTC (rev 5287)
+++ branches/play_plumage/cumin/etc/cumin.conf 2012-04-09 21:21:50 UTC (rev 5288)
@@ -86,6 +86,8 @@
# section based on 'persona'. See additional comments in [data] section.
#datas: data
+# reports: report
+
# ****************************************************
[web]
@@ -176,6 +178,10 @@
# include-classes: [all]
exclude-classes: com.redhat.grid:*, com.redhat.sesame:*
+[report]
+# include-classes: [all]
+# exclude-classes: [none]
+
# *************** Some parameter explanations by section ***************
## [common]
Modified: branches/play_plumage/cumin/model/plumage/plumage.xml
===================================================================
--- branches/play_plumage/cumin/model/plumage/plumage.xml 2012-04-09 19:48:49 UTC (rev 5287)
+++ branches/play_plumage/cumin/model/plumage/plumage.xml 2012-04-09 21:21:50 UTC (rev 5288)
@@ -9,5 +9,6 @@
<statistic name="avail" type="uint32" unit="uint32" />
<statistic name="used" type="uint32" unit="uint32" />
+ <statistic name="update_time" type="absTime" unit="uint32" />
</class>
</schema>
Modified: branches/play_plumage/cumin/python/cumin/config.py
===================================================================
--- branches/play_plumage/cumin/python/cumin/config.py 2012-04-09 19:48:49 UTC (rev 5287)
+++ branches/play_plumage/cumin/python/cumin/config.py 2012-04-09 21:21:50 UTC (rev 5288)
@@ -98,7 +98,7 @@
param.default = "web"
param = ConfigParameter(master, "reports", str)
- param.default = ""
+ param.default = "report"
class CuminConfigSection(ConfigSection):
def __init__(self, config, name, strict_section=False):
Modified: branches/play_plumage/mint/python/mint/plumage/main.py
===================================================================
--- branches/play_plumage/mint/python/mint/plumage/main.py 2012-04-09 19:48:49 UTC (rev 5287)
+++ branches/play_plumage/mint/python/mint/plumage/main.py 2012-04-09 21:21:50 UTC (rev 5288)
@@ -32,8 +32,12 @@
self.print_event_level = 0
- self.packages = []
+ self.packages = set()
+ self.classes = set()
+
+ self.package_filter = ["com.redhat.cumin"]
+
def check(self):
log.info("Checking %s", self)
Modified: branches/play_plumage/mint/python/mint/plumage/session.py
===================================================================
--- branches/play_plumage/mint/python/mint/plumage/session.py 2012-04-09 19:48:49 UTC (rev 5287)
+++ branches/play_plumage/mint/python/mint/plumage/session.py 2012-04-09 21:21:50 UTC (rev 5288)
@@ -1,4 +1,5 @@
import logging
+import time
from mint.util import MintDaemonThread
from time import sleep
@@ -11,7 +12,7 @@
def __init__(self, app, server_url):
self.app = app
self.server_url = server_url
- self.thread = PlumageSessionThread(app, server_url)
+ self.threads = []
def check(self):
log.info("Checking %s", self)
@@ -19,21 +20,52 @@
def init(self):
log.info("Initializing %s", self)
+ # Apply the package filter to the class list
+ if len(self.app.classes):
+ black_list = set()
+ for cls in self.app.classes:
+ if cls._package._name in self.app.package_filter:
+ black_list.add(cls)
+ else:
+ # We're going to have a thread for this...
+ self.threads.append(PlumageSessionThread(
+ self.app,
+ self.server_url,
+ cls))
+
+ # Update our list, minus the black_list
+ self.app.classes.difference_update(black_list)
+
+ else:
+ # Generate the package list from the model, minus
+ # the package filter
+ for pkg in self.app.model._packages:
+ if pkg._name not in self.app.package_filter:
+ self.app.packages.add(pkg)
+ for cls in pkg._classes:
+ self.threads.append(PlumageSessionThread(
+ self.app,
+ self.server_url,
+ cls))
+
def start(self):
log.info("Starting %s", self)
- self.thread.start()
+ for t in self.threads:
+ t.start()
def stop(self):
log.info("Stopping %s", self)
- self.thread.stop()
+ for t in self.threads:
+ t.stop()
def __repr__(self):
return "%s(%s)" % (self.__class__.__name__, self.server_url)
class PlumageSessionThread(MintDaemonThread):
- def __init__(self, app, server_url):
+ def __init__(self, app, server_url, cls):
super(PlumageSessionThread, self).__init__(app)
self.server_url = server_url
+ self.cls = cls
def run(self):
uuid = 1
@@ -44,7 +76,7 @@
# We create objects here. Tag them with the right class,
# probably specified to us from a config option (with a corresponding
# query specification in the xml)
- cls = self.app.model.com_redhat_grid_plumage.OSUtil
+ #cls = self.app.model.com_redhat_grid_plumage.OSUtil
class OSUtil(object):
pass
@@ -55,6 +87,7 @@
os.uuid = str(uuid)
os.osName = "Linux"
+ os.update_time = time.time() * 1000000000
# random, 16 slots
os.avail = randrange(0, 17)
@@ -62,7 +95,7 @@
# some number of available used
os.used = randrange(0, os.avail+1)
- obj = ObjectUpdate(self.app.model, os, cls)
+ obj = ObjectUpdate(self.app.model, os, self.cls)
self.app.update_thread.enqueue(obj)
sleep(1)
12 years
r5287 - in branches/play_plumage: cumin/model/plumage cumin/python/cumin mint/python/mint/plumage rosemary/python/rosemary
by tmckay@fedoraproject.org
Author: tmckay
Date: 2012-04-09 19:48:49 +0000 (Mon, 09 Apr 2012)
New Revision: 5287
Modified:
branches/play_plumage/cumin/model/plumage/plumage.xml
branches/play_plumage/cumin/python/cumin/model.py
branches/play_plumage/cumin/python/cumin/stat.py
branches/play_plumage/mint/python/mint/plumage/session.py
branches/play_plumage/mint/python/mint/plumage/update.py
branches/play_plumage/rosemary/python/rosemary/model.py
branches/play_plumage/rosemary/python/rosemary/sqltype.py
Log:
cumin-report which generates false data, complete with objects/samples in the db
Add obj.get_signature() and cls.get_object_by_signature()
Modified: branches/play_plumage/cumin/model/plumage/plumage.xml
===================================================================
--- branches/play_plumage/cumin/model/plumage/plumage.xml 2012-04-09 16:56:04 UTC (rev 5286)
+++ branches/play_plumage/cumin/model/plumage/plumage.xml 2012-04-09 19:48:49 UTC (rev 5287)
@@ -1,11 +1,13 @@
<?xml version="1.0" encoding="utf-8"?>
-<schema package="com.redhat.grid.plumage">
- <class name="OSUtil" qmf_headers="False">
- <property name="uuid" index="y" type="uuid" access="RC" desc="UUID of System Image"/>
+<schema package="com.redhat.grid.plumage" package_type="plumage">
+ <class name="OSUtil">
+ <property name="uuid" unique="y"
+ type="uuid" access="RC" desc="UUID of System Image"/>
- <property name="osName" type="sstr" access="RO" desc="Operating System Name" />
+ <property name="osName"
+ type="sstr" access="RO" desc="Operating System Name" />
- <statistic name="avail" type="uint32" unit="uint32" />
- <statistic name="used" type="uint32" unit="uint32" />
+ <statistic name="avail" type="uint32" unit="uint32" />
+ <statistic name="used" type="uint32" unit="uint32" />
</class>
</schema>
Modified: branches/play_plumage/cumin/python/cumin/model.py
===================================================================
--- branches/play_plumage/cumin/python/cumin/model.py 2012-04-09 16:56:04 UTC (rev 5286)
+++ branches/play_plumage/cumin/python/cumin/model.py 2012-04-09 19:48:49 UTC (rev 5287)
@@ -320,20 +320,13 @@
class SamplesSqlAdapter(SqlAdapter):
qmf_update_col = '_qmf_update_time'
- def __init__(self, app, table, agent_id, object_id):
- super(SamplesSqlAdapter, self).__init__(app, table)
+ def __init__(self, app, cls, sig):
+ super(SamplesSqlAdapter, self).__init__(app, cls.sql_samples_table)
- assert ";" not in agent_id
- assert ";" not in object_id
+ filters = cls.get_sample_filters_by_signature(sig)
+ for f in filters:
+ self.query.add_filter(f)
- col = self.table._qmf_agent_id
- filter = SqlComparisonFilter(col, "'%s'" % agent_id)
- self.query.add_filter(filter)
-
- col = self.table._qmf_object_id
- filter = SqlComparisonFilter(col, "'%s'" % object_id)
- self.query.add_filter(filter)
-
def get_sql_options(self, options):
return options
Modified: branches/play_plumage/cumin/python/cumin/stat.py
===================================================================
--- branches/play_plumage/cumin/python/cumin/stat.py 2012-04-09 16:56:04 UTC (rev 5286)
+++ branches/play_plumage/cumin/python/cumin/stat.py 2012-04-09 19:48:49 UTC (rev 5287)
@@ -154,10 +154,14 @@
object = self.object.get(session)
params = list()
- # open-flash-chart can't handle the : in the agent id. Net even if
- # it's converted to %3A, so we have to change it
- params.append("agent=%s" % object._qmf_agent_id.replace(":", "|"))
- params.append("object_id=%s" % object._qmf_object_id)
+ # Ask the object for a list of values that uniquely identifies it.
+ # Signature is the name of a ListParameter that will hold the values
+ # as an ordered list. The values can be passed back to the class to
+ # retrieve the object (or its samples)
+ signature = object.get_signature()
+ for s in signature:
+ params.append("signature=%s" % s)
+
params.append("chart_id=%s" % self.render_id(session, None))
params.append("duration=%s" % self.duration.get(session))
params.append("rpkg=%s" % object._class._package._name)
@@ -340,12 +344,10 @@
self.rosemary_package = Parameter(app, "rpkg")
self.add_parameter(self.rosemary_package)
- self.agent_id = ModifiedAgentIdParameter(app, "agent")
- self.add_parameter(self.agent_id)
+ param = StringParameter(app, "sigparam")
+ self.object_signature = ListParameter(app, "signature", param)
+ self.add_parameter(self.object_signature)
- self.object_id = StringParameter(app, "object_id")
- self.add_parameter(self.object_id)
-
param = Parameter(app, "param")
self.stats = ListParameter(app, "stat", param)
self.add_parameter(self.stats)
@@ -389,15 +391,15 @@
return self.samples.get(session) and "text/plain" or "image/png"
def get_object_property(self, session, property):
- agent_id = self.agent_id.get(session)
-
rpackage = self.rosemary_package.get(session)
rclass = self.rosemary_class.get(session)
+ signature = self.object_signature.get(session)
+
rosemary_package = self.app.model._packages_by_name[rpackage]
cls = rosemary_package._classes_by_name[rclass]
-
cursor = self.app.database.get_read_cursor()
- object = cls.get_object(cursor, _qmf_agent_id=agent_id)
+ object = cls.get_object_by_signature(cursor, signature)
+
return object.get_value(property)
def get_adapter_stats(self, session):
@@ -406,15 +408,9 @@
rosemary_package = self.app.model._packages_by_name[rpackage]
rosemary_class = rosemary_package._classes_by_name[rclass]
- table = rosemary_class.sql_samples_table
- agent_id = self.agent_id.get(session)
- object_id = self.object_id.get(session)
+ signature = self.object_signature.get(session)
+ adapter = SamplesSqlAdapter(self.app, rosemary_class, signature)
- assert agent_id
- assert object_id
-
- adapter = SamplesSqlAdapter(self.app, table, agent_id, object_id)
-
stats = [getattr(rosemary_class, x) for x in self.stats.get(session)]
return (adapter, stats)
@@ -480,12 +476,10 @@
self.updater.update_enabled = True
self.add_child(self.updater)
- self.object_id = StringParameter(app, "object_id")
- self.add_parameter(self.object_id)
+ param = StringParameter(app, "sigparam")
+ self.object_signature = ListParameter(app, "signature", param)
+ self.add_parameter(self.object_signature)
- self.agent_id = ModifiedAgentIdParameter(app, "agent")
- self.add_parameter(self.agent_id)
-
self.rosemary_class = Parameter(app, "rcls")
self.add_parameter(self.rosemary_class)
@@ -531,14 +525,13 @@
if not obj:
rpackage = self.widget.rosemary_package.get(session)
rclass = self.widget.rosemary_class.get(session)
+ signature = self.widget.object_signature.get(session)
rosemary_package = self.app.model._packages_by_name[rpackage]
rosemary_class = rosemary_package._classes_by_name[rclass]
- agent_id = self.widget.agent_id.get(session)
cursor = self.app.database.get_read_cursor()
+ obj = rosemary_class.get_object_by_signature(cursor, signature)
- obj = rosemary_class.get_object(cursor, _qmf_agent_id=agent_id)
-
self.set(session, obj)
return obj
Modified: branches/play_plumage/mint/python/mint/plumage/session.py
===================================================================
--- branches/play_plumage/mint/python/mint/plumage/session.py 2012-04-09 16:56:04 UTC (rev 5286)
+++ branches/play_plumage/mint/python/mint/plumage/session.py 2012-04-09 19:48:49 UTC (rev 5287)
@@ -2,6 +2,8 @@
from mint.util import MintDaemonThread
from time import sleep
+from update import ObjectUpdate
+from random import randrange
log = logging.getLogger("mint.plumage.session")
@@ -9,7 +11,8 @@
def __init__(self, app, server_url):
self.app = app
self.server_url = server_url
-
+ self.thread = PlumageSessionThread(app, server_url)
+
def check(self):
log.info("Checking %s", self)
@@ -18,25 +21,57 @@
def start(self):
log.info("Starting %s", self)
+ self.thread.start()
def stop(self):
log.info("Stopping %s", self)
+ self.thread.stop()
def __repr__(self):
return "%s(%s)" % (self.__class__.__name__, self.server_url)
class PlumageSessionThread(MintDaemonThread):
- def __init__(self, app):
- super(PumageSessionThread, self).__init__(app)
+ def __init__(self, app, server_url):
+ super(PlumageSessionThread, self).__init__(app)
+ self.server_url = server_url
def run(self):
+ uuid = 1
while True:
if self.stop_requested:
break
- # We create objects here
- sleep(5)
+ # We create objects here. Tag them with the right class,
+ # probably specified to us from a config option (with a corresponding
+ # query specification in the xml)
+ cls = self.app.model.com_redhat_grid_plumage.OSUtil
+ class OSUtil(object):
+ pass
+
+ # We need to get some data, and we need to turn it into some kind
+ # of update object.
+ os = OSUtil()
+
+ os.uuid = str(uuid)
+ os.osName = "Linux"
+
+ # random, 16 slots
+ os.avail = randrange(0, 17)
+
+ # some number of available used
+ os.used = randrange(0, os.avail+1)
+
+ obj = ObjectUpdate(self.app.model, os, cls)
+ self.app.update_thread.enqueue(obj)
+
+ sleep(1)
+
+ # five machines
+ uuid = (uuid + 1) % 6
+ if uuid == 0:
+ uuid = 1
+
'''
class MintConsole(Console):
def __init__(self, model):
Modified: branches/play_plumage/mint/python/mint/plumage/update.py
===================================================================
--- branches/play_plumage/mint/python/mint/plumage/update.py 2012-04-09 16:56:04 UTC (rev 5286)
+++ branches/play_plumage/mint/python/mint/plumage/update.py 2012-04-09 19:48:49 UTC (rev 5287)
@@ -1,6 +1,7 @@
import copy
import resource
import pickle
+from rosemary.sqltype import qmf_type_code_by_string
from psycopg2 import IntegrityError, TimestampFromTicks
from psycopg2.extensions import cursor as Cursor
@@ -14,6 +15,12 @@
sample_window_min = 60
sample_window_max = 60 * 5
+class UpdateDropped(Exception):
+ pass
+
+class MappingException(Exception):
+ pass
+
class UpdateThread(MintDaemonThread):
def __init__(self, app):
super(UpdateThread, self).__init__(app)
@@ -94,13 +101,14 @@
self.stats.sql_ops += 1
class Update(object):
- def __init__(self, model):
+ def __init__(self, model, cls):
self.model = model
+ self.cls = cls
def process(self, thread):
log.debug("Processing %s", self)
try:
- self.do_process(thread.cursor, thread.stats, thread.app.packages)
+ self.do_process(thread.cursor, thread.stats)
thread.conn.commit()
except UpdateDropped:
log.debug("Update dropped")
@@ -118,55 +126,34 @@
if thread.halt_on_error:
raise
- def do_process(self, cursor, stats, packages):
+ def do_process(self, cursor, stats):
raise Exception("Not implemented")
def __repr__(self):
return self.__class__.__name__
class ObjectUpdate(Update):
- def __init__(self, model, plumage_object):
- super(ObjectUpdate, self).__init__(model)
+ def __init__(self, model, plumage_object, cls):
+ super(ObjectUpdate, self).__init__(model, cls)
self.plumage_object = plumage_object
- def do_process(self, cursor, stats, bound_classes, bound_packages):
- cls = self.get_class()
- object_id = self.get_object_id()
-
+ def do_process(self, cursor, stats):
# Okay, we want to do a lookup by object id here.
- # If we can't find it, we create it. Agent is
- # a mint Agent class.
- # We need a lookup by field on machine.
-
+ # Let the class find the unique identifiers in the
+ # object based on the xml table description and do
+ # the lookup for us. Names in the record must match
+ # the names in the xml, or we need a mapping construct
+ # in the rosemary.xml to override.
+ sig = self.cls.get_signature(self.plumage_object)
try:
- obj = agent.get_object( cursor, cls, object_id)
- except RosemaryNotFound:
- obj = self.create_object(cursor, stats, cls)
+ obj = self.cls.get_object_by_signature(cursor, sig)
+ except:
+ self.create_object(cursor, stats, self.cls)
return
self.update_object(cursor, stats, obj)
- def maybe_drop_sample(self, obj):
- properties = self.qmf_object.getProperties()
- statistics = self.qmf_object.getStatistics()
-
- if not properties and statistics:
- # Just stats; do we want it?
- # if stats.enqueued - stats.dequeued > 500:
-
- now = time.time()
- update = self.qmf_object.getTimestamps()[0] / 1000000000
- sample = obj._sample_time
-
- if update < now - sample_window_max:
- # The sample is too old
- raise UpdateDropped()
-
- if sample and sample > now - sample_window_min:
- # The samples are too fidelitous
- raise UpdateDropped()
-
def create_object(self, cursor, stats, cls):
obj = cls.create_object(cursor)
@@ -177,7 +164,7 @@
object_columns.append(table._id)
- self.process_properties(obj, object_columns, cursor)
+ self.process_properties(obj, object_columns)
self.process_statistics(obj, object_columns, sample_columns)
statements = list()
@@ -204,13 +191,12 @@
object_columns = list()
sample_columns = list()
- self.process_properties(obj, object_columns, cursor)
+ self.process_properties(obj, object_columns)
self.process_statistics(obj, object_columns, sample_columns)
statements = list()
cls = obj._class
- # force a write if it's been too long, even if the values match
if object_columns:
sql = cls.sql_update_object.emit(object_columns)
statements.append(sql)
@@ -232,73 +218,56 @@
stats.objects_updated += 1
#stats.objects_updated_by_class[cls] += 1
- def process_properties(self, obj, columns, cursor):
+ def process_properties(self, obj, columns):
cls = obj._class
- # getProperties comes from console.py
- # list of tuples, first is an instance
- # which can be spoofed, second is a value for the property
-
- for prop, value in self.qmf_object.getProperties():
- try:
- col, nvalue = self.process_value(cls, prop, value)
- except MappingException, e:
- log.debug(e)
+ # get a list of property names from the cls
+ # loop through the list of properties, look for
+ # the value in the object
+ for name, prop in cls._properties_by_name.iteritems():
+ value = getattr(self.plumage_object, name)
+ if value is not None:
+ value = transform_value(prop, value)
+ if value == getattr(obj, prop.sql_column.name):
continue
- # XXX This optimization will be obsolete when QMF does it
- # instead
- if nvalue == getattr(obj, col.name):
- continue
+ setattr(obj, prop.sql_column.name, value)
+ columns.append(prop.sql_column)
- setattr(obj, col.name, nvalue)
- columns.append(col)
-
- def process_value(self, cls, prop, value):
- try:
- col = cls._properties_by_name[prop.name].sql_column
- except KeyError:
- raise MappingException("Property %s is unknown" % prop)
-
- if value is not None:
- value = transform_value(prop, value)
-
- return col, value
-
def process_statistics(self, obj, update_columns, insert_columns):
build_columns = list()
saw_change = False
+ cls = obj._class
-
- # getStatistics comes from console.py
- # list of tuples, first is an instance
- # which can be spoofed, second is a value for the property
-
- for stat, value in self.qmf_object.getStatistics():
- try:
- col = obj._class._statistics_by_name[stat.name].sql_column
- except KeyError:
- log.debug("Statistic %s is unknown", stat)
-
- continue
-
+ # get a list of statistic names from the cls
+ # loop through the list of properties, look for
+ # the value in the object
+ for name, stat in cls._statistics_by_name.iteritems():
+ value = getattr(self.plumage_object, name)
if value is not None:
value = transform_value(stat, value)
# Don't write unchanged values
- #
- # XXX This optimization will be obsolete when QMF does it
- # instead
- if value != getattr(obj, col.name):
- setattr(obj, col.name, value)
- update_columns.append(col)
+ if value != getattr(obj, stat.sql_column.name):
+ setattr(obj, stat.sql_column.name, value)
+ update_columns.append(stat.sql_column)
saw_change = True
# If we do end up seeing a value change, we will
# need to insert an entire row so build it up
- build_columns.append(col)
+ build_columns.append(stat.sql_column)
- if saw_change:
+ if saw_change:
+ # Okay, we saw some changed statistics. We also
+ # need to include here all the property columns marked
+ # unique since they are reflected in the sample tables
+ # in addition to the statistics.
+ key_cols = []
+ for p in cls._properties:
+ if p.unique:
+ key_cols.append(p.sql_column)
+
+ insert_columns.extend(key_cols)
insert_columns.extend(build_columns)
def execute_sql(self, cursor, text, args):
@@ -318,14 +287,14 @@
log.error("object properties:")
# see process_properties
- for item in sorted(self.qmf_object.getProperties()):
- log.error(" %-34s %r", *item)
+ #for item in sorted(self.qmf_object.getProperties()):
+ # log.error(" %-34s %r", *item)
- log.error("object statistics:")
+ #log.error("object statistics:")
# see process statistics
- for item in sorted(self.qmf_object.getStatistics()):
- log.error(" %-34s %r", *item)
+ #for item in sorted(self.qmf_object.getStatistics()):
+ # log.error(" %-34s %r", *item)
raise
@@ -360,4 +329,5 @@
transformers[15] = transform_pickle
def transform_value(attr, value):
- return transformers[attr.type](value)
+ t = qmf_type_code_by_string[attr.type]
+ return transformers[t](value)
Modified: branches/play_plumage/rosemary/python/rosemary/model.py
===================================================================
--- branches/play_plumage/rosemary/python/rosemary/model.py 2012-04-09 16:56:04 UTC (rev 5286)
+++ branches/play_plumage/rosemary/python/rosemary/model.py 2012-04-09 19:48:49 UTC (rev 5287)
@@ -4,6 +4,7 @@
from sqltype import *
import util
from cumin.util import xml_escape
+from parsley.collectionsex import defaultdict
log = logging.getLogger("rosemary.model")
@@ -66,7 +67,8 @@
attr = elem.get("package")
if attr:
- pkg = RosemaryPackage(self, attr)
+ pkg_type = elem.get("package_type", None)
+ pkg = RosemaryPackage(self, attr, pkg_type)
pkg.load(elem)
# This on the other hand is a tag itself. It should
@@ -76,7 +78,8 @@
# stuff
# </package>
for child in elem.findall("package"):
- pkg = RosemaryPackage(self, child.get("name"))
+ pkg_type = child.get("package_type", None)
+ pkg = RosemaryPackage(self, child.get("name"), pkg_type)
pkg.load(child)
def extend(self, elem):
@@ -100,16 +103,20 @@
if len(objects) > 0:
youngest = objects[0]
for obj in objects:
- if obj._qmf_update_time > youngest._qmf_update_time:
+ if obj.get_timestamp() > youngest.get_timestamp():
youngest = obj
except:
pass
return youngest
class RosemaryPackage(object):
- def __init__(self, model, name):
+ def __init__(self, model, name, pkg_type):
self._model = model
self._name = name
+ self.pkg_type = pkg_type
+ if self.pkg_type in (None, ""):
+ self.pkg_type = "qmf"
+ self.xml_class = self.get_xml_class()
# Set in the extension xml file. This marks a package as
# containing persistent data which should not be deleted
@@ -147,9 +154,26 @@
groups_by_name[name] = child
for child in elem.findall("class"):
- cls = RosemaryClass(self, child.get("name"), child.get("qmf_headers"))
+ cls = self.xml_class(self, child.get("name"))
cls.load(child, groups_by_name)
+ def get_xml_class(self):
+ # Split the pkg_type specifier on last .
+ # Last item is the class type to use for this package.
+ # The convention is that type, capitalized and appended with "_class",
+ # names a class corresponding to the xml definition.
+ # Previous items name a module under rosemary.
+ try:
+ mdl, cls = self.pkg_type.rsplit(".", 1)
+ except ValueError:
+ # no dot, just the class is given
+ # default module to 'model'
+ mdl = "model"
+ cls = self.pkg_type
+ m = sys.modules["rosemary." + mdl.lower()]
+ cls = m.__dict__.get(cls.capitalize() + "_class")
+ return cls
+
def extend(self, elem):
persistent = elem.get("persistent")
if persistent is not None:
@@ -172,7 +196,7 @@
return "%s(%s)" % args
class RosemaryClass(object):
- def __init__(self, package, name, qmf_headers):
+ def __init__(self, package, name):
self._package = package
self._name = name
@@ -215,35 +239,20 @@
self._indexes = list()
self._indexes_by_name = dict()
- self._id = RosemaryAttribute(self, "_id")
+ self.add_unique_to_samples = False
- self.qmf_headers = qmf_headers is None or qmf_headers.lower() == "true"
- if self.qmf_headers:
- self.add_headers()
+ # Ditionary of tuples, indicates which
+ # columns are unique (either single columns
+ # or groups of columns)
+ self._unique_col_groups = defaultdict(list)
- def add_headers(self):
- name = "_qmf_agent_id"
- self._qmf_agent_id = RosemaryHeader(self, name, sql_text)
- self._qmf_agent_id.title = "Agent ID"
+ # Similar for samples, indicate which indices
+ # should be generated for the sample tables
+ self._sample_index_col_groups = defaultdict(list)
- name = "_qmf_object_id"
- self._qmf_object_id = RosemaryHeader(self, name, sql_text)
- self._qmf_object_id.title = "Object ID"
+ self._id = RosemaryAttribute(self, "_id")
+ self.qmf_headers = False
- name = "_qmf_create_time"
- self._qmf_create_time = RosemaryHeader(self, name, sql_timestamp)
- self._qmf_create_time.title = "Create Time"
-
- name = "_qmf_update_time"
- self._qmf_update_time = RosemaryHeader(self, name, sql_timestamp)
- self._qmf_update_time.title = "Update Time"
-
- name = "_qmf_delete_time"
- self._qmf_delete_time = RosemaryHeader(self, name, sql_timestamp)
- self._qmf_delete_time.title = "Delete Time"
-
- self._qmf_delete_time.optional = True
-
def load(self, elem, groups_by_name):
log.debug("Loading %s", self)
@@ -253,7 +262,6 @@
for child in elem.findall("group"):
name = child.get("name")
-
try:
self.load_class_components(groups_by_name[name])
except KeyError:
@@ -266,12 +274,10 @@
def load_class_components(self, elem):
for child in elem.findall("property"):
name = child.get("name")
-
if child.get("references"):
attr = RosemaryReference(self, name)
else:
attr = RosemaryProperty(self, name)
-
attr.load(child)
for child in elem.findall("statistic"):
@@ -330,6 +336,10 @@
hdr.init()
for prop in self._properties:
+ # Properties will add themselves to
+ # self._unique_col_groups.
+ # The unique constraints will
+ # be generated in add_sql_contraints.
prop.init()
for stat in self._statistics:
@@ -343,6 +353,7 @@
self.add_sql_constraints()
self.add_sql_operations()
+ self.add_sql_sample_indices()
def init_references(self):
for ref in self._references:
@@ -361,18 +372,16 @@
SqlPrimaryKeyConstraint(self.sql_table, name, (id_col,))
def add_sql_constraints(self):
- if self.qmf_headers:
- name = "%s_qmf_id_uq" % self._name
- cols = (self._qmf_agent_id.sql_column, self._qmf_object_id.sql_column)
- SqlUniqueConstraint(self.sql_table, name, cols)
+ for k,v in self._unique_col_groups.iteritems():
+ SqlUniqueConstraint(self.sql_table, k, v)
+ def add_sql_sample_indices(self):
+ for k,v in self._sample_index_col_groups.iteritems():
+ SqlIndex(self._package.sql_schema, k, v)
+
def add_sql_operations(self):
self.sql_get_new_id = SqlGetNewId(self.sql_table, self.sql_sequence)
self.sql_select_by_id = SqlSelectObject(self.sql_table)
-
- if self.qmf_headers:
- self.sql_select_by_qmf_id = SqlSelectObjectByQmfId(self.sql_table)
-
self.sql_insert_object = SqlInsertObject(self.sql_table)
self.sql_update_object = SqlUpdateObject(self.sql_table)
self.sql_delete_object = SqlDeleteObject(self.sql_table)
@@ -382,20 +391,7 @@
table = SqlTable(self._package.sql_schema, name)
self.sql_samples_table = table
-
- if self.qmf_headers:
- agent_col = SqlColumn(table, "_qmf_agent_id", sql_text)
- object_col = SqlColumn(table, "_qmf_object_id", sql_text)
-
- name = "%s_qmf_id_idx" % table._name
- SqlIndex(self._package.sql_schema, name, (agent_col, object_col))
-
- SqlColumn(table, "_qmf_update_time", sql_timestamp)
-
self.sql_samples_insert = SqlInsertObjectSamples(table, self.qmf_headers)
- if self.qmf_headers:
- # We will need another way to delete non-qmf samples
- self.sql_samples_delete = SqlDeleteObjectSamples(table)
def get_object(self, cursor, **criteria):
sql = SqlSelect(self.sql_table)
@@ -413,7 +409,7 @@
if not record:
return
- obj = RosemaryObject(self, None)
+ obj = self._get_rosemary_object(None)
self._set_object_attributes(obj, self.sql_table._columns, record)
@@ -432,7 +428,7 @@
sql.execute(cursor, criteria)
for record in cursor.fetchall():
- obj = RosemaryObject(self, None)
+ obj = self._get_rosemary_object(None)
self._set_object_attributes(obj, self.sql_table._columns, record)
@@ -451,7 +447,7 @@
sql.execute(cursor, criteria)
for record in cursor.fetchall():
- obj = RosemaryObject(self, None)
+ obj = self._get_rosemary_object(None)
self.set_object_attributes(obj, self.sql_table._columns, record)
@@ -462,20 +458,12 @@
def get_object_by_id(self, cursor, id):
assert id
- obj = RosemaryObject(self, id)
+ obj = self._get_rosemary_object(id)
self.load_object_by_id(cursor, obj)
return obj
- def get_object_by_qmf_id(self, cursor, agent_id, object_id):
- assert self.qmf_headers
- obj = RosemaryObject(self, None)
- obj._qmf_agent_id = agent_id
- obj._qmf_object_id = object_id
- self.load_object_by_qmf_id(cursor, obj)
- return obj
-
def get_new_id(self, cursor):
self.sql_get_new_id.execute(cursor)
@@ -485,11 +473,17 @@
return values[0]
+ # Allow this to be overloaded so that
+ # descendents of RosemaryClass can produce
+ # descendents of RosemaryObject in create_object
+ # and other places
+ def _get_rosemary_object(self, id):
+ return RosemaryObject(self, id)
+
def create_object(self, cursor):
id = self.get_new_id(cursor)
+ return self._get_rosemary_object(id)
- return RosemaryObject(self, id)
-
def load_object_by_id(self, cursor, obj):
assert isinstance(obj, RosemaryObject)
assert obj._id, obj
@@ -503,21 +497,6 @@
self._set_object_attributes(obj, self.sql_table._columns, values)
- def load_object_by_qmf_id(self, cursor, obj):
- assert isinstance(obj, RosemaryObject)
- assert self.qmf_headers
- assert obj._qmf_agent_id, obj
- assert obj._qmf_object_id, obj
-
- self.sql_select_by_qmf_id.execute(cursor, obj.__dict__)
-
- values = cursor.fetchone()
-
- if not values:
- raise RosemaryNotFound()
-
- self._set_object_attributes(obj, self.sql_table._columns, values)
-
def _set_object_attributes(self, obj, columns, values):
assert isinstance(obj, RosemaryObject)
@@ -572,6 +551,182 @@
args = (self.__class__.__name__, self._package._name, self._name)
return "%s(%s,%s)" % args
+class Qmf_class(RosemaryClass):
+ def __init__(self, package, name):
+ super(Qmf_class, self).__init__(package, name)
+
+ self.qmf_headers = True
+ name = "_qmf_agent_id"
+ self._qmf_agent_id = RosemaryHeader(self, name, sql_text)
+ self._qmf_agent_id.title = "Agent ID"
+
+ name = "_qmf_object_id"
+ self._qmf_object_id = RosemaryHeader(self, name, sql_text)
+ self._qmf_object_id.title = "Object ID"
+
+ name = "_qmf_create_time"
+ self._qmf_create_time = RosemaryHeader(self, name, sql_timestamp)
+ self._qmf_create_time.title = "Create Time"
+
+ name = "_qmf_update_time"
+ self._qmf_update_time = RosemaryHeader(self, name, sql_timestamp)
+ self._qmf_update_time.title = "Update Time"
+
+ name = "_qmf_delete_time"
+ self._qmf_delete_time = RosemaryHeader(self, name, sql_timestamp)
+ self._qmf_delete_time.title = "Delete Time"
+ self._qmf_delete_time.optional = True
+
+ def add_sql_operations(self):
+ super(Qmf_class, self).add_sql_operations()
+ self.sql_select_by_qmf_id = SqlSelectObjectByQmfId(self.sql_table)
+
+ def add_sql_constraints(self):
+ # Add agent and object columns as a unique column set
+ name = "%s_qmf_id_uq" % self._name
+ self._unique_col_groups[name] = (self._qmf_agent_id.sql_column,
+ self._qmf_object_id.sql_column)
+ super(Qmf_class, self).add_sql_constraints()
+
+ def add_samples_sql(self):
+ super(Qmf_class, self).add_samples_sql()
+ agent_col = SqlColumn(self.sql_samples_table,
+ "_qmf_agent_id", sql_text)
+
+ object_col = SqlColumn(self.sql_samples_table,
+ "_qmf_object_id", sql_text)
+
+ name = "%s_qmf_id_idx" % self.sql_samples_table._name
+ SqlIndex(self._package.sql_schema, name, (agent_col, object_col))
+
+ SqlColumn(self.sql_samples_table, "_qmf_update_time", sql_timestamp)
+
+ # We will need another way to delete non-qmf samples
+ self.sql_samples_delete = SqlDeleteObjectSamples(self.sql_samples_table)
+
+ def get_object_by_qmf_id(self, cursor, agent_id, object_id):
+ obj = self._get_rosemary_object(None)
+ obj._qmf_agent_id = agent_id
+ obj._qmf_object_id = object_id
+ self.load_object_by_qmf_id(cursor, obj)
+ return obj
+
+ def load_object_by_qmf_id(self, cursor, obj):
+ assert isinstance(obj, RosemaryObject)
+ assert self.qmf_headers
+ assert obj._qmf_agent_id, obj
+ assert obj._qmf_object_id, obj
+
+ self.sql_select_by_qmf_id.execute(cursor, obj.__dict__)
+
+ values = cursor.fetchone()
+
+ if not values:
+ raise RosemaryNotFound()
+
+ self._set_object_attributes(obj, self.sql_table._columns, values)
+
+ def get_signature(self, obj):
+ # Return an ordered set of values that can uniquely
+ # identify the object
+ return [obj._qmf_agent_id.replace(":", "|"),
+ obj._qmf_object_id]
+
+ def get_object_by_signature(self, cursor, signature):
+ # Use the signature to find the object
+ agent_id, obj_id = signature
+ # open-flash-chart can't handle the : in the agent id. Net even if
+ # it's converted to %3A, so we have to change it
+ agent_id = agent_id.replace("|", ":")
+ return self.get_object_by_qmf_id(cursor, agent_id, obj_id)
+
+ def get_sample_filters_by_signature(self, signature):
+ agent_id, obj_id = signature
+ agent_id = agent_id.replace("|", ":")
+ filters = []
+
+ col = self.sql_samples_table._qmf_agent_id
+ filters.append(SqlComparisonFilter(col, "'%s'" % agent_id))
+
+ col = self.sql_samples_table._qmf_object_id
+ filters.append(SqlComparisonFilter(col, "'%s'" % obj_id))
+
+ return filters
+
+ # Allow this to be overloaded so that
+ # descendents of RosemaryClass can produce
+ # descendents of RosemaryObject in create_object
+ # and other places
+ def _get_rosemary_object(self, id):
+ return RosemaryQmfObject(self, id)
+
+class Plumage_class(RosemaryClass):
+ def __init__(self, package, name):
+ super(Plumage_class, self).__init__(package, name)
+
+ # Columns marked unique will be added to the
+ # sample table
+ self.add_unique_to_samples = True
+
+ def init(self):
+ super(Plumage_class, self).init()
+
+ # Store a sorted list of the names of unique
+ # column groups
+ self.col_group_list = self._unique_col_groups.keys()
+ self.col_group_list.sort()
+
+ # Todo: this could be optimized if we use the shortest
+ # set of unique column groups but probably there is only one
+ self.signature_cols = self._unique_col_groups[self.col_group_list[0]]
+
+ # Set up a query that we can use on a unique
+ # column set
+ self.sql_select_signature = SqlSelect(self.sql_table)
+ for c in self.signature_cols:
+ self.sql_select_signature.add_filter(SqlValueFilter(c))
+
+ def load_class_components(self, elem):
+ for child in elem.findall("property"):
+ name = child.get("name")
+ attr = RosemaryProperty(self, name)
+ attr.load(child)
+
+ for child in elem.findall("statistic"):
+ stat = RosemaryStatistic(self, child.get("name"))
+ stat.load(child)
+
+ def get_signature(self, obj):
+ # Return an ordered set of values that can uniquely
+ # identify the object.
+ # For plumage objects, this is going to be the list
+ # of values in the columns marked unique.
+ sig = []
+ for c in self.signature_cols:
+ # For each col, record the value from the object
+ sig.append(getattr(obj, c.name))
+ return sig
+
+ def get_object_by_signature(self, cursor, signature):
+ # signature is an ordered list of values corresponding
+ # to self.signature_cols
+ obj = self._get_rosemary_object(None)
+ sig_idx = 0
+ for c in self.signature_cols:
+ setattr(obj, c.name, signature[sig_idx])
+ sig_idx += 1
+ self.load_object_by_signature(cursor, obj)
+ return obj
+
+ def load_object_by_signature(self, cursor, obj):
+ assert isinstance(obj, RosemaryObject)
+
+ self.sql_select_signature.execute(cursor, obj.__dict__)
+ values = cursor.fetchone()
+ if not values:
+ raise RosemaryNotFound()
+ self._set_object_attributes(obj, self.sql_table._columns, values)
+
class RosemaryAttribute(object):
def __init__(self, cls, name):
self.cls = cls
@@ -598,6 +753,19 @@
self.sql_column = None
+ def process_unique(self, value):
+ # If unique is None, "y" or "n" then it
+ # is a boolean value and the column is not part
+ # of a group, it is simply unique by itself
+ if value is None:
+ return False
+ elif value in ("n", "y"):
+ return value == "y"
+
+ # Otherwise, the column is part of a multi-field
+ # unique set where value is the name of the set
+ return value
+
def load(self, elem):
self.type = elem.get("type")
self.references = elem.get("references")
@@ -605,12 +773,12 @@
self.unit = elem.get("unit")
self.index = elem.get("index", "n") == "y" and True
self.optional = elem.get("optional", "n") == "y" and True
+ self.unique = self.process_unique(elem.get("unique"))
self.description = clean_description(elem.get("desc"))
def extend(self, elem):
self.title = elem.findtext("title")
self.short_title = elem.findtext("short")
- self.unique = elem.get("unique", "n") == "y" and True
formatter = elem.findtext("formatter")
# Allow overloading of optional in the extensions
@@ -620,6 +788,11 @@
if optional is not None:
self.optional = optional == "y"
+ # Same here
+ unique = elem.get("unique")
+ if unique is not None:
+ self.unique = self.process_unique(unique)
+
if formatter:
meth = getattr(util, formatter, None)
@@ -712,10 +885,32 @@
self.sql_column = SqlColumn(self.cls.sql_table, self.name, type)
self.sql_column.nullable = self.optional
- if self.unique:
- name = "%s_%s_uq" % (self.cls._name, self.name)
- SqlUniqueConstraint(self.cls.sql_table, name, (self.sql_column,))
+ # Save away refernces for generation of unique constraints and indices
+ # If it's a single column marked unique, save it under its own name.
+ if self.unique == True:
+ name = "%s_%s_uq" % (self.cls.sql_table._name, self.name)
+ self.cls._unique_col_groups[name] = (self.sql_column,)
+
+ # If it's part of a group, append it to a group of columns named
+ # for the group.
+ elif self.unique:
+ name = "%s_%s_uq" % (self.cls.sql_table._name, self.unique)
+ self.cls._unique_col_groups[name].append(self.sql_column)
+ # For some classes, add unique columns to the sample table, too.
+ # Produce an index for the column(s) based on the 'unique' value.
+ # These columns take the place of qmf_agent_id and qmf_object_id
+ # for non-qmf records
+ if self.cls.add_unique_to_samples and self.unique:
+ col = SqlColumn(self.cls.sql_samples_table, self.name, type)
+ col.nullable = self.optional
+ if self.unique == True:
+ name = "%s_%s_idx" % (self.cls.sql_samples_table._name, self.name)
+ self.cls._sample_index_col_groups[name] = (col,)
+ else:
+ name = "%s_%s_idx" % (self.cls.sql_samples_table._name, self.unique)
+ self.cls._sample_index_col_groups[name].append(col)
+
class RosemaryStatistic(RosemaryAttribute):
def __init__(self, cls, name):
super(RosemaryStatistic, self).__init__(cls, name)
@@ -842,12 +1037,6 @@
def add_sample(self, cursor, columns=None):
self._class.add_object_sample(cursor, self, columns)
- def fake_qmf_values(self):
- self._qmf_agent_id = "__rosemary__"
- self._qmf_object_id = str(datetime.now())
- self._qmf_create_time = datetime.now()
- self._qmf_update_time = datetime.now()
-
def get_title(self):
if self._class._object_title:
return self._class._object_title % self.__dict__
@@ -877,11 +1066,30 @@
return value
+ def get_timestamp(self):
+ raise Exception("Not implemented")
+
def __repr__(self):
name = self.__class__.__name__
args = (name, self._class, self._id, self._save_time)
return "%s(%s,%i,%s)" % args
+class RosemaryQmfObject(RosemaryObject):
+ def fake_qmf_values(self):
+ self._qmf_agent_id = "__rosemary__"
+ self._qmf_object_id = str(datetime.now())
+ self._qmf_create_time = datetime.now()
+ self._qmf_update_time = datetime.now()
+
+ def get_timestamp(self):
+ return self._qmf_update_time
+
+ def get_signature(self):
+ # return an ordered list of values that can uniquely
+ # identify this object. Let the class determine
+ # the list.
+ return self._class.get_signature(self)
+
class RosemaryNotFound(Exception):
pass
Modified: branches/play_plumage/rosemary/python/rosemary/sqltype.py
===================================================================
--- branches/play_plumage/rosemary/python/rosemary/sqltype.py 2012-04-09 16:56:04 UTC (rev 5286)
+++ branches/play_plumage/rosemary/python/rosemary/sqltype.py 2012-04-09 19:48:49 UTC (rev 5287)
@@ -76,3 +76,4 @@
sql_types_by_qmf_type_code = dict([(x[1], x[0]) for x in __mappings])
sql_types_by_qmf_type_string = dict([(x[2], x[0]) for x in __mappings])
+qmf_type_code_by_string = dict([(x[2], x[1]) for x in __mappings])
12 years
r5286 - trunk/sage/python/sage/wallaby
by croberts@fedoraproject.org
Author: croberts
Date: 2012-04-09 16:56:04 +0000 (Mon, 09 Apr 2012)
New Revision: 5286
Modified:
trunk/sage/python/sage/wallaby/wallabyoperations.py
Log:
Additional check for None for BZ 802704
Modified: trunk/sage/python/sage/wallaby/wallabyoperations.py
===================================================================
--- trunk/sage/python/sage/wallaby/wallabyoperations.py 2012-04-09 16:45:49 UTC (rev 5285)
+++ trunk/sage/python/sage/wallaby/wallabyoperations.py 2012-04-09 16:56:04 UTC (rev 5286)
@@ -332,7 +332,7 @@
d = self._cache[which].data.values()
# Here we handle the possible filtering of node names
if which == WBTypes.NODES:
- if valuefilter["nodeName"] != "%%%":
+ if valuefilter is not None and valuefilter["nodeName"] != "%%%":
filter = valuefilter["nodeName"].replace("%", "")
if filter != "":
d = [value for value in d if value.name.find(filter) > -1]
12 years
r5285 - in trunk: cumin/python/cumin/inventory sage/python/sage/wallaby
by croberts@fedoraproject.org
Author: croberts
Date: 2012-04-09 16:45:49 +0000 (Mon, 09 Apr 2012)
New Revision: 5285
Modified:
trunk/cumin/python/cumin/inventory/system.py
trunk/sage/python/sage/wallaby/wallabyoperations.py
Log:
Fixing BZ 802704 by allowing the wallaby node list to be restricted to the same value that the sesame node names are restricted by.
Modified: trunk/cumin/python/cumin/inventory/system.py
===================================================================
--- trunk/cumin/python/cumin/inventory/system.py 2012-04-04 19:36:33 UTC (rev 5284)
+++ trunk/cumin/python/cumin/inventory/system.py 2012-04-09 16:45:49 UTC (rev 5285)
@@ -238,7 +238,7 @@
sqldata = super(WallabyAndSqlAdapter, self).get_data(values, options)
#now get the wallaby data
- wallaby_nodes = self.app.wallaby.get_data(WBTypes.NODES)
+ wallaby_nodes = self.app.wallaby.get_data(WBTypes.NODES, values)
data = list()
if len(sqldata) > 0 and len(wallaby_nodes) > 0: #means that we have both sesame and wallaby data
Modified: trunk/sage/python/sage/wallaby/wallabyoperations.py
===================================================================
--- trunk/sage/python/sage/wallaby/wallabyoperations.py 2012-04-04 19:36:33 UTC (rev 5284)
+++ trunk/sage/python/sage/wallaby/wallabyoperations.py 2012-04-09 16:45:49 UTC (rev 5285)
@@ -315,7 +315,7 @@
finally:
self._condition.release()
- def get_data(self, which):
+ def get_data(self, which, valuefilter=None):
'''
Return a list of cached values for the specified category.
@@ -330,6 +330,12 @@
try:
if which in self._cache:
d = self._cache[which].data.values()
+ # Here we handle the possible filtering of node names
+ if which == WBTypes.NODES:
+ if valuefilter["nodeName"] != "%%%":
+ filter = valuefilter["nodeName"].replace("%", "")
+ if filter != "":
+ d = [value for value in d if value.name.find(filter) > -1]
finally:
self._lock.release()
return d
12 years
r5284 - trunk/wooly/python/wooly
by tmckay@fedoraproject.org
Author: tmckay
Date: 2012-04-04 19:36:33 +0000 (Wed, 04 Apr 2012)
New Revision: 5284
Modified:
trunk/wooly/python/wooly/forms.py
Log:
Add escaping to FormError widgets.
BZ809369
Modified: trunk/wooly/python/wooly/forms.py
===================================================================
--- trunk/wooly/python/wooly/forms.py 2012-04-02 15:53:30 UTC (rev 5283)
+++ trunk/wooly/python/wooly/forms.py 2012-04-04 19:36:33 UTC (rev 5284)
@@ -53,7 +53,7 @@
class FormError(object):
def __init__(self, message):
- self.message = message
+ self.message = xml_escape(message)
def get_message(self, session):
return self.message
12 years, 1 month
r5283 - branches/play_plumage/cumin/python/cumin
by croberts@fedoraproject.org
Author: croberts
Date: 2012-04-02 15:53:30 +0000 (Mon, 02 Apr 2012)
New Revision: 5283
Modified:
branches/play_plumage/cumin/python/cumin/main.py
Log:
Adding plumage to the model_dir to pick up the schema for the plumage/reporting data.
Modified: branches/play_plumage/cumin/python/cumin/main.py
===================================================================
--- branches/play_plumage/cumin/python/cumin/main.py 2012-04-02 15:33:14 UTC (rev 5282)
+++ branches/play_plumage/cumin/python/cumin/main.py 2012-04-02 15:53:30 UTC (rev 5283)
@@ -39,7 +39,7 @@
self.home = home
self.authmech = authmech
- model_dir = [os.path.join(self.home, x) for x in ("model/admin", "model")]
+ model_dir = [os.path.join(self.home, x) for x in ("model/admin", "model", "model/plumage")]
self.model = CuminModel(self, model_dir)
self.session = CuminSession(self, broker_uris)
12 years, 1 month
r5282 - branches/play_plumage/rosemary/bin
by tmckay@fedoraproject.org
Author: tmckay
Date: 2012-04-02 15:33:14 +0000 (Mon, 02 Apr 2012)
New Revision: 5282
Modified:
branches/play_plumage/rosemary/bin/rosemary-model
Log:
Fix up rosemary-model to be more useful with multiple subdirectories
Modified: branches/play_plumage/rosemary/bin/rosemary-model
===================================================================
--- branches/play_plumage/rosemary/bin/rosemary-model 2012-03-30 20:32:08 UTC (rev 5281)
+++ branches/play_plumage/rosemary/bin/rosemary-model 2012-04-02 15:33:14 UTC (rev 5282)
@@ -80,11 +80,16 @@
if __name__ == "__main__":
model = RosemaryModel()
- model.load_model_dir(os.path.join(os.environ["ROSEMARY_HOME"], "model"))
- model.init()
-
if len(sys.argv) == 1:
- print "model, ddl, dml"
+ print "rosemary-model (model | ddl | dml) [model dir under $ROSEMARY_HOME]"
sys.exit(1)
- globals()["do_%s" % sys.argv[1]](sys.argv[2:])
+ if len(sys.argv) > 2:
+ mdir = sys.argv[2]
+ else:
+ mdir = "model"
+
+ model.load_model_dir(os.path.join(os.environ["ROSEMARY_HOME"], mdir))
+ model.init()
+
+ globals()["do_%s" % sys.argv[1]](sys.argv[3:])
12 years, 1 month