https://www.aeolusproject.org/redmine/projects/aeolus/wiki/Allow_Conductor_t...
The document talks about two possible approaches to keeping deployment activity; one is to use an archive table, and another is to use a soft_delete flag. I'd be interested in hearing people's thoughts on both.
Oh, and the document also talks about potentially using dbomatic for periodic history purges. I know there's been some discussion around dbomatic, so if someone had an opinion about that as well. . .
Mainn
On Fri, Mar 30, 2012 at 11:59:11AM -0400, Tzu-Mainn Chen wrote:
https://www.aeolusproject.org/redmine/projects/aeolus/wiki/Allow_Conductor_t...
So I wrote the below email and was about to send it, when I realized that I'd focused on the technical implementation details without ever fully understanding the overall use case. But it's not abundantly clear to me what that entails. What information, exactly, are we intending to keep? I wonder if we should look at stashing core information in the Events table?
The document talks about two possible approaches to keeping deployment activity; one is to use an archive table, and another is to use a soft_delete flag. I'd be interested in hearing people's thoughts on both.
I was going to mention that I'm still fond of acts_as_paranoid, but I just saw your mention on the wiki page that Tomas sent a patch adding archivist six months ago. I'm all for just using that in this case. I think we're likely to need to stitch together associations anyway when we're dealing with deleted objects.
Though, FWIW, using soft delete wouldn't necessarily require any major changes. acts_as_paranoid would override the finder methods of AR to add a "WHERE deleted_at IS NULL" to all queries unless you ran "find_with_deleted". If we did this ourselves, we could probably just use a default scope and a with_deleted method to override it.
Oh, and the document also talks about potentially using dbomatic for periodic history purges. I know there's been some discussion around dbomatic, so if someone had an opinion about that as well. . .
I think I talked to you on IRC about this and didn't note this objection, so I apologize for suddenly raising it. ;) But it occurs to me now that cleaning up old history is something that we want to do perhaps once a month, not every couple of minutes, so maybe we should just use cron. What if we added a rake task / little script that did this, using a configurable time period? Then we could ship with a reasonable default (do a monthly cleanup of entries older than 6 months?), but allow administrators to change it by just editing crontab. (I'm also not entirely convinced that deleting archived entries by default is a good thing... Some users might want to preserve history indefinitely. This way they could at least disable it easily.)
One additional note -- the last sentence of your "Summary" section includes this interesting tidbit: "Finally, historical event data will need to be searched efficiently. In order to do so, we will evaluate Katello's search implementation and see if it is suitable."
That sounds like a really good idea, although I suspect it'll be a good bit of work, enough that I wonder if it should be its own feature. I think we'd need to work with Katello to make sure it was appropriately configured for multi-tenancy, and make sure that Katello and Aeolus both included a dependency on it, plus make sure that we had consistent configuration so that if one project set it up the other project could use it without problems. It's definitely achievable and likely worthwhile; I just think it deserves scoping as a bigger task. (IMHO.)
-- Matt
On 04/02/2012 09:49 AM, Matt Wagner wrote:
On Fri, Mar 30, 2012 at 11:59:11AM -0400, Tzu-Mainn Chen wrote:
https://www.aeolusproject.org/redmine/projects/aeolus/wiki/Allow_Conductor_t...
So I wrote the below email and was about to send it, when I realized that I'd focused on the technical implementation details without ever fully understanding the overall use case. But it's not abundantly clear to me what that entails. What information, exactly, are we intending to keep? I wonder if we should look at stashing core information in the Events table?
I'd imagine Hugh or someone else with a better feel for customer requirements may need to weigh in, but in general part of what we may be dealing with are audit requirements, especially if we ever start modeling cost issues. We may need to be able to produce a report on a user's history/activities, when instances were up, which ones, who changed properties, etc.
The "soft delete" vs. "archiving" is only part of this, though. We need it so that references in events, etc. to deleted items are still valid. However we also need to fully spec out the event/auditing requirements (not part of this task it seems). What changes do we need to record? What history do we need for what object types? Is there some process for purging in-db events/archives to some "long-term" storage/report format?
The document talks about two possible approaches to keeping deployment activity; one is to use an archive table, and another is to use a soft_delete flag. I'd be interested in hearing people's thoughts on both.
I was going to mention that I'm still fond of acts_as_paranoid, but I just saw your mention on the wiki page that Tomas sent a patch adding archivist six months ago. I'm all for just using that in this case. I think we're likely to need to stitch together associations anyway when we're dealing with deleted objects.
Though, FWIW, using soft delete wouldn't necessarily require any major changes. acts_as_paranoid would override the finder methods of AR to add a "WHERE deleted_at IS NULL" to all queries unless you ran "find_with_deleted". If we did this ourselves, we could probably just use a default scope and a with_deleted method to override it.
Yeah -- the document didn't make it clear whether you'd looked into the existing rails plugins for this, but when I looked into this a couple years ago there were options for both soft_delete (i.e. acts_as_paranoid) or archive tables (I forget what I found here). Also, while the finder overrides will make _most_ instances of deleted objects disappear, you've still got to be careful with joins, custom queries, etc. I'd imagine that even with acts_as_paranoid we'll need to be careful to make sure that our code doesn't find deleted stuff when we don't want it -- and that we've got decent automated test coverage to confirm that deleted things don't start appearing with future code changes.
The real pain with deletion isn't the object itself but the associations. Not just for query filtering (w/ soft-delete) but also w/ foreign key constraints (for 'archive tables').
Oh, and the document also talks about potentially using dbomatic for periodic history purges. I know there's been some discussion around dbomatic, so if someone had an opinion about that as well. . .
I think I talked to you on IRC about this and didn't note this objection, so I apologize for suddenly raising it. ;) But it occurs to me now that cleaning up old history is something that we want to do perhaps once a month, not every couple of minutes, so maybe we should just use cron. What if we added a rake task / little script that did this, using a configurable time period? Then we could ship with a reasonable default (do a monthly cleanup of entries older than 6 months?), but allow administrators to change it by just editing crontab. (I'm also not entirely convinced that deleting archived entries by default is a good thing... Some users might want to preserve history indefinitely. This way they could at least disable it easily.)
In addition, there may be other actions that need to occur in conjunction with purging history. Something as simple as a log file logging each deleted item (so there's still some possibility to manually recover old archive data if needed), or perhaps someting more complex. Also the 'deleted item' purging and old events purging should probably done in conjunction with each other.
One additional note -- the last sentence of your "Summary" section includes this interesting tidbit: "Finally, historical event data will need to be searched efficiently. In order to do so, we will evaluate Katello's search implementation and see if it is suitable."
With soft-delete, we've got search -- but perhaps even the standard list UI will help us -- we can provide an 'include deleted' flag for admins to see deleted stuff too.
That sounds like a really good idea, although I suspect it'll be a good bit of work, enough that I wonder if it should be its own feature. I think we'd need to work with Katello to make sure it was appropriately configured for multi-tenancy, and make sure that Katello and Aeolus both included a dependency on it, plus make sure that we had consistent configuration so that if one project set it up the other project could use it without problems. It's definitely achievable and likely worthwhile; I just think it deserves scoping as a bigger task. (IMHO.)
-- Matt
Thanks for the comments! My comments in-line:
----- Original Message -----
From: "Matt Wagner" matt.wagner@redhat.com To: "Tzu-Mainn Chen" tzumainn@redhat.com Cc: aeolus-devel@lists.fedorahosted.org Sent: Monday, April 2, 2012 9:49:02 AM Subject: Re: RFC Feature Planning: Deployment Activity History
On Fri, Mar 30, 2012 at 11:59:11AM -0400, Tzu-Mainn Chen wrote:
https://www.aeolusproject.org/redmine/projects/aeolus/wiki/Allow_Conductor_t...
So I wrote the below email and was about to send it, when I realized that I'd focused on the technical implementation details without ever fully understanding the overall use case. But it's not abundantly clear to me what that entails. What information, exactly, are we intending to keep? I wonder if we should look at stashing core information in the Events table?
To be honest, I assumed that the scope of the task was simply to create a way to store events, and then worry about (potentially) adding events later. I'd hope that the implementation certainly wouldn't preclude that possibility, and expandability is definitely something worth keeping in mind.
The document talks about two possible approaches to keeping deployment activity; one is to use an archive table, and another is to use a soft_delete flag. I'd be interested in hearing people's thoughts on both.
I was going to mention that I'm still fond of acts_as_paranoid, but I just saw your mention on the wiki page that Tomas sent a patch adding archivist six months ago. I'm all for just using that in this case. I think we're likely to need to stitch together associations anyway when we're dealing with deleted objects.
Though, FWIW, using soft delete wouldn't necessarily require any major changes. acts_as_paranoid would override the finder methods of AR to add a "WHERE deleted_at IS NULL" to all queries unless you ran "find_with_deleted". If we did this ourselves, we could probably just use a default scope and a with_deleted method to override it.
I personally prefer the soft delete approach, because my experience with playing with associations is not good; it's possible a bad association took my lunch money in grade school.
Oh, and the document also talks about potentially using dbomatic for periodic history purges. I know there's been some discussion around dbomatic, so if someone had an opinion about that as well. . .
I think I talked to you on IRC about this and didn't note this objection, so I apologize for suddenly raising it. ;) But it occurs to me now that cleaning up old history is something that we want to do perhaps once a month, not every couple of minutes, so maybe we should just use cron. What if we added a rake task / little script that did this, using a configurable time period? Then we could ship with a reasonable default (do a monthly cleanup of entries older than 6 months?), but allow administrators to change it by just editing crontab. (I'm also not entirely convinced that deleting archived entries by default is a good thing... Some users might want to preserve history indefinitely. This way they could at least disable it easily.)
There's a separate task (specified in the document) that specifies that the administrator should be able to configure history purges. I should add details to that section to specify what I think that means. In any case, I disagree that cleaning up old history is a once-a-month task, but the point about cron still stands. What I *do* want to avoid is the sense that Conductor is using multiple schedulers for multiple tasks when one scheduler could do; I don't think that's a pleasant user experience. Ideally, though, the action and the scheduler should be reasonably discrete. I'll make a note of the issue in the documentation.
One additional note -- the last sentence of your "Summary" section includes this interesting tidbit: "Finally, historical event data will need to be searched efficiently. In order to do so, we will evaluate Katello's search implementation and see if it is suitable."
That sounds like a really good idea, although I suspect it'll be a good bit of work, enough that I wonder if it should be its own feature. I think we'd need to work with Katello to make sure it was appropriately configured for multi-tenancy, and make sure that Katello and Aeolus both included a dependency on it, plus make sure that we had consistent configuration so that if one project set it up the other project could use it without problems. It's definitely achievable and likely worthwhile; I just think it deserves scoping as a bigger task. (IMHO.)
-- Matt
----- Original Message -----
From: "Scott Seago" sseago@redhat.com To: aeolus-devel@lists.fedorahosted.org Sent: Monday, April 2, 2012 11:15:47 AM Subject: Re: RFC Feature Planning: Deployment Activity History
On 04/02/2012 09:49 AM, Matt Wagner wrote:
On Fri, Mar 30, 2012 at 11:59:11AM -0400, Tzu-Mainn Chen wrote:
https://www.aeolusproject.org/redmine/projects/aeolus/wiki/Allow_Conductor_t...
So I wrote the below email and was about to send it, when I realized that I'd focused on the technical implementation details without ever fully understanding the overall use case. But it's not abundantly clear to me what that entails. What information, exactly, are we intending to keep? I wonder if we should look at stashing core information in the Events table?
I'd imagine Hugh or someone else with a better feel for customer requirements may need to weigh in, but in general part of what we may be dealing with are audit requirements, especially if we ever start modeling cost issues. We may need to be able to produce a report on a user's history/activities, when instances were up, which ones, who changed properties, etc.
The "soft delete" vs. "archiving" is only part of this, though. We need it so that references in events, etc. to deleted items are still valid. However we also need to fully spec out the event/auditing requirements (not part of this task it seems). What changes do we need to record? What history do we need for what object types? Is there some process for purging in-db events/archives to some "long-term" storage/report format?
The document talks about two possible approaches to keeping deployment activity; one is to use an archive table, and another is to use a soft_delete flag. I'd be interested in hearing people's thoughts on both.
I was going to mention that I'm still fond of acts_as_paranoid, but I just saw your mention on the wiki page that Tomas sent a patch adding archivist six months ago. I'm all for just using that in this case. I think we're likely to need to stitch together associations anyway when we're dealing with deleted objects.
Though, FWIW, using soft delete wouldn't necessarily require any major changes. acts_as_paranoid would override the finder methods of AR to add a "WHERE deleted_at IS NULL" to all queries unless you ran "find_with_deleted". If we did this ourselves, we could probably just use a default scope and a with_deleted method to override it.
Yeah -- the document didn't make it clear whether you'd looked into the existing rails plugins for this, but when I looked into this a couple years ago there were options for both soft_delete (i.e. acts_as_paranoid) or archive tables (I forget what I found here). Also, while the finder overrides will make _most_ instances of deleted objects disappear, you've still got to be careful with joins, custom queries, etc. I'd imagine that even with acts_as_paranoid we'll need to be careful to make sure that our code doesn't find deleted stuff when we don't want it -- and that we've got decent automated test coverage to confirm that deleted things don't start appearing with future code changes.
The real pain with deletion isn't the object itself but the associations. Not just for query filtering (w/ soft-delete) but also w/ foreign key constraints (for 'archive tables').
Yeah, definitely agreed. I'm strongly in favor of soft-deletion for that reason. In my mind, soft-deletion would only have to be added to deployments (with the requisite logic trickling down), while archiving could potentially end up with a diverse ecology of objects needing to be archived.
Another argument in favor of soft deletion is this: if we add a new association to deployments, under archive-tables, we'd potentially have to add an entirely new class of objects to the archive. Under soft deletion, we'd only have to remember to add a filter (which, as Matt says, could be essentially the default mode).
I'll look more closely into existing rails soft-deletion plugins.
Oh, and the document also talks about potentially using dbomatic for periodic history purges. I know there's been some discussion around dbomatic, so if someone had an opinion about that as well. . .
I think I talked to you on IRC about this and didn't note this objection, so I apologize for suddenly raising it. ;) But it occurs to me now that cleaning up old history is something that we want to do perhaps once a month, not every couple of minutes, so maybe we should just use cron. What if we added a rake task / little script that did this, using a configurable time period? Then we could ship with a reasonable default (do a monthly cleanup of entries older than 6 months?), but allow administrators to change it by just editing crontab. (I'm also not entirely convinced that deleting archived entries by default is a good thing... Some users might want to preserve history indefinitely. This way they could at least disable it easily.)
In addition, there may be other actions that need to occur in conjunction with purging history. Something as simple as a log file logging each deleted item (so there's still some possibility to manually recover old archive data if needed), or perhaps someting more complex.
In my opinion, this shouldn't be Conductor's problem; database backups are common and (I think) expected for any critical application. But I could be wrong :)
Also the 'deleted item' purging and old events purging should probably done in conjunction with each other.
Agreed, I'll add further specification to the documentation.
One additional note -- the last sentence of your "Summary" section includes this interesting tidbit: "Finally, historical event data will need to be searched efficiently. In order to do so, we will evaluate Katello's search implementation and see if it is suitable."
With soft-delete, we've got search -- but perhaps even the standard list UI will help us -- we can provide an 'include deleted' flag for admins to see deleted stuff too.
That sounds like a really good idea, although I suspect it'll be a good bit of work, enough that I wonder if it should be its own feature. I think we'd need to work with Katello to make sure it was appropriately configured for multi-tenancy, and make sure that Katello and Aeolus both included a dependency on it, plus make sure that we had consistent configuration so that if one project set it up the other project could use it without problems. It's definitely achievable and likely worthwhile; I just think it deserves scoping as a bigger task. (IMHO.)
-- Matt
On Tue, Apr 03, 2012 at 12:05:48AM -0400, Tzu-Mainn Chen wrote:
Thanks for the comments! My comments in-line:
----- Original Message -----
From: "Matt Wagner" matt.wagner@redhat.com To: "Tzu-Mainn Chen" tzumainn@redhat.com Cc: aeolus-devel@lists.fedorahosted.org Sent: Monday, April 2, 2012 9:49:02 AM Subject: Re: RFC Feature Planning: Deployment Activity History
On Fri, Mar 30, 2012 at 11:59:11AM -0400, Tzu-Mainn Chen wrote:
https://www.aeolusproject.org/redmine/projects/aeolus/wiki/Allow_Conductor_t...
So I wrote the below email and was about to send it, when I realized that I'd focused on the technical implementation details without ever fully understanding the overall use case. But it's not abundantly clear to me what that entails. What information, exactly, are we intending to keep? I wonder if we should look at stashing core information in the Events table?
To be honest, I assumed that the scope of the task was simply to create a way to store events, and then worry about (potentially) adding events later. I'd hope that the implementation certainly wouldn't preclude that possibility, and expandability is definitely something worth keeping in mind.
So I certainly don't mean to point the finger at you here, because I think you're right about this task only referring to the technical changes. But I will note that this is the second major "Events" task to be entered without any user stories about what history we want to save, or why. Normally I'm not too big of a stickler about user stories, but in the case of storing events, I think it's really pretty important to know *what* we want to save. I suspect you've done the best that can be done without more background, but I think we could make more progress if we knew what it is that we're building.
The document talks about two possible approaches to keeping deployment activity; one is to use an archive table, and another is to use a soft_delete flag. I'd be interested in hearing people's thoughts on both.
I was going to mention that I'm still fond of acts_as_paranoid, but I just saw your mention on the wiki page that Tomas sent a patch adding archivist six months ago. I'm all for just using that in this case. I think we're likely to need to stitch together associations anyway when we're dealing with deleted objects.
Though, FWIW, using soft delete wouldn't necessarily require any major changes. acts_as_paranoid would override the finder methods of AR to add a "WHERE deleted_at IS NULL" to all queries unless you ran "find_with_deleted". If we did this ourselves, we could probably just use a default scope and a with_deleted method to override it.
I personally prefer the soft delete approach, because my experience with playing with associations is not good; it's possible a bad association took my lunch money in grade school.
has_no :lunch_money ?
I think this is a good point, though. We have some code to check if things have the "last reference" to something on has_many relationships, which would probably result in things being deleted if we switched to archive tables.
Although, actually, we're still going to have to deal with this. Take deployments, which can't be deleted if they still have instances. (I'm going from memory here, so if that's not true, pretend it is.) Now if you "delete" all the instances, that code needs to be changed to look for non-deleted instances only, and to do a soft-delete on the deployment in that case. So I'm left wondering if maybe we're going to need to shape some code to suit our needs whether we use archive tables or soft deletes.
(Note, though, that I'm personally in the "soft delete" camp myself, I'm just thinking of ways to not reinvent the wheel after Tomas already sent out an archive table patch a few months ago.)
Oh, and the document also talks about potentially using dbomatic for periodic history purges. I know there's been some discussion around dbomatic, so if someone had an opinion about that as well. . .
I think I talked to you on IRC about this and didn't note this objection, so I apologize for suddenly raising it. ;) But it occurs to me now that cleaning up old history is something that we want to do perhaps once a month, not every couple of minutes, so maybe we should just use cron. What if we added a rake task / little script that did this, using a configurable time period? Then we could ship with a reasonable default (do a monthly cleanup of entries older than 6 months?), but allow administrators to change it by just editing crontab. (I'm also not entirely convinced that deleting archived entries by default is a good thing... Some users might want to preserve history indefinitely. This way they could at least disable it easily.)
There's a separate task (specified in the document) that specifies that the administrator should be able to configure history purges. I should add details to that section to specify what I think that means. In any case, I disagree that cleaning up old history is a once-a-month task, but the point about cron still stands. What I *do* want to avoid is the sense that Conductor is using multiple schedulers for multiple tasks when one scheduler could do; I don't think that's a pleasant user experience. Ideally, though, the action and the scheduler should be reasonably discrete. I'll make a note of the issue in the documentation.
Oh, hmm -- does a "configurable policy" mean that you should be able to change it through the web interface? Because that potentially means that we can't "just use cron" here. Though I can see how being able to change this on the fly would actually be a bad thing -- someone accidentally changes the archive period to 1 hour, saves, loses almost all history, and then changes it back to 90 days and everyone is left wondering where the history went. (That said, I'm really not a big fan of disabling features because someone could misuse them. But it makes me wonder if maybe the interval should just be set in a config file, not on-the-fly.)
-- Matt
----- Original Message -----
From: "Matt Wagner" matt.wagner@redhat.com To: "Tzu-Mainn Chen" tzumainn@redhat.com Cc: aeolus-devel@lists.fedorahosted.org Sent: Tuesday, April 3, 2012 11:53:22 AM Subject: Re: RFC Feature Planning: Deployment Activity History
On Tue, Apr 03, 2012 at 12:05:48AM -0400, Tzu-Mainn Chen wrote:
Thanks for the comments! My comments in-line:
----- Original Message -----
From: "Matt Wagner" matt.wagner@redhat.com To: "Tzu-Mainn Chen" tzumainn@redhat.com Cc: aeolus-devel@lists.fedorahosted.org Sent: Monday, April 2, 2012 9:49:02 AM Subject: Re: RFC Feature Planning: Deployment Activity History
On Fri, Mar 30, 2012 at 11:59:11AM -0400, Tzu-Mainn Chen wrote:
https://www.aeolusproject.org/redmine/projects/aeolus/wiki/Allow_Conductor_t...
So I wrote the below email and was about to send it, when I realized that I'd focused on the technical implementation details without ever fully understanding the overall use case. But it's not abundantly clear to me what that entails. What information, exactly, are we intending to keep? I wonder if we should look at stashing core information in the Events table?
To be honest, I assumed that the scope of the task was simply to create a way to store events, and then worry about (potentially) adding events later. I'd hope that the implementation certainly wouldn't preclude that possibility, and expandability is definitely something worth keeping in mind.
So I certainly don't mean to point the finger at you here, because I think you're right about this task only referring to the technical changes. But I will note that this is the second major "Events" task to be entered without any user stories about what history we want to save, or why. Normally I'm not too big of a stickler about user stories, but in the case of storing events, I think it's really pretty important to know *what* we want to save. I suspect you've done the best that can be done without more background, but I think we could make more progress if we knew what it is that we're building.
No, that's a fair point. I'll update the doc and seek clarification.
The document talks about two possible approaches to keeping deployment activity; one is to use an archive table, and another is to use a soft_delete flag. I'd be interested in hearing people's thoughts on both.
I was going to mention that I'm still fond of acts_as_paranoid, but I just saw your mention on the wiki page that Tomas sent a patch adding archivist six months ago. I'm all for just using that in this case. I think we're likely to need to stitch together associations anyway when we're dealing with deleted objects.
Though, FWIW, using soft delete wouldn't necessarily require any major changes. acts_as_paranoid would override the finder methods of AR to add a "WHERE deleted_at IS NULL" to all queries unless you ran "find_with_deleted". If we did this ourselves, we could probably just use a default scope and a with_deleted method to override it.
I personally prefer the soft delete approach, because my experience with playing with associations is not good; it's possible a bad association took my lunch money in grade school.
has_no :lunch_money ?
I think this is a good point, though. We have some code to check if things have the "last reference" to something on has_many relationships, which would probably result in things being deleted if we switched to archive tables.
Although, actually, we're still going to have to deal with this. Take deployments, which can't be deleted if they still have instances. (I'm going from memory here, so if that's not true, pretend it is.) Now if you "delete" all the instances, that code needs to be changed to look for non-deleted instances only, and to do a soft-delete on the deployment in that case. So I'm left wondering if maybe we're going to need to shape some code to suit our needs whether we use archive tables or soft deletes.
(Note, though, that I'm personally in the "soft delete" camp myself, I'm just thinking of ways to not reinvent the wheel after Tomas already sent out an archive table patch a few months ago.)
Oh, and the document also talks about potentially using dbomatic for periodic history purges. I know there's been some discussion around dbomatic, so if someone had an opinion about that as well. . .
I think I talked to you on IRC about this and didn't note this objection, so I apologize for suddenly raising it. ;) But it occurs to me now that cleaning up old history is something that we want to do perhaps once a month, not every couple of minutes, so maybe we should just use cron. What if we added a rake task / little script that did this, using a configurable time period? Then we could ship with a reasonable default (do a monthly cleanup of entries older than 6 months?), but allow administrators to change it by just editing crontab. (I'm also not entirely convinced that deleting archived entries by default is a good thing... Some users might want to preserve history indefinitely. This way they could at least disable it easily.)
There's a separate task (specified in the document) that specifies that the administrator should be able to configure history purges. I should add details to that section to specify what I think that means. In any case, I disagree that cleaning up old history is a once-a-month task, but the point about cron still stands. What I *do* want to avoid is the sense that Conductor is using multiple schedulers for multiple tasks when one scheduler could do; I don't think that's a pleasant user experience. Ideally, though, the action and the scheduler should be reasonably discrete. I'll make a note of the issue in the documentation.
Oh, hmm -- does a "configurable policy" mean that you should be able to change it through the web interface? Because that potentially means that we can't "just use cron" here. Though I can see how being able to change this on the fly would actually be a bad thing -- someone accidentally changes the archive period to 1 hour, saves, loses almost all history, and then changes it back to 90 days and everyone is left wondering where the history went. (That said, I'm really not a big fan of disabling features because someone could misuse them. But it makes me wonder if maybe the interval should just be set in a config file, not on-the-fly.)
This might be a wider philosophical point, is the current standard to use a config file?
-- Matt
I've updated the wiki documentation:
https://www.aeolusproject.org/redmine/projects/aeolus/wiki/Allow_Conductor_t...
Things of note:
a) added in user scenarios that Angus created b) argued a bit more forcefully for the "soft delete" approach c) added a note about graphical views of the data, along with a link to a partial evaluation of graphical packages performed by some other group in Red Hat. Does anyone have personal experience dealing with graphical ruby plugins?
Mainn
----- Original Message -----
From: "Tzu-Mainn Chen" tzumainn@redhat.com To: "Matt Wagner" matt.wagner@redhat.com Cc: aeolus-devel@lists.fedorahosted.org Sent: Tuesday, April 3, 2012 12:04:19 PM Subject: Re: RFC Feature Planning: Deployment Activity History
----- Original Message -----
From: "Matt Wagner" matt.wagner@redhat.com To: "Tzu-Mainn Chen" tzumainn@redhat.com Cc: aeolus-devel@lists.fedorahosted.org Sent: Tuesday, April 3, 2012 11:53:22 AM Subject: Re: RFC Feature Planning: Deployment Activity History
On Tue, Apr 03, 2012 at 12:05:48AM -0400, Tzu-Mainn Chen wrote:
Thanks for the comments! My comments in-line:
----- Original Message -----
From: "Matt Wagner" matt.wagner@redhat.com To: "Tzu-Mainn Chen" tzumainn@redhat.com Cc: aeolus-devel@lists.fedorahosted.org Sent: Monday, April 2, 2012 9:49:02 AM Subject: Re: RFC Feature Planning: Deployment Activity History
On Fri, Mar 30, 2012 at 11:59:11AM -0400, Tzu-Mainn Chen wrote:
https://www.aeolusproject.org/redmine/projects/aeolus/wiki/Allow_Conductor_t...
So I wrote the below email and was about to send it, when I realized that I'd focused on the technical implementation details without ever fully understanding the overall use case. But it's not abundantly clear to me what that entails. What information, exactly, are we intending to keep? I wonder if we should look at stashing core information in the Events table?
To be honest, I assumed that the scope of the task was simply to create a way to store events, and then worry about (potentially) adding events later. I'd hope that the implementation certainly wouldn't preclude that possibility, and expandability is definitely something worth keeping in mind.
So I certainly don't mean to point the finger at you here, because I think you're right about this task only referring to the technical changes. But I will note that this is the second major "Events" task to be entered without any user stories about what history we want to save, or why. Normally I'm not too big of a stickler about user stories, but in the case of storing events, I think it's really pretty important to know *what* we want to save. I suspect you've done the best that can be done without more background, but I think we could make more progress if we knew what it is that we're building.
No, that's a fair point. I'll update the doc and seek clarification.
The document talks about two possible approaches to keeping deployment activity; one is to use an archive table, and another is to use a soft_delete flag. I'd be interested in hearing people's thoughts on both.
I was going to mention that I'm still fond of acts_as_paranoid, but I just saw your mention on the wiki page that Tomas sent a patch adding archivist six months ago. I'm all for just using that in this case. I think we're likely to need to stitch together associations anyway when we're dealing with deleted objects.
Though, FWIW, using soft delete wouldn't necessarily require any major changes. acts_as_paranoid would override the finder methods of AR to add a "WHERE deleted_at IS NULL" to all queries unless you ran "find_with_deleted". If we did this ourselves, we could probably just use a default scope and a with_deleted method to override it.
I personally prefer the soft delete approach, because my experience with playing with associations is not good; it's possible a bad association took my lunch money in grade school.
has_no :lunch_money ?
I think this is a good point, though. We have some code to check if things have the "last reference" to something on has_many relationships, which would probably result in things being deleted if we switched to archive tables.
Although, actually, we're still going to have to deal with this. Take deployments, which can't be deleted if they still have instances. (I'm going from memory here, so if that's not true, pretend it is.) Now if you "delete" all the instances, that code needs to be changed to look for non-deleted instances only, and to do a soft-delete on the deployment in that case. So I'm left wondering if maybe we're going to need to shape some code to suit our needs whether we use archive tables or soft deletes.
(Note, though, that I'm personally in the "soft delete" camp myself, I'm just thinking of ways to not reinvent the wheel after Tomas already sent out an archive table patch a few months ago.)
Oh, and the document also talks about potentially using dbomatic for periodic history purges. I know there's been some discussion around dbomatic, so if someone had an opinion about that as well. . .
I think I talked to you on IRC about this and didn't note this objection, so I apologize for suddenly raising it. ;) But it occurs to me now that cleaning up old history is something that we want to do perhaps once a month, not every couple of minutes, so maybe we should just use cron. What if we added a rake task / little script that did this, using a configurable time period? Then we could ship with a reasonable default (do a monthly cleanup of entries older than 6 months?), but allow administrators to change it by just editing crontab. (I'm also not entirely convinced that deleting archived entries by default is a good thing... Some users might want to preserve history indefinitely. This way they could at least disable it easily.)
There's a separate task (specified in the document) that specifies that the administrator should be able to configure history purges. I should add details to that section to specify what I think that means. In any case, I disagree that cleaning up old history is a once-a-month task, but the point about cron still stands. What I *do* want to avoid is the sense that Conductor is using multiple schedulers for multiple tasks when one scheduler could do; I don't think that's a pleasant user experience. Ideally, though, the action and the scheduler should be reasonably discrete. I'll make a note of the issue in the documentation.
Oh, hmm -- does a "configurable policy" mean that you should be able to change it through the web interface? Because that potentially means that we can't "just use cron" here. Though I can see how being able to change this on the fly would actually be a bad thing -- someone accidentally changes the archive period to 1 hour, saves, loses almost all history, and then changes it back to 90 days and everyone is left wondering where the history went. (That said, I'm really not a big fan of disabling features because someone could misuse them. But it makes me wonder if maybe the interval should just be set in a config file, not on-the-fly.)
This might be a wider philosophical point, is the current standard to use a config file?
-- Matt
On Wed, Apr 04, 2012 at 12:09:12PM -0400, Tzu-Mainn Chen wrote:
I've updated the wiki documentation:
https://www.aeolusproject.org/redmine/projects/aeolus/wiki/Allow_Conductor_t...
Things of note:
a) added in user scenarios that Angus created
Thanks for doing this! And I think it was quite valuable, because I had been viewing this as simply a tabular log of events. The uses cases help me see some use cases I hadn't anticipated before, e.g.:
"As a user, I want the option to filter that view by pool, pool family, front end realm, image or hardware profile" which implies a good deal of filtering, and ensuring that a lot of data is persisted as things get deleted.
Also: "I want an aggregated view which shows my total resource usage during a customisable period" implies that this "history of deployment activity" includes summarization and customized reporting.
As for the above use case, it's actually much larger: "As a user, I want an aggregated view which shows my total resource usage during a customisable period, represented in graphs which show the frequency of deployment launches and the total number of compute hours consumed, optionally filtered by deployable, by image, by hardware profile or by pool"
This task is much larger than I had expected. It's not just saving some data when things get deleted.
b) argued a bit more forcefully for the "soft delete" approach
I'm all for this, personally.
c) added a note about graphical views of the data, along with a link to a partial evaluation of graphical packages performed by some other group in Red Hat. Does anyone have personal experience dealing with graphical ruby plugins?
I used lazy_high_charts on a previous project. (The "lazy" and "high" are not both meant as adjectives, lest you think this is a stoner's graphing library. It's a "lazy" Ruby implementation for drawing Highcharts graphs, which the page you link to suggests is non-free.) So that's probably out.
I think D3[1] is the current rage. I've seen some graphs in Katello; we should see what they're using.
On 03/30/2012 05:59 PM, Tzu-Mainn Chen wrote:
https://www.aeolusproject.org/redmine/projects/aeolus/wiki/Allow_Conductor_t...
The document talks about two possible approaches to keeping deployment activity; one is to use an archive table, and another is to use a soft_delete flag. I'd be interested in hearing people's thoughts on both.
Oh, and the document also talks about potentially using dbomatic for periodic history purges. I know there's been some discussion around dbomatic, so if someone had an opinion about that as well. . .
Mainn
Hi Mainn, one nit which wasn't probably mentioned (or I missed it) is potential problem with unique keys: - primary keys - I think we use everywhere 'id' so it shouldn't be problem (as long as we don't use for example 'name' as primary key) - 'validates_uniqueness_of' for all models which use soft deletion will have to be updated (acts_as_paranoid doesn't re-implement this method)
Jan
Ah, thanks for pointing that out! I'll make sure it still works.
Mainn
----- Original Message -----
From: "Jan Provazník" jprovazn@redhat.com To: "Tzu-Mainn Chen" tzumainn@redhat.com Cc: aeolus-devel@lists.fedorahosted.org Sent: Friday, April 20, 2012 4:17:07 AM Subject: Re: RFC Feature Planning: Deployment Activity History
On 03/30/2012 05:59 PM, Tzu-Mainn Chen wrote:
https://www.aeolusproject.org/redmine/projects/aeolus/wiki/Allow_Conductor_t...
The document talks about two possible approaches to keeping deployment activity; one is to use an archive table, and another is to use a soft_delete flag. I'd be interested in hearing people's thoughts on both.
Oh, and the document also talks about potentially using dbomatic for periodic history purges. I know there's been some discussion around dbomatic, so if someone had an opinion about that as well. . .
Mainn
Hi Mainn, one nit which wasn't probably mentioned (or I missed it) is potential problem with unique keys:
- primary keys - I think we use everywhere 'id' so it shouldn't be
problem (as long as we don't use for example 'name' as primary key)
- 'validates_uniqueness_of' for all models which use soft deletion
will have to be updated (acts_as_paranoid doesn't re-implement this method)
Jan
On 04/20/2012 04:17 AM, Jan Provazník wrote:
On 03/30/2012 05:59 PM, Tzu-Mainn Chen wrote:
https://www.aeolusproject.org/redmine/projects/aeolus/wiki/Allow_Conductor_t...
The document talks about two possible approaches to keeping deployment activity; one is to use an archive table, and another is to use a soft_delete flag. I'd be interested in hearing people's thoughts on both.
Oh, and the document also talks about potentially using dbomatic for periodic history purges. I know there's been some discussion around dbomatic, so if someone had an opinion about that as well. . .
Mainn
Hi Mainn, one nit which wasn't probably mentioned (or I missed it) is potential problem with unique keys:
- primary keys - I think we use everywhere 'id' so it shouldn't be
problem (as long as we don't use for example 'name' as primary key)
- 'validates_uniqueness_of' for all models which use soft deletion
will have to be updated (acts_as_paranoid doesn't re-implement this method)
Jan
Jan, yes I'd forgotten entirely about that. Unique fields and deleted items were a big problem last time I worked with an in-house-designed soft-delete model. We ended up having to mangle names, etc. as part of the 'delete' action. You could get around this by hacking validates_uniqueness_of, but that's probably really painful. In addition, that won't work if you ever have a uniqueness constraint on the underlying db table, or if you have other custom validation methods that also have a problem with uniqueness.
Scott
On Fri, Apr 20, 2012 at 10:40:20AM -0400, Scott Seago wrote:
On 04/20/2012 04:17 AM, Jan Provazník wrote:
On 03/30/2012 05:59 PM, Tzu-Mainn Chen wrote:
https://www.aeolusproject.org/redmine/projects/aeolus/wiki/Allow_Conductor_t...
The document talks about two possible approaches to keeping deployment activity; one is to use an archive table, and another is to use a soft_delete flag. I'd be interested in hearing people's thoughts on both.
Oh, and the document also talks about potentially using dbomatic for periodic history purges. I know there's been some discussion around dbomatic, so if someone had an opinion about that as well. . .
Mainn
Hi Mainn, one nit which wasn't probably mentioned (or I missed it) is potential problem with unique keys:
- primary keys - I think we use everywhere 'id' so it shouldn't be
problem (as long as we don't use for example 'name' as primary key)
- 'validates_uniqueness_of' for all models which use soft deletion
will have to be updated (acts_as_paranoid doesn't re-implement this method)
Jan
Jan, yes I'd forgotten entirely about that. Unique fields and deleted items were a big problem last time I worked with an in-house-designed soft-delete model. We ended up having to mangle names, etc. as part of the 'delete' action. You could get around this by hacking validates_uniqueness_of, but that's probably really painful. In addition, that won't work if you ever have a uniqueness constraint on the underlying db table, or if you have other custom validation methods that also have a problem with uniqueness.
If we don't intend to allow people to un-delete records, I think we could make it work pretty easily with something like this:
validates_uniqueness_of :name, :unless => deleted_at
That does imply that you couldn't have a unique constraint in the database on :name, but I think that's an inevitable cost of doing this. (Though you could kind of hack it with a compound key, but it would be ugly and bad.)
-- Matt
----- Original Message -----
From: "Scott Seago" sseago@redhat.com To: aeolus-devel@lists.fedorahosted.org Sent: Friday, April 20, 2012 10:40:20 AM Subject: Re: RFC Feature Planning: Deployment Activity History
On 04/20/2012 04:17 AM, Jan Provazník wrote:
On 03/30/2012 05:59 PM, Tzu-Mainn Chen wrote:
https://www.aeolusproject.org/redmine/projects/aeolus/wiki/Allow_Conductor_t...
The document talks about two possible approaches to keeping deployment activity; one is to use an archive table, and another is to use a soft_delete flag. I'd be interested in hearing people's thoughts on both.
Oh, and the document also talks about potentially using dbomatic for periodic history purges. I know there's been some discussion around dbomatic, so if someone had an opinion about that as well. . .
Mainn
Hi Mainn, one nit which wasn't probably mentioned (or I missed it) is potential problem with unique keys:
- primary keys - I think we use everywhere 'id' so it shouldn't be
problem (as long as we don't use for example 'name' as primary key)
- 'validates_uniqueness_of' for all models which use soft deletion
will have to be updated (acts_as_paranoid doesn't re-implement this method)
Jan
Jan, yes I'd forgotten entirely about that. Unique fields and deleted items were a big problem last time I worked with an in-house-designed soft-delete model. We ended up having to mangle names, etc. as part of the 'delete' action. You could get around this by hacking validates_uniqueness_of, but that's probably really painful. In addition, that won't work if you ever have a uniqueness constraint on the underlying db table, or if you have other custom validation methods that also have a problem with uniqueness.
Scott
Actually, there seems to be a pretty simple solution:
validates_uniqueness_of :name, :scope => [:pool_id, :deleted_at]
Mainn
aeolus-devel@lists.fedorahosted.org