Hi,
to get some experience I tried how some migration to github might look like. I did that with my personal GitHb account, so this is not available in github/rhq-project (yet)
** First I basically followed the answer to this question on stack overflow to pull stuff from FH and pushed to GH
http://stackoverflow.com/questions/5181845/git-push-existing-repo-to-a-new-a...
I think for a real migration we would first do a
git fetch --all git remote add github <github url> and then a git push --mirror github
This should copy the whole repository inc. all branches and tags.
We may consider pruning old tags and branches here - especially as the FH repo is not going away.
** Then I went and cloned the repo locally and followed http://stackoverflow.com/a/359759/100957
to separate out the modules/plugins directory into its own repo. Created a new repo on github rhq-plugins and pushed the plugins to this new repo
** In the third step I cloned the github rhq repository and followed the first comment to http://stackoverflow.com/a/955793/100957 to clean out the plugins from above rhq repository and tried to push the result to github. This whole locally cleaning "took ages"
The push failed, as the local repo and the remote have diverged. One can use -f here to forcibly push (which I've done) Or in an earlier step not push the rhq repo until after the split-off of the plugins. Or in the main repo just rm -r && git rm on the plugins and have the removal recorded in the history. This may even be faster than pruning that directory hierarchy out.
Results are now at https://github.com/pilhuhn/rhq and https://github.com/pilhuhn/rhq-plugins
Heiko
So does this allow someone to do plugin development just by checking out https://github.com/pilhuhn/rhq-plugins ? That should be a goal right?
If these two repo's can be developed separately, then do we need to consider changes to our versioning schemes? i.e. decouple the versions of the plugins from the version of the plugin container, plugin api etc which comes from https://github.com/pilhuhn/rhq ?
I presume this wouldn't impact older (i.e. before the split) branches/tags that got brought across to https://github.com/pilhuhn/rhq say, you would still be able to do a regular maven build and build everything on those older branches/tags?
Thanks Charles
----- Original Message -----
Hi,
to get some experience I tried how some migration to github might look like. I did that with my personal GitHb account, so this is not available in github/rhq-project (yet)
** First I basically followed the answer to this question on stack overflow to pull stuff from FH and pushed to GH
http://stackoverflow.com/questions/5181845/git-push-existing-repo-to-a-new-a...
I think for a real migration we would first do a
git fetch --all git remote add github <github url> and then a git push --mirror github
This should copy the whole repository inc. all branches and tags.
We may consider pruning old tags and branches here - especially as the FH repo is not going away.
** Then I went and cloned the repo locally and followed http://stackoverflow.com/a/359759/100957
to separate out the modules/plugins directory into its own repo. Created a new repo on github rhq-plugins and pushed the plugins to this new repo
** In the third step I cloned the github rhq repository and followed the first comment to http://stackoverflow.com/a/955793/100957 to clean out the plugins from above rhq repository and tried to push the result to github. This whole locally cleaning "took ages"
The push failed, as the local repo and the remote have diverged. One can use -f here to forcibly push (which I've done) Or in an earlier step not push the rhq repo until after the split-off of the plugins. Or in the main repo just rm -r && git rm on the plugins and have the removal recorded in the history. This may even be faster than pruning that directory hierarchy out.
Results are now at https://github.com/pilhuhn/rhq and https://github.com/pilhuhn/rhq-plugins
Heiko
-- Reg. Adresse: Red Hat GmbH, Technopark II, Haus C, Werner-von-Siemens-Ring 14, D-85630 Grasbrunn Handelsregister: Amtsgericht München HRB 153243 Geschaeftsführer: Mark Hegarty, Charlie Peters, Michael Cunningham, Charles Cachera
rhq-devel mailing list rhq-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/rhq-devel
Hey Charles,
Am 24.09.2012 um 18:32 schrieb Charles Crouch:
So does this allow someone to do plugin development just by checking out https://github.com/pilhuhn/rhq-plugins ? That should be a goal right?
Exactly. Which requires the user to have the dependencies either in the local repository or to download them from central. This has an interesting implication, that we need to publish the artifacts to maven, which happens for non-snapshot ones. So the dependencies of the plugins (mostly modules/core/ stuff) need to be published as non-snapshots.
If these two repo's can be developed separately, then do we need to consider changes to our versioning schemes? i.e. decouple the versions of the plugins from the version of the plugin container, plugin api etc which comes from https://github.com/pilhuhn/rhq ?
Yes, this is what I mean. We need to be more strict with the dependencies and not have plugins depend on -SNAPSHOT versions anymore (as they will never be available from central).
I presume this wouldn't impact older (i.e. before the split) branches/tags that got brought across to https://github.com/pilhuhn/rhq say, you would still be able to do a regular maven build and build everything on those older branches/tags?
First I would expect that they would still be available as-is from fedorahosted.
Having said that: if we do not remove the plugins from the main project, as I have done it, but just rm -r && git rm them, then they will be available in older revisions as they should be, so I guess the answer to your question is yes.
Heiko
Am 25.09.2012 um 11:47 schrieb Heiko W.Rupp:
This has an interesting implication, that we need to publish the artifacts to maven, which happens for non-snapshot ones. So the dependencies of the plugins (mostly modules/core/ stuff) need to be published as non-snapshots.
So this means that the stuff in github.com/pilhuhn/rhq-plugins would need to depend on rhq-4.5.0 artifacts, as they will be published for rhq 4.5.
----- Original Message -----
Hey Charles,
Am 24.09.2012 um 18:32 schrieb Charles Crouch:
So does this allow someone to do plugin development just by checking out https://github.com/pilhuhn/rhq-plugins ? That should be a goal right?
Exactly. Which requires the user to have the dependencies either in the local repository or to download them from central. This has an interesting implication, that we need to publish the artifacts to maven, which happens for non-snapshot ones. So the dependencies of the plugins (mostly modules/core/ stuff) need to be published as non-snapshots.
If these two repo's can be developed separately, then do we need to consider changes to our versioning schemes? i.e. decouple the versions of the plugins from the version of the plugin container, plugin api etc which comes from https://github.com/pilhuhn/rhq ?
Yes, this is what I mean. We need to be more strict with the dependencies and not have plugins depend on -SNAPSHOT versions anymore (as they will never be available from central).
+1 this sounds like it should be part of the "plan" Lukas mentions in his mail.
I presume this wouldn't impact older (i.e. before the split) branches/tags that got brought across to https://github.com/pilhuhn/rhq say, you would still be able to do a regular maven build and build everything on those older branches/tags?
First I would expect that they would still be available as-is from fedorahosted.
Maybe I missed this, but will there be an interaction between these two repos: github and fedorahosted? Will fedorahosted become read-only?
Having said that: if we do not remove the plugins from the main project, as I have done it, but just rm -r && git rm them, then they will be available in older revisions as they should be, so I guess the answer to your question is yes.
Heiko
-- Reg. Adresse: Red Hat GmbH, Technopark II, Haus C, Werner-von-Siemens-Ring 14, D-85630 Grasbrunn Handelsregister: Amtsgericht München HRB 153243 Geschaeftsführer: Mark Hegarty, Charlie Peters, Michael Cunningham, Charles Cachera
rhq-devel mailing list rhq-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/rhq-devel
If we are publishing artifacts, would it even be necessary to check out the rhq-plugins repo to do plugin development? I wouldn't think so. On Sep 25, 2012, at 3:02 PM, Charles Crouch ccrouch@redhat.com wrote:
----- Original Message -----
Hey Charles,
Am 24.09.2012 um 18:32 schrieb Charles Crouch:
So does this allow someone to do plugin development just by checking out https://github.com/pilhuhn/rhq-plugins ? That should be a goal right?
Exactly. Which requires the user to have the dependencies either in the local repository or to download them from central. This has an interesting implication, that we need to publish the artifacts to maven, which happens for non-snapshot ones. So the dependencies of the plugins (mostly modules/core/ stuff) need to be published as non-snapshots.
If these two repo's can be developed separately, then do we need to consider changes to our versioning schemes? i.e. decouple the versions of the plugins from the version of the plugin container, plugin api etc which comes from https://github.com/pilhuhn/rhq ?
Yes, this is what I mean. We need to be more strict with the dependencies and not have plugins depend on -SNAPSHOT versions anymore (as they will never be available from central).
+1 this sounds like it should be part of the "plan" Lukas mentions in his mail.
I presume this wouldn't impact older (i.e. before the split) branches/tags that got brought across to https://github.com/pilhuhn/rhq say, you would still be able to do a regular maven build and build everything on those older branches/tags?
First I would expect that they would still be available as-is from fedorahosted.
Maybe I missed this, but will there be an interaction between these two repos: github and fedorahosted? Will fedorahosted become read-only?
Having said that: if we do not remove the plugins from the main project, as I have done it, but just rm -r && git rm them, then they will be available in older revisions as they should be, so I guess the answer to your question is yes.
Heiko
-- Reg. Adresse: Red Hat GmbH, Technopark II, Haus C, Werner-von-Siemens-Ring 14, D-85630 Grasbrunn Handelsregister: Amtsgericht München HRB 153243 Geschaeftsführer: Mark Hegarty, Charlie Peters, Michael Cunningham, Charles Cachera
rhq-devel mailing list rhq-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/rhq-devel
rhq-devel mailing list rhq-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/rhq-devel
Am 26.09.2012 um 14:22 schrieb John Sanda:
If we are publishing artifacts, would it even be necessary to check out the rhq-plugins repo to do plugin development? I wouldn't think so.
To write a new plugin from scratch you don't need the plugin repo, the published artifacts are enough.
Am 25.09.2012 um 21:02 schrieb Charles Crouch:
Maybe I missed this, but will there be an interaction between these two repos: github and fedorahosted?
I would at least for RHQ expect everyone to interact with GitHub. If we do not split the repo, it looks like some mirroring from GitHub to Fedorahosted would be possible: http://stackoverflow.com/questions/11370239/creating-an-official-github-mirr...
Will fedorahosted become read-only?
We may decide to do some development / bugfixing there only. But I would not want users to push to GH and FH for the same branch and hope that the changes magically show up in the other one.
What is the status with respect to the migration? Is there going to be some discussion, formalization around the steps involved for people to migrate over, what/if any impact it has on community contributions, and so on. Heiko took a great first step with splitting the existing repo, and one of the discussions that definitely has to occur is around the plugin repo(s). I am of the opinion that we should additional plugin repos. There are plugins that are included in JON and that are officially supported by Red Hat. Then there are plugins that might have a reasonable amount of activity but are not included in JON. Lastly, there are plugins that have little to know activity, and there are more than enough of these. For plugins in this last group, I would prefer never to have to build them again.
For my own, self-serving motives I would love to get these issues ironed out and get this migration well on its way. I am working in two branches - feature/cassandra-backend and jsanda/cassandra-plugin. The cassandra-backend branch was created from the AS 7 branch. It will be a while before work in this branch makes its way back to master. The cassandra-plugin branch though was created off of master, and I could merge it back into master at any time. The problem is I need to be in both branches at the same time. It does not make sense to do the cassandra plugin work in the cassandra-backend branch. The plugin work does not depend on it and will be ready for inclusion in a community release much sooner. Working on it in the cassandra-backend brach would only complicate that. The easiest and cleanest solution is for the cassandra-plugin branch to exist in a separate repo. Then I can load code from both branches side by side in my IDE, making changes to both without having any dependency from an upstream branch to worry about.
- John
Hello,
We run RHQ 4.4 in our company and our db admin checked the server's activity on the database.
He advised the followings:
- regular reindex and manual VACUUM runs are not required. There's no reason to run reindex on a every day basis; reindex is flooding the Backup. And there's an autovacuum Daemon since Postgres 8.3.
- create missing indexes would help much more The query below is run very often: DELETE FROM RHQ_AVAILABILITY WHERE END_TIME < '1317423602458'; There's an index missing on this column.
Further missing index: rhq_measurement_sched.definition
- You should check all the join conditions for missing indexes too.
What do you think about it ?
Best regards, Vlad Craciunoiu - 1&1 Internet AG
I thought there was an option in Administration>SystemSettings that lets you tell the server not to perform daily DB maintenance (which is that vacuum).
As for indexes, I believe we purposefully chose not to add that index on RHQ_AVAILABILITY to limit the affect on writes. I can't remember the details. We have performed some DB analysis and opted to not put indexes on some tables, but its been a while and I can't remember all that was decided.
You should write up a Bugzilla issue and document the indexs you think we should need to perhaps some data reports like query plans that you've seen. That could help.
----- Original Message -----
Hello,
We run RHQ 4.4 in our company and our db admin checked the server's activity on the database.
He advised the followings:
- regular reindex and manual VACUUM runs are not required. There's no
reason to run reindex on a every day basis; reindex is flooding the Backup. And there's an autovacuum Daemon since Postgres 8.3.
- create missing indexes would help much more
The query below is run very often: DELETE FROM RHQ_AVAILABILITY WHERE END_TIME < '1317423602458'; There's an index missing on this column.
Further missing index: rhq_measurement_sched.definition
- You should check all the join conditions for missing indexes too.
What do you think about it ?
Best regards, Vlad Craciunoiu - 1&1 Internet AG
rhq-devel mailing list rhq-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/rhq-devel
On Thu, 2012-10-04 at 09:33 -0400, John Mazzitelli wrote:
I thought there was an option in Administration>SystemSettings that lets you tell the server not to perform daily DB maintenance (which is that vacuum).
If RHQ is running vacuum, that is a VERY SCARY thought! Database maintenance should be left up to the DBA and they should have full control over what happens to their production databases and when. The PostgreSQL documentation also states this. If the user wants more aggressive vacuuming, then they should modify their PostgreSQL vacuum daemon or the manual jobs run by cron. Or they can issue the command themselves from a PostgreSQL admin console. Although vacuuming is designed to run in parallel with a production database, vacuum does create a disk and CPU bottleneck on the entire database -- not just the RHQ tables -- which can cause poor performance for other active sessions.
If vacuuming is happening in RHQ on its own, this sounds like a pretty big bug and from the code, it appears that this may have been introduced a year ago. Ouch!
As for indexes, I believe we purposefully chose not to add that index on RHQ_AVAILABILITY to limit the affect on writes. I can't remember the details. We have performed some DB analysis and opted to not put indexes on some tables, but its been a while and I can't remember all that was decided.
Do we have any of the results from this analysis posted anywhere? I couldn't find it on the wiki and was curious. I would imagine that an index on a time stamp column would be ideal if it is what we primarily use in criteria. I do know that in cases where the table may only contain a few thousand rows, an index can add a few milliseconds to writes but once you exceed tens of thousands or rows, the performance gains become favorable.
You should write up a Bugzilla issue and document the indexs you think we should need to perhaps some data reports like query plans that you've seen. That could help.
----- Original Message -----
Hello,
We run RHQ 4.4 in our company and our db admin checked the server's activity on the database.
He advised the followings:
- regular reindex and manual VACUUM runs are not required. There's no
reason to run reindex on a every day basis; reindex is flooding the Backup. And there's an autovacuum Daemon since Postgres 8.3.
- create missing indexes would help much more
The query below is run very often: DELETE FROM RHQ_AVAILABILITY WHERE END_TIME < '1317423602458'; There's an index missing on this column.
Further missing index: rhq_measurement_sched.definition
- You should check all the join conditions for missing indexes too.
What do you think about it ?
Best regards, Vlad Craciunoiu - 1&1 Internet AG
rhq-devel mailing list rhq-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/rhq-devel
rhq-devel mailing list rhq-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/rhq-devel
I've created this:
https://bugzilla.redhat.com/show_bug.cgi?id=863169
Best regards, Vlad
________________________________ From: Larry O'Leary loleary@redhat.com To: rhq-devel@lists.fedorahosted.org Sent: Thursday, 4 October 2012, 17:31 Subject: Re: db settings
On Thu, 2012-10-04 at 09:33 -0400, John Mazzitelli wrote:
I thought there was an option in Administration>SystemSettings that lets you tell the server not to perform daily DB maintenance (which is that vacuum).
If RHQ is running vacuum, that is a VERY SCARY thought! Database maintenance should be left up to the DBA and they should have full control over what happens to their production databases and when. The PostgreSQL documentation also states this. If the user wants more aggressive vacuuming, then they should modify their PostgreSQL vacuum daemon or the manual jobs run by cron. Or they can issue the command themselves from a PostgreSQL admin console. Although vacuuming is designed to run in parallel with a production database, vacuum does create a disk and CPU bottleneck on the entire database -- not just the RHQ tables -- which can cause poor performance for other active sessions.
If vacuuming is happening in RHQ on its own, this sounds like a pretty big bug and from the code, it appears that this may have been introduced a year ago. Ouch!
As for indexes, I believe we purposefully chose not to add that index on RHQ_AVAILABILITY to limit the affect on writes. I can't remember the details. We have performed some DB analysis and opted to not put indexes on some tables, but its been a while and I can't remember all that was decided.
Do we have any of the results from this analysis posted anywhere? I couldn't find it on the wiki and was curious. I would imagine that an index on a time stamp column would be ideal if it is what we primarily use in criteria. I do know that in cases where the table may only contain a few thousand rows, an index can add a few milliseconds to writes but once you exceed tens of thousands or rows, the performance gains become favorable.
You should write up a Bugzilla issue and document the indexs you think we should need to perhaps some data reports like query plans that you've seen. That could help.
----- Original Message -----
Hello,
We run RHQ 4.4 in our company and our db admin checked the server's activity on the database.
He advised the followings:
- regular reindex and manual VACUUM runs are not required. There's no
reason to run reindex on a every day basis; reindex is flooding the Backup. And there's an autovacuum Daemon since Postgres 8.3.
- create missing indexes would help much more
The query below is run very often: DELETE FROM RHQ_AVAILABILITY WHERE END_TIME < '1317423602458'; There's an index missing on this column.
Further missing index: rhq_measurement_sched.definition
- You should check all the join conditions for missing indexes too.
What do you think about it ?
Best regards, Vlad Craciunoiu - 1&1 Internet AG
rhq-devel mailing list rhq-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/rhq-devel
rhq-devel mailing list rhq-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/rhq-devel
On Wed, Oct 3, 2012 at 7:39 PM, John Sanda jsanda@redhat.com wrote:
I am of the opinion that we should additional plugin repos. There are plugins that are included in JON and that are officially supported by Red Hat. Then there are plugins that might have a reasonable amount of activity but are not included in JON. Lastly, there are plugins that have little to know activity, and there are more than enough of these. For plugins in this last group, I would prefer never to have to build them again.
Rather than divide the plugins based on which product bundles what, I could see something more divided up like: * core, including platform, JMX, script, agent, etc. * admin (grub, postfix, iptables) * alert (or server) plugins like irc, twitter * jboss, including jboss project plugins like JBossCache (might have to be part of core?) * database (mysql, oracle, postgres) * network (snmp, net services) * experimental (ones that don't get built/distributed by default)
and maybe a few others.
Looking at the rhq/modules/plugins/pom.xml I see a couple of profiles as maybe a suggested set of repositories.
Am 04.10.2012 um 04:39 schrieb John Sanda:
has to occur is around the plugin repo(s). I am of the opinion that we should additional plugin repos. There are plugins that are included in JON and that are officially supported by Red Hat. Then there are plugins that might have a reasonable amount of activity but are not included in JON. Lastly, there are plugins that have little to know activity, and there are more than enough of these. For plugins in this last group, I would prefer never to have to build them again.
I would split the plugins rather in a way that those that are building are in a plugin-repo and that others like my OneWire plugin would be moved to e.g. the samples repo. If such a plugin comes alive again, it can still be moved back in the main repo. And then we have some profiles that determine which plugins are built, so that you really only build what you need (I make the error myself often enough to have all plugins built and installed and am then disabling the augeas ones on my Mac).
Heiko
Maybe we should watch our fellows at JBoss Tools doing their SVN migration and repository splitting so that we can learn from their mistakes ;)
https://github.com/maxandersen/jbosstools-gitmigration
Also, I do believe the split should be more granular than plugins and the rest of the codebase - at least core, agent plugins, server plugins, scripting, server.
Then we have the opportunity for example to move dbutils out of core where it really doesn't belong, because nothing but server needs it and potentially other changes.
IMHO, the logistics of the migration is going to be much simpler than the preparation steps of modifying the build and actually deciding on the granularity of the repo split. This is a discussion we did not have yet though - at least not one where we'd come to an agreement on an action plan ;)
Lukas
On Friday, September 21, 2012 16:44:03 Heiko W.Rupp wrote:
Hi,
to get some experience I tried how some migration to github might look like. I did that with my personal GitHb account, so this is not available in github/rhq-project (yet)
** First I basically followed the answer to this question on stack overflow to pull stuff from FH and pushed to GH
http://stackoverflow.com/questions/5181845/git-push-existing-repo-to-a-new-a nd-different-remote-repo-server
I think for a real migration we would first do a
git fetch --all git remote add github <github url> and then a git push --mirror github
This should copy the whole repository inc. all branches and tags.
We may consider pruning old tags and branches here - especially as the FH repo is not going away.
** Then I went and cloned the repo locally and followed http://stackoverflow.com/a/359759/100957
to separate out the modules/plugins directory into its own repo. Created a new repo on github rhq-plugins and pushed the plugins to this new repo
** In the third step I cloned the github rhq repository and followed the first comment to http://stackoverflow.com/a/955793/100957 to clean out the plugins from above rhq repository and tried to push the result to github. This whole locally cleaning "took ages"
The push failed, as the local repo and the remote have diverged. One can use -f here to forcibly push (which I've done) Or in an earlier step not push the rhq repo until after the split-off of the plugins. Or in the main repo just rm -r && git rm on the plugins and have the removal recorded in the history. This may even be faster than pruning that directory hierarchy out.
Results are now at https://github.com/pilhuhn/rhq and https://github.com/pilhuhn/rhq-plugins
Heiko
----- Original Message -----
Maybe we should watch our fellows at JBoss Tools doing their SVN migration and repository splitting so that we can learn from their mistakes ;)
https://github.com/maxandersen/jbosstools-gitmigration
Also, I do believe the split should be more granular than plugins and the rest of the codebase - at least core, agent plugins, server plugins, scripting, server.
Then we have the opportunity for example to move dbutils out of core where it really doesn't belong, because nothing but server needs it and potentially other changes.
How much of this reorganizing should we bite off as part of the github migration? Is it easier/harder to do this now versus wait until we've moved over?
IMHO, the logistics of the migration is going to be much simpler than the preparation steps of modifying the build and actually deciding on the granularity of the repo split. This is a discussion we did not have yet though
- at least not one where we'd come to an agreement on an action plan
;)
+1 I agree we need a plan.
Lukas
On Friday, September 21, 2012 16:44:03 Heiko W.Rupp wrote:
Hi,
to get some experience I tried how some migration to github might look like. I did that with my personal GitHb account, so this is not available in github/rhq-project (yet)
** First I basically followed the answer to this question on stack overflow to pull stuff from FH and pushed to GH
http://stackoverflow.com/questions/5181845/git-push-existing-repo-to-a-new-a nd-different-remote-repo-server
I think for a real migration we would first do a
git fetch --all git remote add github <github url> and then a git push --mirror github
This should copy the whole repository inc. all branches and tags.
We may consider pruning old tags and branches here - especially as the FH repo is not going away.
** Then I went and cloned the repo locally and followed http://stackoverflow.com/a/359759/100957
to separate out the modules/plugins directory into its own repo. Created a new repo on github rhq-plugins and pushed the plugins to this new repo
** In the third step I cloned the github rhq repository and followed the first comment to http://stackoverflow.com/a/955793/100957 to clean out the plugins from above rhq repository and tried to push the result to github. This whole locally cleaning "took ages"
The push failed, as the local repo and the remote have diverged. One can use -f here to forcibly push (which I've done) Or in an earlier step not push the rhq repo until after the split-off of the plugins. Or in the main repo just rm -r && git rm on the plugins and have the removal recorded in the history. This may even be faster than pruning that directory hierarchy out.
Results are now at https://github.com/pilhuhn/rhq and https://github.com/pilhuhn/rhq-plugins
Heiko
rhq-devel mailing list rhq-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/rhq-devel
Am 25.09.2012 um 20:56 schrieb Charles Crouch:
How much of this reorganizing should we bite off as part of the github migration? Is it easier/harder to do this now versus wait until we've moved over?
The following in modules/helpers are not core to RHQ and could be moved at any time:
* bundleGen -- bundle generator script. Could perhaps be totally erased if we have something better * pluginAnnotations -- annotations to decorate (application) code with to create plugin artifacts with * pluginGen -- standalone plugin generator that asks questions and creates a plugin skeleton * rest-docs-generator -- the annotation scanner used to create an xml file from the swagger annotations to create our REST-api docs via XSLT transformation into docbook * jeeGen -- A tool to generate JEE test applications
and then there is * inventory-serializer, that looks like a tool to print the current inventory from the DB
The two RtFilter modules are needed in RHQ.
Heiko
Am 22.10.2012 um 17:28 schrieb Heiko W.Rupp:
Am 25.09.2012 um 20:56 schrieb Charles Crouch:
How much of this reorganizing should we bite off as part of the github migration? Is it easier/harder to do this now versus wait until we've moved over?
The following in modules/helpers are not core to RHQ and could be moved at any time:
- bundleGen -- bundle generator script. Could perhaps be totally erased if we have something better
- pluginAnnotations -- annotations to decorate (application) code with to create plugin artifacts with
- pluginGen -- standalone plugin generator that asks questions and creates a plugin skeleton
- rest-docs-generator -- the annotation scanner used to create an xml file from the swagger annotations
to create our REST-api docs via XSLT transformation into docbook
- jeeGen -- A tool to generate JEE test applications
and then there is
- inventory-serializer, that looks like a tool to print the current inventory from the DB
I have copied those (in master) over to https://github.com/rhq-project/helpers and am ready to remove those (in master) from the main git repository at any time.
Heiko
Hi,
as a followup I have Am 21.09.2012 um 16:44 schrieb Heiko W.Rupp:
Hi,
to get some experience I tried how some migration to github might look like. I did that with my personal GitHb account, so this is not available in github/rhq-project (yet)
** First I basically followed the answer to this question on stack overflow to pull stuff from FH and pushed to GH
http://stackoverflow.com/questions/5181845/git-push-existing-repo-to-a-new-a...
I think for a real migration we would first do a
git fetch --all git remote add github <github url> and then a git push --mirror github
This should copy the whole repository inc. all branches and tags.
This seems to copy all tags, but not all branches.
One can probably just loop over all branches like e.g.
$ git checkout -t -b rhq-on-as7 origin/rhq-on-as7 $ git push github rhq-on-as7:rhq-on-as7
One has to be careful here, as a) the correct branch needs to be taken and b) the name on the remote should be just rhq-on-as7 and not origin/rhq-on-as7
Something like
BRANCHES=`git branch -r | grep -v "upstream/" | sed -e 's/^ *origin///'` for B in $BRANCHES do echo $B git checkout -b origin/$B git push upstream $B:$B done
There may be a better way, than individually pushing them
Heiko
I took a look at https://github.com/pilhuhn/rhq/tree/master/modules/ and see that the plugins are still there. Will the plugins subdirectory get purged since you split them out into a separate repo? If I push a branch to the rhq-plugins repo, will there be any issues since the source trees will be different? In other words, if I push a branch that exists in the fedora hosted repo over to the rhq-plugins repo, will that wind up pulling in parts of the source tree outside of the plugins directory?
On Oct 17, 2012, at 4:18 PM, "Heiko W.Rupp" hrupp@redhat.com wrote:
Hi,
as a followup I have Am 21.09.2012 um 16:44 schrieb Heiko W.Rupp:
Hi,
to get some experience I tried how some migration to github might look like. I did that with my personal GitHb account, so this is not available in github/rhq-project (yet)
** First I basically followed the answer to this question on stack overflow to pull stuff from FH and pushed to GH
http://stackoverflow.com/questions/5181845/git-push-existing-repo-to-a-new-a...
I think for a real migration we would first do a
git fetch --all git remote add github <github url> and then a git push --mirror github
This should copy the whole repository inc. all branches and tags.
This seems to copy all tags, but not all branches.
One can probably just loop over all branches like e.g.
$ git checkout -t -b rhq-on-as7 origin/rhq-on-as7 $ git push github rhq-on-as7:rhq-on-as7
One has to be careful here, as a) the correct branch needs to be taken and b) the name on the remote should be just rhq-on-as7 and not origin/rhq-on-as7
Something like
BRANCHES=`git branch -r | grep -v "upstream/" | sed -e 's/^ *origin///'` for B in $BRANCHES do echo $B git checkout -b origin/$B git push upstream $B:$B done
There may be a better way, than individually pushing them
Heiko
-- Reg. Adresse: Red Hat GmbH, Technopark II, Haus C, Werner-von-Siemens-Ring 14, D-85630 Grasbrunn Handelsregister: Amtsgericht München HRB 153243 Geschaeftsführer: Mark Hegarty, Charlie Peters, Michael Cunningham, Charles Cachera
rhq-devel mailing list rhq-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/rhq-devel
John,
Am 17.10.2012 um 22:46 schrieb John Sanda:
I took a look at https://github.com/pilhuhn/rhq/tree/master/modules/ and see that the plugins are still there. Will the plugins subdirectory get purged since you
I stopped there yesterday as I think I made a mistake -- my email actually shows an updated script from what I've tried. The end goal is to have them vanish from the main rhq repo.
Also I think my rhq-plugins repo suffers from the same issue I made when initially copying over the main one.
Please note that pilhuhn/rhq* is my private playground to investigate the steps needed to move (*) and will not be the official one. The official ones will then live at github.com/rhq-project/
split them out into a separate repo? If I push a branch to the rhq-plugins repo, will there be any issues since the source trees will be different? In other words, if I push a branch that exists in the fedora hosted repo over to the rhq-plugins repo, will that wind up pulling in parts of the source tree outside of the plugins directory?
I have no idea, but that is possible.
My idea for a schedule here is to: 1) sync all work to FH 1a) freeze the git.FH repo and perhaps even revoke write rights for (almost) everyone - see below *3) 2) sync the whole tree over to GH - we may filter out old branches that we don't want to keep 3) split out the plugins into their own repo - same for other parts of the source that we want to split out - remove those parts from the main repo 4) have every developer throw away the old FH based single repo 5) have every developer clone the repos from GH - no usage of submodules 6) change plugins + other repos (if applicable) to build against the latest published maven artifacts *2) 7) Change jenkins jobs to build the repos -- making sure that e.g. plugins also only use published artifacts 8) have every developer only push to GH into the respective repo. As e.g. the plugin repo has no classes from core, there will be no cross-talking
Steps 2+3 seem to keep all the tags, so we should be on the safe side here.
Heiko
*) JBossTools did a migation to github from svn, so their story is not 1:1 applicable to us. *2) working against snapshots is no good idea, as external parties may want to retrieve the dependencies from jboss nexus or central without the need to compile them; we may add some repo setting in the plugins pom to also allow snapshots from jboss nexus, but it must be clear that for compiling plugins, a user must never be forced to checkout the source of modules/core or any other part of RHQ. *3) There may be a way that GitHub mirrors all commits into FH to give us some air when transitioning e.g. Jenkins. But then I don't think that can work with split repos.
On Oct 17, 2012, at 4:18 PM, "Heiko W.Rupp" hrupp@redhat.com wrote:
Hi,
as a followup I have Am 21.09.2012 um 16:44 schrieb Heiko W.Rupp:
Hi,
to get some experience I tried how some migration to github might look like. I did that with my personal GitHb account, so this is not available in github/rhq-project (yet)
** First I basically followed the answer to this question on stack overflow to pull stuff from FH and pushed to GH
http://stackoverflow.com/questions/5181845/git-push-existing-repo-to-a-new-a...
I think for a real migration we would first do a
git fetch --all git remote add github <github url> and then a git push --mirror github
This should copy the whole repository inc. all branches and tags.
This seems to copy all tags, but not all branches.
One can probably just loop over all branches like e.g.
$ git checkout -t -b rhq-on-as7 origin/rhq-on-as7 $ git push github rhq-on-as7:rhq-on-as7
One has to be careful here, as a) the correct branch needs to be taken and b) the name on the remote should be just rhq-on-as7 and not origin/rhq-on-as7
Something like
BRANCHES=`git branch -r | grep -v "upstream/" | sed -e 's/^ *origin///'` for B in $BRANCHES do echo $B git checkout -b origin/$B git push upstream $B:$B done
There may be a better way, than individually pushing them
Heiko
-- Reg. Adresse: Red Hat GmbH, Technopark II, Haus C, Werner-von-Siemens-Ring 14, D-85630 Grasbrunn Handelsregister: Amtsgericht München HRB 153243 Geschaeftsführer: Mark Hegarty, Charlie Peters, Michael Cunningham, Charles Cachera
rhq-devel mailing list rhq-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/rhq-devel
rhq-devel mailing list rhq-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/rhq-devel
An update to the github migration script.
We should definitively decide which branches to migrate over and not migrate "one-offs" or feature-branches that have long been merged into master
Corrected migration script:
BRANCHES=`git branch -r | grep -v "upstream/" | sed -e 's/^ *origin///'` for B in $BRANCHES do echo $B git checkout -t -b $B origin/$B # git push upstream $B:$B done
this fails on
+ git checkout -t -b release-3.0.1 origin/release-3.0.1 warning: refname 'origin/release-3.0.1' is ambiguous. warning: refname 'origin/release-3.0.1' is ambiguous. fatal: Ambiguous object name: 'origin/release-3.0.1'.
Probably because of
snert:/tmp/rhq hrupp$ git branch -r | grep 3.0.1 origin/origin/release-3.0.1 origin/release-3.0.1 [...]
so origin/release-3.0.1 needs to be filtered out above
After the script has run one can add the upstream remote and then push all branches at once: git push -v --all upstream
Now that this is done we can pull out the plugins
1) create the plugin repo on github 2) filter down the plugins
$ cd /tmp $ git clone /tmp/rhq rhq-plugins
run this script to "nail" the branches": BRANCHES=`git branch -r | grep -v "upstream/" | sed -e 's/^ *origin///'` for B in $BRANCHES do echo $B git branch -t $B origin/$B done
now filter down the plugins
git filter-branch --tag-name-filter cat --prune-empty --subdirectory-filter modules/plugins -- --all
Took another 10-15 mins locally
Now continue with step 5 of the git stackoverflow entry ( http://stackoverflow.com/a/359759/100957 ) , which is
$ git reset --hard $ git for-each-ref --format="%(refname)" refs/original/ | xargs -n 1 git update-ref -d $ git reflog expire --expire=now --all $ git gc --aggressive --prune=now
And then push to GH:
$ git remote add upstream <github url> $ git push --all upstream
3) Now get rid of the plugins in the first ("main") repository
NOTE: best is to check out the main tree from github and then do this. Or make sure to remove origin from the remotes, so that this can not mess up fedorahosted (as I accidentally did)
BRANCHES=`git branch -r | grep -v "upstream/" | grep -v "origin/release-3.0.1" | sed -e 's/^ *origin///'` for B in $BRANCHES do echo $B git checkout -f $B git rm -r --cached modules/plugins git commit -m "removed plugins" done
b) Remove the origin remote $ git remote remove origin
c) Send updates to github: $git push --all upstream
d) get rid of the (now untracked) plugins $ rm -r modules/plugins
e) adjust modules/pom.xml to no longer list plugins (in all branches)
4) in rhq-plugins repository a) copy over the .mailmap file from the main repo and commit b) fix the pom.xml to use a 4.5.1 parent pom (for master at least) c) fix the dev-container relative path within the pom.xml d) add a .gitignore file
Results are at : https://github.com/pilhuhn/rhq-plugins and https://github.com/pilhuhn/rhq/tree/master/modules
NOTES: - It may make sense to populate a file with the branches to copy once and then use this subsequently $ git branch -r | grep -v "upstream/" | grep -v "origin/release-3.0.1" | sed -e 's/^ *origin///' > branch-list.txt $ vi branch-list.txt ### only leave branches in we are interested in moving over $ BRANCHES=`cat branch-list.txt`
The current list of branches is attached
If we would like to filter out the branches that have been merged into master, you can get a list of those like this:
git branch -a --merged master
Of course this will not find branches that have been completely cherry-picked over to master (instead of merged) but I think it is good starting point still.
Lukas
On Friday, October 19, 2012 13:04:44 Heiko W.Rupp wrote:
An update to the github migration script.
We should definitively decide which branches to migrate over and not migrate "one-offs" or feature-branches that have long been merged into master
Corrected migration script:
BRANCHES=`git branch -r | grep -v "upstream/" | sed -e 's/^ *origin///'` for B in $BRANCHES do echo $B git checkout -t -b $B origin/$B # git push upstream $B:$B done
this fails on
- git checkout -t -b release-3.0.1 origin/release-3.0.1
warning: refname 'origin/release-3.0.1' is ambiguous. warning: refname 'origin/release-3.0.1' is ambiguous. fatal: Ambiguous object name: 'origin/release-3.0.1'.
Probably because of
snert:/tmp/rhq hrupp$ git branch -r | grep 3.0.1 origin/origin/release-3.0.1 origin/release-3.0.1 [...]
so origin/release-3.0.1 needs to be filtered out above
After the script has run one can add the upstream remote and then push all branches at once: git push -v --all upstream
Now that this is done we can pull out the plugins
- create the plugin repo on github
- filter down the plugins
$ cd /tmp $ git clone /tmp/rhq rhq-plugins
run this script to "nail" the branches": BRANCHES=`git branch -r | grep -v "upstream/" | sed -e 's/^ *origin///'` for B in $BRANCHES do echo $B git branch -t $B origin/$B done
now filter down the plugins
git filter-branch --tag-name-filter cat --prune-empty --subdirectory-filter modules/plugins -- --all
Took another 10-15 mins locally
Now continue with step 5 of the git stackoverflow entry ( http://stackoverflow.com/a/359759/100957 ) , which is
$ git reset --hard $ git for-each-ref --format="%(refname)" refs/original/ | xargs -n 1 git update-ref -d $ git reflog expire --expire=now --all $ git gc --aggressive --prune=now
And then push to GH:
$ git remote add upstream <github url> $ git push --all upstream
- Now get rid of the plugins in the first ("main") repository
NOTE: best is to check out the main tree from github and then do this. Or make sure to remove origin from the remotes, so that this can not mess up fedorahosted (as I accidentally did)
BRANCHES=`git branch -r | grep -v "upstream/" | grep -v "origin/release-3.0.1" | sed -e 's/^ *origin///'` for B in $BRANCHES do echo $B git checkout -f $B git rm -r --cached modules/plugins git commit -m "removed plugins" done
b) Remove the origin remote $ git remote remove origin
c) Send updates to github: $git push --all upstream
d) get rid of the (now untracked) plugins $ rm -r modules/plugins
e) adjust modules/pom.xml to no longer list plugins (in all branches)
- in rhq-plugins repository
a) copy over the .mailmap file from the main repo and commit b) fix the pom.xml to use a 4.5.1 parent pom (for master at least) c) fix the dev-container relative path within the pom.xml d) add a .gitignore file
Results are at : https://github.com/pilhuhn/rhq-plugins and https://github.com/pilhuhn/rhq/tree/master/modules
NOTES:
- It may make sense to populate a file with the branches to copy once and
then use this subsequently $ git branch -r | grep -v "upstream/" | grep -v "origin/release-3.0.1" | sed -e 's/^ *origin///' > branch-list.txt $ vi branch-list.txt ### only leave branches in we are interested in moving over $ BRANCHES=`cat branch-list.txt`
The current list of branches is attached
rhq-devel@lists.fedorahosted.org