Hey folks!
There's a pretty big DB schema change that I'd like to do in HyperKitty, and since it involves a primary key I expect a downtime of several hours, possibly a day (more precise testing going on)
The good news it that mailman will keep running, only the web UIs will be down (HyperKitty & Postorius, the admin UI).
It's a change that I've put off for a while, but the more we wait the longer the downtime will be (because it depends on the size of the email table).
What time do you think would be the less worse for such a change? After freeze break next week? After General Availability (whenever that is) ? Another time?
I have a feeling that after freeze break is better because the folks who use the web UIs to read email may also be those who will need to communicate more after the GA, to discuss communication and reviews.
I don't mind starting the migration on a weekend of course, to impact less people.
Thanks for your input.
Aurélien
On Fri, Jun 09, 2017 at 12:39:23PM +0200, Aurelien Bompard wrote:
I have a feeling that after freeze break is better because the folks who use the web UIs to read email may also be those who will need to communicate more after the GA, to discuss communication and reviews.
Yes, I agree. If this can't be put off until a couple of weeks after GA, let's do it soon.
I assume the mitigation plan if something goes wrong is to restore from backups? Is it not possible to do the migration on a staging copy of the db and then swap it in?
On 9 June 2017 at 06:39, Aurelien Bompard abompard@fedoraproject.org wrote:
Hey folks!
There's a pretty big DB schema change that I'd like to do in HyperKitty, and since it involves a primary key I expect a downtime of several hours, possibly a day (more precise testing going on)
The good news it that mailman will keep running, only the web UIs will be down (HyperKitty & Postorius, the admin UI).
It's a change that I've put off for a while, but the more we wait the longer the downtime will be (because it depends on the size of the email table).
What time do you think would be the less worse for such a change? After freeze break next week? After General Availability (whenever that is) ? Another time?
I have a feeling that after freeze break is better because the folks who use the web UIs to read email may also be those who will need to communicate more after the GA, to discuss communication and reviews.
I don't mind starting the migration on a weekend of course, to impact less people.
Thanks for your input.
Aurélien
infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-leave@lists.fedoraproject.org
OK I would say we can not make any decision until next week, probably during the Infrastructure meeting. Kevin and Patrick are out and they are the closest we have to DBAs and similar to help work out the pros and cons of any plan.
Are you thinking of something like the following (is it even possible?)?
0. Show that we can bring back the website from backups in case we have to restore because something didn't work. 1. Duplicate the current data into staging. 2. Run the upgrade in staging on that data with a time set. 3. Get an idea if we could move this over and 'rerun' data in the meantime to catch it up. 4. If we can dump current tables, roll in the tables from staging and rerun 'inbetween data?' 5. If we can not, then schedule a day downtime and do this. I would go with a non-workday (Sat/Sun).
Hey folks!
Some feedback on this weekend's HyperKitty schema upgrade. It lasted about twice the time I measured in staging. I had provisionned for more time, but not that much, so the update went over the scheduled downtime, sorry about that. My migration was basically 3 short SQL queries so all the work happens in the DB server. The only thing I can think of for a reason is that the prod DB is under much higher load than the staging DB (I anticipated that somewhat but not enough). I think it can be a useful info for the next people who want to estimate this sort of thing.
Anyway, it's done now and everything seems as functional as before. Can someone in the sysadmin-main or sysadmin-noc groups bring status.fp.o back to the green state?
Thanks, and an particularly huge amount of thanks to Kevin and Patrick who helped me on a Saturday morning/afternoon. I owe you a drink, and thankfully the Great Drinks Redeeming Conference is coming up in a couple weeks.
Cheers!
Aurélien
infrastructure@lists.fedoraproject.org