so a few weeks ago my boss (Jason Brooks) asked me if I could help
Fedora Magazine to be moved from the current hosting (a cloud instance
on fedora cloud) to a WPengine account that is sponsored by RH, and
managed by my team. He did the setup of the blog, and was discussing
with Paul, but delegated to me the infra part and part of the
So the plan was simple:
- we set a instance, configure it and we do transfer data (using export
and import) and see how it go
- if that go well, we sync again, do what magic is needed fedora infra
side and that's it
Obviously, a plan that simple couldn't go without issues, and indeed,
went with some issues. I will skip over the Fedora infra ones we faced
(like me not having my token, not having access to logs or infra, or
the blogs on both side, etc), cause they got fixed fast enough, and did
just delay the migration due to bad timing.
On the Wordpress side, we faced a few issues, some that could have been
avoidable, some not. In retrospect, I kinda think we should have done
the migration another way.
Issue 1: The curious case of duplicate buttons
The first issue I did notice is that after the sync, the theme did
start to have duplicate buttons for social media (the round one on top
right, for g+, rss, facebook, etc). The fix was to go on
and remove the
duplicate. 5 buttons, not a big problem (unlike another blog I did
migrate, where the whole menu structure of 30 items got duplicated and
I had to remove them 1 by 1 using the not so fast web interface).
I guess that the solution would have been to test on a staging site
first, and do the import on a empty site for prod.
Issue 2: Raiders of the lost sessions
Then another issue that Paul did bring was that people lost their
session (so folks lost the document they had open). While the migration
did happen on friday night (I was planning sooner), I didn't really
anticipate that people would keep document in their browser open (while
that's a valid workflow). In retrospect, I should have sent a email,
but I got lost in details (and Jenkins fighting), so that's my fault
for not communicating more.
Paul did poke the db, not sure if he managed to recover things.
Solution: communicate more with people, especially for a successful
website as fedora magazine.
Issue 3: The hunt for red comments
Then yesterday, Paul also told me that at least one comment did
disappear. Upon looking, we found that the comment was in the database
and the admin UI, but didn't appear on the website. And the comment was
counted on the top counter (11 comments), but not the one that bottom
of the article (10 comments). For folks wanting to see, it was Colin
comment on https://fedoramagazine.org/introducing-fedora-coreos/
started to dig, and it seems that the comment was a answer to another
comment, but for some reason, it was answering to a comment on another
After trying to understand what happened, it seems that while the
export xml is self consistent but ids of post/comments got changed with
the 2nd export, but just for some comments. How the code did cause that
problem is still a mystery. In the end, I just went in the DB and fixed
So if you see a page with the same problem, please tell me and I will
take a look. If a comment did disappear, please tell too.
Solution: restarting the sync from scratch would have likely mitigated
Issue 4: With a Friend Like misc
Then after migrating commblog, I did notice that a few posts (around
10) had their author field changed to be my account. I couldn't find a
common denominator for that, so I did fixed them one by one. It was
faster once I found "quick edit" in the UI, which, as the name
indicate, is quicker.
I was then able to eliminate all the problematic posts.
So I started to look at fedora magazine, and it seems that there is a
cause I got one post:
But that's a copy of:
I guess I will have to clean that. Interestingly, WP UI say there is 14
posts for me. Not sure what happen to the 13 others, since it only show
Restarting the sync from scratch would have likely prevented the issue.
Issue 5: They link
Ryan Lerch did found today that among the post, there is some who are
different, and that at least 1 post had wrong links:
Upon looking at the export file, the link was correct, but it seems
that WP is fixing things on import, or something like that. I need to
see if that's in the DB or a setting somewhere else.
I suspect the domain name was different at the time of the migration,
in the wordpress configuration, so it tried to make things correct. I
guess we need to make sure the server is 100% correct before we import
posts. or it could also be a caching issue, but we did reset the cache
I will see how many post are impacted with a sql query and fix stuff.
Issue 6: Invasion of the post snatchers
On irc, Gregbartholomew did signal that one on the article they wrote
got duplicated, the same for Ryan. I guess that was a side effect of
the 2nd sync, especially if the name of the post got changed (so it got
created as a new post). Given that the windows was quite small (2
days), people can just get ride of the postpeop^W posts.
Again, likely a side effect of not cleaning the db before syncing.
As Paul told, the revision are disabled. I will see with Jason to
enable them. The reasoning of wpengine is on
2) direct db access
As a admin on the main wpengine account, I do have phpmyadmin access
for the various instances. But I do not have enough access to the main
account to give others people account or access. Again, this will wait
3) warning about "doing a backup"
If a admin try to upgrade plugins, the UX will ask them if they want to
do a backup, or say that they did. Again, that require elevated access
that we can't give for now. And there is automated backup, so in doubt
people can say "yes I did". And if things go wrong, we can try to help.
But we can't disable the warning.
I think while that went well, a lot of issues could have been avoided
by using a separate test install, then ask to people to check, and then
do the migration in prod.
Unfortunately, the timing wasn't great, between some unexpected
interruptions, some roadblocks and summertime (so people taking PTO).
There was also mix-up regarding communication, as I thought that was
taken care of, but I didn't verify, so my fault for not telling people
And while I speak of "restarting from scratch", I am not sure exactly
how to do that in practice with a hosted solution like wpengine (if it
was self hosted, the answer would have been "ansible", which is a
answer to a lot of stuff). I can surely have erased all posts, all
comments, all users, but maybe I am not removing everything, or
removing too much. Removing the whole site would have been another
solution, but then, I had no idea on how it was setup in the first
place, not what config was done (especially for openid integration, or
how to import a theme).
Anyway, if people see any more issues, please tell us on the list, we
will take a look. (no need to name the issues, just tell what is wrong)
Sysadmin, Community Infrastructure