On Wed, 16 Oct 2019 at 18:35, Kevin Fenzi <kevin(a)scrye.com>
wrote:
>
> Greetings folks.
>
> As you may know, currently database backups on db-koji01 are causing
> very heavy load, disrupting our users builds (
>
https://pagure.io/fedora-infrastructure/issue/8292 )
> so, they are currently disabled.
>
> However, not having current backups is not a good thing, IMHO.
>
> So, I am considering the idea of adding a db-koji02 vm (also rhel7 using
> the same postgres version that db-koji01 is) and enabling streaming
> replication from db-koji01 -> db-koji02 and then once thats working, run
> the database backups on db-koji02.
>
> It turns out this doesn't require that many changes on db-koji01:
> * adding a replication user
> * Setting 3 new lines of postgresql config and restarting:
> wal_level = 'hot_standby'
> max_wal_senders = 10
> wal_keep_segments = 100
> (May need to adjust senders and keep segments)
>
> All the other changes are on db-koji02:
> * create/setup the vm
> * run pg_basebackup to pull all the current data from 01
> * setup postgresql.conf and recovery.conf files
> * start server and confirm it keeps up with 01
> * run pg_dump and confirm it keeps up with 01
>
> This is, of course a really big change in a freeze to a criticial
> service, so I'd like to get thoughts from others about it.
>
> Should we wait until after freeze and do without backups until then
> (note, that we have never had to restore this db from backups in the
> past, although we have dumped/restored it to move to newer postgres
> versions)?
>
> Is there something else we can do thats easier to mitigate the issues
> right now?
>
> Thoughts? ideas? Rotten fruit?
I think this would be a good thing to do. I guess we can put this on
virthost04 once we get it to actually boot versus hang on tang?
Well, it needs to be a bvirthost, so I was going to use bvirthost12.
It's pretty lightly loaded and has lots of memory free.
kevin