Is there a way to implement an artificial capacity limit that would prevent processes from exhausting the overlay so that the reserve might be used for recording the event and rebooting back to a safer state?
The easiest way is to play with the partition size (set in your kickstart file). There are two things that can stop a file being written: either the overlay is full, or the filesystem is full. If you set the partition size to a smaller value, you'll get filesystem errors, which are probably going to be less severe.
Of course, if you have a small partition size and a huge overlay, then most of your overlay will not ever be usable, so you want to play with the two things together. As a rough idea, the free space in your partition (after everything's been installed) should be a bit less than the overlay size.
There are potential pitfalls here, because free filesystem space doesn't quite equate with free overlay space. I don't know what happens if you boot up and delete a large file that was installed in the squashfs--it might increase the free space as far as the filesystem is concerned, but it obviously won't buy you any extra overlay space.
James