ok. After a bunch of debugging and digging today it sure looks like the
underlying problem was squid operating in smp mode.
We switched the squid servers back to non smp mode around 00:00 and I've
not seen any of these failures since then.
I sure hope everything is back to normal now.
kevin