Solved
I removed
<smartcard mode='passthrough' type='spicevmc'>
<address type='ccid' controller='0' slot='0'/>
</smartcard>
and the cpu usage dropped to match the guest.
FWIW, to track this down, I created a Fedora server using virt-manager and another with
Boxes and reproduced the issue. I compared their ps -lf outputs and tried removing the
extra components, and removing smartcard made the difference.