It is likely anti-aliased fonts doing it. Trying to hint every character for the tons of text scrolling is cpu intensive. I have mentioned it to mharris before and he said something like the anti-alias font code isn't
It isnt the CPU that is the problem. A lot of code does compositing against server side image data, and the X servers (except the Nvidia proprietary one and the Vesa server) implement this in a totally screwball manner still. The resut is a lot of ping pong bus traffic and on some chipsets it seems to hurt the throughout of disks, latency of audio, video capture etc.