Cleaning up leftover bootstrap directories on builders