I've physically swapped Fedora ARM SD cards between RPi3 and RPi4 devices in my Pi
cluster without any problems.
Are you certain your RPi4 4GB device has sufficient power? I've seen Pi devices with
different USB peripherals not boot reliably if I don't have a good 5V 3A power supply
connected to it.