I've physically swapped Fedora ARM SD cards between RPi3 and RPi4 devices in my Pi cluster without any problems.
Are you certain your RPi4 4GB device has sufficient power? I've seen Pi devices with different USB peripherals not boot reliably if I don't have a good 5V 3A power supply connected to it.