do you know if there's any reliable and widely available way how to
measure memory usage on Linux by user space application (Firefox in
this case) and detect low-memory state?
gmemusage is a tool to show memory usage per userspace application.
top can show low memory state.