The alloc function in the LZMA SDK allows you to pass it a pointer
to
your own allocater function. (I don't know whether or not the xz library works
like that. [Ed: Yes, it does.]) Would that be enough for UPX?
Maybe. Some changes would be required to UPX, on both the compression
and decompression sides. Decompression is highly machine-dependent:
machine-language instructions, bare stack, no libraries. Ten architectures
are supported today.
For decompression, current UPX [pre-]allocates all space (input, output, and
temporary) because that results in smaller code size, which is important.
Tens of bytes can make a difference.
An allocator function also requires effectively-static storage for the
'next' pointer, which can be cumbersome depending on architecture (such
as -fPIE, x86_64, ARM) and interference from SELinux (exec_mod, etc.)
--