I think https://btrfs.wiki.kernel.org/index.php/Manpage/btrfs-qgroup and https://btrfs.wiki.kernel.org/index.php/Quota_support are what you are looking for.