Hi,
Can someone provide the recipes for firewalling libvirt guests? I see talk about it but no clear answers how to correctly implement it.
I would like to be able to enable the following typical scenarios: 1. Guest restricted to connecting to existing vpn (privateinternetaccess.com) established by PIA app on a given branch. No in or out connections to LAN resources. No connections to other guests. 2. Direct connection to WAN via gateway. Bypasses VPN. No connections to LAN. No connections to other guests. 3. Guest connection to resources provided by VM host only.
I know this is a big ask, but I've been reading posts here & elsewhere and its a bit confusing. Previously I had this kluged together through a series of iptables scripts but it was an awful hack and I'd like to do better this time. This is on kubuntu 20.10 (firewalld 0.9.3, libvirt 7.6.0, QEMU 6.0.0), pretty close to stock setup.
Thanks!
+ Laine, from the libvirt project
On Tue, Oct 26, 2021 at 05:37:41AM -0000, David Geise wrote:
Hi,
Can someone provide the recipes for firewalling libvirt guests? I see talk about it but no clear answers how to correctly implement it.
I can try.
Firewalld v0.9.0+ has policy objects which allow forward filtering. I think that will help in many areas you point to below.
This blog may also be useful: https://firewalld.org/2020/09/policy-objects-filtering-container-and-vm-traf...
I would like to be able to enable the following typical scenarios:
- Guest restricted to connecting to existing vpn (privateinternetaccess.com) established by PIA app on a given branch. No in or out connections to LAN resources. No connections to other guests.
Newer libvirt uses a "libvirt" zone. You should put your VPN interface into a separate zone, e.g. "vpn". Then you can create one policy to allow forwarding to the VPN and second policy to deny everything else.
A policy to allow forward from libvirt --> VPN would look like this (untested):
# firewall-cmd --permanent --new-policy libvirtToVpn # firewall-cmd --permanent --policy libvirtToVpn --priority -100 # firewall-cmd --permanent --policy libvirtToVpn \ --add-ingress-zone libvirt # firewall-cmd --permanent --policy libvirtToVpn \ --add-egress-zone vpn # firewall-cmd --permanent --policy libvirtToVpn \ --set-target ACCEPT
A policy to deny forward from libvirt --> world would look like this (untested):
# firewall-cmd --permanent --new-policy libvirtToWorld # firewall-cmd --permanent --policy libvirtToWorld --priority -99 # firewall-cmd --permanent --policy libvirtToWorld \ --add-ingress-zone libvirt # firewall-cmd --permanent --policy libvirtToWorld \ --add-egress-zone ANY # firewall-cmd --permanent --policy libvirtToWorld \ --set-target REJECT
Note: libvirtToWorld uses a lower precedence priority value so it will execute _after_ libvirtToVpn.
Note 2: Since firewalld is stateful the return traffic will implicitly be allowed.
- Direct connection to WAN via gateway. Bypasses VPN. No connections to LAN. No connections to other guests.
Similar to above, but put your gateway into a separate zone, e.g. "external".
- Guest connection to resources provided by VM host only.
I don't understand what you mean by this.
I know this is a big ask, but I've been reading posts here & elsewhere and its a bit confusing.
No problem. This is why the mailing list exists. :)
Previously I had this kluged together through a series of iptables scripts but it was an awful hack and I'd like to do better this time. This is on kubuntu 20.10 (firewalld 0.9.3, libvirt 7.6.0, QEMU 6.0.0), pretty close to stock setup.
Hopefully the above points you in the correct direction.
Eric.
On 10/26/21 2:27 PM, Eric Garver wrote:
- Laine, from the libvirt project
On Tue, Oct 26, 2021 at 05:37:41AM -0000, David Geise wrote:
Hi,
Can someone provide the recipes for firewalling libvirt guests? I see talk about it but no clear answers how to correctly implement it.
I can try.
Firewalld v0.9.0+ has policy objects which allow forward filtering. I think that will help in many areas you point to below.
This blog may also be useful: https://firewalld.org/2020/09/policy-objects-filtering-container-and-vm-traf...
I would like to be able to enable the following typical scenarios:
- Guest restricted to connecting to existing vpn (privateinternetaccess.com) established by PIA app on a given branch. No in or out connections to LAN resources. No connections to other guests.
Newer libvirt uses a "libvirt" zone. You should put your VPN interface into a separate zone, e.g. "vpn". Then you can create one policy to allow forwarding to the VPN and second policy to deny everything else.
A policy to allow forward from libvirt --> VPN would look like this (untested):
# firewall-cmd --permanent --new-policy libvirtToVpn # firewall-cmd --permanent --policy libvirtToVpn --priority -100 # firewall-cmd --permanent --policy libvirtToVpn \ --add-ingress-zone libvirt # firewall-cmd --permanent --policy libvirtToVpn \ --add-egress-zone vpn # firewall-cmd --permanent --policy libvirtToVpn \ --set-target ACCEPT
A policy to deny forward from libvirt --> world would look like this (untested):
# firewall-cmd --permanent --new-policy libvirtToWorld # firewall-cmd --permanent --policy libvirtToWorld --priority -99 # firewall-cmd --permanent --policy libvirtToWorld \ --add-ingress-zone libvirt # firewall-cmd --permanent --policy libvirtToWorld \ --add-egress-zone ANY # firewall-cmd --permanent --policy libvirtToWorld \ --set-target REJECT
But if the VPN interface was intended to be the default route for all traffic from VMs, and the "normal" (non-VPN) ethernet on the host was intended to be the default route for traffic from the host (and maybe from some other VMs), then you'd need some sort of policy *routing*, which would result in a different next-hop for traffic coming from the VM's tap device / IP address. Is there policy-based routing underneath the --add-egress-zone/--policy options in your example?
Normally, once the packets come out of the VM's tap device and into the host networking stack, they will be routed according to their destination address and nothing else, so if the default route on the host was via enp2s0, then both host and VM traffic would go out that interface.
Note that libvirt does have a *very* primitive method of limiting traffic outbound from one of its virtual networks to the outside - in the <forward> element for the network, you can set the "dev" attribute, and that will limit outbound/inbound traffic for any VM connected to that network to using the specified interface. So for example, if your libvirt network definition had
<forward mode='nat' dev='vpn0'/>
then all traffic from guests that ended up being routed out a different interface from "vpn0" would be rejected. But this suffers from the problem I mention above - it doesn't magically cause all the traffic to egress via vpn0, it just rejects all the traffic that doesn't end up doing that (it's still up to the IP routing on the host to figure out that the traffic needs to egress via vpn0).
Note: libvirtToWorld uses a lower precedence priority value so it will execute _after_ libvirtToVpn.
Note 2: Since firewalld is stateful the return traffic will implicitly be allowed.
- Direct connection to WAN via gateway. Bypasses VPN. No connections to LAN. No connections to other guests.
Similar to above, but put your gateway into a separate zone, e.g. "external".
This implies to me that there *is* policy-based routing being setup by firewalld, otherwise I don't understand how it could work...
- Guest connection to resources provided by VM host only.
I don't understand what you mean by this.
I think he's asking for the equivalent of what libvirt calls and "isolated" network - the guests on the network can communicate with each other and with the host, but nothing else. This can easily be done in libvirt by defining an isolated network and connecting to that, as outlined here:
https://libvirt.org/formatnetwork.html#examplesPrivate
(you can additionally prevent the VMs connected to this network from communicating with each other by adding <port isolated='yes'/> to the network definition - this sets a bit for each port on the bridge device. so doesn't even involve iptables).
I know this is a big ask, but I've been reading posts here & elsewhere and its a bit confusing.
No problem. This is why the mailing list exists. :)
Previously I had this kluged together through a series of iptables scripts but it was an awful hack and I'd like to do better this time. This is on kubuntu 20.10 (firewalld 0.9.3, libvirt 7.6.0, QEMU 6.0.0), pretty close to stock setup.
Hopefully the above points you in the correct direction.
Note that all of my comments point in a completely *different* direction, since I'm coming from the PoV of libvirt. Eric's suggestions are likely more modern and flexible. You may find that some of the things I suggest are simpler, and could be adequate for certain cases, but may not do everything you need (in particular, the facilities built into libvirt have no way to do policy routing so that some VMs have a default route pointing to the VPN and some to the normal local network's default router).
Eric.
On Wed, Oct 27, 2021 at 04:46:35PM -0400, Laine Stump wrote:
On 10/26/21 2:27 PM, Eric Garver wrote:
- Laine, from the libvirt project
On Tue, Oct 26, 2021 at 05:37:41AM -0000, David Geise wrote:
Hi,
Can someone provide the recipes for firewalling libvirt guests? I see talk about it but no clear answers how to correctly implement it.
I can try.
Firewalld v0.9.0+ has policy objects which allow forward filtering. I think that will help in many areas you point to below.
This blog may also be useful: https://firewalld.org/2020/09/policy-objects-filtering-container-and-vm-traf...
I would like to be able to enable the following typical scenarios:
- Guest restricted to connecting to existing vpn (privateinternetaccess.com) established by PIA app on a given branch. No in or out connections to LAN resources. No connections to other guests.
Newer libvirt uses a "libvirt" zone. You should put your VPN interface into a separate zone, e.g. "vpn". Then you can create one policy to allow forwarding to the VPN and second policy to deny everything else.
A policy to allow forward from libvirt --> VPN would look like this (untested):
# firewall-cmd --permanent --new-policy libvirtToVpn # firewall-cmd --permanent --policy libvirtToVpn --priority -100 # firewall-cmd --permanent --policy libvirtToVpn \ --add-ingress-zone libvirt # firewall-cmd --permanent --policy libvirtToVpn \ --add-egress-zone vpn # firewall-cmd --permanent --policy libvirtToVpn \ --set-target ACCEPT
A policy to deny forward from libvirt --> world would look like this (untested):
# firewall-cmd --permanent --new-policy libvirtToWorld # firewall-cmd --permanent --policy libvirtToWorld --priority -99 # firewall-cmd --permanent --policy libvirtToWorld \ --add-ingress-zone libvirt # firewall-cmd --permanent --policy libvirtToWorld \ --add-egress-zone ANY # firewall-cmd --permanent --policy libvirtToWorld \ --set-target REJECT
But if the VPN interface was intended to be the default route for all traffic from VMs, and the "normal" (non-VPN) ethernet on the host was intended to be the default route for traffic from the host (and maybe from some other VMs), then you'd need some sort of policy *routing*, which would result in a different next-hop for traffic coming from the VM's tap device / IP address.
Agree. You would need help from ip-route.
Is there policy-based routing underneath the --add-egress-zone/--policy options in your example?
No. Firewalld doesn't touch routing.
Normally, once the packets come out of the VM's tap device and into the host networking stack, they will be routed according to their destination address and nothing else, so if the default route on the host was via enp2s0, then both host and VM traffic would go out that interface.
Note that libvirt does have a *very* primitive method of limiting traffic outbound from one of its virtual networks to the outside - in the <forward> element for the network, you can set the "dev" attribute, and that will limit outbound/inbound traffic for any VM connected to that network to using the specified interface. So for example, if your libvirt network definition had
<forward mode='nat' dev='vpn0'/>
then all traffic from guests that ended up being routed out a different interface from "vpn0" would be rejected. But this suffers from the problem I mention above - it doesn't magically cause all the traffic to egress via vpn0, it just rejects all the traffic that doesn't end up doing that (it's still up to the IP routing on the host to figure out that the traffic needs to egress via vpn0).
This sounds a lot like what I suggested with the above firewalld config. So libvirt user have two options.
Note: libvirtToWorld uses a lower precedence priority value so it will execute _after_ libvirtToVpn.
Note 2: Since firewalld is stateful the return traffic will implicitly be allowed.
- Direct connection to WAN via gateway. Bypasses VPN. No connections to LAN. No connections to other guests.
Similar to above, but put your gateway into a separate zone, e.g. "external".
This implies to me that there *is* policy-based routing being setup by firewalld, otherwise I don't understand how it could work...
I glossed over the routing. I guess I should have been explicit that routing will also have to be configured by the user (out-of-band of firewalld).
- Guest connection to resources provided by VM host only.
I don't understand what you mean by this.
I think he's asking for the equivalent of what libvirt calls and "isolated" network - the guests on the network can communicate with each other and with the host, but nothing else. This can easily be done in libvirt by defining an isolated network and connecting to that, as outlined here:
https://libvirt.org/formatnetwork.html#examplesPrivate
(you can additionally prevent the VMs connected to this network from communicating with each other by adding <port isolated='yes'/> to the network definition - this sets a bit for each port on the bridge device. so doesn't even involve iptables).
I know this is a big ask, but I've been reading posts here & elsewhere and its a bit confusing.
No problem. This is why the mailing list exists. :)
Previously I had this kluged together through a series of iptables scripts but it was an awful hack and I'd like to do better this time. This is on kubuntu 20.10 (firewalld 0.9.3, libvirt 7.6.0, QEMU 6.0.0), pretty close to stock setup.
Hopefully the above points you in the correct direction.
Note that all of my comments point in a completely *different* direction, since I'm coming from the PoV of libvirt. Eric's suggestions are likely more modern and flexible. You may find that some of the things I suggest are simpler, and could be adequate for certain cases, but may not do everything you need (in particular, the facilities built into libvirt have no way to do policy routing so that some VMs have a default route pointing to the VPN and some to the normal local network's default router).
If we have handful of use cases I'll happily write a blog/tutorial and post it to https://firewalld.org/blog.
firewalld-users@lists.fedorahosted.org
-
David Geise -
Eric Garver -
Laine Stump