On Tue, Sep 08, 2020 at 06:35:05AM -0400, Gunnar Niels wrote:
Hello,
I have docker and firewalld both installed on my Arch Linux machine and have been
happy with them so far, although I've recently run into an issue where I've
broken
what seems to be DNS for internal running containers, and I'm trying to isolate the
reason for that. Recently I created a bridge with my primary static ip address
and added my physical NIC as a slave on the bridge so that I'd be able to run
libvirt VMs that appeared to be machines directly on my local network. That
has all worked as expected:
---
# ipb link
lo UNKNOWN 00:00:00:00:00:00 <LOOPBACK,UP,LOWER_UP>
enp0s25 UP d0:50:99:a5:08:49
<BROADCAST,MULTICAST,UP,LOWER_UP>
exbr UP 9a:5f:b2:1e:94:a7
<BROADCAST,MULTICAST,UP,LOWER_UP>
virbr0 DOWN 52:54:00:4f:ca:69
<NO-CARRIER,BROADCAST,MULTICAST,UP>
virbr0-nic DOWN 52:54:00:4f:ca:69 <BROADCAST,MULTICAST>
docker0 DOWN 02:42:dd:21:67:40
<NO-CARRIER,BROADCAST,MULTICAST,UP>
# bridge link show
2: enp0s25: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master exbr state forwarding
priority 32 cost 100
5: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 master virbr0 state disabled priority
32 cost 100
# ipb a
lo UNKNOWN 127.0.0.1/8 ::1/128
enp0s25 UP
exbr UP 192.168.2.97/24 fe80::985f:b2ff:fe1e:94a7/64
virbr0 DOWN 192.168.100.1/24
virbr0-nic DOWN
docker0 DOWN 172.17.0.1/16 fe80::46e8:567:b3cf:1aa1/64
---
My problem has suddenly become: My docker containers do not appear to be capable
of dns requests unless I *disable* firwalld, something I obviously do not want to
do. So, I've taken the approach of turning off the firewalld, restarting docker,
and capturing the known good iptables dump, and then tried to compare it to a known
bad dump that is produced after I turn on firewalld and restart docker. Interestingly
*they are identical* as far as I can tell [1]. Additionally, I tried to TRACE the
traffic, and I'm seeing some very confusing info out of the kernel logs:
Looks like you're using a firewalld version that defaults to the
nftables backend. Check "nft list ruleset".
When things are configured in the "bad" state, the last log
I see is:
Sep 08 06:08:49 baldur kernel: TRACE: filter:FORWARD:rule:5 IN=docker0
OUT=exbrPHYSIN=veth7880ce1 MAC=02:42:dd:21:67:40:02:42:ac:11:00:02:08:00 SRC=172.17.0.2
DST=192.168.2.1 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=45314 DF PROTO=UDP SPT=45834 DPT=53
LEN=40
In my attached gist, you'll see the full iptables output dump for a "bad"
configuration,
and according to rule 5 on the filter FOREWARD chain....that's an accept all rule?
So I'm very configured why this would be a problem.
iptables and nftables are _independent_. That means it must pass through
both firewalls. If a packets is accepted in iptables it still has to
pass through the nftables (firewalld) rules.
In the known working case, here are my traces:
Failed trace output on filter rule 5
Sep 08 06:08:49 baldur kernel: TRACE: filter:DOCKER-ISOLATION-STAGE-1:return:2 IN=docker0
OUT=exbr PHYSIN=veth7880ce1 MAC=02:42:dd:21:67:40:02:42:ac:11:00:02:08:00 SRC=172.17.0.2
DST=192.168.2.1 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=45314 DF PROTO=UDP SPT=45834 DPT=53
LEN=40
Sep 08 06:08:49 baldur kernel: TRACE: filter:FORWARD:rule:5 IN=docker0 OUT=exbr
PHYSIN=veth7880ce1 MAC=02:42:dd:21:67:40:02:42:ac:11:00:02:08:00 SRC=172.17.0.2
DST=192.168.2.1 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=45314 DF PROTO=UDP SPT=45834 DPT=53
LEN=40
Sep 08 06:08:49 baldur kernel: TRACE: mangle:POSTROUTING:policy:1 IN=docker0 OUT=exbr
PHYSIN=veth7880ce1 MAC=02:42:dd:21:67:40:02:42:ac:11:00:02:08:00 SRC=172.17.0.2
DST=192.168.2.1 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=45314 DF PROTO=UDP SPT=45834 DPT=53
LEN=40
So obviously something about disabling firewalld allows for the packet to pass
through the forward chain and hit the mangle POSTROUTING. It's completely unclear
to me how to diagnose what might be going on here, any tips appreciated. I have the
fully verbose outputs in this gist:
Try adding the docker bridge to the "trusted" firewalld zone.
# firewall-cmd --zone trusted --add-interface docker0
docker (moby) very recently gained integration with firewalld [2].
[2]:
https://github.com/moby/libnetwork/pull/2548