I'm having trouble getting a gateway/router working.
Context: - I have two gateways, both to broadband, both requiring PPPoE - both are running Fedora 38 and use firewalld-1.3.1-1.fc38.noarch - one works and one doesn't work reliably: some web sites time out
I think that the problem is with MTU. PPPoE doesn't allow 1500 byte packets through: the limit is 1492.
The "proof" is that if I set the MTU to 1492 on a client's ethernet interface, the client can use the bad gateway with no problem.
firewalld.policy(5) says that there is an option:
tcp-mss-clamp Is an optional empty-element tag and can be used several times. If left empty maximum segment size is set to 'pmtu'. This tag has exactly one optional attribute:
value="string" Value can set maximum segment size to 'pmtu' (Path Maximum Transmission Unit) or a user-defined value that is greater than or equal to 536.
I took this as meaning that I could add <tcp-mss-clamp/> to the policy file, like this:
<?xml version="1.0" encoding="utf-8"?> <policy priority="100" target="ACCEPT"> <ingress-zone name="FedoraServer"/> <egress-zone name="external"/> <tcp-mss-clamp/> </policy>
When I do so, firewalld throws a python error (see below).
==> am I writing this option correctly? Should I have used <tcp-mss-clamp value="1492"/>?
==> is there a way of expressing this in firewall-cmd? (None is documented in firewall-cmd(1).)
==> is this a reasonable diagnostic? (I don't think so.)
==> bonus questions:
Why would some web sites be blocked by this MTU problem with one ISP and not the other?
The working ISP is providing VDSL, the last mile provided by Bell Canada. The failing ISP is Bell Canada itself, providing FTTH.
Why some sites (eg. news.banggood.com) and not others?
What is the correct fix for my problem? Somehow Path MTU discovery isn't solving the problem.
================ log of firewalld python error ================ Aug 06 02:29:40 my-hostname systemd[1]: Starting firewalld.service - firewalld - dynamic firewall daemon... Aug 06 02:29:40 my-hostname systemd[1]: Started firewalld.service - firewalld - dynamic firewall daemon. Aug 06 02:29:40 my-hostname audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=firewalld comm="systemd" exe="/usr/lib> Aug 06 02:29:40 my-hostname firewalld[17576]: ERROR: Failed to load user configuration. Falling back to full stock configuration. Aug 06 02:29:40 my-hostname audit[17576]: NETFILTER_CFG table=firewalld:13 family=1 entries=1 op=nft_register_table pid=17576 subj=system_u:system_r:firewalld_t:s0 comm="firewa> Aug 06 02:29:40 my-hostname audit[17576]: NETFILTER_CFG table=firewalld:14 family=1 entries=222 op=nft_register_chain pid=17576 subj=system_u:system_r:firewalld_t:s0 comm="fire> Aug 06 02:29:40 my-hostname firewalld[17576]: Traceback (most recent call last): File "/usr/lib/python3.11/site-packages/firewall/server/decorators.py", line 51, in _impl return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.11/site-packages/firewall/server/firewalld.py", line 91, in start return self.fw.start() ^^^^^^^^^^^^^^^ File "/usr/lib/python3.11/site-packages/firewall/core/fw.py", line 656, in start raise original_ex File "/usr/lib/python3.11/site-packages/firewall/core/fw.py", line 633, in start self._start() File "/usr/lib/python3.11/site-packages/firewall/core/fw.py", line 584, in _start self._start_load_user_config() File "/usr/lib/python3.11/site-packages/firewall/core/fw.py", line 410, in _start_load_user_config self._loader_policies(config.ETC_FIREWALLD_POLICIES) File "/usr/lib/python3.11/site-packages/firewall/core/fw.py", line 716, in _loader_policies obj = policy_reader(filename, path) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.11/site-packages/firewall/core/io/policy.py", line 1075, in policy_reader parser.parse(source) File "/usr/lib64/python3.11/xml/sax/expatreader.py", line 111, in parse xmlreader.IncrementalParser.parse(self, source) File "/usr/lib64/python3.11/xml/sax/xmlreader.py", line 125, in parse self.feed(buffer) File "/usr/lib64/python3.11/xml/sax/expatreader.py", line 217, in feed self._parser.Parse(data, isFinal) File "/builddir/build/BUILD/Python-3.11.4/Modules/pyexpat.c", line 416, in StartElement File "/usr/lib64/python3.11/xml/sax/expatreader.py", line 333, in start_element self._cont_handler.startElement(name, AttributesImpl(attrs)) File "/usr/lib/python3.11/site-packages/firewall/core/io/policy.py", line 993, in startElement if common_startElement(self, name, attrs): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.11/site-packages/firewall/core/io/policy.py", line 117, in common_startElement attrs["value"]) ~~~~~^^^^^^^^^ File "/usr/lib64/python3.11/xml/sax/xmlreader.py", line 318, in __getitem__ return self._attrs[name] ~~~~~~~~~~~^^^^^^ KeyError: 'value'
On 06.08.2023 10:30, D. Hugh Redelmeier wrote:
I'm having trouble getting a gateway/router working.
Context:
- I have two gateways, both to broadband, both requiring PPPoE
- both are running Fedora 38 and use firewalld-1.3.1-1.fc38.noarch
- one works and one doesn't work reliably: some web sites time out
I think that the problem is with MTU. PPPoE doesn't allow 1500 byte packets through: the limit is 1492.
The "proof" is that if I set the MTU to 1492 on a client's ethernet interface, the client can use the bad gateway with no problem.
firewalld.policy(5) says that there is an option:
tcp-mss-clamp Is an optional empty-element tag and can be used several times. If left empty maximum segment size is set to 'pmtu'. This tag has exactly one optional attribute:
value="string" Value can set maximum segment size to 'pmtu' (Path Maximum Transmission Unit) or a user-defined value that is greater than or equal to 536.I took this as meaning that I could add <tcp-mss-clamp/> to the policy file, like this:
<?xml version="1.0" encoding="utf-8"?>
<policy priority="100" target="ACCEPT"> <ingress-zone name="FedoraServer"/> <egress-zone name="external"/> <tcp-mss-clamp/> </policy>
It sounds like the documentation bug. For all I can tell, tcp-mss-clamp can only be used in rich rules and all firewalld regression tests also use it only in rich rules.
| From: Andrei Borzenkov arvidjaar@gmail.com
| It sounds like the documentation bug. For all I can tell, tcp-mss-clamp can | only be used in rich rules and all firewalld regression tests also use it only | in rich rules.
Thanks for the quick reply.
(That still leaves the problem of Firewalld crashing with a python stack dump.)
Speaking without a lot of knowledge:
I try to avoid Rich Rules since they feel like a kind of failure of the model of Firewalld.
I think clamping is required when forwarding is done: the application cannot just ask the (local) interface what its MTU is. So it seems natural to put this in a Policy Object. I guess not all forwarding is inter-zonal, but all mine is. Not all interfaces in a zone need to have the same MTU, but all mine do.
In the mean time, I will go look at Rich Rules again.
This might not be the whole picture. For example, I don't know:
- why PMTU discovery doesn't solve this problem
- how clamping can work in the face of end-to-end encryption.
- why my one router works and my other doesn't. They both run Fedora 38 with Firewalld and limit the upstream MTU to 1492.
On Sun, Aug 06, 2023 at 11:56:33AM -0400, D. Hugh Redelmeier wrote:
| From: Andrei Borzenkov arvidjaar@gmail.com
| It sounds like the documentation bug. For all I can tell, tcp-mss-clamp can | only be used in rich rules and all firewalld regression tests also use it only | in rich rules.
Thanks for the quick reply.
It's a doc bug. tcp-mss-clamp is only valid inside of a `<rich>` element.
(That still leaves the problem of Firewalld crashing with a python stack dump.)
Check `firewall-cmd --state`. I bet it's in failed state because your configuration is invalid.
The stack dump is from the failed start. A few lines above you can see that firewalld performed a fallback to the safe defaults. That being said, it's an ugly failure in the code and should be throwing an intelligible error. Feel free to file an issue on github to improve this.
The log indicates your configuration is broken.
Aug 06 02:29:40 my-hostname firewalld[17576]: ERROR: Failed to load user configuration. Falling back to full stock configuration.
You could also try `firewall-cmd --check-config`.
Speaking without a lot of knowledge:
I try to avoid Rich Rules since they feel like a kind of failure of the model of Firewalld.
Agree. But for uncommon/advanced things it's compromise. We either muddy the CLI with tons of options.. or put the advanced features in rich rules.
I think clamping is required when forwarding is done: the application cannot just ask the (local) interface what its MTU is. So it seems natural to put this in a Policy Object. I guess not all forwarding is inter-zonal, but all mine is. Not all interfaces in a zone need to have the same MTU, but all mine do.
In the mean time, I will go look at Rich Rules again.
This might not be the whole picture. For example, I don't know:
- why PMTU discovery doesn't solve this problem
It should.
- how clamping can work in the face of end-to-end encryption.
The outer IP/TCP is where the MSS negotiation happens. The inner part of the packet is encrypted, not the outer part.
- why my one router works and my other doesn't. They both run Fedora 38 with Firewalld and limit the upstream MTU to 1492.
I don't know without more information: configs, etc.
| From: Eric Garver egarver@redhat.com
Thanks for the useful reply. I hope the project benefits from naive questions.
| It's a doc bug. tcp-mss-clamp is only valid inside of a `<rich>` | element.
Is <rule> the same as <rich>? firewalld.policy(5) doesn't mention <rich>.
This is accepted in my policy XML file
<?xml version="1.0" encoding="utf-8"?> <policy priority="100" target="ACCEPT"> <ingress-zone name="FedoraServer"/> <egress-zone name="external"/> <rule> <tcp-mss-clamp value="1492"/> </rule> </policy>
This is not allowed by the syntax described in firewalld.policy(5).
I don't know that it is doing anything. It doesn't fix my particular problem.
I tried this to see if the clamping showed up in the netfilter configuration: nft list ruleset | grep -i mss It lists nothing. I'm not adept at nft, so this might not be the right test.
firewalld.policy(5) gives the syntax for rich rules twice. It would be better to only have one description. This wold be less confusing and it would prevent divergence. I think that it should include clamping.
Perhaps rich rules should be described only in their own man page. This is assuming the rules are the same whether they are for a zone or a policy object. firewalld.richlang(5) would seem to be the right place but it doesn't talk about XML. It says to look at firewalld.zone(5) and doesn't mention firewalld.policy(5) (that seems like a bug).
firewalld.richlang(5) lists Tcp-Mss-Clamp but the syntax doesn't seem to have a place for it. I guess that it is syntactically like masquerade but that isn't what the page says.
Why is Tcp-Mss-Clamp capitalized this way in filrewalld.richlanguage(5)? The natural way to capitalize it would be TCP-MSS-clamp. But all lower case is probably more humane. I notice that ICMP is all-caps.
On Tue, Aug 08, 2023 at 01:09:12PM -0400, D. Hugh Redelmeier wrote:
| From: Eric Garver egarver@redhat.com
Thanks for the useful reply. I hope the project benefits from naive questions.
| It's a doc bug. tcp-mss-clamp is only valid inside of a `<rich>` | element.
Is <rule> the same as <rich>? firewalld.policy(5) doesn't mention <rich>.
It should be "<rule>". My bad. :)
This is accepted in my policy XML file
<?xml version="1.0" encoding="utf-8"?>
<policy priority="100" target="ACCEPT"> <ingress-zone name="FedoraServer"/> <egress-zone name="external"/> <rule> <tcp-mss-clamp value="1492"/> </rule> </policy>
This is not allowed by the syntax described in firewalld.policy(5).
That's a doc bug. It's valid.
I don't know that it is doing anything. It doesn't fix my particular problem.
I tried this to see if the clamping showed up in the netfilter configuration: nft list ruleset | grep -i mss It lists nothing. I'm not adept at nft, so this might not be the right test.
Perhaps these above zones are not actually active (have interfaces/sources).
Can you show --get-active-zones ?
firewalld.policy(5) gives the syntax for rich rules twice. It would be better to only have one description. This wold be less confusing and it would prevent divergence. I think that it should include clamping.
Perhaps rich rules should be described only in their own man page. This is assuming the rules are the same whether they are for a zone or a policy object. firewalld.richlang(5) would seem to be the right place but it doesn't talk about XML. It says to look at firewalld.zone(5) and doesn't mention firewalld.policy(5) (that seems like a bug).
firewalld.richlang(5) lists Tcp-Mss-Clamp but the syntax doesn't seem to have a place for it. I guess that it is syntactically like masquerade but that isn't what the page says.
Why is Tcp-Mss-Clamp capitalized this way in filrewalld.richlanguage(5)? The natural way to capitalize it would be TCP-MSS-clamp. But all lower case is probably more humane. I notice that ICMP is all-caps.
Sorry about all these. Please open a github issue for these doc bugs.
firewalld-users@lists.fedorahosted.org
-
Andrei Borzenkov -
D. Hugh Redelmeier -
Eric Garver