Hi Nobin,
just wanted to let you know that I backported the fix to nm-1-38, nm-1-36 and nm-1-34 branches. You can check it here: https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/tree/nm-1-34
Let me know if you need anything else.
Thanks, Fernando.
On Mon, Aug 29, 2022 at 11:08 AM Nobin Mathew nobin.mathew@ericsson.com wrote:
Thanks Fernando for the quick response...
It will be great if you can provide a backported patch for 1.34, otherwise please share the patch which fixed this problem, I will try to backport.
-Nobin
-----Original Message----- From: Fernando Fernandez Mancera ferferna@redhat.com Sent: Monday, August 29, 2022 1:06 PM To: Nobin Mathew nobin.mathew@ericsson.com Cc: nmstate-devel@lists.fedorahosted.org Subject: Re: [nmstate-devel] OVS missing some interfaces on node reboot or NetworkManager retstart
Hi!
On Mon, Aug 29, 2022 at 7:36 AM Nobin Mathew nobin.mathew@ericsson.com wrote:
Hi,
We r running
######
/home/ # NetworkManager -V
1.34.0
/home/ # ovs-vsctl -V
ovs-vsctl (Open vSwitch) 2.14.2
DB Schema 8.2.0
/home/ # uname -a
Linux 5.3.18-150300.59.87-default SMP Thu Jul 21 14:31:28 UTC 2022 (cc90276) x86_64 x86_64 x86_64 GNU/Linux
this is SLES15 sp3 machine.
pool1-bm25c5:~> nmstatectl --version
1.2.0
pool1-bm25c5:~>
######
when we reboot node or restart NetworkManager(systemctl restart NetworkManager) we are loosing some of the ovs interfaces (randomly, not on all restarts).
I think this issue is related to this bug: https://bugzilla.redhat.com/show_bug.cgi?id=2080236
It is a NetworkManager issue and it was fixed in NetworkManager-1.39.6, could you check it out? Would you need an upstream backport to 1.34?
Thanks! Fernando.
logs are here:
Missing interfaces:
https://protect2.fireeye.com/v1/url?k=31323334-501d5122-313273af-45444 5555731-93530804397fe1ff&q=1&e=f9884072-555f-4847-9f8b-466f9b806b2a&u= https%3A%2F%2Fgist.github.com%2Fnmathew%2Fd7d472b938f389c4efbadd01ef78 d466%23file-missing-interfaces
NetworkManager trace log for restart duration
https://protect2.fireeye.com/v1/url?k=31323334-501d5122-313273af-45444 5555731-cfdd31dcdea77648&q=1&e=f9884072-555f-4847-9f8b-466f9b806b2a&u= https%3A%2F%2Fgist.github.com%2Fnmathew%2Fd7d472b938f389c4efbadd01ef78 d466%23file-networkmanager-logs-with-trace
/etc/NetworkManager/system-connections
https://protect2.fireeye.com/v1/url?k=31323334-501d5122-313273af-45444 5555731-3474c8a3dfe835ee&q=1&e=f9884072-555f-4847-9f8b-466f9b806b2a&u= https%3A%2F%2Fgist.github.com%2Fnmathew%2Fd7d472b938f389c4efbadd01ef78 d466%23file-connection-conf
ovsdb-client-dump for non working case(or when interfaces are missing)
https://protect2.fireeye.com/v1/url?k=31323334-501d5122-313273af-45444 5555731-289bfcbe422b0355&q=1&e=f9884072-555f-4847-9f8b-466f9b806b2a&u= https%3A%2F%2Fgist.github.com%2Fnmathew%2Fd7d472b938f389c4efbadd01ef78 d466%23file-non-working-case-ovsdb-client-dump
ovsdb-client-dump for working case(or when all interfaces are present)
https://protect2.fireeye.com/v1/url?k=31323334-501d5122-313273af-45444 5555731-b1efb31f6496469e&q=1&e=f9884072-555f-4847-9f8b-466f9b806b2a&u= https%3A%2F%2Fgist.github.com%2Fnmathew%2Fd7d472b938f389c4efbadd01ef78 d466%23file-on-working-ovsdb-client-dump
ovs-vsctl-show when interfaces are missing
https://protect2.fireeye.com/v1/url?k=31323334-501d5122-313273af-45444 5555731-75a8fd4d20fe10f6&q=1&e=f9884072-555f-4847-9f8b-466f9b806b2a&u= https%3A%2F%2Fgist.github.com%2Fnmathew%2Fd7d472b938f389c4efbadd01ef78 d466%23file-ovs-vsctl-show
ovs-vswitchd-log for the restart duration
https://protect2.fireeye.com/v1/url?k=31323334-501d5122-313273af-45444 5555731-8c324322ae3a8e8f&q=1&e=f9884072-555f-4847-9f8b-466f9b806b2a&u= https%3A%2F%2Fgist.github.com%2Fnmathew%2Fd7d472b938f389c4efbadd01ef78 d466%23file-ovs-vswitchd-log
ovsdb-tool-show-log for the restart duration
https://protect2.fireeye.com/v1/url?k=31323334-501d5122-313273af-45444 5555731-c5b663cb0344f371&q=1&e=f9884072-555f-4847-9f8b-466f9b806b2a&u= https%3A%2F%2Fgist.github.com%2Fnmathew%2Fd7d472b938f389c4efbadd01ef78 d466%23file-sudo-ovsdb-tool-mmmmmm-show-log
In this particular instance of run, we lost two interfaces.
we have a script which sets "autoconnect-slaves" to 0, which was introduced for an older version of NM. grep -lF -e type=ovs-bridge -e type=ovs-interface -e type=ovs-port /etc/NetworkManager/system-connections/*|xargs -I % nmcli connection modify filename % connection.autoconnect-slaves 0 this scripts also runs currently, that why all "autoconnect-slaves" set to zero.
when we are losing interfaces that port additions are not coming to ovsdb:
2022-08-26T22:38:26.176Z|00179|bridge|INFO|bridge data_bridge: added interface traffic5_iface on port 1
2022-08-26T22:38:26.348Z|00182|bridge|INFO|bridge data_bridge: added interface traffic1_iface on port 2
2022-08-26T22:38:26.376Z|00183|bridge|INFO|bridge data_bridge: added interface bm25c_ext_iface on port 3
2022-08-26T22:38:26.376Z|00184|bridge|INFO|bridge data_bridge: added interface bond0 on port 4
Only 4 interfaces are getting added back, two are missing, "ovsdb-tool -mmmmmm show-log " shows the same.
Any pointers where it is going wrong?
-Nobin
nmstate-devel mailing list -- nmstate-devel@lists.fedorahosted.org To unsubscribe send an email to nmstate-devel-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://protect2.fireeye.com/v1/url?k=31323334-501d5122-313273af-45444 5555731-09b7aeb775896a89&q=1&e=f9884072-555f-4847-9f8b-466f9b806b2a&u= https%3A%2F%2Flists.fedorahosted.org%2Farchives%2Flist%2Fnmstate-devel %40lists.fedorahosted.org Do not reply to spam, report it: https://protect2.fireeye.com/v1/url?k=31323334-501d5122-313273af-45444 5555731-1df4bd1e5f7cb906&q=1&e=f9884072-555f-4847-9f8b-466f9b806b2a&u= https%3A%2F%2Fpagure.io%2Ffedora-infrastructure%2Fnew_issue