Hi Nobin,
just wanted to let you know that I backported the fix to nm-1-38,
nm-1-36 and nm-1-34 branches. You can check it here:
https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/tree/nm-1-34
Let me know if you need anything else.
Thanks,
Fernando.
On Mon, Aug 29, 2022 at 11:08 AM Nobin Mathew <nobin.mathew(a)ericsson.com> wrote:
>
> Thanks Fernando for the quick response...
>
> It will be great if you can provide a backported patch for 1.34, otherwise please share the patch which fixed this problem, I will try to backport.
>
> -Nobin
>
> -----Original Message-----
> From: Fernando Fernandez Mancera <ferferna(a)redhat.com>
> Sent: Monday, August 29, 2022 1:06 PM
> To: Nobin Mathew <nobin.mathew(a)ericsson.com>
> Cc: nmstate-devel(a)lists.fedorahosted.org
> Subject: Re: [nmstate-devel] OVS missing some interfaces on node reboot or NetworkManager retstart
>
> Hi!
>
> On Mon, Aug 29, 2022 at 7:36 AM Nobin Mathew <nobin.mathew(a)ericsson.com> wrote:
> >
> > Hi,
> >
> > We r running
> >
> > ######
> >
> > /home/ # NetworkManager -V
> >
> > 1.34.0
> >
> > /home/ # ovs-vsctl -V
> >
> > ovs-vsctl (Open vSwitch) 2.14.2
> >
> > DB Schema 8.2.0
> >
> >
> >
> > /home/ # uname -a
> >
> > Linux 5.3.18-150300.59.87-default SMP Thu Jul 21 14:31:28 UTC 2022
> > (cc90276) x86_64 x86_64 x86_64 GNU/Linux
> >
> >
> >
> > this is SLES15 sp3 machine.
> >
> >
> >
> > pool1-bm25c5:~> nmstatectl --version
> >
> > 1.2.0
> >
> > pool1-bm25c5:~>
> >
> >
> >
> > ######
> >
> > when we reboot node or restart NetworkManager(systemctl restart NetworkManager) we are loosing some of the ovs interfaces (randomly, not on all restarts).
> >
>
> I think this issue is related to this bug:
> https://bugzilla.redhat.com/show_bug.cgi?id=2080236
>
> It is a NetworkManager issue and it was fixed in NetworkManager-1.39.6, could you check it out? Would you need an upstream backport to 1.34?
>
> Thanks!
> Fernando.
>
> > logs are here:
> >
> > Missing interfaces:
> >
> > https://protect2.fireeye.com/v1/url?k=31323334-501d5122-313273af-45444
> > 5555731-93530804397fe1ff&q=1&e=f9884072-555f-4847-9f8b-466f9b806b2a&u=
> > https%3A%2F%2Fgist.github.com%2Fnmathew%2Fd7d472b938f389c4efbadd01ef78
> > d466%23file-missing-interfaces
> >
> > NetworkManager trace log for restart duration
> >
> > https://protect2.fireeye.com/v1/url?k=31323334-501d5122-313273af-45444
> > 5555731-cfdd31dcdea77648&q=1&e=f9884072-555f-4847-9f8b-466f9b806b2a&u=
> > https%3A%2F%2Fgist.github.com%2Fnmathew%2Fd7d472b938f389c4efbadd01ef78
> > d466%23file-networkmanager-logs-with-trace
> >
> > /etc/NetworkManager/system-connections
> >
> > https://protect2.fireeye.com/v1/url?k=31323334-501d5122-313273af-45444
> > 5555731-3474c8a3dfe835ee&q=1&e=f9884072-555f-4847-9f8b-466f9b806b2a&u=
> > https%3A%2F%2Fgist.github.com%2Fnmathew%2Fd7d472b938f389c4efbadd01ef78
> > d466%23file-connection-conf
> >
> > ovsdb-client-dump for non working case(or when interfaces are missing)
> >
> > https://protect2.fireeye.com/v1/url?k=31323334-501d5122-313273af-45444
> > 5555731-289bfcbe422b0355&q=1&e=f9884072-555f-4847-9f8b-466f9b806b2a&u=
> > https%3A%2F%2Fgist.github.com%2Fnmathew%2Fd7d472b938f389c4efbadd01ef78
> > d466%23file-non-working-case-ovsdb-client-dump
> >
> > ovsdb-client-dump for working case(or when all interfaces are present)
> >
> > https://protect2.fireeye.com/v1/url?k=31323334-501d5122-313273af-45444
> > 5555731-b1efb31f6496469e&q=1&e=f9884072-555f-4847-9f8b-466f9b806b2a&u=
> > https%3A%2F%2Fgist.github.com%2Fnmathew%2Fd7d472b938f389c4efbadd01ef78
> > d466%23file-on-working-ovsdb-client-dump
> >
> > ovs-vsctl-show when interfaces are missing
> >
> > https://protect2.fireeye.com/v1/url?k=31323334-501d5122-313273af-45444
> > 5555731-75a8fd4d20fe10f6&q=1&e=f9884072-555f-4847-9f8b-466f9b806b2a&u=
> > https%3A%2F%2Fgist.github.com%2Fnmathew%2Fd7d472b938f389c4efbadd01ef78
> > d466%23file-ovs-vsctl-show
> >
> > ovs-vswitchd-log for the restart duration
> >
> > https://protect2.fireeye.com/v1/url?k=31323334-501d5122-313273af-45444
> > 5555731-8c324322ae3a8e8f&q=1&e=f9884072-555f-4847-9f8b-466f9b806b2a&u=
> > https%3A%2F%2Fgist.github.com%2Fnmathew%2Fd7d472b938f389c4efbadd01ef78
> > d466%23file-ovs-vswitchd-log
> >
> > ovsdb-tool-show-log for the restart duration
> >
> > https://protect2.fireeye.com/v1/url?k=31323334-501d5122-313273af-45444
> > 5555731-c5b663cb0344f371&q=1&e=f9884072-555f-4847-9f8b-466f9b806b2a&u=
> > https%3A%2F%2Fgist.github.com%2Fnmathew%2Fd7d472b938f389c4efbadd01ef78
> > d466%23file-sudo-ovsdb-tool-mmmmmm-show-log
> >
> > In this particular instance of run, we lost two interfaces.
> >
> > we have a script which sets "autoconnect-slaves" to 0, which was introduced for an older version of NM. grep -lF -e type=ovs-bridge -e type=ovs-interface -e type=ovs-port /etc/NetworkManager/system-connections/*|xargs -I % nmcli connection modify filename % connection.autoconnect-slaves 0 this scripts also runs currently, that why all "autoconnect-slaves" set to zero.
> >
> > when we are losing interfaces that port additions are not coming to ovsdb:
> >
> > 2022-08-26T22:38:26.176Z|00179|bridge|INFO|bridge data_bridge: added
> > interface traffic5_iface on port 1
> >
> > 2022-08-26T22:38:26.348Z|00182|bridge|INFO|bridge data_bridge: added
> > interface traffic1_iface on port 2
> >
> > 2022-08-26T22:38:26.376Z|00183|bridge|INFO|bridge data_bridge: added
> > interface bm25c_ext_iface on port 3
> >
> > 2022-08-26T22:38:26.376Z|00184|bridge|INFO|bridge data_bridge: added
> > interface bond0 on port 4
> >
> > Only 4 interfaces are getting added back, two are missing, "ovsdb-tool -mmmmmm show-log " shows the same.
> >
> > Any pointers where it is going wrong?
> >
> > -Nobin
> >
> >
> >
> > _______________________________________________
> > nmstate-devel mailing list -- nmstate-devel(a)lists.fedorahosted.org
> > To unsubscribe send an email to
> > nmstate-devel-leave(a)lists.fedorahosted.org
> > Fedora Code of Conduct:
> > https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > List Guidelines:
> > https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives:
> > https://protect2.fireeye.com/v1/url?k=31323334-501d5122-313273af-45444
> > 5555731-09b7aeb775896a89&q=1&e=f9884072-555f-4847-9f8b-466f9b806b2a&u=
> > https%3A%2F%2Flists.fedorahosted.org%2Farchives%2Flist%2Fnmstate-devel
> > %40lists.fedorahosted.org Do not reply to spam, report it:
> > https://protect2.fireeye.com/v1/url?k=31323334-501d5122-313273af-45444
> > 5555731-1df4bd1e5f7cb906&q=1&e=f9884072-555f-4847-9f8b-466f9b806b2a&u=
> > https%3A%2F%2Fpagure.io%2Ffedora-infrastructure%2Fnew_issue
Hi!
On Mon, Aug 29, 2022 at 7:36 AM Nobin Mathew <nobin.mathew(a)ericsson.com> wrote:
>
> Hi,
>
> We r running
>
> ######
>
> /home/ # NetworkManager -V
>
> 1.34.0
>
> /home/ # ovs-vsctl -V
>
> ovs-vsctl (Open vSwitch) 2.14.2
>
> DB Schema 8.2.0
>
>
>
> /home/ # uname -a
>
> Linux 5.3.18-150300.59.87-default SMP Thu Jul 21 14:31:28 UTC 2022 (cc90276) x86_64 x86_64 x86_64 GNU/Linux
>
>
>
> this is SLES15 sp3 machine.
>
>
>
> pool1-bm25c5:~> nmstatectl --version
>
> 1.2.0
>
> pool1-bm25c5:~>
>
>
>
> ######
>
> when we reboot node or restart NetworkManager(systemctl restart NetworkManager) we are loosing some of the ovs interfaces (randomly, not on all restarts).
>
I think this issue is related to this bug:
https://bugzilla.redhat.com/show_bug.cgi?id=2080236
It is a NetworkManager issue and it was fixed in
NetworkManager-1.39.6, could you check it out? Would you need an
upstream backport to 1.34?
Thanks!
Fernando.
> logs are here:
>
> Missing interfaces:
>
> https://gist.github.com/nmathew/d7d472b938f389c4efbadd01ef78d466#file-missi…
>
> NetworkManager trace log for restart duration
>
> https://gist.github.com/nmathew/d7d472b938f389c4efbadd01ef78d466#file-netwo…
>
> /etc/NetworkManager/system-connections
>
> https://gist.github.com/nmathew/d7d472b938f389c4efbadd01ef78d466#file-conne…
>
> ovsdb-client-dump for non working case(or when interfaces are missing)
>
> https://gist.github.com/nmathew/d7d472b938f389c4efbadd01ef78d466#file-non-w…
>
> ovsdb-client-dump for working case(or when all interfaces are present)
>
> https://gist.github.com/nmathew/d7d472b938f389c4efbadd01ef78d466#file-on-wo…
>
> ovs-vsctl-show when interfaces are missing
>
> https://gist.github.com/nmathew/d7d472b938f389c4efbadd01ef78d466#file-ovs-v…
>
> ovs-vswitchd-log for the restart duration
>
> https://gist.github.com/nmathew/d7d472b938f389c4efbadd01ef78d466#file-ovs-v…
>
> ovsdb-tool-show-log for the restart duration
>
> https://gist.github.com/nmathew/d7d472b938f389c4efbadd01ef78d466#file-sudo-…
>
> In this particular instance of run, we lost two interfaces.
>
> we have a script which sets "autoconnect-slaves" to 0, which was introduced for an older version of NM. grep -lF -e type=ovs-bridge -e type=ovs-interface -e type=ovs-port /etc/NetworkManager/system-connections/*|xargs -I % nmcli connection modify filename % connection.autoconnect-slaves 0 this scripts also runs currently, that why all "autoconnect-slaves" set to zero.
>
> when we are losing interfaces that port additions are not coming to ovsdb:
>
> 2022-08-26T22:38:26.176Z|00179|bridge|INFO|bridge data_bridge: added interface traffic5_iface on port 1
>
> 2022-08-26T22:38:26.348Z|00182|bridge|INFO|bridge data_bridge: added interface traffic1_iface on port 2
>
> 2022-08-26T22:38:26.376Z|00183|bridge|INFO|bridge data_bridge: added interface bm25c_ext_iface on port 3
>
> 2022-08-26T22:38:26.376Z|00184|bridge|INFO|bridge data_bridge: added interface bond0 on port 4
>
> Only 4 interfaces are getting added back, two are missing, "ovsdb-tool -mmmmmm show-log " shows the same.
>
> Any pointers where it is going wrong?
>
> -Nobin
>
>
>
> _______________________________________________
> nmstate-devel mailing list -- nmstate-devel(a)lists.fedorahosted.org
> To unsubscribe send an email to nmstate-devel-leave(a)lists.fedorahosted.org
> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedorahosted.org/archives/list/nmstate-devel@lists.fedorahost…
> Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Hi,
We r running
######
/home/ # NetworkManager -V
1.34.0
/home/ # ovs-vsctl -V
ovs-vsctl (Open vSwitch) 2.14.2
DB Schema 8.2.0
/home/ # uname -a
Linux 5.3.18-150300.59.87-default SMP Thu Jul 21 14:31:28 UTC 2022 (cc90276) x86_64 x86_64 x86_64 GNU/Linux
this is SLES15 sp3 machine.
pool1-bm25c5:~> nmstatectl --version
1.2.0
pool1-bm25c5:~>
######
when we reboot node or restart NetworkManager(systemctl restart NetworkManager) we are loosing some of the ovs interfaces (randomly, not on all restarts).
logs are here:
Missing interfaces:
https://gist.github.com/nmathew/d7d472b938f389c4efbadd01ef78d466#file-missi…
NetworkManager trace log for restart duration
https://gist.github.com/nmathew/d7d472b938f389c4efbadd01ef78d466#file-netwo…
/etc/NetworkManager/system-connections
https://gist.github.com/nmathew/d7d472b938f389c4efbadd01ef78d466#file-conne…
ovsdb-client-dump for non working case(or when interfaces are missing)
https://gist.github.com/nmathew/d7d472b938f389c4efbadd01ef78d466#file-non-w…
ovsdb-client-dump for working case(or when all interfaces are present)
https://gist.github.com/nmathew/d7d472b938f389c4efbadd01ef78d466#file-on-wo…
ovs-vsctl-show when interfaces are missing
https://gist.github.com/nmathew/d7d472b938f389c4efbadd01ef78d466#file-ovs-v…
ovs-vswitchd-log for the restart duration
https://gist.github.com/nmathew/d7d472b938f389c4efbadd01ef78d466#file-ovs-v…
ovsdb-tool-show-log for the restart duration
https://gist.github.com/nmathew/d7d472b938f389c4efbadd01ef78d466#file-sudo-…
In this particular instance of run, we lost two interfaces.
we have a script which sets "autoconnect-slaves" to 0, which was introduced for an older version of NM. grep -lF -e type=ovs-bridge -e type=ovs-interface -e type=ovs-port /etc/NetworkManager/system-connections/*|xargs -I % nmcli connection modify filename % connection.autoconnect-slaves 0 this scripts also runs currently, that why all "autoconnect-slaves" set to zero.
when we are losing interfaces that port additions are not coming to ovsdb:
2022-08-26T22:38:26.176Z|00179|bridge|INFO|bridge data_bridge: added interface traffic5_iface on port 1
2022-08-26T22:38:26.348Z|00182|bridge|INFO|bridge data_bridge: added interface traffic1_iface on port 2
2022-08-26T22:38:26.376Z|00183|bridge|INFO|bridge data_bridge: added interface bm25c_ext_iface on port 3
2022-08-26T22:38:26.376Z|00184|bridge|INFO|bridge data_bridge: added interface bond0 on port 4
Only 4 interfaces are getting added back, two are missing, "ovsdb-tool -mmmmmm show-log " shows the same.
Any pointers where it is going wrong?
-Nobin
Hi all,
The kubevirt networking team is working on a new cmd/lib tool to generate
dynamic NMState configurations.
When networking configuration for a cluster is needed and all the details
are common between the nodes in the cluster a NMState yaml configuration is
enough.
Problems arise when some of the network configuration details are different
between nodes and depend on the current node network state.
For that a different NMState yaml configuration needs to be generated per
node and that's not convenient for big clusters and also at scale up
scenarios.
The NMPolicy goal is to solve this problem.
Given a node network state and a network configuration policy (common to
the cluster), the NMPolicy tool will generate a node specific desired
network state.
NMPolicy was already integrated to kubernetes-nmstate and a usage example
can be found here
<https://github.com/nmstate/kubernetes-nmstate/blob/main/test/e2e/handler/de…>
.
*CLI* Usage example -
Simulated --help output
./nmpolicyctl -h
This tool helps you to generate dynamic NMState configurations.
The generated NMState configuration is written to *STDOUT*.
Usage of ./*nmpolicyctl*:
Commands:
*gen* Generates NMState by policy filename
Flags:
-s, --current-state=current-state.yaml: input file path to current
NMState. If not specified, stdin is used.
-c, --captured-state-output=captured-state-output.yaml: output file
path to the emitted captured states. If not specified,
~/.cache/nmpolicy/cache.yaml will be used.
*Creating a new bridge with the base interface as its port. The bridge will
get the mac address of the base interface.*
# echo """
capture:
*default-gw: routes.running.destination=="0.0.0.0/0 <http://0.0.0.0/0>"*
* base-iface: interfaces.name
<http://interfaces.name>==capture.default-gw.routes.running.0.next-hop-interface*
desiredState:
interfaces:
- name: br1
description: Linux bridge with base interface as a port
type: linux-bridge
state: up
mac-address: "{{ *capture.base-iface.interfaces.**0.mac-address* }}"
ipv4:
dhcp: true
enabled: true
bridge:
options:
stp:
enabled: false
port:
- name: "{{ *capture.base-iface.interfaces.0.name
<http://capture.base-iface.interfaces.0.name>* }}"
""" > *policy.yaml*
*# nmstatectl show | nmpolicyctl gen policy.yaml | nmstatectl apply*
*Output of - nmstatectl show | nmpolicyctl gen policy.yaml*
~/.cache/nmpolicy/*cache*.yaml (the default captured states output file)
base-iface:
metaInfo:
time: "2021-12-15T13:45:40Z"
state:
interfaces:
- accept-all-mac-addresses: false
ethernet:
auto-negotiation: false
ethtool:
feature:
rx-gro: true
rx-gro-list: false
rx-udp-gro-forwarding: false
tx-checksum-ip-generic: true
tx-generic-segmentation: true
tx-nocache-copy: false
tx-tcp-ecn-segmentation: true
tx-tcp-mangleid-segmentation: false
tx-tcp-segmentation: true
tx-tcp6-segmentation: true
ring:
rx: 256
tx: 256
ipv4:
address:
- ip: 192.168.66.101
prefix-length: 24
auto-dns: true
auto-gateway: true
auto-route-table-id: 0
auto-routes: true
dhcp: true
enabled: true
ipv6:
address:
- ip: fd00::101
prefix-length: 128
- ip: fe80::2d55:7c37:8090:7261
prefix-length: 64
auto-dns: true
auto-gateway: true
auto-route-table-id: 0
auto-routes: true
autoconf: true
dhcp: true
enabled: true
lldp:
enabled: false
*mac-address: 52:55:00:D1:55:01*
mtu: 1500
*name: eth0*
state: up
type: ethernet
default-gw:
metaInfo:
time: "2021-12-15T13:45:40Z"
state:
routes:
running:
- *destination: 0.0.0.0/0 <http://0.0.0.0/0>*
metric: 100
next-hop-address: 192.168.66.2
*next-hop-interface: eth0*
table-id: 254
*Stdout* (the *desired state* to be applied by nmstatectl)
interfaces:
- bridge:
options:
stp:
enabled: false
port:
- *name: eth0*
vlan:
mode: trunk
trunk-tags:
- id-range:
max: 4094
min: 2
description: Linux bridge with base interface as a port
ipv4:
dhcp: true
enabled: true
*mac-address: 52:55:00:D1:55:01*
*name: br1*
state: up
type: linux-bridge
For more details, please review the design doc
<https://docs.google.com/document/d/1k8gljRJoGtCGXoQ3YkTpDmekLAIeUWZ4P3-oPfj…>
and share your feedback about it.
Thanks,
Alona.
Hello everyone!
Nmstate community has decided to move our IRC channel #nmstate to
libera.chat due to the issues recently reported by the former freenode
staff [1]. The freenode channel will be kept for a while but we ask
you to please communicate via libera.chat.
In order to register your account and join the new server you can
follow these steps [2].
Thanks,
Fernando.
[1] https://www.kline.sh/
[2] https://libera.chat/guides
Hello everyone!
As reported on https://bugzilla.redhat.com/show_bug.cgi?id=1950734 the
current nmstate-0.3.6, 0.3.7, 1.0.2, 1.0.3 packages are broken. It is
not possible to update/remove the package. This was caused by a
missing BuildRequires dependency.
This issue has been fixed but in order to remove the current broken
package you should use the following command:
`dnf remove --setopt=tsflags=noscripts nmstate`
Please, let me know if you have any questions.
Thanks,
Fernando.