Hi All, After upgrading from 3.10.104 to kernel 4.14 I see constant drops (around 10 packets/sec) on team interface in active-backup mode, increasing txqueuelen on a physical port helps, however, drops still occur from time to time. I've added debug printk in team driver where dev_kfree_skb_any is called, however, I do not see messages in dmesg until I reset physical interface by ethtool, therefore packets are being dropped somewhere else, however, for now, I have no clue how to find the exact cause. Do you have any ideas/suggestions on how to debug this? Thanks!
net7 Link encap:Ethernet HWaddr 00:1E:67:B5:7F:76 inet addr:192.168.221.203 Bcast:0.0.0.0 Mask:255.255.255.0 inet6 addr: fe80::21e:67ff:feb5:7f76/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:263 errors:0 dropped:151 overruns:0 frame:0 TX packets:83944096 errors:0 dropped:57692 overruns:0 carrier:0 collisions:0 txqueuelen:10000 RX bytes:27531 (26.8 KiB) TX bytes:113995992408 (106.1 GiB)
# teamnl net7 ports 5: eth6: up 1000Mbit FD
# ethtool -S eth6 NIC statistics: rx_packets: 113 tx_packets: 86938454 rx_bytes: 22848 tx_bytes: 118410084622 rx_broadcast: 68 tx_broadcast: 6 rx_multicast: 45 tx_multicast: 86938448 multicast: 45 collisions: 0 rx_crc_errors: 0 rx_no_buffer_count: 0 rx_missed_errors: 0 tx_aborted_errors: 0 tx_carrier_errors: 0 tx_window_errors: 0 tx_abort_late_coll: 0 tx_deferred_ok: 0 tx_single_coll_ok: 0 tx_multi_coll_ok: 0 tx_timeout_count: 0 rx_long_length_errors: 0 rx_short_length_errors: 0 rx_align_errors: 0 tx_tcp_seg_good: 0 tx_tcp_seg_failed: 0 rx_flow_control_xon: 68 rx_flow_control_xoff: 96 tx_flow_control_xon: 0 tx_flow_control_xoff: 0 rx_long_byte_count: 22848 tx_dma_out_of_sync: 0 lro_aggregated: 0 lro_flushed: 0 tx_smbus: 0 rx_smbus: 0 dropped_smbus: 0 os2bmc_rx_by_bmc: 0 os2bmc_tx_by_bmc: 0 os2bmc_tx_by_host: 0 os2bmc_rx_by_host: 0 tx_hwtstamp_timeouts: 0 rx_hwtstamp_cleared: 0 rx_errors: 0 tx_errors: 0 tx_dropped: 0 rx_length_errors: 0 rx_over_errors: 0 rx_frame_errors: 0 rx_fifo_errors: 0 tx_fifo_errors: 0 tx_heartbeat_errors: 0 tx_queue_0_packets: 86938440 tx_queue_0_bytes: 118062311560 tx_queue_0_restart: 0 rx_queue_0_packets: 277 rx_queue_0_bytes: 32236 rx_queue_0_drops: 0 rx_queue_0_csum_err: 0 rx_queue_0_alloc_failed: 0
# teamdctl net7 s d { "ports": { "eth6": { "ifinfo": { "dev_addr": "00:1e:67:b5:7f:76", "dev_addr_len": 6, "ifindex": 5, "ifname": "eth6" }, "link": { "duplex": "full", "speed": 1000, "up": true }, "link_watches": { "list": { "link_watch_0": { "delay_down": 0, "delay_up": 0, "down_count": 1, "name": "ethtool", "up": true } }, "up": true } } }, "runner": { "active_port": "eth6" }, "setup": { "daemonized": true, "dbus_enabled": false, "debug_level": 0, "kernel_team_mode_name": "activebackup", "pid": 5533, "pid_file": "/var/run/teamd/net7.pid", "runner_name": "activebackup", "zmq_enabled": false }, "team_device": { "ifinfo": { "dev_addr": "00:1e:67:b5:7f:76", "dev_addr_len": 6, "ifindex": 13, "ifname": "net7" } } }
This seems more like a difference in the operation of the underlying slave than a difference in teaming.
You could try the slave on its own, or try the slave with bonding, to rule out teaming as a cause.
You can find calls to kfree_skb easier with the dropwatch utility, or with SystemTap's example script dropwatch.stp. I find it easier to add a timestamp to the stap script's 5-second loop so you know when the packets were freed. You can then correspond a difference in dropwatch output with timestamped collection of "ip -s link" or similar.
You could possibly avoid this by increasing the device hardware tx buffer to max, eg: ethtool --set-ring ethX tx 4096
For possible underlying reasons:
Make sure you have irqbalance running or are balancing IRQ channels across real cores (not HyperThreads) all in the same NUMA node, and in the same NUMA node as the NIC if the NIC has NUMA locality. See my notes and video here for a little more in-depth explanation: http://jbainbri.github.io/
Make sure you aren't doing too much expensive traffic handling in the transmit path, like passing traffic through tens of thousands of firewall or tc rules.
Jamie
On Sun, 22 Sep 2019 at 12:43, Yevhen Kondrashyn e.kondrashyn@gmail.com wrote:
Hi All, After upgrading from 3.10.104 to kernel 4.14 I see constant drops (around 10 packets/sec) on team interface in active-backup mode, increasing txqueuelen on a physical port helps, however, drops still occur from time to time. I've added debug printk in team driver where dev_kfree_skb_any is called, however, I do not see messages in dmesg until I reset physical interface by ethtool, therefore packets are being dropped somewhere else, however, for now, I have no clue how to find the exact cause. Do you have any ideas/suggestions on how to debug this? Thanks!
net7 Link encap:Ethernet HWaddr 00:1E:67:B5:7F:76 inet addr:192.168.221.203 Bcast:0.0.0.0 Mask:255.255.255.0 inet6 addr: fe80::21e:67ff:feb5:7f76/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:263 errors:0 dropped:151 overruns:0 frame:0 TX packets:83944096 errors:0 dropped:57692 overruns:0 carrier:0 collisions:0 txqueuelen:10000 RX bytes:27531 (26.8 KiB) TX bytes:113995992408 (106.1 GiB)
# teamnl net7 ports 5: eth6: up 1000Mbit FD
# ethtool -S eth6 NIC statistics: rx_packets: 113 tx_packets: 86938454 rx_bytes: 22848 tx_bytes: 118410084622 rx_broadcast: 68 tx_broadcast: 6 rx_multicast: 45 tx_multicast: 86938448 multicast: 45 collisions: 0 rx_crc_errors: 0 rx_no_buffer_count: 0 rx_missed_errors: 0 tx_aborted_errors: 0 tx_carrier_errors: 0 tx_window_errors: 0 tx_abort_late_coll: 0 tx_deferred_ok: 0 tx_single_coll_ok: 0 tx_multi_coll_ok: 0 tx_timeout_count: 0 rx_long_length_errors: 0 rx_short_length_errors: 0 rx_align_errors: 0 tx_tcp_seg_good: 0 tx_tcp_seg_failed: 0 rx_flow_control_xon: 68 rx_flow_control_xoff: 96 tx_flow_control_xon: 0 tx_flow_control_xoff: 0 rx_long_byte_count: 22848 tx_dma_out_of_sync: 0 lro_aggregated: 0 lro_flushed: 0 tx_smbus: 0 rx_smbus: 0 dropped_smbus: 0 os2bmc_rx_by_bmc: 0 os2bmc_tx_by_bmc: 0 os2bmc_tx_by_host: 0 os2bmc_rx_by_host: 0 tx_hwtstamp_timeouts: 0 rx_hwtstamp_cleared: 0 rx_errors: 0 tx_errors: 0 tx_dropped: 0 rx_length_errors: 0 rx_over_errors: 0 rx_frame_errors: 0 rx_fifo_errors: 0 tx_fifo_errors: 0 tx_heartbeat_errors: 0 tx_queue_0_packets: 86938440 tx_queue_0_bytes: 118062311560 tx_queue_0_restart: 0 rx_queue_0_packets: 277 rx_queue_0_bytes: 32236 rx_queue_0_drops: 0 rx_queue_0_csum_err: 0 rx_queue_0_alloc_failed: 0
# teamdctl net7 s d { "ports": { "eth6": { "ifinfo": { "dev_addr": "00:1e:67:b5:7f:76", "dev_addr_len": 6, "ifindex": 5, "ifname": "eth6" }, "link": { "duplex": "full", "speed": 1000, "up": true }, "link_watches": { "list": { "link_watch_0": { "delay_down": 0, "delay_up": 0, "down_count": 1, "name": "ethtool", "up": true } }, "up": true } } }, "runner": { "active_port": "eth6" }, "setup": { "daemonized": true, "dbus_enabled": false, "debug_level": 0, "kernel_team_mode_name": "activebackup", "pid": 5533, "pid_file": "/var/run/teamd/net7.pid", "runner_name": "activebackup", "zmq_enabled": false }, "team_device": { "ifinfo": { "dev_addr": "00:1e:67:b5:7f:76", "dev_addr_len": 6, "ifindex": 13, "ifname": "net7" } } } _______________________________________________ libteam mailing list -- libteam@lists.fedorahosted.org To unsubscribe send an email to libteam-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/libteam@lists.fedorahosted.org
Thank you, Jamie!
I have already maxed out ring buffer on my NIC. and unfortunately, I'm unable to use SystemTap on my system, so I stick to dropwatch.
After filtering dopwatch output for unique elements, I found 7 functions generating drops, however, there are no bugs reported in these exact functions.
__udp4_lib_rcv+6b0 net/ipv4/udp.c +2015
ip_forward+98 net/ipv4/ip_forward.c + 165
kfree_skb_list+13 net/core/skbuff.c +669
sk_stream_kill_queues+4a include/linux/skbuff.h +2580
tcp_v4_do_rcv+154 net/ipv4/tcp_ipv4.c +1501
tcp_v4_rcv+1de net/ipv4/tcp_ipv4.c +1771
unix_stream_connect+3b4 net/unix/af_inet.c +1373
I've also checked the /proc/interrupts, it turns out that most queues are assigned to cpu0, and it gets much better after deploying irqbalance as you suggested. Interestingly enough, this same configuration somehow used to work ok on 3.10.
So, I think the issue is not related to libteam and more interrupt distribution / interrupt handling in kernel problem. Nonetheless, thank you very much for your answer, I find your videos very helpful as well!
Yevhen
пн, 23 сент. 2019 г. в 05:28, Jamie Bainbridge jamie.bainbridge@gmail.com:
This seems more like a difference in the operation of the underlying slave than a difference in teaming.
You could try the slave on its own, or try the slave with bonding, to rule out teaming as a cause.
You can find calls to kfree_skb easier with the dropwatch utility, or with SystemTap's example script dropwatch.stp. I find it easier to add a timestamp to the stap script's 5-second loop so you know when the packets were freed. You can then correspond a difference in dropwatch output with timestamped collection of "ip -s link" or similar.
You could possibly avoid this by increasing the device hardware tx buffer to max, eg: ethtool --set-ring ethX tx 4096
For possible underlying reasons:
Make sure you have irqbalance running or are balancing IRQ channels across real cores (not HyperThreads) all in the same NUMA node, and in the same NUMA node as the NIC if the NIC has NUMA locality. See my notes and video here for a little more in-depth explanation: http://jbainbri.github.io/
Make sure you aren't doing too much expensive traffic handling in the transmit path, like passing traffic through tens of thousands of firewall or tc rules.
Jamie
On Sun, 22 Sep 2019 at 12:43, Yevhen Kondrashyn e.kondrashyn@gmail.com wrote:
Hi All, After upgrading from 3.10.104 to kernel 4.14 I see constant drops (around 10 packets/sec) on team interface in active-backup mode, increasing txqueuelen on a physical port helps, however, drops still occur from time to time. I've added debug printk in team driver where dev_kfree_skb_any is called, however, I do not see messages in dmesg until I reset physical interface by ethtool, therefore packets are being dropped somewhere else, however, for now, I have no clue how to find the exact cause. Do you have any ideas/suggestions on how to debug this? Thanks!
net7 Link encap:Ethernet HWaddr 00:1E:67:B5:7F:76 inet addr:192.168.221.203 Bcast:0.0.0.0 Mask:255.255.255.0 inet6 addr: fe80::21e:67ff:feb5:7f76/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:263 errors:0 dropped:151 overruns:0 frame:0 TX packets:83944096 errors:0 dropped:57692 overruns:0 carrier:0 collisions:0 txqueuelen:10000 RX bytes:27531 (26.8 KiB) TX bytes:113995992408 (106.1 GiB)
# teamnl net7 ports 5: eth6: up 1000Mbit FD
# ethtool -S eth6 NIC statistics: rx_packets: 113 tx_packets: 86938454 rx_bytes: 22848 tx_bytes: 118410084622 rx_broadcast: 68 tx_broadcast: 6 rx_multicast: 45 tx_multicast: 86938448 multicast: 45 collisions: 0 rx_crc_errors: 0 rx_no_buffer_count: 0 rx_missed_errors: 0 tx_aborted_errors: 0 tx_carrier_errors: 0 tx_window_errors: 0 tx_abort_late_coll: 0 tx_deferred_ok: 0 tx_single_coll_ok: 0 tx_multi_coll_ok: 0 tx_timeout_count: 0 rx_long_length_errors: 0 rx_short_length_errors: 0 rx_align_errors: 0 tx_tcp_seg_good: 0 tx_tcp_seg_failed: 0 rx_flow_control_xon: 68 rx_flow_control_xoff: 96 tx_flow_control_xon: 0 tx_flow_control_xoff: 0 rx_long_byte_count: 22848 tx_dma_out_of_sync: 0 lro_aggregated: 0 lro_flushed: 0 tx_smbus: 0 rx_smbus: 0 dropped_smbus: 0 os2bmc_rx_by_bmc: 0 os2bmc_tx_by_bmc: 0 os2bmc_tx_by_host: 0 os2bmc_rx_by_host: 0 tx_hwtstamp_timeouts: 0 rx_hwtstamp_cleared: 0 rx_errors: 0 tx_errors: 0 tx_dropped: 0 rx_length_errors: 0 rx_over_errors: 0 rx_frame_errors: 0 rx_fifo_errors: 0 tx_fifo_errors: 0 tx_heartbeat_errors: 0 tx_queue_0_packets: 86938440 tx_queue_0_bytes: 118062311560 tx_queue_0_restart: 0 rx_queue_0_packets: 277 rx_queue_0_bytes: 32236 rx_queue_0_drops: 0 rx_queue_0_csum_err: 0 rx_queue_0_alloc_failed: 0
# teamdctl net7 s d { "ports": { "eth6": { "ifinfo": { "dev_addr": "00:1e:67:b5:7f:76", "dev_addr_len": 6, "ifindex": 5, "ifname": "eth6" }, "link": { "duplex": "full", "speed": 1000, "up": true }, "link_watches": { "list": { "link_watch_0": { "delay_down": 0, "delay_up": 0, "down_count": 1, "name": "ethtool", "up": true } }, "up": true } } }, "runner": { "active_port": "eth6" }, "setup": { "daemonized": true, "dbus_enabled": false, "debug_level": 0, "kernel_team_mode_name": "activebackup", "pid": 5533, "pid_file": "/var/run/teamd/net7.pid", "runner_name": "activebackup", "zmq_enabled": false }, "team_device": { "ifinfo": { "dev_addr": "00:1e:67:b5:7f:76", "dev_addr_len": 6, "ifindex": 13, "ifname": "net7" } } } _______________________________________________ libteam mailing list -- libteam@lists.fedorahosted.org To unsubscribe send an email to libteam-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/libteam@lists.fedorahosted.org
libteam@lists.fedorahosted.org
-
Jamie Bainbridge -
Yevhen Kondrashyn