This seems more like a difference in the operation of the underlying slave than a difference in teaming.
You could try the slave on its own, or try the slave with bonding, to rule out teaming as a cause.
You can find calls to kfree_skb easier with the dropwatch utility, or with SystemTap's example script dropwatch.stp. I find it easier to add a timestamp to the stap script's 5-second loop so you know when the packets were freed. You can then correspond a difference in dropwatch output with timestamped collection of "ip -s link" or similar.
You could possibly avoid this by increasing the device hardware tx buffer to max, eg: ethtool --set-ring ethX tx 4096
For possible underlying reasons:
Make sure you have irqbalance running or are balancing IRQ channels across real cores (not HyperThreads) all in the same NUMA node, and in the same NUMA node as the NIC if the NIC has NUMA locality. See my notes and video here for a little more in-depth explanation: http://jbainbri.github.io/
Make sure you aren't doing too much expensive traffic handling in the transmit path, like passing traffic through tens of thousands of firewall or tc rules.
Jamie
On Sun, 22 Sep 2019 at 12:43, Yevhen Kondrashyn e.kondrashyn@gmail.com wrote:
Hi All, After upgrading from 3.10.104 to kernel 4.14 I see constant drops (around 10 packets/sec) on team interface in active-backup mode, increasing txqueuelen on a physical port helps, however, drops still occur from time to time. I've added debug printk in team driver where dev_kfree_skb_any is called, however, I do not see messages in dmesg until I reset physical interface by ethtool, therefore packets are being dropped somewhere else, however, for now, I have no clue how to find the exact cause. Do you have any ideas/suggestions on how to debug this? Thanks!
net7 Link encap:Ethernet HWaddr 00:1E:67:B5:7F:76 inet addr:192.168.221.203 Bcast:0.0.0.0 Mask:255.255.255.0 inet6 addr: fe80::21e:67ff:feb5:7f76/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:263 errors:0 dropped:151 overruns:0 frame:0 TX packets:83944096 errors:0 dropped:57692 overruns:0 carrier:0 collisions:0 txqueuelen:10000 RX bytes:27531 (26.8 KiB) TX bytes:113995992408 (106.1 GiB)
# teamnl net7 ports 5: eth6: up 1000Mbit FD
# ethtool -S eth6 NIC statistics: rx_packets: 113 tx_packets: 86938454 rx_bytes: 22848 tx_bytes: 118410084622 rx_broadcast: 68 tx_broadcast: 6 rx_multicast: 45 tx_multicast: 86938448 multicast: 45 collisions: 0 rx_crc_errors: 0 rx_no_buffer_count: 0 rx_missed_errors: 0 tx_aborted_errors: 0 tx_carrier_errors: 0 tx_window_errors: 0 tx_abort_late_coll: 0 tx_deferred_ok: 0 tx_single_coll_ok: 0 tx_multi_coll_ok: 0 tx_timeout_count: 0 rx_long_length_errors: 0 rx_short_length_errors: 0 rx_align_errors: 0 tx_tcp_seg_good: 0 tx_tcp_seg_failed: 0 rx_flow_control_xon: 68 rx_flow_control_xoff: 96 tx_flow_control_xon: 0 tx_flow_control_xoff: 0 rx_long_byte_count: 22848 tx_dma_out_of_sync: 0 lro_aggregated: 0 lro_flushed: 0 tx_smbus: 0 rx_smbus: 0 dropped_smbus: 0 os2bmc_rx_by_bmc: 0 os2bmc_tx_by_bmc: 0 os2bmc_tx_by_host: 0 os2bmc_rx_by_host: 0 tx_hwtstamp_timeouts: 0 rx_hwtstamp_cleared: 0 rx_errors: 0 tx_errors: 0 tx_dropped: 0 rx_length_errors: 0 rx_over_errors: 0 rx_frame_errors: 0 rx_fifo_errors: 0 tx_fifo_errors: 0 tx_heartbeat_errors: 0 tx_queue_0_packets: 86938440 tx_queue_0_bytes: 118062311560 tx_queue_0_restart: 0 rx_queue_0_packets: 277 rx_queue_0_bytes: 32236 rx_queue_0_drops: 0 rx_queue_0_csum_err: 0 rx_queue_0_alloc_failed: 0
# teamdctl net7 s d { "ports": { "eth6": { "ifinfo": { "dev_addr": "00:1e:67:b5:7f:76", "dev_addr_len": 6, "ifindex": 5, "ifname": "eth6" }, "link": { "duplex": "full", "speed": 1000, "up": true }, "link_watches": { "list": { "link_watch_0": { "delay_down": 0, "delay_up": 0, "down_count": 1, "name": "ethtool", "up": true } }, "up": true } } }, "runner": { "active_port": "eth6" }, "setup": { "daemonized": true, "dbus_enabled": false, "debug_level": 0, "kernel_team_mode_name": "activebackup", "pid": 5533, "pid_file": "/var/run/teamd/net7.pid", "runner_name": "activebackup", "zmq_enabled": false }, "team_device": { "ifinfo": { "dev_addr": "00:1e:67:b5:7f:76", "dev_addr_len": 6, "ifindex": 13, "ifname": "net7" } } } _______________________________________________ libteam mailing list -- libteam@lists.fedorahosted.org To unsubscribe send an email to libteam-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/libteam@lists.fedorahosted.org