This seems more like a difference in the operation of the underlying
slave than a difference in teaming.
You could try the slave on its own, or try the slave with bonding, to
rule out teaming as a cause.
You can find calls to kfree_skb easier with the dropwatch utility, or
with SystemTap's example script dropwatch.stp. I find it easier to add
a timestamp to the stap script's 5-second loop so you know when the
packets were freed. You can then correspond a difference in dropwatch
output with timestamped collection of "ip -s link" or similar.
You could possibly avoid this by increasing the device hardware tx
buffer to max, eg: ethtool --set-ring ethX tx 4096
For possible underlying reasons:
Make sure you have irqbalance running or are balancing IRQ channels
across real cores (not HyperThreads) all in the same NUMA node, and in
the same NUMA node as the NIC if the NIC has NUMA locality. See my
notes and video here for a little more in-depth explanation:
http://jbainbri.github.io/
Make sure you aren't doing too much expensive traffic handling in the
transmit path, like passing traffic through tens of thousands of
firewall or tc rules.
Jamie
On Sun, 22 Sep 2019 at 12:43, Yevhen Kondrashyn <e.kondrashyn(a)gmail.com> wrote:
>
> Hi All,
> After upgrading from 3.10.104 to kernel 4.14 I see constant drops
> (around 10 packets/sec) on team interface in active-backup mode,
> increasing txqueuelen on a physical port helps, however, drops still
> occur from time to time.
> I've added debug printk in team driver where dev_kfree_skb_any is
> called, however, I do not see messages in dmesg until I reset physical
> interface by ethtool, therefore packets are being dropped somewhere
> else, however, for now, I have no clue how to find the exact cause.
> Do you have any ideas/suggestions on how to debug this?
> Thanks!
>
> net7 Link encap:Ethernet HWaddr 00:1E:67:B5:7F:76
> inet addr:192.168.221.203 Bcast:0.0.0.0 Mask:255.255.255.0
> inet6 addr: fe80::21e:67ff:feb5:7f76/64 Scope:Link
> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
> RX packets:263 errors:0 dropped:151 overruns:0 frame:0
> TX packets:83944096 errors:0 dropped:57692 overruns:0 carrier:0
> collisions:0 txqueuelen:10000
> RX bytes:27531 (26.8 KiB) TX bytes:113995992408 (106.1 GiB)
>
> # teamnl net7 ports
> 5: eth6: up 1000Mbit FD
>
> # ethtool -S eth6
> NIC statistics:
> rx_packets: 113
> tx_packets: 86938454
> rx_bytes: 22848
> tx_bytes: 118410084622
> rx_broadcast: 68
> tx_broadcast: 6
> rx_multicast: 45
> tx_multicast: 86938448
> multicast: 45
> collisions: 0
> rx_crc_errors: 0
> rx_no_buffer_count: 0
> rx_missed_errors: 0
> tx_aborted_errors: 0
> tx_carrier_errors: 0
> tx_window_errors: 0
> tx_abort_late_coll: 0
> tx_deferred_ok: 0
> tx_single_coll_ok: 0
> tx_multi_coll_ok: 0
> tx_timeout_count: 0
> rx_long_length_errors: 0
> rx_short_length_errors: 0
> rx_align_errors: 0
> tx_tcp_seg_good: 0
> tx_tcp_seg_failed: 0
> rx_flow_control_xon: 68
> rx_flow_control_xoff: 96
> tx_flow_control_xon: 0
> tx_flow_control_xoff: 0
> rx_long_byte_count: 22848
> tx_dma_out_of_sync: 0
> lro_aggregated: 0
> lro_flushed: 0
> tx_smbus: 0
> rx_smbus: 0
> dropped_smbus: 0
> os2bmc_rx_by_bmc: 0
> os2bmc_tx_by_bmc: 0
> os2bmc_tx_by_host: 0
> os2bmc_rx_by_host: 0
> tx_hwtstamp_timeouts: 0
> rx_hwtstamp_cleared: 0
> rx_errors: 0
> tx_errors: 0
> tx_dropped: 0
> rx_length_errors: 0
> rx_over_errors: 0
> rx_frame_errors: 0
> rx_fifo_errors: 0
> tx_fifo_errors: 0
> tx_heartbeat_errors: 0
> tx_queue_0_packets: 86938440
> tx_queue_0_bytes: 118062311560
> tx_queue_0_restart: 0
> rx_queue_0_packets: 277
> rx_queue_0_bytes: 32236
> rx_queue_0_drops: 0
> rx_queue_0_csum_err: 0
> rx_queue_0_alloc_failed: 0
>
> # teamdctl net7 s d
> {
> "ports": {
> "eth6": {
> "ifinfo": {
> "dev_addr": "00:1e:67:b5:7f:76",
> "dev_addr_len": 6,
> "ifindex": 5,
> "ifname": "eth6"
> },
> "link": {
> "duplex": "full",
> "speed": 1000,
> "up": true
> },
> "link_watches": {
> "list": {
> "link_watch_0": {
> "delay_down": 0,
> "delay_up": 0,
> "down_count": 1,
> "name": "ethtool",
> "up": true
> }
> },
> "up": true
> }
> }
> },
> "runner": {
> "active_port": "eth6"
> },
> "setup": {
> "daemonized": true,
> "dbus_enabled": false,
> "debug_level": 0,
> "kernel_team_mode_name": "activebackup",
> "pid": 5533,
> "pid_file": "/var/run/teamd/net7.pid",
> "runner_name": "activebackup",
> "zmq_enabled": false
> },
> "team_device": {
> "ifinfo": {
> "dev_addr": "00:1e:67:b5:7f:76",
> "dev_addr_len": 6,
> "ifindex": 13,
> "ifname": "net7"
> }
> }
> }
> _______________________________________________
> libteam mailing list -- libteam(a)lists.fedorahosted.org
> To unsubscribe send an email to libteam-leave(a)lists.fedorahosted.org
> Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines:
https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
https://lists.fedorahosted.org/archives/list/libteam@lists.fedorahosted.org