Thank you, Jamie!
I have already maxed out ring buffer on my NIC. and unfortunately, I'm
unable to use SystemTap on my system, so I stick to dropwatch.
After filtering dopwatch output for unique elements, I found 7
functions generating drops, however, there are no bugs reported in
these exact functions.
__udp4_lib_rcv+6b0
net/ipv4/udp.c +2015
ip_forward+98
net/ipv4/ip_forward.c + 165
kfree_skb_list+13
net/core/skbuff.c +669
sk_stream_kill_queues+4a
include/linux/skbuff.h +2580
tcp_v4_do_rcv+154
net/ipv4/tcp_ipv4.c +1501
tcp_v4_rcv+1de
net/ipv4/tcp_ipv4.c +1771
unix_stream_connect+3b4
net/unix/af_inet.c +1373
I've also checked the /proc/interrupts, it turns out that most queues
are assigned to cpu0, and it gets much better after deploying
irqbalance as you suggested. Interestingly enough, this same
configuration somehow used to work ok on 3.10.
So, I think the issue is not related to libteam and more interrupt
distribution / interrupt handling in kernel problem.
Nonetheless, thank you very much for your answer, I find your videos
very helpful as well!
Yevhen
пн, 23 сент. 2019 г. в 05:28, Jamie Bainbridge <jamie.bainbridge(a)gmail.com>:
>
> This seems more like a difference in the operation of the underlying
> slave than a difference in teaming.
>
> You could try the slave on its own, or try the slave with bonding, to
> rule out teaming as a cause.
>
> You can find calls to kfree_skb easier with the dropwatch utility, or
> with SystemTap's example script dropwatch.stp. I find it easier to add
> a timestamp to the stap script's 5-second loop so you know when the
> packets were freed. You can then correspond a difference in dropwatch
> output with timestamped collection of "ip -s link" or similar.
>
> You could possibly avoid this by increasing the device hardware tx
> buffer to max, eg: ethtool --set-ring ethX tx 4096
>
> For possible underlying reasons:
>
> Make sure you have irqbalance running or are balancing IRQ channels
> across real cores (not HyperThreads) all in the same NUMA node, and in
> the same NUMA node as the NIC if the NIC has NUMA locality. See my
> notes and video here for a little more in-depth explanation:
>
http://jbainbri.github.io/
>
> Make sure you aren't doing too much expensive traffic handling in the
> transmit path, like passing traffic through tens of thousands of
> firewall or tc rules.
>
> Jamie
>
> On Sun, 22 Sep 2019 at 12:43, Yevhen Kondrashyn <e.kondrashyn(a)gmail.com>
wrote:
> >
> > Hi All,
> > After upgrading from 3.10.104 to kernel 4.14 I see constant drops
> > (around 10 packets/sec) on team interface in active-backup mode,
> > increasing txqueuelen on a physical port helps, however, drops still
> > occur from time to time.
> > I've added debug printk in team driver where dev_kfree_skb_any is
> > called, however, I do not see messages in dmesg until I reset physical
> > interface by ethtool, therefore packets are being dropped somewhere
> > else, however, for now, I have no clue how to find the exact cause.
> > Do you have any ideas/suggestions on how to debug this?
> > Thanks!
> >
> > net7 Link encap:Ethernet HWaddr 00:1E:67:B5:7F:76
> > inet addr:192.168.221.203 Bcast:0.0.0.0 Mask:255.255.255.0
> > inet6 addr: fe80::21e:67ff:feb5:7f76/64 Scope:Link
> > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
> > RX packets:263 errors:0 dropped:151 overruns:0 frame:0
> > TX packets:83944096 errors:0 dropped:57692 overruns:0 carrier:0
> > collisions:0 txqueuelen:10000
> > RX bytes:27531 (26.8 KiB) TX bytes:113995992408 (106.1 GiB)
> >
> > # teamnl net7 ports
> > 5: eth6: up 1000Mbit FD
> >
> > # ethtool -S eth6
> > NIC statistics:
> > rx_packets: 113
> > tx_packets: 86938454
> > rx_bytes: 22848
> > tx_bytes: 118410084622
> > rx_broadcast: 68
> > tx_broadcast: 6
> > rx_multicast: 45
> > tx_multicast: 86938448
> > multicast: 45
> > collisions: 0
> > rx_crc_errors: 0
> > rx_no_buffer_count: 0
> > rx_missed_errors: 0
> > tx_aborted_errors: 0
> > tx_carrier_errors: 0
> > tx_window_errors: 0
> > tx_abort_late_coll: 0
> > tx_deferred_ok: 0
> > tx_single_coll_ok: 0
> > tx_multi_coll_ok: 0
> > tx_timeout_count: 0
> > rx_long_length_errors: 0
> > rx_short_length_errors: 0
> > rx_align_errors: 0
> > tx_tcp_seg_good: 0
> > tx_tcp_seg_failed: 0
> > rx_flow_control_xon: 68
> > rx_flow_control_xoff: 96
> > tx_flow_control_xon: 0
> > tx_flow_control_xoff: 0
> > rx_long_byte_count: 22848
> > tx_dma_out_of_sync: 0
> > lro_aggregated: 0
> > lro_flushed: 0
> > tx_smbus: 0
> > rx_smbus: 0
> > dropped_smbus: 0
> > os2bmc_rx_by_bmc: 0
> > os2bmc_tx_by_bmc: 0
> > os2bmc_tx_by_host: 0
> > os2bmc_rx_by_host: 0
> > tx_hwtstamp_timeouts: 0
> > rx_hwtstamp_cleared: 0
> > rx_errors: 0
> > tx_errors: 0
> > tx_dropped: 0
> > rx_length_errors: 0
> > rx_over_errors: 0
> > rx_frame_errors: 0
> > rx_fifo_errors: 0
> > tx_fifo_errors: 0
> > tx_heartbeat_errors: 0
> > tx_queue_0_packets: 86938440
> > tx_queue_0_bytes: 118062311560
> > tx_queue_0_restart: 0
> > rx_queue_0_packets: 277
> > rx_queue_0_bytes: 32236
> > rx_queue_0_drops: 0
> > rx_queue_0_csum_err: 0
> > rx_queue_0_alloc_failed: 0
> >
> > # teamdctl net7 s d
> > {
> > "ports": {
> > "eth6": {
> > "ifinfo": {
> > "dev_addr": "00:1e:67:b5:7f:76",
> > "dev_addr_len": 6,
> > "ifindex": 5,
> > "ifname": "eth6"
> > },
> > "link": {
> > "duplex": "full",
> > "speed": 1000,
> > "up": true
> > },
> > "link_watches": {
> > "list": {
> > "link_watch_0": {
> > "delay_down": 0,
> > "delay_up": 0,
> > "down_count": 1,
> > "name": "ethtool",
> > "up": true
> > }
> > },
> > "up": true
> > }
> > }
> > },
> > "runner": {
> > "active_port": "eth6"
> > },
> > "setup": {
> > "daemonized": true,
> > "dbus_enabled": false,
> > "debug_level": 0,
> > "kernel_team_mode_name": "activebackup",
> > "pid": 5533,
> > "pid_file": "/var/run/teamd/net7.pid",
> > "runner_name": "activebackup",
> > "zmq_enabled": false
> > },
> > "team_device": {
> > "ifinfo": {
> > "dev_addr": "00:1e:67:b5:7f:76",
> > "dev_addr_len": 6,
> > "ifindex": 13,
> > "ifname": "net7"
> > }
> > }
> > }
> > _______________________________________________
> > libteam mailing list -- libteam(a)lists.fedorahosted.org
> > To unsubscribe send an email to libteam-leave(a)lists.fedorahosted.org
> > Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > List Guidelines:
https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives:
https://lists.fedorahosted.org/archives/list/libteam@lists.fedorahosted.org