Hi Jiri,

Unfortunately, this is the output of the "teamnl team0 options" command. [root@rtr ~]# teamnl team0 options lb_port_stats (port:enp5s0f1) \00\00\00\00\00\00\00\00 queue_id (port:enp5s0f1) 0 priority (port:enp5s0f1) 100 user_linkup_enabled (port:enp5s0f1) false user_linkup (port:enp5s0f1) true enabled (port:enp5s0f1) true lb_port_stats (port:enp5s0f0) \00\00\00\00\00\00\00\00 queue_id (port:enp5s0f0) 0 priority (port:enp5s0f0) 100 user_linkup_enabled (port:enp5s0f0) false user_linkup (port:enp5s0f0) true enabled (port:enp5s0f0) true lb_port_stats (port:enp4s0f1) \00\00\00\00\00\00\00\00 queue_id (port:enp4s0f1) 0 priority (port:enp4s0f1) 100 user_linkup_enabled (port:enp4s0f1) false user_linkup (port:enp4s0f1) true enabled (port:enp4s0f1) true lb_port_stats (port:enp4s0f0) \00\00\00\00\00\00\00\00 queue_id (port:enp4s0f0) 0 priority (port:enp4s0f0) 100 user_linkup_enabled (port:enp4s0f0) false user_linkup (port:enp4s0f0) true enabled (port:enp4s0f0) true lb_stats_refresh_interval 0 lb_hash_stats (arridx:255) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:254) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:253) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:252) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:251) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:250) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:249) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:248) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:247) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:246) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:245) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:244) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:243) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:242) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:241) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:240) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:239) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:238) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:237) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:236) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:235) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:234) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:233) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:232) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:231) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:230) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:229) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:228) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:227) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:226) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:225) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:224) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:223) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:222) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:221) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:220) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:219) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:218) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:217) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:216) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:215) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:214) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:213) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:212) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:211) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:210) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:209) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:208) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:207) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:206) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:205) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:204) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:203) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:202) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:201) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:200) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:199) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:198) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:197) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:196) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:195) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:194) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:193) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:192) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:191) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:190) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:189) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:188) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:187) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:186) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:185) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:184) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:183) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:182) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:181) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:180) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:179) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:178) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:177) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:176) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:175) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:174) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:173) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:172) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:171) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:170) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:169) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:168) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:167) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:166) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:165) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:164) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:163) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:162) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:161) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:160) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:159) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:158) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:157) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:156) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:155) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:154) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:153) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:152) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:151) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:150) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:149) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:148) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:147) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:146) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:145) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:144) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:143) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:142) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:141) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:140) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:139) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:138) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:137) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:136) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:135) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:134) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:133) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:132) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:131) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:130) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:129) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:128) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:127) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:126) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:125) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:124) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:123) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:122) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:121) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:120) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:119) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:118) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:117) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:116) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:115) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:114) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:113) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:112) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:111) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:110) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:109) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:108) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:107) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:106) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:105) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:104) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:103) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:102) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:101) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:100) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:99) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:98) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:97) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:96) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:95) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:94) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:93) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:92) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:91) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:90) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:89) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:88) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:87) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:86) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:85) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:84) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:83) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:82) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:81) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:80) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:79) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:78) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:77) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:76) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:75) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:74) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:73) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:72) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:71) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:70) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:69) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:68) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:67) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:66) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:65) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:64) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:63) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:62) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:61) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:60) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:59) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:58) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:57) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:56) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:55) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:54) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:53) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:52) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:51) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:50) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:49) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:48) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:47) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:46) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:45) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:44) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:43) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:42) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:41) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:40) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:39) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:38) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:37) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:36) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:35) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:34) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:33) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:32) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:31) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:30) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:29) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:28) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:27) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:26) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:25) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:24) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:23) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:22) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:21) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:20) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:19) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:18) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:17) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:16) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:15) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:14) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:13) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:12) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:11) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:10) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:9) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:8) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:7) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:6) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:5) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:4) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:3) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:2) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:1) \00\00\00\00\00\00\00\00 lb_hash_stats (arridx:0) \00\00\00\00\00\00\00\00 lb_tx_hash_to_port_mapping (arridx:255) 0 lb_tx_hash_to_port_mapping (arridx:254) 0 lb_tx_hash_to_port_mapping (arridx:253) 0 lb_tx_hash_to_port_mapping (arridx:252) 0 lb_tx_hash_to_port_mapping (arridx:251) 0 lb_tx_hash_to_port_mapping (arridx:250) 0 lb_tx_hash_to_port_mapping (arridx:249) 0 lb_tx_hash_to_port_mapping (arridx:248) 0 lb_tx_hash_to_port_mapping (arridx:247) 0 lb_tx_hash_to_port_mapping (arridx:246) 0 lb_tx_hash_to_port_mapping (arridx:245) 0 lb_tx_hash_to_port_mapping (arridx:244) 0 lb_tx_hash_to_port_mapping (arridx:243) 0 lb_tx_hash_to_port_mapping (arridx:242) 0 lb_tx_hash_to_port_mapping (arridx:241) 0 lb_tx_hash_to_port_mapping (arridx:240) 0 lb_tx_hash_to_port_mapping (arridx:239) 0 lb_tx_hash_to_port_mapping (arridx:238) 0 lb_tx_hash_to_port_mapping (arridx:237) 0 lb_tx_hash_to_port_mapping (arridx:236) 0 lb_tx_hash_to_port_mapping (arridx:235) 0 lb_tx_hash_to_port_mapping (arridx:234) 0 lb_tx_hash_to_port_mapping (arridx:233) 0 lb_tx_hash_to_port_mapping (arridx:232) 0 lb_tx_hash_to_port_mapping (arridx:231) 0 lb_tx_hash_to_port_mapping (arridx:230) 0 lb_tx_hash_to_port_mapping (arridx:229) 0 lb_tx_hash_to_port_mapping (arridx:228) 0 lb_tx_hash_to_port_mapping (arridx:227) 0 lb_tx_hash_to_port_mapping (arridx:226) 0 lb_tx_hash_to_port_mapping (arridx:225) 0 lb_tx_hash_to_port_mapping (arridx:224) 0 lb_tx_hash_to_port_mapping (arridx:223) 0 lb_tx_hash_to_port_mapping (arridx:222) 0 lb_tx_hash_to_port_mapping (arridx:221) 0 lb_tx_hash_to_port_mapping (arridx:220) 0 lb_tx_hash_to_port_mapping (arridx:219) 0 lb_tx_hash_to_port_mapping (arridx:218) 0 lb_tx_hash_to_port_mapping (arridx:217) 0 lb_tx_hash_to_port_mapping (arridx:216) 0 lb_tx_hash_to_port_mapping (arridx:215) 0 lb_tx_hash_to_port_mapping (arridx:214) 0 lb_tx_hash_to_port_mapping (arridx:213) 0 lb_tx_hash_to_port_mapping (arridx:212) 0 lb_tx_hash_to_port_mapping (arridx:211) 0 lb_tx_hash_to_port_mapping (arridx:210) 0 lb_tx_hash_to_port_mapping (arridx:209) 0 lb_tx_hash_to_port_mapping (arridx:208) 0 lb_tx_hash_to_port_mapping (arridx:207) 0 lb_tx_hash_to_port_mapping (arridx:206) 0 lb_tx_hash_to_port_mapping (arridx:205) 0 lb_tx_hash_to_port_mapping (arridx:204) 0 lb_tx_hash_to_port_mapping (arridx:203) 0 lb_tx_hash_to_port_mapping (arridx:202) 0 lb_tx_hash_to_port_mapping (arridx:201) 0 lb_tx_hash_to_port_mapping (arridx:200) 0 lb_tx_hash_to_port_mapping (arridx:199) 0 lb_tx_hash_to_port_mapping (arridx:198) 0 lb_tx_hash_to_port_mapping (arridx:197) 0 lb_tx_hash_to_port_mapping (arridx:196) 0 lb_tx_hash_to_port_mapping (arridx:195) 0 lb_tx_hash_to_port_mapping (arridx:194) 0 lb_tx_hash_to_port_mapping (arridx:193) 0 lb_tx_hash_to_port_mapping (arridx:192) 0 lb_tx_hash_to_port_mapping (arridx:191) 0 lb_tx_hash_to_port_mapping (arridx:190) 0 lb_tx_hash_to_port_mapping (arridx:189) 0 lb_tx_hash_to_port_mapping (arridx:188) 0 lb_tx_hash_to_port_mapping (arridx:187) 0 lb_tx_hash_to_port_mapping (arridx:186) 0 lb_tx_hash_to_port_mapping (arridx:185) 0 lb_tx_hash_to_port_mapping (arridx:184) 0 lb_tx_hash_to_port_mapping (arridx:183) 0 lb_tx_hash_to_port_mapping (arridx:182) 0 lb_tx_hash_to_port_mapping (arridx:181) 0 lb_tx_hash_to_port_mapping (arridx:180) 0 lb_tx_hash_to_port_mapping (arridx:179) 0 lb_tx_hash_to_port_mapping (arridx:178) 0 lb_tx_hash_to_port_mapping (arridx:177) 0 lb_tx_hash_to_port_mapping (arridx:176) 0 lb_tx_hash_to_port_mapping (arridx:175) 0 lb_tx_hash_to_port_mapping (arridx:174) 0 lb_tx_hash_to_port_mapping (arridx:173) 0 lb_tx_hash_to_port_mapping (arridx:172) 0 lb_tx_hash_to_port_mapping (arridx:171) 0 lb_tx_hash_to_port_mapping (arridx:170) 0 lb_tx_hash_to_port_mapping (arridx:169) 0 lb_tx_hash_to_port_mapping (arridx:168) 0 lb_tx_hash_to_port_mapping (arridx:167) 0 lb_tx_hash_to_port_mapping (arridx:166) 0 lb_tx_hash_to_port_mapping (arridx:165) 0 lb_tx_hash_to_port_mapping (arridx:164) 0 lb_tx_hash_to_port_mapping (arridx:163) 0 lb_tx_hash_to_port_mapping (arridx:162) 0 lb_tx_hash_to_port_mapping (arridx:161) 0 lb_tx_hash_to_port_mapping (arridx:160) 0 lb_tx_hash_to_port_mapping (arridx:159) 0 lb_tx_hash_to_port_mapping (arridx:158) 0 lb_tx_hash_to_port_mapping (arridx:157) 0 lb_tx_hash_to_port_mapping (arridx:156) 0 lb_tx_hash_to_port_mapping (arridx:155) 0 lb_tx_hash_to_port_mapping (arridx:154) 0 lb_tx_hash_to_port_mapping (arridx:153) 0 lb_tx_hash_to_port_mapping (arridx:152) 0 lb_tx_hash_to_port_mapping (arridx:151) 0 lb_tx_hash_to_port_mapping (arridx:150) 0 lb_tx_hash_to_port_mapping (arridx:149) 0 lb_tx_hash_to_port_mapping (arridx:148) 0 lb_tx_hash_to_port_mapping (arridx:147) 0 lb_tx_hash_to_port_mapping (arridx:146) 0 lb_tx_hash_to_port_mapping (arridx:145) 0 lb_tx_hash_to_port_mapping (arridx:144) 0 lb_tx_hash_to_port_mapping (arridx:143) 0 lb_tx_hash_to_port_mapping (arridx:142) 0 lb_tx_hash_to_port_mapping (arridx:141) 0 lb_tx_hash_to_port_mapping (arridx:140) 0 lb_tx_hash_to_port_mapping (arridx:139) 0 lb_tx_hash_to_port_mapping (arridx:138) 0 lb_tx_hash_to_port_mapping (arridx:137) 0 lb_tx_hash_to_port_mapping (arridx:136) 0 lb_tx_hash_to_port_mapping (arridx:135) 0 lb_tx_hash_to_port_mapping (arridx:134) 0 lb_tx_hash_to_port_mapping (arridx:133) 0 lb_tx_hash_to_port_mapping (arridx:132) 0 lb_tx_hash_to_port_mapping (arridx:131) 0 lb_tx_hash_to_port_mapping (arridx:130) 0 lb_tx_hash_to_port_mapping (arridx:129) 0 lb_tx_hash_to_port_mapping (arridx:128) 0 lb_tx_hash_to_port_mapping (arridx:127) 0 lb_tx_hash_to_port_mapping (arridx:126) 0 lb_tx_hash_to_port_mapping (arridx:125) 0 lb_tx_hash_to_port_mapping (arridx:124) 0 lb_tx_hash_to_port_mapping (arridx:123) 0 lb_tx_hash_to_port_mapping (arridx:122) 0 lb_tx_hash_to_port_mapping (arridx:121) 0 lb_tx_hash_to_port_mapping (arridx:120) 0 lb_tx_hash_to_port_mapping (arridx:119) 0 lb_tx_hash_to_port_mapping (arridx:118) 0 lb_tx_hash_to_port_mapping (arridx:117) 0 lb_tx_hash_to_port_mapping (arridx:116) 0 lb_tx_hash_to_port_mapping (arridx:115) 0 lb_tx_hash_to_port_mapping (arridx:114) 0 lb_tx_hash_to_port_mapping (arridx:113) 0 lb_tx_hash_to_port_mapping (arridx:112) 0 lb_tx_hash_to_port_mapping (arridx:111) 0 lb_tx_hash_to_port_mapping (arridx:110) 0 lb_tx_hash_to_port_mapping (arridx:109) 0 lb_tx_hash_to_port_mapping (arridx:108) 0 lb_tx_hash_to_port_mapping (arridx:107) 0 lb_tx_hash_to_port_mapping (arridx:106) 0 lb_tx_hash_to_port_mapping (arridx:105) 0 lb_tx_hash_to_port_mapping (arridx:104) 0 lb_tx_hash_to_port_mapping (arridx:103) 0 lb_tx_hash_to_port_mapping (arridx:102) 0 lb_tx_hash_to_port_mapping (arridx:101) 0 lb_tx_hash_to_port_mapping (arridx:100) 0 lb_tx_hash_to_port_mapping (arridx:99) 0 lb_tx_hash_to_port_mapping (arridx:98) 0 lb_tx_hash_to_port_mapping (arridx:97) 0 lb_tx_hash_to_port_mapping (arridx:96) 0 lb_tx_hash_to_port_mapping (arridx:95) 0 lb_tx_hash_to_port_mapping (arridx:94) 0 lb_tx_hash_to_port_mapping (arridx:93) 0 lb_tx_hash_to_port_mapping (arridx:92) 0 lb_tx_hash_to_port_mapping (arridx:91) 0 lb_tx_hash_to_port_mapping (arridx:90) 0 lb_tx_hash_to_port_mapping (arridx:89) 0 lb_tx_hash_to_port_mapping (arridx:88) 0 lb_tx_hash_to_port_mapping (arridx:87) 0 lb_tx_hash_to_port_mapping (arridx:86) 0 lb_tx_hash_to_port_mapping (arridx:85) 0 lb_tx_hash_to_port_mapping (arridx:84) 0 lb_tx_hash_to_port_mapping (arridx:83) 0 lb_tx_hash_to_port_mapping (arridx:82) 0 lb_tx_hash_to_port_mapping (arridx:81) 0 lb_tx_hash_to_port_mapping (arridx:80) 0 lb_tx_hash_to_port_mapping (arridx:79) 0 lb_tx_hash_to_port_mapping (arridx:78) 0 lb_tx_hash_to_port_mapping (arridx:77) 0 lb_tx_hash_to_port_mapping (arridx:76) 0 lb_tx_hash_to_port_mapping (arridx:75) 0 lb_tx_hash_to_port_mapping (arridx:74) 0 lb_tx_hash_to_port_mapping (arridx:73) 0 lb_tx_hash_to_port_mapping (arridx:72) 0 lb_tx_hash_to_port_mapping (arridx:71) 0 lb_tx_hash_to_port_mapping (arridx:70) 0 lb_tx_hash_to_port_mapping (arridx:69) 0 lb_tx_hash_to_port_mapping (arridx:68) 0 lb_tx_hash_to_port_mapping (arridx:67) 0 lb_tx_hash_to_port_mapping (arridx:66) 0 lb_tx_hash_to_port_mapping (arridx:65) 0 lb_tx_hash_to_port_mapping (arridx:64) 0 lb_tx_hash_to_port_mapping (arridx:63) 0 lb_tx_hash_to_port_mapping (arridx:62) 0 lb_tx_hash_to_port_mapping (arridx:61) 0 lb_tx_hash_to_port_mapping (arridx:60) 0 lb_tx_hash_to_port_mapping (arridx:59) 0 lb_tx_hash_to_port_mapping (arridx:58) 0 lb_tx_hash_to_port_mapping (arridx:57) 0 lb_tx_hash_to_port_mapping (arridx:56) 0 lb_tx_hash_to_port_mapping (arridx:55) 0 lb_tx_hash_to_port_mapping (arridx:54) 0 lb_tx_hash_to_port_mapping (arridx:53) 0 lb_tx_hash_to_port_mapping (arridx:52) 0 lb_tx_hash_to_port_mapping (arridx:51) 0 lb_tx_hash_to_port_mapping (arridx:50) 0 lb_tx_hash_to_port_mapping (arridx:49) 0 lb_tx_hash_to_port_mapping (arridx:48) 0 lb_tx_hash_to_port_mapping (arridx:47) 0 lb_tx_hash_to_port_mapping (arridx:46) 0 lb_tx_hash_to_port_mapping (arridx:45) 0 lb_tx_hash_to_port_mapping (arridx:44) 0 lb_tx_hash_to_port_mapping (arridx:43) 0 lb_tx_hash_to_port_mapping (arridx:42) 0 lb_tx_hash_to_port_mapping (arridx:41) 0 lb_tx_hash_to_port_mapping (arridx:40) 0 lb_tx_hash_to_port_mapping (arridx:39) 0 lb_tx_hash_to_port_mapping (arridx:38) 0 lb_tx_hash_to_port_mapping (arridx:37) 0 lb_tx_hash_to_port_mapping (arridx:36) 0 lb_tx_hash_to_port_mapping (arridx:35) 0 lb_tx_hash_to_port_mapping (arridx:34) 0 lb_tx_hash_to_port_mapping (arridx:33) 0 lb_tx_hash_to_port_mapping (arridx:32) 0 lb_tx_hash_to_port_mapping (arridx:31) 0 lb_tx_hash_to_port_mapping (arridx:30) 0 lb_tx_hash_to_port_mapping (arridx:29) 0 lb_tx_hash_to_port_mapping (arridx:28) 0 lb_tx_hash_to_port_mapping (arridx:27) 0 lb_tx_hash_to_port_mapping (arridx:26) 0 lb_tx_hash_to_port_mapping (arridx:25) 0 lb_tx_hash_to_port_mapping (arridx:24) 0 lb_tx_hash_to_port_mapping (arridx:23) 0 lb_tx_hash_to_port_mapping (arridx:22) 0 lb_tx_hash_to_port_mapping (arridx:21) 0 lb_tx_hash_to_port_mapping (arridx:20) 0 lb_tx_hash_to_port_mapping (arridx:19) 0 lb_tx_hash_to_port_mapping (arridx:18) 0 lb_tx_hash_to_port_mapping (arridx:17) 0 lb_tx_hash_to_port_mapping (arridx:16) 0 lb_tx_hash_to_port_mapping (arridx:15) 0 lb_tx_hash_to_port_mapping (arridx:14) 0 lb_tx_hash_to_port_mapping (arridx:13) 0 lb_tx_hash_to_port_mapping (arridx:12) 0 lb_tx_hash_to_port_mapping (arridx:11) 0 lb_tx_hash_to_port_mapping (arridx:10) 0 lb_tx_hash_to_port_mapping (arridx:9) 0 lb_tx_hash_to_port_mapping (arridx:8) 0 lb_tx_hash_to_port_mapping (arridx:7) 0 lb_tx_hash_to_port_mapping (arridx:6) 0 lb_tx_hash_to_port_mapping (arridx:5) 0 lb_tx_hash_to_port_mapping (arridx:4) 0 lb_tx_hash_to_port_mapping (arridx:3) 0 lb_tx_hash_to_port_mapping (arridx:2) 0 lb_tx_hash_to_port_mapping (arridx:1) 0 lb_tx_hash_to_port_mapping (arridx:0) 0 lb_tx_method hash bpf_hash_func \20\00\00\00\02\00\00\00\30\00\00\00\28\f0\ff\ff\07\00\00\00\00\00\00\00\28\00\00\00\00\00\00\00\30\00\0 <trunc> mcast_rejoin_interval 0 mcast_rejoin_count 0 notify_peers_interval 0 notify_peers_count 0 mode loadbalance

It is very hard for me to believe that every I hash has "00\00\00\00\00\00\00\00" in it ! What is balancing the traffic ?  :-)

First let me just answer you on couple of ideas you mentioned. 1. We are the Internet Exchange so the probability that one TCP stream between two of our customers is the cause of that much disbalance is very small.. You can see from the graphics I posted to forum we are talking about 1-2 Gb/sec od traffic that is not balanced between ports while the whole trafic is between 3-5 Gb/sec! Couple of the test show graphs that are such that you can see practically all of the traffic is on single port !!!

2. Also (from the output of the teamnl team0 options) something is VERY wrong because I am not seeing any different hashes been calculated !!! There are two teamd proceses in the "ps -AL" [root@rtr ~]# ps -AL | grep team 3433 3433 ? 00:12:32 teamd 6187 6187 ? 00:00:00 teamd

3. If there is some more data you need please let me know.

Best regards G.Slavic

-----Original Message-----

From: "Jiri Pirko" jiri@resnulli.us To: "Goran Slavić" gslavic@sox.rs Cc: libteam@lists.fedorahosted.org Date: 06/04/15 08:13 Subject: Re: Problems with teaming ...

Wed, Jun 03, 2015 at 10:13:16PM CEST, gslavic@sox.rs wrote:

...

Dear Mr Jiri Pirko,

My name is Goran Slavić, and I work for Serbian Open eXchange. We have setup the teaming on 4x10Gb/sec ports of our routing server. Unfortunetly load balancing between teamed ports doesn't work very well (we have extremely UN-balanced traffic on teamed ports). Traffic analysis suggests that the balancing is only done with MAC addresses and the rest of configuration directives ("ipv4, ipv6, l4" ...) is ignored/not taken in consideration when load balancing traffic between ports !

You can find a detailed explanation (what we configured on ports/teaming software), our configuration details and graph that are the result of our testing on this link (forum where we first shown the problem and asked people to help us):

https://www.centos.org/forums/viewtopic.php?f=50&t=52121

We would highly appreciate any help, advice or suggestion you can give us regarding this problem.

Hello Goran.

It's always good idea to cc libteam mailing list for reports such as yours.

TX port selection is done based on hash (0-255) computed by BPF function. Then, the hash is either used directly to determine tx port (hash%number_of_ports), or there is a hash-port mapping where every hash is mapped to a tx port. The second case is yours.

You can see the mapping either in teamd debug output, or if you list team options directly by "teamnl team0 options". With mapping there are also statistics for each hash there.

So in your case, there might be 3 issues:

1. your traffic is not differenciated enough, so most of the packets

falls into sigle hash. That would happen for example for one tcp stream between 2 ip addresses. 2) hash function fails to produce good hash according to the packets coming through. You could try to dump the BPF function and look at the inctructions by hand. Or another thing would be to assebmle your own hash function and push it into kernel manually (using teamnl team0                               setoption bpf_hash_func                               BFPCODE) 3) mapping is wrong. Balancer might do the mapping wrongly. To figure this out, the debug messages or options dump will help you to find out if this is the case.

Please gather more info so we can proceed to debug this.

Thanks.

Jiri