On Thu, 4 Jul 2019 at 07:03, Daniel Arena daniel.arena@smartplay.com wrote:

Hello,

I have a loadbalance runner for 2 interfaces setup through NetworkManager on fedora 30. I noticed in the system journal the teamd messages have been quite spammy with stats updates/rebalance information from teamd. See example log lines below. I tried looking at `man teamd.conf` but only found the debug_level setting, which is defaulted to 0. Is there some other way to turn these messages off? I tried adjusting NetworkManager's log levels also, but even after shutting off all logging, the messages from teamd still get through. Is this actually an issue for NetworkManager, maybe it should be capturing these log messages and filtering them with NetworkManager's log levels?

My teamd config looks like this:

{"runner": { "name": "loadbalance", "tx_hash": ["eth","ipv4", "ipv6"], "tx_balancer": {"name": "basic"} }, "ports":{"eno1": {}, "eno2": {}} }

Jul 03 15:41:57 server1 teamd_nm-team[32311]: stats update for port eno2: "3257841132330". Jul 03 15:41:57 server1 teamd_nm-team[32311]: stats update for port eno1: "6585330648". Jul 03 15:41:57 server1 teamd_nm-team[32311]: stats update for hash "21": "1242456011". Jul 03 15:41:57 server1 teamd_nm-team[32311]: stats update for hash "16": "1270449453". Jul 03 15:41:57 server1 teamd_nm-team[32311]: stats update for hash "9": "3761543798". Jul 03 15:41:57 server1 teamd_nm-team[32311]: stats update for hash "1": "626647946". Jul 03 15:41:57 server1 teamd_nm-team[32311]: Remapped hash "21" (delta 447647) to port eno2. Jul 03 15:41:57 server1 teamd_nm-team[32311]: Remapped hash "16" (delta 5301) to port eno1. Jul 03 15:41:57 server1 teamd_nm-team[32311]: Port eno2 rebalanced, delta: 447647 Jul 03 15:41:57 server1 teamd_nm-team[32311]: Port eno1 rebalanced, delta: 6205

These are debug-level logs, so it looks like something is enabling more logging than default.

If you run "nmcli general logging" do you see any other logging levels? The default is:

LEVEL DOMAINS INFO PLATFORM,<many other things>

But if you see output like:

LEVEL DOMAINS DEBUG TEAM

That would explain it within NM.

If you've previously had teamd debug logging enabled, you may need to put the team interface down and up to restart teamd to disable the debug logging. The debug logging is applied at daemon start with -g. I don't know if the logging level can be changed by Netlink message off the top of my head.

If you run "ps aux | egrep teamd" and you see the "-g" flag then you know teamd has been started with debug logging enabled.

Jamie

Tue, Jul 09, 2019 at 02:42:11PM CEST, daniel.arena@smartplay.com wrote:

On Mon, Jul 8, 2019 at 10:22 PM Jamie Bainbridge jamie.bainbridge@gmail.com wrote:

...

If you run "nmcli general logging" do you see any other logging levels? The default is:

LEVEL DOMAINS INFO PLATFORM,<many other things>

But if you see output like:

LEVEL DOMAINS DEBUG TEAM

Hi Jamie,

No, it says INFO on the left, then lists all of the domains.TEAM is not in the list though, but BOND is.

...

If you run "ps aux | egrep teamd" and you see the "-g" flag then you know teamd has been started with debug logging enabled.

Jamie

There is not a "-g" there. The teamd process was started as: /usr/bin/teamd -o -n -U -D -N -t nm-team -c {"runner": ... }

That is odd. The messages you mentioned in the original email are debug messages printed out by teamd_log_dbg->daemon_log(LOG_DEBUG,..)

Please check the config for "debug_level" option.

Another thing is, it might be good idea to move these messages to: teamd_log_dbgx(ctx, 2, ..)

Thanks, Dan _______________________________________________ libteam mailing list -- libteam@lists.fedorahosted.org To unsubscribe send an email to libteam-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/libteam@lists.fedorahosted.org

On Thu, Jul 11, 2019 at 10:47 AM Daniel Arena daniel.arena@smartplay.com wrote:

On Thu, Jul 11, 2019 at 6:01 AM Jiri Pirko jiri@resnulli.us wrote:

...

That is odd. The messages you mentioned in the original email are debug messages printed out by teamd_log_dbg->daemon_log(LOG_DEBUG,..)

Please check the config for "debug_level" option.

Another thing is, it might be good idea to move these messages to: teamd_log_dbgx(ctx, 2, ..)

Looks like it is disabled... here is output of teamdctl state dump

"setup": { "daemonized": false, "dbus_enabled": true, "debug_level": 0, "kernel_team_mode_name": "loadbalance", "pid": 32311, "pid_file": "/var/run/teamd/nm-team.pid", "runner_name": "loadbalance", "zmq_enabled": false },

Thanks, Dan

FYI, if I run `journalctl -pinfo` then they are filtered out since they are logged with debug priority. It looks like these messages could indeed be moved to teamd_log_dbgx, and then they would not land in the journal at all unless debug_level is enabled. At least they don't also land in /var/log/messages since my rsyslogd config defaults to only logging up to info level, not debug.

Thanks, Dan

On Sat, 20 Jul 2019 at 13:47, Jiri Pirko jiri@resnulli.us wrote:

That is odd. I don't see why they are there...

I also got a report about this issue. The reason is libdaemon do not filter logs if use DAEMON_LOG_SYSLOG, which looks intend to.

dlog.c: --- 35 static int daemon_verbosity_level = LOG_INFO;

46 void daemon_logv(int prio, const char* template, va_list arglist) { 47 int saved_errno; 48 49 saved_errno = errno; 50 51 if (daemon_log_use & DAEMON_LOG_SYSLOG) { 52 openlog(daemon_log_ident ? daemon_log_ident : "UNKNOWN", LOG_PID, LOG_DAEMON); 53 vsyslog(prio | LOG_DAEMON, template, arglist); 54 } 55 56 if (prio > daemon_verbosity_level) 57 goto end_daemon_logv; ---

dlog.h: --- 94 /** Setter for the verbosity level of standard output. 95 * 96 * @param verbosity_prio Minimum priority level for messages to output 97 * on standard output/error 98 * 99 * Allows to decide which messages to output on standard output/error 100 * streams. All messages are logged to syslog and this setting does 101 * not influence that. 102 * 103 * The default value is LOG_WARNING. 104 * 105 * @since 0.14 106 * @see DAEMON_SET_VERBOSITY_AVAILABLE

And after commit e47d5db538736 ("teamd: add an option to force log output to stdout, stderr or syslog"). NetworkManager will force teamd use DAEMON_LOG_SYSLOG by default. So either the user force to use option "-l stderr" or we need to implement an own filter ourselves.

Thanks Hangbin