#StackBounty: #networking #kernel #20.04 #syslog Syslog and kern.log keeps filling up and network stops working

Bounty: 50

The syslog and kern.log files keep filling in my system and it seems that the network (wifi stops working, not able to connect to VPN, etc) stops working. Reading the logs it seems like some network issue. The problem is that when these log files start filling up then sudo also stops working. If I enter any command with sudo it doesn’t even ask for password and just hangs up. To get the log entries I had to use su -. After restarting and deleting the log files everything goes back to normal – wifi starts working, sudo also starts working, able to connect to VPN. Currently, it is like I have to restart and delete the logs every half an hour which is really frustrating. I tried googling for the solution but failed to find any cases resonating with mine.

Here are the logs that I was able to save from the syslog which keeps repeating and eats up the disk (same logs in kern.log):

Jun 19 18:16:37 nb-ab kernel: [ 1334.288099] CPU: 2 PID: 170 Comm: kworker/2:2 Tainted: G        W         5.4.0-37-generic #41-Ubuntu
Jun 19 18:16:37 nb-ab kernel: [ 1334.288099] Hardware name: LENOVO 20NSS2CC00/20NSS2CC00, BIOS R10ET37W (1.22 ) 07/09/2019
Jun 19 18:16:37 nb-ab kernel: [ 1334.288109] Workqueue: events_freezable ieee80211_restart_work [mac80211]
Jun 19 18:16:37 nb-ab kernel: [ 1334.288118] RIP: 0010:drv_sta_state+0x254/0x3f0 [mac80211]
Jun 19 18:16:37 nb-ab kernel: [ 1334.288120] Code: 00 45 31 ed e9 66 fe ff ff 48 8b 83 78 04 00 00 48 8d b3 98 04 00 00 48 c7 c7 c8 18 0d c1 48 85 c0 48 0f 45 f0 e8 47 11 25 de <0f> 0b 41 bd fb ff ff ff e9 3d fe ff ff 65 8b 05 e8 fc fb 3e 89 c0
Jun 19 18:16:37 nb-ab kernel: [ 1334.288120] RSP: 0018:ffffa639c04ab840 EFLAGS: 00010286
Jun 19 18:16:37 nb-ab kernel: [ 1334.288121] RAX: 0000000000000000 RBX: ffff99785494a8c0 RCX: 0000000000000006
Jun 19 18:16:37 nb-ab kernel: [ 1334.288122] RDX: 0000000000000007 RSI: 0000000000000092 RDI: ffff99785f4978c0
Jun 19 18:16:37 nb-ab kernel: [ 1334.288123] RBP: ffffa639c04ab878 R08: 000000000093f251 R09: 0000000000000004
Jun 19 18:16:37 nb-ab kernel: [ 1334.288124] R10: 0000000000000000 R11: 0000000000000001 R12: ffff9978584d87a0
Jun 19 18:16:37 nb-ab kernel: [ 1334.288124] R13: ffff9978584d87a0 R14: 0000000000000004 R15: ffff9978584d8d68
Jun 19 18:16:37 nb-ab kernel: [ 1334.288125] FS:  0000000000000000(0000) GS:ffff99785f480000(0000) knlGS:0000000000000000
Jun 19 18:16:37 nb-ab kernel: [ 1334.288126] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jun 19 18:16:37 nb-ab kernel: [ 1334.288127] CR2: 00007fed0b240000 CR3: 00000002fab26006 CR4: 00000000003606e0
Jun 19 18:16:37 nb-ab kernel: [ 1334.288128] Call Trace:
Jun 19 18:16:37 nb-ab kernel: [ 1334.288138]  sta_info_move_state+0x276/0x370 [mac80211]
Jun 19 18:16:37 nb-ab kernel: [ 1334.288147]  __sta_info_destroy_part2+0x2f/0x180 [mac80211]
Jun 19 18:16:37 nb-ab kernel: [ 1334.288156]  __sta_info_flush+0x128/0x180 [mac80211]
Jun 19 18:16:37 nb-ab kernel: [ 1334.288171]  ieee80211_set_disassoc+0xc0/0x5f0 [mac80211]
Jun 19 18:16:37 nb-ab kernel: [ 1334.288185]  ieee80211_mgd_deauth+0x104/0x490 [mac80211]
Jun 19 18:16:37 nb-ab kernel: [ 1334.288197]  ieee80211_deauth+0x18/0x20 [mac80211]
Jun 19 18:16:37 nb-ab kernel: [ 1334.288210]  cfg80211_mlme_deauth+0xb6/0x1e0 [cfg80211]
Jun 19 18:16:37 nb-ab kernel: [ 1334.288221]  cfg80211_mlme_down+0x66/0x80 [cfg80211]
Jun 19 18:16:37 nb-ab kernel: [ 1334.288233]  cfg80211_disconnect+0x127/0x1e0 [cfg80211]
Jun 19 18:16:37 nb-ab kernel: [ 1334.288235]  ? _raw_spin_unlock_bh+0x1e/0x20
Jun 19 18:16:37 nb-ab kernel: [ 1334.288243]  __cfg80211_leave+0x133/0x1b0 [cfg80211]
Jun 19 18:16:37 nb-ab kernel: [ 1334.288252]  cfg80211_leave+0x2c/0x40 [cfg80211]
Jun 19 18:16:37 nb-ab kernel: [ 1334.288260]  cfg80211_netdev_notifier_call+0x1b2/0x590 [cfg80211]
Jun 19 18:16:37 nb-ab kernel: [ 1334.288262]  ? wake_up_klogd.part.0+0x34/0x40
Jun 19 18:16:37 nb-ab kernel: [ 1334.288263]  ? vprintk_emit+0x18c/0x290
Jun 19 18:16:37 nb-ab kernel: [ 1334.288265]  ? vprintk_default+0x29/0x50
Jun 19 18:16:37 nb-ab kernel: [ 1334.288267]  ? vprintk_func+0x4c/0xbc
Jun 19 18:16:37 nb-ab kernel: [ 1334.288281]  ? ieee80211_reconfig+0xd3/0x1170 [mac80211]
Jun 19 18:16:37 nb-ab kernel: [ 1334.288282]  ? rtnl_is_locked+0x15/0x20
Jun 19 18:16:37 nb-ab kernel: [ 1334.288284]  ? inetdev_event+0x47/0x560
Jun 19 18:16:37 nb-ab kernel: [ 1334.288286]  ? skb_dequeue+0x5a/0x70
Jun 19 18:16:37 nb-ab kernel: [ 1334.288290]  notifier_call_chain+0x55/0x80
Jun 19 18:16:37 nb-ab kernel: [ 1334.288291]  raw_notifier_call_chain+0x16/0x20
Jun 19 18:16:37 nb-ab kernel: [ 1334.288293]  call_netdevice_notifiers_info+0x2e/0x60
Jun 19 18:16:37 nb-ab kernel: [ 1334.288295]  __dev_close_many+0x63/0x120
Jun 19 18:16:37 nb-ab kernel: [ 1334.288296]  dev_close_many+0x91/0x150
Jun 19 18:16:37 nb-ab kernel: [ 1334.288298]  dev_close.part.0+0x4a/0x70
Jun 19 18:16:37 nb-ab kernel: [ 1334.288300]  dev_close+0x18/0x20
Jun 19 18:16:37 nb-ab kernel: [ 1334.288308]  cfg80211_shutdown_all_interfaces+0x77/0xd0 [cfg80211]
Jun 19 18:16:37 nb-ab kernel: [ 1334.288321]  ieee80211_handle_reconfig_failure+0x9d/0xb0 [mac80211]
Jun 19 18:16:37 nb-ab kernel: [ 1334.288334]  ieee80211_reconfig+0xa0/0x1170 [mac80211]
Jun 19 18:16:37 nb-ab kernel: [ 1334.288342]  ieee80211_restart_work+0xc3/0xf0 [mac80211]
Jun 19 18:16:37 nb-ab kernel: [ 1334.288344]  process_one_work+0x1eb/0x3b0
Jun 19 18:16:37 nb-ab kernel: [ 1334.288348]  worker_thread+0x4d/0x400
Jun 19 18:16:37 nb-ab kernel: [ 1334.288349]  kthread+0x104/0x140
Jun 19 18:16:37 nb-ab kernel: [ 1334.288351]  ? process_one_work+0x3b0/0x3b0
Jun 19 18:16:37 nb-ab kernel: [ 1334.288352]  ? kthread_park+0x90/0x90
Jun 19 18:16:37 nb-ab kernel: [ 1334.288354]  ret_from_fork+0x1f/0x40
Jun 19 18:16:37 nb-ab kernel: [ 1334.288355] ---[ end trace 114453926a486a7c ]---
Jun 19 18:16:37 nb-ab kernel: [ 1334.288373] ------------[ cut here ]------------
Jun 19 18:16:37 nb-ab kernel: [ 1334.288373] wlp0s20f3:  Failed check-sdata-in-driver check, flags: 0x4
Jun 19 18:16:37 nb-ab kernel: [ 1334.288398] WARNING: CPU: 2 PID: 170 at net/mac80211/driver-ops.h:17 drv_sta_state+0x254/0x3f0 [mac80211]
Jun 19 18:16:37 nb-ab kernel: [ 1334.288398] Modules linked in: rfcomm veth xt_nat nf_conntrack_netlink xfrm_user xfrm_algo xt_addrtype br_netfilter ccm xt_CHECKSUM xt_MASQUERADE xt_conntrack ipt_REJECT nf_reject_ipv4 xt_tcpudp ip6table_mangle ip6table_nat iptable_mangle iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c nf_tables nfnetlink ip6table_filter ip6_tables iptable_filter bpfilter bridge stp llc aufs cmac algif_hash algif_skcipher af_alg overlay bnep nls_iso8859_1 snd_hda_codec_hdmi snd_sof_pci snd_sof_intel_hda_common snd_soc_hdac_hda x86_pkg_temp_thermal snd_sof_intel_hda intel_powerclamp snd_sof_intel_byt mei_hdcp snd_sof_intel_ipc coretemp joydev snd_sof intel_rapl_msr snd_sof_xtensa_dsp kvm_intel snd_hda_ext_core snd_hda_codec_realtek snd_soc_acpi_intel_match snd_hda_codec_generic kvm snd_soc_acpi snd_soc_core snd_compress crct10dif_pclmul ac97_bus snd_pcm_dmaengine ghash_clmulni_intel iwlmvm snd_hda_intel uvcvideo mac80211 snd_seq_midi snd_intel_dspcfg aesni_intel snd_seq_midi_event
Jun 19 18:16:37 nb-ab kernel: [ 1334.288414]  videobuf2_vmalloc crypto_simd snd_hda_codec libarc4 videobuf2_memops cryptd snd_rawmidi glue_helper snd_hda_core videobuf2_v4l2 intel_cstate i915 hid_sensor_accel_3d btusb snd_hwdep snd_seq hid_sensor_trigger videobuf2_common btrtl thinkpad_acpi industrialio_triggered_buffer btbcm kfifo_buf iwlwifi intel_rapl_perf snd_pcm videodev rtsx_pci_ms nvram btintel hid_sensor_iio_common input_leds snd_seq_device wmi_bmof serio_raw elan_i2c memstick intel_wmi_thunderbolt ledtrig_audio mc drm_kms_helper bluetooth industrialio snd_timer i2c_algo_bit mei_me fb_sys_fops cfg80211 mei syscopyarea snd sysfillrect processor_thermal_device ecdh_generic cros_ec_ishtp ecc sysimgblt intel_rapl_common raydium_i2c_ts soundcore cros_ec intel_soc_dts_iosf i2c_multi_instantiate int3403_thermal intel_pch_thermal int3400_thermal int340x_thermal_zone acpi_thermal_rel mac_hid acpi_pad sch_fq_codel parport_pc ppdev lp parport drm ip_tables x_tables autofs4 hid_sensor_custom hid_sensor_hub hid_generic
Jun 19 18:16:37 nb-ab kernel: [ 1334.288430]  intel_ishtp_loader intel_ishtp_hid rtsx_pci_sdmmc crc32_pclmul psmouse rtsx_pci e1000e intel_lpss_pci ahci intel_lpss i2c_i801 idma64 libahci intel_ish_ipc virt_dma intel_ishtp i2c_hid wmi hid pinctrl_cannonlake video pinctrl_intel

Here is the process that has the PID 170:

root         170       2 18 17:54 ?        00:04:42 [kworker/2:2+events_freezable]

Output of top command when the logs start filling up with above-mentioned messages:

PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
170 root      20   0       0      0      0 R 100,0   0,0   5:38.02 kworker/2:2+events_freezable
868 syslog    20   0  298540   6980   3800 S  87,7   0,0   4:52.64 rsyslogd
374 root      19  -1  325532 214880 213136 R  49,5   1,3   4:57.52 systemd-journal

Output of journalctl xe:

-- Kernel messages have been lost as the journal system has been unable
-- to process them quickly enough.
Jun 19 18:21:56 nb-ab kernel: Modules linked in: rfcomm veth xt_nat nf_conntrack_netlink xfrm_user xfrm_algo xt_addrtype br_netfilter ccm xt_CHECKSUM xt_MASQUERADE xt_conntrack ipt_REJECT nf_reject_ipv4 xt_tcpu>
Jun 19 18:21:56 nb-ab kernel:  videobuf2_vmalloc crypto_simd snd_hda_codec libarc4 videobuf2_memops cryptd snd_rawmidi glue_helper snd_hda_core videobuf2_v4l2 intel_cstate i915 hid_sensor_accel_3d btusb snd_hwd>
Jun 19 18:21:56 nb-ab kernel:  intel_ishtp_loader intel_ishtp_hid rtsx_pci_sdmmc crc32_pclmul psmouse rtsx_pci e1000e intel_lpss_pci ahci intel_lpss i2c_i801 idma64 libahci intel_ish_ipc virt_dma intel_ishtp i2>
Jun 19 18:21:56 nb-ab kernel: CPU: 2 PID: 170 Comm: kworker/2:2 Tainted: G        W         5.4.0-37-generic #41-Ubuntu
Jun 19 18:21:56 nb-ab kernel: Hardware name: LENOVO 20NSS2CC00/20NSS2CC00, BIOS R10ET37W (1.22 ) 07/09/2019
Jun 19 18:21:56 nb-ab kernel: Workqueue: events_freezable ieee80211_restart_work [mac80211]
Jun 19 18:21:56 nb-ab kernel: RIP: 0010:drv_sta_state+0x254/0x3f0 [mac80211]
Jun 19 18:21:56 nb-ab systemd-journald[374]: Missed 54 kernel messages
-- Subject: Journal messages have been missed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support

I am totally confused about this situation and don’t know how to proceed. Any help will be appreciated.

EDIT 1

As asked this is the output of sudo lshw -C network:

*-network:0               
     description: Wireless interface
     product: Cannon Point-LP CNVi [Wireless-AC]
     vendor: Intel Corporation
     physical id: 14.3
     bus info: pci@0000:00:14.3
     logical name: wlp0s20f3
     version: 11
     serial: 40:74:e0:14:c0:2b
     width: 64 bits
     clock: 33MHz
     capabilities: pm msi pciexpress msix bus_master cap_list ethernet physical wireless
     configuration: broadcast=yes driver=iwlwifi driverversion=5.4.0-37-generic firmware=46.6bf1df06.0 ip=192.168.0.149 latency=0 link=yes multicast=yes wireless=IEEE 802.11
     resources: irq:16 memory:a1238000-a123bfff
*-network:1
     description: Ethernet interface
     product: Ethernet Connection (6) I219-V
     vendor: Intel Corporation
     physical id: 1f.6
     bus info: pci@0000:00:1f.6
     logical name: enp0s31f6
     version: 11
     serial: 48:2a:e3:57:35:4a
     capacity: 1Gbit/s
     width: 32 bits
     clock: 33MHz
     capabilities: pm msi bus_master cap_list ethernet physical tp 10bt 10bt-fd 100bt 100bt-fd 1000bt-fd autonegotiation
     configuration: autonegotiation=on broadcast=yes driver=e1000e driverversion=3.2.6-k firmware=0.5-4 latency=0 link=no multicast=yes port=twisted pair
     resources: irq:127 memory:a1200000-a121ffff
*-network:0
     description: Ethernet interface
     physical id: 2
     logical name: vethc2a65a3
     serial: fa:40:3c:f6:b2:12
     size: 10Gbit/s
     capabilities: ethernet physical
     configuration: autonegotiation=off broadcast=yes driver=veth driverversion=1.0 duplex=full link=yes multicast=yes port=twisted pair speed=10Gbit/s
*-network:1
     description: Ethernet interface
     physical id: 3
     logical name: docker0
     serial: 02:42:0d:09:df:51
     capabilities: ethernet physical
     configuration: broadcast=yes driver=bridge driverversion=2.3 firmware=N/A ip=172.17.0.1 link=no multicast=yes
*-network:2
     description: Ethernet interface
     physical id: 4
     logical name: veth27830f8
     serial: 12:8a:75:6e:fe:84
     size: 10Gbit/s
     capabilities: ethernet physical
     configuration: autonegotiation=off broadcast=yes driver=veth driverversion=1.0 duplex=full link=yes multicast=yes port=twisted pair speed=10Gbit/s
*-network:3
     description: Ethernet interface
     physical id: 5
     logical name: br-040ef3748cb5
     serial: 02:42:f7:fe:24:3e
     capabilities: ethernet physical
     configuration: broadcast=yes driver=bridge driverversion=2.3 firmware=N/A ip=172.18.0.1 link=yes multicast=yes
*-network:4 DISABLED
     description: Ethernet interface
     physical id: 6
     logical name: virbr0-nic
     serial: 52:54:00:c3:2c:b3
     size: 10Mbit/s
     capabilities: ethernet physical
     configuration: autonegotiation=off broadcast=yes driver=tun driverversion=1.6 duplex=full link=no multicast=yes port=twisted pair speed=10Mbit/s
*-network:5
     description: Ethernet interface
     physical id: 7
     logical name: virbr0
     serial: 52:54:00:c3:2c:b3
     capabilities: ethernet physical
     configuration: broadcast=yes driver=bridge driverversion=2.3 firmware=N/A ip=192.168.122.1 link=no multicast=yes
*-network:6
     description: Ethernet interface
     physical id: 8
     logical name: vethcda64a0
     serial: a2:09:6e:c2:74:8e
     size: 10Gbit/s
     capabilities: ethernet physical
     configuration: autonegotiation=off broadcast=yes driver=veth driverversion=1.0 duplex=full link=yes multicast=yes port=twisted pair speed=10Gbit/s

Using dkms status gave no output.


Get this bounty!!!

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.