#StackBounty: #linux #cpu #cpu-usage #turbo-boost #cpufreq Since upgrade from linux 4.6 to (4.7, 4.8), %CPU is always 0; the frequency …

Bounty: 100

Since the upgrade to linux kernel 4.7 (Debian Stretch), it seems that my system (Aurora-R4 i7 3820) is a bit slower (short hangs sometimes) and the most strange thing is that I can’t see anymore the percentages on the CPU cores with top, KSysGuard, etc, …

top (sort by %CPU):

Tasks: 263 total,   1 running, 262 sleeping,   0 stopped,   0 zombie
%Cpu(s):  7.0 us,  1.9 sy,  8.9 ni, 81.5 id,  0.6 wa,  0.0 hi,  0.1 si,  0.0 st
KiB Mem :  8095452 total,  4514552 free,  1361576 used,  2219324 buff/cache
KiB Swap:  8301564 total,  8301564 free,        0 used.  6390680 avail Mem 

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                                                                                                                        
    1 root      20   0  203000   7504   5156 S   0.0  0.1   0:00.11 systemd                                                                                                                                                                        
    2 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kthreadd                                                                                                                                                                       
    3 root      20   0       0      0      0 S   0.0  0.0   0:00.00 ksoftirqd/0                                                                                                                                                                    
    4 root      20   0       0      0      0 S   0.0  0.0  31:41.17 kworker/0:0                                                                                                                                                                    
    5 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/0:0H                                                                                                                                                                   
    7 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcu_sched                                                                                                                                                                      
    8 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcu_bh                                                                                                                                                                         
    9 root      rt   0       0      0      0 S   0.0  0.0   0:00.00 migration/0                                                                                                                                                                    
   10 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 lru-add-drain                                                                                                                                                                  
   11 root      rt   0       0      0      0 S   0.0  0.0   0:00.00 watchdog/0                                                                                                                                                                     
   12 root      20   0       0      0      0 S   0.0  0.0   0:00.00 cpuhp/0                                                                                                                                                                        
   13 root      20   0       0      0      0 S   0.0  0.0   0:00.00 cpuhp/1

But I can see something with mpstat:

$ mpstat -P ALL
Linux 4.7.0-1-amd64 (alienium)  23. 10. 16      _x86_64_        (8 CPU)

14:37:02     CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest  %gnice   %idle
14:37:02     all    5.58    5.02    1.48    0.40    0.00    0.06    0.00    0.00    0.00   87.46
14:37:02       0    6.73    5.03    1.71    0.12    0.00    0.00    0.00    0.00    0.00   86.41
14:37:02       1    6.35    5.15    1.69    0.37    0.00    0.05    0.00    0.00    0.00   86.39
14:37:02       2    4.73    4.98    1.68    0.22    0.00    0.15    0.00    0.00    0.00   88.24
14:37:02       3    6.78    5.18    1.62    0.12    0.00    0.00    0.00    0.00    0.00   86.31
14:37:02       4    7.43    4.96    1.75    2.26    0.00    0.19    0.00    0.00    0.00   83.41
14:37:02       5    3.61    4.83    1.22    0.06    0.00    0.02    0.00    0.00    0.00   90.26
14:37:02       6    5.07    5.06    1.16    0.04    0.00    0.03    0.00    0.00    0.00   88.63
14:37:02       7    3.96    5.01    0.97    0.04    0.00    0.00    0.00    0.00    0.00   90.03

cpuinfo (just the first core)

processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 45
model name      : Intel(R) Core(TM) i7-3820 CPU @ 3.60GHz
stepping        : 7
microcode       : 0x710
cpu MHz         : 3600.045
cache size      : 10240 KB
physical id     : 0
siblings        : 8
core id         : 0
cpu cores       : 4
apicid          : 0
initial apicid  : 0
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx lahf_lm tpr_shadow vnmi flexpriority ept vpid xsaveopt dtherm ida arat pln pts
bugs            :
bogomips        : 7200.09
clflush size    : 64
cache_alignment : 64
address sizes   : 46 bits physical, 48 bits virtual
power management:

uname:

Linux alienium 4.7.0-1-amd64 #1 SMP Debian 4.7.8-1 (2016-10-19) x86_64 GNU/Linux

My only solution is to use the kernel 4.6, then everything is fine.

Any ideas ?

Thank you

Edit 1

The problem seems really intel_pstate.

cpupower frequency-info
analyse du CPU 0 :
  driver: intel_pstate
  CPUs which run at the same hardware frequency: 0
  CPUs which need to have their frequency coordinated by software: 0
  maximum transition latency:  Cannot determine or is not supported.
  limitation matérielle : 1.20 GHz - 4.00 GHz
  régulateurs disponibles : performance powersave
  tactique actuelle : la fréquence doit être comprise entre 1.20 GHz et 4.00 GHz.
                  Le régulateur "powersave" est libre de choisir la vitesse
                  dans cette plage de fréquences.
  current CPU frequency: Unable to call hardware
  current CPU frequency:  Unable to call to kernel
  boost state support:
    Supported: yes
    Active: yes
    4000 MHz max turbo 4 active cores
    4000 MHz max turbo 3 active cores
    4000 MHz max turbo 2 active cores
    4000 MHz max turbo 1 active cores

Here we can see that’s it’s unable to call hardware and to kernel. But sometimes (or after a very long time, I’m not sure); this command returns correctly the CPU frequency.

Edit 2

Still not working with kernel 4.8.5, the frequency stucks at 3.6 GHz according to /proc/cpuinfo.

The frequency is unknown:

sudo cat /sys/devices/system/cpu/cpu0/cpufreq/cpuinfo_cur_freq
<unknown>

Note that my system seems running slower; then I think that it’s using the minimal CPU frequency (1.2 GHz). My watercooling seems not working hard.

Edit 3

I’ve tried kernel 4.9-rc5, it looks like 4.6 but there is a problem. I noticed that on 4.9 and 4.6 it’s not really working fine. The first core stucks always at the same frequency (and even one thread on the second core with 4.9):

With 4.6 and stress

$ cat /proc/cpuinfo | grep MHz
cpu MHz : 3600.045
cpu MHz : 3600.045
cpu MHz : 3899.953
cpu MHz : 3899.953
cpu MHz : 3899.953
cpu MHz : 3899.953
cpu MHz : 3899.953
cpu MHz : 3899.953

And 4.9 and stress

$ cat /proc/cpuinfo | grep MHz
cpu MHz : 3600.045
cpu MHz : 3600.045
cpu MHz : 3600.045
cpu MHz : 3899.953
cpu MHz : 3899.953
cpu MHz : 3899.953
cpu MHz : 3899.953
cpu MHz : 3899.953

If I disable pstate, then the problem persists but CPU0, 1 and 2 stucks at different frequencies. Only 3-7 are working correctly. I will try with linux 4.4.

Edit 4

I build Linux 4.4.33 (LTS) and everything is working perfectly. All cores are changing the frequencies as expected. I think that it was fine with Linux 4.5 too (but I’m a bit discouraged by building 20 times the kernel in one day). I should search what commit breaks on my system but it’s a bit difficult to build, install, reboot, test for each poential commit that potentially breaks something between Linux 4.4 and 4.6. It takes too much time.


Get this bounty!!!

#StackBounty: #cpu #gpu #gaming #lag #fps GPU usage drops causing FPS drop spike

Bounty: 50

I’ve been experiencing some stuttering/micro freezes while playing a certain videogame, BF4 to be precise. When running at different video qualities, ranging from low to ultra, the game scores 60fps smoothly, just to drop to below 10fps for like micro seconds, and goes back to 60fps. This issue is restricted to BF4, other games have not this issue.

Specs

Intel i7 6700HQ 2.6Ghz
nVidia 960M 1GB
16GB RAM
OS Windows 10 creator update
everthing is stock and not overclocked

Using “High performance” power plan, with min and max states of the cpu set at 100% in the advanced settings.

Here’s the usage graph got from MSI afterburner.
MSI afterburner usage graph

Does anyone have some ideas how to troubleshoot this issue and know what’s causing the GPU usage to drop.


Get this bounty!!!

#StackBounty: #windows-10 #cpu #performance #hyper-threading How does the Windows 10 scheduler deal with Hyper Threading since Core Par…

Bounty: 50

I’m running Windows 10 (1607) on an Intel Xeon E3-1231v3 CPU (Haswell, 4 physical cores, 8 logical cores).

When I first had Windows 7 installed on this machine, I could observe that four out of eight logical cores were parked until an application needed more than 4 threads. One can check with Windows resource monitor whether cores are parked or not (example).
As far as I understand, this is an important technique to keep the threads balanced across physical cores, as explained on the Microsoft website: “the Core Parking algorithm and infrastructure is also used to balance processor performance between logical processors on Windows 7 client systems with processors that include Intel Hyper-Threading Technology.

However after upgrading to Windows 10, I noticed that there is no core parking. All logical cores are active all the time and when you run an application using less than four threads you can see how the scheduler equally distributes them across all logical cpu cores. Microsoft employees have confirmed that Core Parking is disabled in Windows 10.

But I wonder why? What was the reason for this? Is there a replacement and if yes, how does it look like? Has Microsoft implemented a new scheduler strategy that made core parking obsolete?


Appendix:

Here is an example on how core parking introduced in Windows 7 can benefit performance (in comparison to Vista which didn’t have core parking feature yet). What you can see is that on Vista, HT (Hyper Threading) harms performance while on Windows 7 it doesn’t:

enter image description here

enter image description here

(source)

I tried to enable Core Parking as mentioned here, but what I observed was that the Core Parking algorithm isn’t Hyper Threading aware anymore. It parked cores 4,5,6,7, while it should have parked core 1,3,5,7 to avoid that threads are assigned to the same physical core. Windows enumerates cores in such a way that two successive indices belong to the same physical core. Very strange. It seems Microsoft has messed this up fundamentally. And no one noticed…

Furthermore, I did some CPU benchmarks using exactly 4 threads.

CPU affinity set to all cores (Windows defualt):

Average running time: 17.094498, standard deviation: 2.472625

CPU affinity set to every other core (so that it runs on different physical cores, best possible scheduling):

Average running time: 15.014045, standard deviation: 1.302473

CPU affinity set to the worst possible scheduling (four logical cores on two physical cores):

Average running time: 20.811493, standard deviation: 1.405621

So there is a performance difference. And you can see that the Windows defualt scheduling ranks between the best and worst possible scheduling, as we would expect it to happen with a non-hyperthreading aware scheduler. However, as pointed out in the comments, there may be other causes responsible for this, like fewer context switches, inference by monitoring applications, etc. So we still don’t have a definitive answer here.

Source code for my benchmark:

#include <stdlib.h>
#include <Windows.h>
#include <math.h>

double runBenchmark(int num_cores) {
  int size = 1000;
  double** source = new double*[size];
  for (int x = 0; x < size; x++) {
    source[x] = new double[size];
  }
  double** target = new double*[size * 2];
  for (int x = 0; x < size * 2; x++) {
    target[x] = new double[size * 2];
  }
  #pragma omp parallel for num_threads(num_cores)
  for (int x = 0; x < size; x++) {
    for (int y = 0; y < size; y++) {
      source[y][x] = rand();
    }
  }
  #pragma omp parallel for num_threads(num_cores)
  for (int x = 0; x < size-1; x++) {
    for (int y = 0; y < size-1; y++) {
      target[x * 2][y * 2] = 0.25 * (source[x][y] + source[x + 1][y] + source[x][y + 1] + source[x + 1][y + 1]);
    }
  }
  double result = target[rand() % size][rand() % size];
  for (int x = 0; x < size * 2; x++) delete[] target[x];
  for (int x = 0; x < size; x++) delete[] source[x];
  delete[] target;
  delete[] source;
  return result;
}

int main(int argc, char** argv)
{
  int num_cores = 4;
  system("pause");  // So we can set cpu affinity before the benchmark starts 
  const int iters = 1000;
  double avgElapsedTime = 0.0;
  double elapsedTimes[iters];
  for (int i = 0; i < iters; i++) {
    LARGE_INTEGER frequency;
    LARGE_INTEGER t1, t2;
    QueryPerformanceFrequency(&frequency);
    QueryPerformanceCounter(&t1);
    runBenchmark(num_cores);
    QueryPerformanceCounter(&t2);
    elapsedTimes[i] = (t2.QuadPart - t1.QuadPart) * 1000.0 / frequency.QuadPart;
    avgElapsedTime += elapsedTimes[i];
  }
  avgElapsedTime = avgElapsedTime / iters;
  double variance = 0;
  for (int i = 0; i < iters; i++) {
    variance += (elapsedTimes[i] - avgElapsedTime) * (elapsedTimes[i] - avgElapsedTime);
  }
  variance = sqrt(variance / iters);
  printf("Average running time: %f, standard deviation: %f", avgElapsedTime, variance);
  return 0;
}


Get this bounty!!!

#StackBounty: #ubuntu #kernel #linux-kernel #cpu Kworker is at 100% – I think I've tried everything!

Bounty: 100

Thanks for checking this one out.

jonathan@melange:~$ top

top - 05:21:08 up 44 min,  2 users,  load average: 1.21, 1.68, 1.98
Tasks: 351 total,   2 running, 349 sleeping,   0 stopped,   0 zombie
%Cpu(s):  4.3 us, 14.0 sy,  2.1 ni, 70.4 id,  8.9 wa,  0.0 hi,  0.3 si,  0.0 st
GiB Mem :   15.579 total,    0.173 free,    4.141 used,   11.264 buff/cache
GiB Swap:   15.910 total,   15.868 free,    0.042 used.   11.014 avail Mem 

  PID  PPID   UID USER     RUSER    TTY          TIME+  %CPU %MEM S COMMAND                                                                                                                                   
   67     2     0 root     root     ?         22:22.40 100.0  0.0 R kworker/0:1 

The setup – ubuntu 16.10. 4.8.0-41-generic. Modern intel based laptop with Nvidia drivers and not quite perfect wifi. Let me know and I can provide you with whatever info you need. I have these working acceptably and I don’t see any reason to believe these are involved in this issue.

I’ve actually already asked this on askubuntu & a couple of times over at Freenode` #ubuntu over the last week but no one will even respond to my question 🙁

I’ve taken some perf reports with

sudo perf record -a -g sleep 10
sudo perf report

With some results

Samples: 92K of event 'cycles:ppp', Event count (approx.): 58330337004406                                                                                                                                     
  Children      Self  Command          Shared Object                        Symbol                                                                                                                           ◆
+   94.27%     0.00%  swapper          [kernel.kallsyms]                    [k] cpu_startup_entry                                                                                                            ▒
+   94.27%     0.00%  swapper          [kernel.kallsyms]                    [k] start_secondary                                                                                                              ▒
+   77.29%     0.00%  swapper          [kernel.kallsyms]                    [k] schedule_preempt_disabled                                                                                                    ▒
-   77.29%    77.29%  swapper          [kernel.kallsyms]                    [k] __schedule                                                                                                                   ▒
     77.29% start_secondary                                                                                                                                                                                  ▒
        cpu_startup_entry                                                                                                                                                                                    ▒
      - schedule_preempt_disabled                                                                                                                                                                            ▒
         - 77.29% schedule                                                                                                                                                                                   ▒
              __schedule                                                                                                                                                                                     ▒
+   77.29%     0.00%  swapper          [kernel.kallsyms]                    [k] schedule                                                                                                                     ▒
+   16.99%     0.00%  swapper          [kernel.kallsyms]                    [k] call_cpuidle                                                                                                                 ▒
+   16.99%     0.00%  swapper          [kernel.kallsyms]                    [k] cpuidle_enter                                                                                                                ▒
+   16.99%     0.00%  swapper          [kernel.kallsyms]                    [k] cpuidle_enter_state                                                                                                          ▒
-   16.99%    16.99%  swapper          [kernel.kallsyms]                    [k] intel_idle                                                                                                                   ▒
     16.98% start_secondary                                                                                                                                                                                  ▒
        cpu_startup_entry                                                                                                                                                                                    ▒
        call_cpuidle                                                                                                                                                                                         ▒
      - cpuidle_enter                                                                                                                                                                                        ▒
         - 16.98% cpuidle_enter_state                                                                                                                                                                        ▒
              intel_idle                                                                                                                                                                                     ▒
+    5.65%     0.00%  pool             [unknown]                            [.] 0000000000000000                                                                                                             ▒
+    5.65%     5.65%  pool             libc-2.24.so                         [.] re_compile_internal                                                                                                          ▒
+    5.65%     0.00%  pool             [unknown]                            [.] 0x00007f049804d628                                                                                                           ▒
+    5.65%     0.00%  pool             [unknown]                            [.] 0x00007f049804d6a8                                                                                                           ▒
+    5.65%     0.00%  pool             [unknown]                            [.] 0x00007f049804d3d8                                                                                                           ▒
+    5.65%     0.00%  pool             [unknown]                            [.] 0x00007f049804d768                                                                                                           ▒
Cannot load tips.txt file, please install perf!

I’ve checked dmesg, over heating messages (thats why I’m here) and some other messages about MSFT0101:00 which I believe is something todo with the kernel not recognising my bios enabled TPM module. I think that this should be insignificant in this matter.

There is another question about kworker threads suggesting the following as per this thread

$ echo workqueue:workqueue_queue_work > /sys/kernel/debug/tracing/set_event
$ cat /sys/kernel/debug/tracing/trace_pipe > out.txt
(wait a few secs)
^C

but it doesn’t work!

jonathan@melange:~$ sudo mount -t debugfs nodev /sys/kernel/debug
mount: nodev is already mounted or /sys/kernel/debug busy
jonathan@melange:~$ sudo echo workqueue:workqueue_queue_work > /sys/kernel/debug/tracing/set_event
bash: /sys/kernel/debug/tracing/set_event: Permission denied
jonathan@melange:~$ sudo cat /proc/67/stack
[<ffffffffffffffff>] 0xffffffffffffffff

Any ideas?!

Update

Before submitting this question I had been using Kworker, what is it and why is it hogging so much CPU? as reference. So I had tried disabling/uninstalling long running processes such as dropbox, insync (google drive), crashplan, keybase, Variety background, multiload indicator, psensor, guake. (I feel like I have a pretty slick setup most of the time…) but nothing seemed to help.

There had been other questions lurking around suggesting malfunctioning wifi, nvidia drivers or usb drivers. But nothing in my logs were suggesting this either. Somewhat thankful as almost always the solution in those was simply find newer nvidia drivers, update the kernel, or “Deal with it.” My laptop is pretty up to date already, I have no enterprise reason to stay on 16.04 and I already have the nvidia ppa activated, as with the intel drivers, so this wasn’t much help.

Perhaps the kworker was actually the result of the laptop overheating -> cpu throttling + cpu fan management. Not the cause. As suggested by Stop cpu from overheating So I’ve just used some compressed air to clean out the fans (didn’t think this would be a problem on a laptop only 9 months old yet there was actually a bit of dust) and investigating the thermal-conf.xml which suggests that the fan kicks in at 55°C (although still working on what I can do here)

Thinking this may actually be the solution. Will report back soon.

Update 2

So doing the Acer bios update totally ruined everything related to my secureboot setup and corrupted the the efi files so it took me a few days to work out how to regenerate the ubuntu efi keys and the and windows efi keys.

I tried cleaning out the dust, and it definitely helped for the two days until I started with the bios issues.

But the kworker is back (and yes it is the same as far as I can tell). I also have some more information now. I can see that the cpu is not throttling down, but rather staying at the maximum. The fan is running, but the device is only sitting around the 60degree mark, so i wouldn’t call this serious over heating.

The commands from the other thread require raising to the root user, not just using sudo. so sudo su and then getting the stack trace gives the following.

[<ffffffff98a9dcea>] worker_thread+0xca/0x500
[<ffffffff98aa40d8>] kthread+0xd8/0xf0
[<ffffffff992a071f>] ret_from_fork+0x1f/0x40
[<ffffffffffffffff>] 0xffffffffffffffff

Doesn’t look particularly helpful to me.


Get this bounty!!!

#StackBounty: #ubuntu #kernel #linux-kernel #cpu Kworker is at 100% – I think I've tried everything!

Bounty: 100

Thanks for checking this one out.

jonathan@melange:~$ top

top - 05:21:08 up 44 min,  2 users,  load average: 1.21, 1.68, 1.98
Tasks: 351 total,   2 running, 349 sleeping,   0 stopped,   0 zombie
%Cpu(s):  4.3 us, 14.0 sy,  2.1 ni, 70.4 id,  8.9 wa,  0.0 hi,  0.3 si,  0.0 st
GiB Mem :   15.579 total,    0.173 free,    4.141 used,   11.264 buff/cache
GiB Swap:   15.910 total,   15.868 free,    0.042 used.   11.014 avail Mem 

  PID  PPID   UID USER     RUSER    TTY          TIME+  %CPU %MEM S COMMAND                                                                                                                                   
   67     2     0 root     root     ?         22:22.40 100.0  0.0 R kworker/0:1 

The setup – ubuntu 16.10. 4.8.0-41-generic. Modern intel based laptop with Nvidia drivers and not quite perfect wifi. Let me know and I can provide you with whatever info you need. I have these working acceptably and I don’t see any reason to believe these are involved in this issue.

I’ve actually already asked this on askubuntu & a couple of times over at Freenode` #ubuntu over the last week but no one will even respond to my question 🙁

I’ve taken some perf reports with

sudo perf record -a -g sleep 10
sudo perf report

With some results

Samples: 92K of event 'cycles:ppp', Event count (approx.): 58330337004406                                                                                                                                     
  Children      Self  Command          Shared Object                        Symbol                                                                                                                           ◆
+   94.27%     0.00%  swapper          [kernel.kallsyms]                    [k] cpu_startup_entry                                                                                                            ▒
+   94.27%     0.00%  swapper          [kernel.kallsyms]                    [k] start_secondary                                                                                                              ▒
+   77.29%     0.00%  swapper          [kernel.kallsyms]                    [k] schedule_preempt_disabled                                                                                                    ▒
-   77.29%    77.29%  swapper          [kernel.kallsyms]                    [k] __schedule                                                                                                                   ▒
     77.29% start_secondary                                                                                                                                                                                  ▒
        cpu_startup_entry                                                                                                                                                                                    ▒
      - schedule_preempt_disabled                                                                                                                                                                            ▒
         - 77.29% schedule                                                                                                                                                                                   ▒
              __schedule                                                                                                                                                                                     ▒
+   77.29%     0.00%  swapper          [kernel.kallsyms]                    [k] schedule                                                                                                                     ▒
+   16.99%     0.00%  swapper          [kernel.kallsyms]                    [k] call_cpuidle                                                                                                                 ▒
+   16.99%     0.00%  swapper          [kernel.kallsyms]                    [k] cpuidle_enter                                                                                                                ▒
+   16.99%     0.00%  swapper          [kernel.kallsyms]                    [k] cpuidle_enter_state                                                                                                          ▒
-   16.99%    16.99%  swapper          [kernel.kallsyms]                    [k] intel_idle                                                                                                                   ▒
     16.98% start_secondary                                                                                                                                                                                  ▒
        cpu_startup_entry                                                                                                                                                                                    ▒
        call_cpuidle                                                                                                                                                                                         ▒
      - cpuidle_enter                                                                                                                                                                                        ▒
         - 16.98% cpuidle_enter_state                                                                                                                                                                        ▒
              intel_idle                                                                                                                                                                                     ▒
+    5.65%     0.00%  pool             [unknown]                            [.] 0000000000000000                                                                                                             ▒
+    5.65%     5.65%  pool             libc-2.24.so                         [.] re_compile_internal                                                                                                          ▒
+    5.65%     0.00%  pool             [unknown]                            [.] 0x00007f049804d628                                                                                                           ▒
+    5.65%     0.00%  pool             [unknown]                            [.] 0x00007f049804d6a8                                                                                                           ▒
+    5.65%     0.00%  pool             [unknown]                            [.] 0x00007f049804d3d8                                                                                                           ▒
+    5.65%     0.00%  pool             [unknown]                            [.] 0x00007f049804d768                                                                                                           ▒
Cannot load tips.txt file, please install perf!

I’ve checked dmesg, over heating messages (thats why I’m here) and some other messages about MSFT0101:00 which I believe is something todo with the kernel not recognising my bios enabled TPM module. I think that this should be insignificant in this matter.

There is another question about kworker threads suggesting the following as per this thread

$ echo workqueue:workqueue_queue_work > /sys/kernel/debug/tracing/set_event
$ cat /sys/kernel/debug/tracing/trace_pipe > out.txt
(wait a few secs)
^C

but it doesn’t work!

jonathan@melange:~$ sudo mount -t debugfs nodev /sys/kernel/debug
mount: nodev is already mounted or /sys/kernel/debug busy
jonathan@melange:~$ sudo echo workqueue:workqueue_queue_work > /sys/kernel/debug/tracing/set_event
bash: /sys/kernel/debug/tracing/set_event: Permission denied
jonathan@melange:~$ sudo cat /proc/67/stack
[<ffffffffffffffff>] 0xffffffffffffffff

Any ideas?!

Update

Before submitting this question I had been using Kworker, what is it and why is it hogging so much CPU? as reference. So I had tried disabling/uninstalling long running processes such as dropbox, insync (google drive), crashplan, keybase, Variety background, multiload indicator, psensor, guake. (I feel like I have a pretty slick setup most of the time…) but nothing seemed to help.

There had been other questions lurking around suggesting malfunctioning wifi, nvidia drivers or usb drivers. But nothing in my logs were suggesting this either. Somewhat thankful as almost always the solution in those was simply find newer nvidia drivers, update the kernel, or “Deal with it.” My laptop is pretty up to date already, I have no enterprise reason to stay on 16.04 and I already have the nvidia ppa activated, as with the intel drivers, so this wasn’t much help.

Perhaps the kworker was actually the result of the laptop overheating -> cpu throttling + cpu fan management. Not the cause. As suggested by Stop cpu from overheating So I’ve just used some compressed air to clean out the fans (didn’t think this would be a problem on a laptop only 9 months old yet there was actually a bit of dust) and investigating the thermal-conf.xml which suggests that the fan kicks in at 55°C (although still working on what I can do here)

Thinking this may actually be the solution. Will report back soon.

Update 2

So doing the Acer bios update totally ruined everything related to my secureboot setup and corrupted the the efi files so it took me a few days to work out how to regenerate the ubuntu efi keys and the and windows efi keys.

I tried cleaning out the dust, and it definitely helped for the two days until I started with the bios issues.

But the kworker is back (and yes it is the same as far as I can tell). I also have some more information now. I can see that the cpu is not throttling down, but rather staying at the maximum. The fan is running, but the device is only sitting around the 60degree mark, so i wouldn’t call this serious over heating.

The commands from the other thread require raising to the root user, not just using sudo. so sudo su and then getting the stack trace gives the following.

[<ffffffff98a9dcea>] worker_thread+0xca/0x500
[<ffffffff98aa40d8>] kthread+0xd8/0xf0
[<ffffffff992a071f>] ret_from_fork+0x1f/0x40
[<ffffffffffffffff>] 0xffffffffffffffff

Doesn’t look particularly helpful to me.


Get this bounty!!!

#StackBounty: #ubuntu #kernel #linux-kernel #cpu Kworker is at 100% – I think I've tried everything!

Bounty: 100

Thanks for checking this one out.

jonathan@melange:~$ top

top - 05:21:08 up 44 min,  2 users,  load average: 1.21, 1.68, 1.98
Tasks: 351 total,   2 running, 349 sleeping,   0 stopped,   0 zombie
%Cpu(s):  4.3 us, 14.0 sy,  2.1 ni, 70.4 id,  8.9 wa,  0.0 hi,  0.3 si,  0.0 st
GiB Mem :   15.579 total,    0.173 free,    4.141 used,   11.264 buff/cache
GiB Swap:   15.910 total,   15.868 free,    0.042 used.   11.014 avail Mem 

  PID  PPID   UID USER     RUSER    TTY          TIME+  %CPU %MEM S COMMAND                                                                                                                                   
   67     2     0 root     root     ?         22:22.40 100.0  0.0 R kworker/0:1 

The setup – ubuntu 16.10. 4.8.0-41-generic. Modern intel based laptop with Nvidia drivers and not quite perfect wifi. Let me know and I can provide you with whatever info you need. I have these working acceptably and I don’t see any reason to believe these are involved in this issue.

I’ve actually already asked this on askubuntu & a couple of times over at Freenode` #ubuntu over the last week but no one will even respond to my question 🙁

I’ve taken some perf reports with

sudo perf record -a -g sleep 10
sudo perf report

With some results

Samples: 92K of event 'cycles:ppp', Event count (approx.): 58330337004406                                                                                                                                     
  Children      Self  Command          Shared Object                        Symbol                                                                                                                           ◆
+   94.27%     0.00%  swapper          [kernel.kallsyms]                    [k] cpu_startup_entry                                                                                                            ▒
+   94.27%     0.00%  swapper          [kernel.kallsyms]                    [k] start_secondary                                                                                                              ▒
+   77.29%     0.00%  swapper          [kernel.kallsyms]                    [k] schedule_preempt_disabled                                                                                                    ▒
-   77.29%    77.29%  swapper          [kernel.kallsyms]                    [k] __schedule                                                                                                                   ▒
     77.29% start_secondary                                                                                                                                                                                  ▒
        cpu_startup_entry                                                                                                                                                                                    ▒
      - schedule_preempt_disabled                                                                                                                                                                            ▒
         - 77.29% schedule                                                                                                                                                                                   ▒
              __schedule                                                                                                                                                                                     ▒
+   77.29%     0.00%  swapper          [kernel.kallsyms]                    [k] schedule                                                                                                                     ▒
+   16.99%     0.00%  swapper          [kernel.kallsyms]                    [k] call_cpuidle                                                                                                                 ▒
+   16.99%     0.00%  swapper          [kernel.kallsyms]                    [k] cpuidle_enter                                                                                                                ▒
+   16.99%     0.00%  swapper          [kernel.kallsyms]                    [k] cpuidle_enter_state                                                                                                          ▒
-   16.99%    16.99%  swapper          [kernel.kallsyms]                    [k] intel_idle                                                                                                                   ▒
     16.98% start_secondary                                                                                                                                                                                  ▒
        cpu_startup_entry                                                                                                                                                                                    ▒
        call_cpuidle                                                                                                                                                                                         ▒
      - cpuidle_enter                                                                                                                                                                                        ▒
         - 16.98% cpuidle_enter_state                                                                                                                                                                        ▒
              intel_idle                                                                                                                                                                                     ▒
+    5.65%     0.00%  pool             [unknown]                            [.] 0000000000000000                                                                                                             ▒
+    5.65%     5.65%  pool             libc-2.24.so                         [.] re_compile_internal                                                                                                          ▒
+    5.65%     0.00%  pool             [unknown]                            [.] 0x00007f049804d628                                                                                                           ▒
+    5.65%     0.00%  pool             [unknown]                            [.] 0x00007f049804d6a8                                                                                                           ▒
+    5.65%     0.00%  pool             [unknown]                            [.] 0x00007f049804d3d8                                                                                                           ▒
+    5.65%     0.00%  pool             [unknown]                            [.] 0x00007f049804d768                                                                                                           ▒
Cannot load tips.txt file, please install perf!

I’ve checked dmesg, over heating messages (thats why I’m here) and some other messages about MSFT0101:00 which I believe is something todo with the kernel not recognising my bios enabled TPM module. I think that this should be insignificant in this matter.

There is another question about kworker threads suggesting the following as per this thread

$ echo workqueue:workqueue_queue_work > /sys/kernel/debug/tracing/set_event
$ cat /sys/kernel/debug/tracing/trace_pipe > out.txt
(wait a few secs)
^C

but it doesn’t work!

jonathan@melange:~$ sudo mount -t debugfs nodev /sys/kernel/debug
mount: nodev is already mounted or /sys/kernel/debug busy
jonathan@melange:~$ sudo echo workqueue:workqueue_queue_work > /sys/kernel/debug/tracing/set_event
bash: /sys/kernel/debug/tracing/set_event: Permission denied
jonathan@melange:~$ sudo cat /proc/67/stack
[<ffffffffffffffff>] 0xffffffffffffffff

Any ideas?!

Update

Before submitting this question I had been using Kworker, what is it and why is it hogging so much CPU? as reference. So I had tried disabling/uninstalling long running processes such as dropbox, insync (google drive), crashplan, keybase, Variety background, multiload indicator, psensor, guake. (I feel like I have a pretty slick setup most of the time…) but nothing seemed to help.

There had been other questions lurking around suggesting malfunctioning wifi, nvidia drivers or usb drivers. But nothing in my logs were suggesting this either. Somewhat thankful as almost always the solution in those was simply find newer nvidia drivers, update the kernel, or “Deal with it.” My laptop is pretty up to date already, I have no enterprise reason to stay on 16.04 and I already have the nvidia ppa activated, as with the intel drivers, so this wasn’t much help.

Perhaps the kworker was actually the result of the laptop overheating -> cpu throttling + cpu fan management. Not the cause. As suggested by Stop cpu from overheating So I’ve just used some compressed air to clean out the fans (didn’t think this would be a problem on a laptop only 9 months old yet there was actually a bit of dust) and investigating the thermal-conf.xml which suggests that the fan kicks in at 55°C (although still working on what I can do here)

Thinking this may actually be the solution. Will report back soon.

Update 2

So doing the Acer bios update totally ruined everything related to my secureboot setup and corrupted the the efi files so it took me a few days to work out how to regenerate the ubuntu efi keys and the and windows efi keys.

I tried cleaning out the dust, and it definitely helped for the two days until I started with the bios issues.

But the kworker is back (and yes it is the same as far as I can tell). I also have some more information now. I can see that the cpu is not throttling down, but rather staying at the maximum. The fan is running, but the device is only sitting around the 60degree mark, so i wouldn’t call this serious over heating.

The commands from the other thread require raising to the root user, not just using sudo. so sudo su and then getting the stack trace gives the following.

[<ffffffff98a9dcea>] worker_thread+0xca/0x500
[<ffffffff98aa40d8>] kthread+0xd8/0xf0
[<ffffffff992a071f>] ret_from_fork+0x1f/0x40
[<ffffffffffffffff>] 0xffffffffffffffff

Doesn’t look particularly helpful to me.


Get this bounty!!!

#StackBounty: #ubuntu #kernel #linux-kernel #cpu Kworker is at 100% – I think I've tried everything!

Bounty: 100

Thanks for checking this one out.

jonathan@melange:~$ top

top - 05:21:08 up 44 min,  2 users,  load average: 1.21, 1.68, 1.98
Tasks: 351 total,   2 running, 349 sleeping,   0 stopped,   0 zombie
%Cpu(s):  4.3 us, 14.0 sy,  2.1 ni, 70.4 id,  8.9 wa,  0.0 hi,  0.3 si,  0.0 st
GiB Mem :   15.579 total,    0.173 free,    4.141 used,   11.264 buff/cache
GiB Swap:   15.910 total,   15.868 free,    0.042 used.   11.014 avail Mem 

  PID  PPID   UID USER     RUSER    TTY          TIME+  %CPU %MEM S COMMAND                                                                                                                                   
   67     2     0 root     root     ?         22:22.40 100.0  0.0 R kworker/0:1 

The setup – ubuntu 16.10. 4.8.0-41-generic. Modern intel based laptop with Nvidia drivers and not quite perfect wifi. Let me know and I can provide you with whatever info you need. I have these working acceptably and I don’t see any reason to believe these are involved in this issue.

I’ve actually already asked this on askubuntu & a couple of times over at Freenode` #ubuntu over the last week but no one will even respond to my question 🙁

I’ve taken some perf reports with

sudo perf record -a -g sleep 10
sudo perf report

With some results

Samples: 92K of event 'cycles:ppp', Event count (approx.): 58330337004406                                                                                                                                     
  Children      Self  Command          Shared Object                        Symbol                                                                                                                           ◆
+   94.27%     0.00%  swapper          [kernel.kallsyms]                    [k] cpu_startup_entry                                                                                                            ▒
+   94.27%     0.00%  swapper          [kernel.kallsyms]                    [k] start_secondary                                                                                                              ▒
+   77.29%     0.00%  swapper          [kernel.kallsyms]                    [k] schedule_preempt_disabled                                                                                                    ▒
-   77.29%    77.29%  swapper          [kernel.kallsyms]                    [k] __schedule                                                                                                                   ▒
     77.29% start_secondary                                                                                                                                                                                  ▒
        cpu_startup_entry                                                                                                                                                                                    ▒
      - schedule_preempt_disabled                                                                                                                                                                            ▒
         - 77.29% schedule                                                                                                                                                                                   ▒
              __schedule                                                                                                                                                                                     ▒
+   77.29%     0.00%  swapper          [kernel.kallsyms]                    [k] schedule                                                                                                                     ▒
+   16.99%     0.00%  swapper          [kernel.kallsyms]                    [k] call_cpuidle                                                                                                                 ▒
+   16.99%     0.00%  swapper          [kernel.kallsyms]                    [k] cpuidle_enter                                                                                                                ▒
+   16.99%     0.00%  swapper          [kernel.kallsyms]                    [k] cpuidle_enter_state                                                                                                          ▒
-   16.99%    16.99%  swapper          [kernel.kallsyms]                    [k] intel_idle                                                                                                                   ▒
     16.98% start_secondary                                                                                                                                                                                  ▒
        cpu_startup_entry                                                                                                                                                                                    ▒
        call_cpuidle                                                                                                                                                                                         ▒
      - cpuidle_enter                                                                                                                                                                                        ▒
         - 16.98% cpuidle_enter_state                                                                                                                                                                        ▒
              intel_idle                                                                                                                                                                                     ▒
+    5.65%     0.00%  pool             [unknown]                            [.] 0000000000000000                                                                                                             ▒
+    5.65%     5.65%  pool             libc-2.24.so                         [.] re_compile_internal                                                                                                          ▒
+    5.65%     0.00%  pool             [unknown]                            [.] 0x00007f049804d628                                                                                                           ▒
+    5.65%     0.00%  pool             [unknown]                            [.] 0x00007f049804d6a8                                                                                                           ▒
+    5.65%     0.00%  pool             [unknown]                            [.] 0x00007f049804d3d8                                                                                                           ▒
+    5.65%     0.00%  pool             [unknown]                            [.] 0x00007f049804d768                                                                                                           ▒
Cannot load tips.txt file, please install perf!

I’ve checked dmesg, over heating messages (thats why I’m here) and some other messages about MSFT0101:00 which I believe is something todo with the kernel not recognising my bios enabled TPM module. I think that this should be insignificant in this matter.

There is another question about kworker threads suggesting the following as per this thread

$ echo workqueue:workqueue_queue_work > /sys/kernel/debug/tracing/set_event
$ cat /sys/kernel/debug/tracing/trace_pipe > out.txt
(wait a few secs)
^C

but it doesn’t work!

jonathan@melange:~$ sudo mount -t debugfs nodev /sys/kernel/debug
mount: nodev is already mounted or /sys/kernel/debug busy
jonathan@melange:~$ sudo echo workqueue:workqueue_queue_work > /sys/kernel/debug/tracing/set_event
bash: /sys/kernel/debug/tracing/set_event: Permission denied
jonathan@melange:~$ sudo cat /proc/67/stack
[<ffffffffffffffff>] 0xffffffffffffffff

Any ideas?!

Update

Before submitting this question I had been using Kworker, what is it and why is it hogging so much CPU? as reference. So I had tried disabling/uninstalling long running processes such as dropbox, insync (google drive), crashplan, keybase, Variety background, multiload indicator, psensor, guake. (I feel like I have a pretty slick setup most of the time…) but nothing seemed to help.

There had been other questions lurking around suggesting malfunctioning wifi, nvidia drivers or usb drivers. But nothing in my logs were suggesting this either. Somewhat thankful as almost always the solution in those was simply find newer nvidia drivers, update the kernel, or “Deal with it.” My laptop is pretty up to date already, I have no enterprise reason to stay on 16.04 and I already have the nvidia ppa activated, as with the intel drivers, so this wasn’t much help.

Perhaps the kworker was actually the result of the laptop overheating -> cpu throttling + cpu fan management. Not the cause. As suggested by Stop cpu from overheating So I’ve just used some compressed air to clean out the fans (didn’t think this would be a problem on a laptop only 9 months old yet there was actually a bit of dust) and investigating the thermal-conf.xml which suggests that the fan kicks in at 55°C (although still working on what I can do here)

Thinking this may actually be the solution. Will report back soon.

Update 2

So doing the Acer bios update totally ruined everything related to my secureboot setup and corrupted the the efi files so it took me a few days to work out how to regenerate the ubuntu efi keys and the and windows efi keys.

I tried cleaning out the dust, and it definitely helped for the two days until I started with the bios issues.

But the kworker is back (and yes it is the same as far as I can tell). I also have some more information now. I can see that the cpu is not throttling down, but rather staying at the maximum. The fan is running, but the device is only sitting around the 60degree mark, so i wouldn’t call this serious over heating.

The commands from the other thread require raising to the root user, not just using sudo. so sudo su and then getting the stack trace gives the following.

[<ffffffff98a9dcea>] worker_thread+0xca/0x500
[<ffffffff98aa40d8>] kthread+0xd8/0xf0
[<ffffffff992a071f>] ret_from_fork+0x1f/0x40
[<ffffffffffffffff>] 0xffffffffffffffff

Doesn’t look particularly helpful to me.


Get this bounty!!!

#StackBounty: #ubuntu #kernel #linux-kernel #cpu Kworker is at 100% – I think I've tried everything!

Bounty: 100

Thanks for checking this one out.

jonathan@melange:~$ top

top - 05:21:08 up 44 min,  2 users,  load average: 1.21, 1.68, 1.98
Tasks: 351 total,   2 running, 349 sleeping,   0 stopped,   0 zombie
%Cpu(s):  4.3 us, 14.0 sy,  2.1 ni, 70.4 id,  8.9 wa,  0.0 hi,  0.3 si,  0.0 st
GiB Mem :   15.579 total,    0.173 free,    4.141 used,   11.264 buff/cache
GiB Swap:   15.910 total,   15.868 free,    0.042 used.   11.014 avail Mem 

  PID  PPID   UID USER     RUSER    TTY          TIME+  %CPU %MEM S COMMAND                                                                                                                                   
   67     2     0 root     root     ?         22:22.40 100.0  0.0 R kworker/0:1 

The setup – ubuntu 16.10. 4.8.0-41-generic. Modern intel based laptop with Nvidia drivers and not quite perfect wifi. Let me know and I can provide you with whatever info you need. I have these working acceptably and I don’t see any reason to believe these are involved in this issue.

I’ve actually already asked this on askubuntu & a couple of times over at Freenode` #ubuntu over the last week but no one will even respond to my question 🙁

I’ve taken some perf reports with

sudo perf record -a -g sleep 10
sudo perf report

With some results

Samples: 92K of event 'cycles:ppp', Event count (approx.): 58330337004406                                                                                                                                     
  Children      Self  Command          Shared Object                        Symbol                                                                                                                           ◆
+   94.27%     0.00%  swapper          [kernel.kallsyms]                    [k] cpu_startup_entry                                                                                                            ▒
+   94.27%     0.00%  swapper          [kernel.kallsyms]                    [k] start_secondary                                                                                                              ▒
+   77.29%     0.00%  swapper          [kernel.kallsyms]                    [k] schedule_preempt_disabled                                                                                                    ▒
-   77.29%    77.29%  swapper          [kernel.kallsyms]                    [k] __schedule                                                                                                                   ▒
     77.29% start_secondary                                                                                                                                                                                  ▒
        cpu_startup_entry                                                                                                                                                                                    ▒
      - schedule_preempt_disabled                                                                                                                                                                            ▒
         - 77.29% schedule                                                                                                                                                                                   ▒
              __schedule                                                                                                                                                                                     ▒
+   77.29%     0.00%  swapper          [kernel.kallsyms]                    [k] schedule                                                                                                                     ▒
+   16.99%     0.00%  swapper          [kernel.kallsyms]                    [k] call_cpuidle                                                                                                                 ▒
+   16.99%     0.00%  swapper          [kernel.kallsyms]                    [k] cpuidle_enter                                                                                                                ▒
+   16.99%     0.00%  swapper          [kernel.kallsyms]                    [k] cpuidle_enter_state                                                                                                          ▒
-   16.99%    16.99%  swapper          [kernel.kallsyms]                    [k] intel_idle                                                                                                                   ▒
     16.98% start_secondary                                                                                                                                                                                  ▒
        cpu_startup_entry                                                                                                                                                                                    ▒
        call_cpuidle                                                                                                                                                                                         ▒
      - cpuidle_enter                                                                                                                                                                                        ▒
         - 16.98% cpuidle_enter_state                                                                                                                                                                        ▒
              intel_idle                                                                                                                                                                                     ▒
+    5.65%     0.00%  pool             [unknown]                            [.] 0000000000000000                                                                                                             ▒
+    5.65%     5.65%  pool             libc-2.24.so                         [.] re_compile_internal                                                                                                          ▒
+    5.65%     0.00%  pool             [unknown]                            [.] 0x00007f049804d628                                                                                                           ▒
+    5.65%     0.00%  pool             [unknown]                            [.] 0x00007f049804d6a8                                                                                                           ▒
+    5.65%     0.00%  pool             [unknown]                            [.] 0x00007f049804d3d8                                                                                                           ▒
+    5.65%     0.00%  pool             [unknown]                            [.] 0x00007f049804d768                                                                                                           ▒
Cannot load tips.txt file, please install perf!

I’ve checked dmesg, over heating messages (thats why I’m here) and some other messages about MSFT0101:00 which I believe is something todo with the kernel not recognising my bios enabled TPM module. I think that this should be insignificant in this matter.

There is another question about kworker threads suggesting the following as per this thread

$ echo workqueue:workqueue_queue_work > /sys/kernel/debug/tracing/set_event
$ cat /sys/kernel/debug/tracing/trace_pipe > out.txt
(wait a few secs)
^C

but it doesn’t work!

jonathan@melange:~$ sudo mount -t debugfs nodev /sys/kernel/debug
mount: nodev is already mounted or /sys/kernel/debug busy
jonathan@melange:~$ sudo echo workqueue:workqueue_queue_work > /sys/kernel/debug/tracing/set_event
bash: /sys/kernel/debug/tracing/set_event: Permission denied
jonathan@melange:~$ sudo cat /proc/67/stack
[<ffffffffffffffff>] 0xffffffffffffffff

Any ideas?!

Update

Before submitting this question I had been using Kworker, what is it and why is it hogging so much CPU? as reference. So I had tried disabling/uninstalling long running processes such as dropbox, insync (google drive), crashplan, keybase, Variety background, multiload indicator, psensor, guake. (I feel like I have a pretty slick setup most of the time…) but nothing seemed to help.

There had been other questions lurking around suggesting malfunctioning wifi, nvidia drivers or usb drivers. But nothing in my logs were suggesting this either. Somewhat thankful as almost always the solution in those was simply find newer nvidia drivers, update the kernel, or “Deal with it.” My laptop is pretty up to date already, I have no enterprise reason to stay on 16.04 and I already have the nvidia ppa activated, as with the intel drivers, so this wasn’t much help.

Perhaps the kworker was actually the result of the laptop overheating -> cpu throttling + cpu fan management. Not the cause. As suggested by Stop cpu from overheating So I’ve just used some compressed air to clean out the fans (didn’t think this would be a problem on a laptop only 9 months old yet there was actually a bit of dust) and investigating the thermal-conf.xml which suggests that the fan kicks in at 55°C (although still working on what I can do here)

Thinking this may actually be the solution. Will report back soon.

Update 2

So doing the Acer bios update totally ruined everything related to my secureboot setup and corrupted the the efi files so it took me a few days to work out how to regenerate the ubuntu efi keys and the and windows efi keys.

I tried cleaning out the dust, and it definitely helped for the two days until I started with the bios issues.

But the kworker is back (and yes it is the same as far as I can tell). I also have some more information now. I can see that the cpu is not throttling down, but rather staying at the maximum. The fan is running, but the device is only sitting around the 60degree mark, so i wouldn’t call this serious over heating.

The commands from the other thread require raising to the root user, not just using sudo. so sudo su and then getting the stack trace gives the following.

[<ffffffff98a9dcea>] worker_thread+0xca/0x500
[<ffffffff98aa40d8>] kthread+0xd8/0xf0
[<ffffffff992a071f>] ret_from_fork+0x1f/0x40
[<ffffffffffffffff>] 0xffffffffffffffff

Doesn’t look particularly helpful to me.


Get this bounty!!!

#StackBounty: #ubuntu #kernel #linux-kernel #cpu Kworker is at 100% – I think I've tried everything!

Bounty: 100

Thanks for checking this one out.

jonathan@melange:~$ top

top - 05:21:08 up 44 min,  2 users,  load average: 1.21, 1.68, 1.98
Tasks: 351 total,   2 running, 349 sleeping,   0 stopped,   0 zombie
%Cpu(s):  4.3 us, 14.0 sy,  2.1 ni, 70.4 id,  8.9 wa,  0.0 hi,  0.3 si,  0.0 st
GiB Mem :   15.579 total,    0.173 free,    4.141 used,   11.264 buff/cache
GiB Swap:   15.910 total,   15.868 free,    0.042 used.   11.014 avail Mem 

  PID  PPID   UID USER     RUSER    TTY          TIME+  %CPU %MEM S COMMAND                                                                                                                                   
   67     2     0 root     root     ?         22:22.40 100.0  0.0 R kworker/0:1 

The setup – ubuntu 16.10. 4.8.0-41-generic. Modern intel based laptop with Nvidia drivers and not quite perfect wifi. Let me know and I can provide you with whatever info you need. I have these working acceptably and I don’t see any reason to believe these are involved in this issue.

I’ve actually already asked this on askubuntu & a couple of times over at Freenode` #ubuntu over the last week but no one will even respond to my question 🙁

I’ve taken some perf reports with

sudo perf record -a -g sleep 10
sudo perf report

With some results

Samples: 92K of event 'cycles:ppp', Event count (approx.): 58330337004406                                                                                                                                     
  Children      Self  Command          Shared Object                        Symbol                                                                                                                           ◆
+   94.27%     0.00%  swapper          [kernel.kallsyms]                    [k] cpu_startup_entry                                                                                                            ▒
+   94.27%     0.00%  swapper          [kernel.kallsyms]                    [k] start_secondary                                                                                                              ▒
+   77.29%     0.00%  swapper          [kernel.kallsyms]                    [k] schedule_preempt_disabled                                                                                                    ▒
-   77.29%    77.29%  swapper          [kernel.kallsyms]                    [k] __schedule                                                                                                                   ▒
     77.29% start_secondary                                                                                                                                                                                  ▒
        cpu_startup_entry                                                                                                                                                                                    ▒
      - schedule_preempt_disabled                                                                                                                                                                            ▒
         - 77.29% schedule                                                                                                                                                                                   ▒
              __schedule                                                                                                                                                                                     ▒
+   77.29%     0.00%  swapper          [kernel.kallsyms]                    [k] schedule                                                                                                                     ▒
+   16.99%     0.00%  swapper          [kernel.kallsyms]                    [k] call_cpuidle                                                                                                                 ▒
+   16.99%     0.00%  swapper          [kernel.kallsyms]                    [k] cpuidle_enter                                                                                                                ▒
+   16.99%     0.00%  swapper          [kernel.kallsyms]                    [k] cpuidle_enter_state                                                                                                          ▒
-   16.99%    16.99%  swapper          [kernel.kallsyms]                    [k] intel_idle                                                                                                                   ▒
     16.98% start_secondary                                                                                                                                                                                  ▒
        cpu_startup_entry                                                                                                                                                                                    ▒
        call_cpuidle                                                                                                                                                                                         ▒
      - cpuidle_enter                                                                                                                                                                                        ▒
         - 16.98% cpuidle_enter_state                                                                                                                                                                        ▒
              intel_idle                                                                                                                                                                                     ▒
+    5.65%     0.00%  pool             [unknown]                            [.] 0000000000000000                                                                                                             ▒
+    5.65%     5.65%  pool             libc-2.24.so                         [.] re_compile_internal                                                                                                          ▒
+    5.65%     0.00%  pool             [unknown]                            [.] 0x00007f049804d628                                                                                                           ▒
+    5.65%     0.00%  pool             [unknown]                            [.] 0x00007f049804d6a8                                                                                                           ▒
+    5.65%     0.00%  pool             [unknown]                            [.] 0x00007f049804d3d8                                                                                                           ▒
+    5.65%     0.00%  pool             [unknown]                            [.] 0x00007f049804d768                                                                                                           ▒
Cannot load tips.txt file, please install perf!

I’ve checked dmesg, over heating messages (thats why I’m here) and some other messages about MSFT0101:00 which I believe is something todo with the kernel not recognising my bios enabled TPM module. I think that this should be insignificant in this matter.

There is another question about kworker threads suggesting the following as per this thread

$ echo workqueue:workqueue_queue_work > /sys/kernel/debug/tracing/set_event
$ cat /sys/kernel/debug/tracing/trace_pipe > out.txt
(wait a few secs)
^C

but it doesn’t work!

jonathan@melange:~$ sudo mount -t debugfs nodev /sys/kernel/debug
mount: nodev is already mounted or /sys/kernel/debug busy
jonathan@melange:~$ sudo echo workqueue:workqueue_queue_work > /sys/kernel/debug/tracing/set_event
bash: /sys/kernel/debug/tracing/set_event: Permission denied
jonathan@melange:~$ sudo cat /proc/67/stack
[<ffffffffffffffff>] 0xffffffffffffffff

Any ideas?!

Update

Before submitting this question I had been using Kworker, what is it and why is it hogging so much CPU? as reference. So I had tried disabling/uninstalling long running processes such as dropbox, insync (google drive), crashplan, keybase, Variety background, multiload indicator, psensor, guake. (I feel like I have a pretty slick setup most of the time…) but nothing seemed to help.

There had been other questions lurking around suggesting malfunctioning wifi, nvidia drivers or usb drivers. But nothing in my logs were suggesting this either. Somewhat thankful as almost always the solution in those was simply find newer nvidia drivers, update the kernel, or “Deal with it.” My laptop is pretty up to date already, I have no enterprise reason to stay on 16.04 and I already have the nvidia ppa activated, as with the intel drivers, so this wasn’t much help.

Perhaps the kworker was actually the result of the laptop overheating -> cpu throttling + cpu fan management. Not the cause. As suggested by Stop cpu from overheating So I’ve just used some compressed air to clean out the fans (didn’t think this would be a problem on a laptop only 9 months old yet there was actually a bit of dust) and investigating the thermal-conf.xml which suggests that the fan kicks in at 55°C (although still working on what I can do here)

Thinking this may actually be the solution. Will report back soon.

Update 2

So doing the Acer bios update totally ruined everything related to my secureboot setup and corrupted the the efi files so it took me a few days to work out how to regenerate the ubuntu efi keys and the and windows efi keys.

I tried cleaning out the dust, and it definitely helped for the two days until I started with the bios issues.

But the kworker is back (and yes it is the same as far as I can tell). I also have some more information now. I can see that the cpu is not throttling down, but rather staying at the maximum. The fan is running, but the device is only sitting around the 60degree mark, so i wouldn’t call this serious over heating.

The commands from the other thread require raising to the root user, not just using sudo. so sudo su and then getting the stack trace gives the following.

[<ffffffff98a9dcea>] worker_thread+0xca/0x500
[<ffffffff98aa40d8>] kthread+0xd8/0xf0
[<ffffffff992a071f>] ret_from_fork+0x1f/0x40
[<ffffffffffffffff>] 0xffffffffffffffff

Doesn’t look particularly helpful to me.


Get this bounty!!!

#StackBounty: #ubuntu #kernel #linux-kernel #cpu Kworker is at 100% – I think I've tried everything!

Bounty: 100

Thanks for checking this one out.

jonathan@melange:~$ top

top - 05:21:08 up 44 min,  2 users,  load average: 1.21, 1.68, 1.98
Tasks: 351 total,   2 running, 349 sleeping,   0 stopped,   0 zombie
%Cpu(s):  4.3 us, 14.0 sy,  2.1 ni, 70.4 id,  8.9 wa,  0.0 hi,  0.3 si,  0.0 st
GiB Mem :   15.579 total,    0.173 free,    4.141 used,   11.264 buff/cache
GiB Swap:   15.910 total,   15.868 free,    0.042 used.   11.014 avail Mem 

  PID  PPID   UID USER     RUSER    TTY          TIME+  %CPU %MEM S COMMAND                                                                                                                                   
   67     2     0 root     root     ?         22:22.40 100.0  0.0 R kworker/0:1 

The setup – ubuntu 16.10. 4.8.0-41-generic. Modern intel based laptop with Nvidia drivers and not quite perfect wifi. Let me know and I can provide you with whatever info you need. I have these working acceptably and I don’t see any reason to believe these are involved in this issue.

I’ve actually already asked this on askubuntu & a couple of times over at Freenode` #ubuntu over the last week but no one will even respond to my question 🙁

I’ve taken some perf reports with

sudo perf record -a -g sleep 10
sudo perf report

With some results

Samples: 92K of event 'cycles:ppp', Event count (approx.): 58330337004406                                                                                                                                     
  Children      Self  Command          Shared Object                        Symbol                                                                                                                           ◆
+   94.27%     0.00%  swapper          [kernel.kallsyms]                    [k] cpu_startup_entry                                                                                                            ▒
+   94.27%     0.00%  swapper          [kernel.kallsyms]                    [k] start_secondary                                                                                                              ▒
+   77.29%     0.00%  swapper          [kernel.kallsyms]                    [k] schedule_preempt_disabled                                                                                                    ▒
-   77.29%    77.29%  swapper          [kernel.kallsyms]                    [k] __schedule                                                                                                                   ▒
     77.29% start_secondary                                                                                                                                                                                  ▒
        cpu_startup_entry                                                                                                                                                                                    ▒
      - schedule_preempt_disabled                                                                                                                                                                            ▒
         - 77.29% schedule                                                                                                                                                                                   ▒
              __schedule                                                                                                                                                                                     ▒
+   77.29%     0.00%  swapper          [kernel.kallsyms]                    [k] schedule                                                                                                                     ▒
+   16.99%     0.00%  swapper          [kernel.kallsyms]                    [k] call_cpuidle                                                                                                                 ▒
+   16.99%     0.00%  swapper          [kernel.kallsyms]                    [k] cpuidle_enter                                                                                                                ▒
+   16.99%     0.00%  swapper          [kernel.kallsyms]                    [k] cpuidle_enter_state                                                                                                          ▒
-   16.99%    16.99%  swapper          [kernel.kallsyms]                    [k] intel_idle                                                                                                                   ▒
     16.98% start_secondary                                                                                                                                                                                  ▒
        cpu_startup_entry                                                                                                                                                                                    ▒
        call_cpuidle                                                                                                                                                                                         ▒
      - cpuidle_enter                                                                                                                                                                                        ▒
         - 16.98% cpuidle_enter_state                                                                                                                                                                        ▒
              intel_idle                                                                                                                                                                                     ▒
+    5.65%     0.00%  pool             [unknown]                            [.] 0000000000000000                                                                                                             ▒
+    5.65%     5.65%  pool             libc-2.24.so                         [.] re_compile_internal                                                                                                          ▒
+    5.65%     0.00%  pool             [unknown]                            [.] 0x00007f049804d628                                                                                                           ▒
+    5.65%     0.00%  pool             [unknown]                            [.] 0x00007f049804d6a8                                                                                                           ▒
+    5.65%     0.00%  pool             [unknown]                            [.] 0x00007f049804d3d8                                                                                                           ▒
+    5.65%     0.00%  pool             [unknown]                            [.] 0x00007f049804d768                                                                                                           ▒
Cannot load tips.txt file, please install perf!

I’ve checked dmesg, over heating messages (thats why I’m here) and some other messages about MSFT0101:00 which I believe is something todo with the kernel not recognising my bios enabled TPM module. I think that this should be insignificant in this matter.

There is another question about kworker threads suggesting the following as per this thread

$ echo workqueue:workqueue_queue_work > /sys/kernel/debug/tracing/set_event
$ cat /sys/kernel/debug/tracing/trace_pipe > out.txt
(wait a few secs)
^C

but it doesn’t work!

jonathan@melange:~$ sudo mount -t debugfs nodev /sys/kernel/debug
mount: nodev is already mounted or /sys/kernel/debug busy
jonathan@melange:~$ sudo echo workqueue:workqueue_queue_work > /sys/kernel/debug/tracing/set_event
bash: /sys/kernel/debug/tracing/set_event: Permission denied
jonathan@melange:~$ sudo cat /proc/67/stack
[<ffffffffffffffff>] 0xffffffffffffffff

Any ideas?!

Update

Before submitting this question I had been using Kworker, what is it and why is it hogging so much CPU? as reference. So I had tried disabling/uninstalling long running processes such as dropbox, insync (google drive), crashplan, keybase, Variety background, multiload indicator, psensor, guake. (I feel like I have a pretty slick setup most of the time…) but nothing seemed to help.

There had been other questions lurking around suggesting malfunctioning wifi, nvidia drivers or usb drivers. But nothing in my logs were suggesting this either. Somewhat thankful as almost always the solution in those was simply find newer nvidia drivers, update the kernel, or “Deal with it.” My laptop is pretty up to date already, I have no enterprise reason to stay on 16.04 and I already have the nvidia ppa activated, as with the intel drivers, so this wasn’t much help.

Perhaps the kworker was actually the result of the laptop overheating -> cpu throttling + cpu fan management. Not the cause. As suggested by Stop cpu from overheating So I’ve just used some compressed air to clean out the fans (didn’t think this would be a problem on a laptop only 9 months old yet there was actually a bit of dust) and investigating the thermal-conf.xml which suggests that the fan kicks in at 55°C (although still working on what I can do here)

Thinking this may actually be the solution. Will report back soon.

Update 2

So doing the Acer bios update totally ruined everything related to my secureboot setup and corrupted the the efi files so it took me a few days to work out how to regenerate the ubuntu efi keys and the and windows efi keys.

I tried cleaning out the dust, and it definitely helped for the two days until I started with the bios issues.

But the kworker is back (and yes it is the same as far as I can tell). I also have some more information now. I can see that the cpu is not throttling down, but rather staying at the maximum. The fan is running, but the device is only sitting around the 60degree mark, so i wouldn’t call this serious over heating.

The commands from the other thread require raising to the root user, not just using sudo. so sudo su and then getting the stack trace gives the following.

[<ffffffff98a9dcea>] worker_thread+0xca/0x500
[<ffffffff98aa40d8>] kthread+0xd8/0xf0
[<ffffffff992a071f>] ret_from_fork+0x1f/0x40
[<ffffffffffffffff>] 0xffffffffffffffff

Doesn’t look particularly helpful to me.


Get this bounty!!!