https://marktaguiad.dev/tags/troubleshoot/ Recent content in Troubleshoot on marktaguiad.dev Hugo en-us marktaguiad@marktaguiad.dev (Mark Taguiad) marktaguiad@marktaguiad.dev (Mark Taguiad) marktaguiad.dev Wed, 13 Aug 2025 00:00:00 +0000 https://marktaguiad.dev/post/linux-troubleshooting/ Wed, 13 Aug 2025 00:00:00 +0000marktaguiad@marktaguiad.dev (Mark Taguiad) https://marktaguiad.dev/post/linux-troubleshooting/ <h1 id="table-of-contents">Table of Contents</h1> <nav id="TableOfContents"> <ol> <li><a href="#cpu-and-processes">CPU and Processes</a> <ol> <li><a href="#understanding-top-command">Understanding top command</a> <ol> <li><a href="#header-line-system-status">Header line (system status)</a></li> <li><a href="#tasks-process-summary">Tasks (process summary)</a></li> <li><a href="#cpu-usage">CPU usage</a></li> <li><a href="#memory-usage">Memory usage</a></li> <li><a href="#process-list">Process list</a></li> <li><a href="#some-top-command---while-running">Some top command - while running</a></li> </ol> </li> <li><a href="#process-states">Process States</a></li> <li><a href="#top-cpu-user">Top CPU User</a> <ol> <li><a href="#status">Status</a></li> <li><a href="#multi-threaded">Multi-threaded</a></li> </ol> </li> <li><a href="#pidstat">PIDSTAT</a></li> <li><a href="#perf">perf</a></li> <li><a href="#signals">Signals</a> <ol> <li><a href="#sigstop">SIGSTOP</a></li> <li><a href="#sigcont">SIGCONT</a></li> <li><a href="#sigint">SIGINT</a></li> <li><a href="#sigterm-and-sigquit">SIGTERM and SIGQUIT</a></li> <li><a href="#sigkill">SIGKILL</a></li> </ol> </li> <li><a href="#cpu--io">CPU &amp; I/O</a> <ol> <li><a href="#vmstat">vmstat</a></li> </ol> </li> <li><a href="#disk-level-io">Disk level I/O</a> <ol> <li><a href="#iostat">iostat</a></li> </ol> </li> <li><a href="#identify-processes-stuck-in-io">Identify Processes Stuck in I/O</a></li> <li><a href="#priority-and-nice-values">Priority and Nice Values</a> <ol> <li><a href="#status-1">Status</a></li> <li><a href="#lower-priority">Lower priority.</a></li> <li><a href="#raise-priority">Raise priority</a></li> </ol> </li> <li><a href="#zombie-processes">Zombie Processes</a></li> <li><a href="#ulimit">ulimit</a> <ol> <li><a href="#soft-limits">Soft Limits</a></li> <li><a href="#hard-limits">Hard Limits</a></li> <li><a href="#process-number">Process Number</a></li> <li><a href="#file-size">File Size</a></li> <li><a href="#virtual-memory">Virtual Memory</a></li> <li><a href="#opened-files">Opened Files</a></li> <li><a href="#other-limit">Other limit</a></li> </ol> </li> <li><a href="#finding-pid-using-port">Finding PID using Port</a> <ol> <li><a href="#netstat">netstat</a></li> <li><a href="#ss">ss</a></li> <li><a href="#lsof">lsof</a></li> </ol> </li> <li><a href="#get-process-id">Get Process ID</a> <ol> <li><a href="#show-background-process">Show Background Process</a></li> <li><a href="#by-application-name">By Application Name</a></li> <li><a href="#by-file">By File</a></li> </ol> </li> <li><a href="#killing-a-running-application">Killing a running application</a> <ol> <li><a href="#kill-by-application-name">Kill by application name</a></li> <li><a href="#terminate-the-process-using-its-pid">Terminate the process using its PID</a></li> <li><a href="#forceful-termination">Forceful Termination</a></li> <li><a href="#kill-all-application-run-by-user">Kill all application run by user</a></li> <li><a href="#kill-process-tree">Kill process tree.</a></li> <li><a href="#kill-by-port">Kill by Port</a></li> </ol> </li> <li><a href="#finding-who-killed-the-process">Finding who killed the process</a> <ol> <li><a href="#kernel">Kernel</a></li> <li><a href="#oomkill">OOMKILL</a></li> </ol> </li> </ol> </li> <li><a href="#memory">Memory</a> <ol> <li><a href="#usage">Usage</a></li> <li><a href="#hogs">Hogs</a> <ol> <li><a href="#smem">smem</a></li> </ol> </li> <li><a href="#leak">Leak</a></li> <li><a href="#oom-score">OOM Score</a></li> <li><a href="#swap">Swap</a></li> </ol> </li> <li><a href="#disk">Disk</a> <ol> <li><a href="#status-2">Status</a></li> <li><a href="#disk-io-slowness--system-freezing">Disk I/O slowness / system freezing</a></li> <li><a href="#disk-health">Disk health</a> <ol> <li><a href="#smartctl">smartctl</a></li> <li><a href="#fsck">fsck</a></li> </ol> </li> <li><a href="#trim">trim</a></li> <li><a href="#logs">Logs</a> <ol> <li><a href="#logrotate">logrotate</a></li> <li><a href="#journald">journald</a></li> </ol> </li> <li><a href="#docker-system-usage">Docker System Usage</a> <ol> <li><a href="#imagebuildervolume">image/builder/volume</a></li> <li><a href="#logs-1">logs</a></li> </ol> </li> </ol> </li> <li><a href="#docker">Docker</a> <ol> <li><a href="#container-overloading-the-system">Container overloading the system</a></li> </ol> </li> </ol> </nav> <h3 id="cpu-and-processes">CPU and Processes</h3> <h4 id="understanding-top-command">Understanding top command</h4> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="ln"> 1</span><span class="cl">$ top </span></span><span class="line"><span class="ln"> 2</span><span class="cl"> </span></span><span class="line"><span class="ln"> 3</span><span class="cl">top - 18:47:04 up 22:45, <span class="m">2</span> users, load average: 0.46, 0.74, 0.85 </span></span><span class="line"><span class="ln"> 4</span><span class="cl">Tasks: <span class="m">410</span> total, <span class="m">1</span> running, <span class="m">407</span> sleeping, <span class="m">0</span> stopped, <span class="m">2</span> zombie </span></span><span class="line"><span class="ln"> 5</span><span class="cl">%Cpu<span class="o">(</span>s<span class="o">)</span>: 7.1 us, 2.4 sy, 4.1 ni, 86.2 id, 0.1 wa, 0.0 hi, 0.0 si, 0.0 st </span></span><span class="line"><span class="ln"> 6</span><span class="cl">MiB Mem : 47961.0 total, 32654.2 free, 11221.4 used, 7702.5 buff/cache </span></span><span class="line"><span class="ln"> 7</span><span class="cl">MiB Swap: 20479.5 total, 20479.4 free, 0.1 used. 36739.6 avail Mem </span></span><span class="line"><span class="ln"> 8</span><span class="cl"> </span></span><span class="line"><span class="ln"> 9</span><span class="cl"> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND </span></span><span class="line"><span class="ln">10</span><span class="cl"> <span class="m">22671</span> mcbtagu+ <span class="m">14</span> -6 12.2g <span class="m">968172</span> <span class="m">437988</span> S 47.2 2.0 171:20.29 firefox-bin </span></span></code></pre></div><h5 id="header-line-system-status">Header line (system status)</h5> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="ln">1</span><span class="cl">top - 18:47:04 up 22:45, <span class="m">2</span> users, load average: 0.46, 0.74, 0.85 </span></span></code></pre></div><ul> <li><strong>18:47:04</strong> - current time</li> <li><strong>up 22:45</strong> - machine has been running for 22 hours 45 minutes</li> <li><strong>2 users</strong> - two logged-in users</li> <li><strong>load average: 0.46, 0.74, 0.85</strong> <ul> <li>Average number of runnable processes over: <ul> <li>last 1 min</li> <li>last 5 min</li> <li>last 15 min</li> </ul> </li> <li>On a multi-core system, these numbers are low - system is not stressed</li> </ul> </li> </ul> <h5 id="tasks-process-summary">Tasks (process summary)</h5> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-fallback" data-lang="fallback"><span class="line"><span class="ln">1</span><span class="cl">Tasks: 410 total, 1 running, 407 sleeping, 0 stopped, 2 zombie </span></span></code></pre></div><ul> <li><strong>410</strong> total - total processes/threads</li> <li><strong>1</strong> running - only one actively using CPU right now</li> <li><strong>407</strong> sleeping - normal; most processes wait for work</li> <li><strong>2</strong> zombie - processes that exited but haven’t been cleaned up</li> </ul> <h5 id="cpu-usage">CPU usage</h5> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-fallback" data-lang="fallback"><span class="line"><span class="ln">1</span><span class="cl">%Cpu(s): 7.1 us, 2.4 sy, 4.1 ni, 86.2 id, 0.1 wa, 0.0 hi, 0.0 si, 0.0 st </span></span></code></pre></div><ul> <li><strong>us (7.1%)</strong> - user programs</li> <li><strong>sy (2.4%)</strong> - kernel/system work</li> <li><strong>ni (4.1%)</strong> - low-priority (nice) processes</li> <li><strong>id (86.2%)</strong> - idle CPU</li> <li><strong>wa (0.1%)</strong> - waiting on disk I/O</li> <li><strong>hi / si</strong> - hardware/software interrupts</li> <li><strong>st</strong> - stolen by hypervisor (VMs)</li> </ul> <h5 id="memory-usage">Memory usage</h5> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-fallback" data-lang="fallback"><span class="line"><span class="ln">1</span><span class="cl">MiB Mem : 47961.0 total, 32654.2 free, 11221.4 used, 7702.5 buff/cache </span></span></code></pre></div><ul> <li><strong>Total</strong>: ~48 GB</li> <li><strong>Free</strong>: ~32 GB (very high)</li> <li><strong>Used</strong>: ~11 GB (actual app usage)</li> <li>b<strong>uff/cache</strong>: ~7.7 GB (filesystem cache, reclaimable)</li> </ul> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-fallback" data-lang="fallback"><span class="line"><span class="ln">1</span><span class="cl">MiB Swap: 20479.5 total, 20479.4 free, 0.1 used. 36739.6 avail Mem </span></span></code></pre></div><ul> <li>Swap almost unused - excellent</li> <li>avail Mem ~36 GB - memory pressure is basically zero</li> </ul> <h5 id="process-list">Process list</h5> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-fallback" data-lang="fallback"><span class="line"><span class="ln">1</span><span class="cl">PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND </span></span><span class="line"><span class="ln">2</span><span class="cl">22671 mcbtag+ 14 -6 12.2g 968172 437988 S 47.2 2.0 171:20.29 </span></span></code></pre></div><p><strong>Identity</strong></p>