major.io https://major.io/ en-us Major Hayden (C) 2019 2019-04-05 00:00:00 +0000 UTC Inspecting OpenShift cgroups from inside the pod https://major.io/2019/04/05/inspecting-openshift-cgroups-from-inside-the-pod/ Fri, 05 Apr 2019 00:00:00 UTC Major Hayden https://major.io/2019/04/05/inspecting-openshift-cgroups-from-inside-the-pod/ <p><img src="../images/2019-04-05-inspecting-cgroups.jpg" alt="walking_through_rock_valley" /></p> <p>My team at Red Hat builds a lot of kernels in OpenShift pods as part of our work with the <a href="https://cki-project.org/">Continuous Kernel Integration (CKI)</a> project. We have lots of different pod sizes depending on the type of work we are doing and our GitLab runners spawn these pods based on the tags in our GitLab CI pipeline.</p> <h2 id="compiling-with-make">Compiling with <code>make</code></h2> <p>When you compile a large software project, such as the Linux kernel, you can use multiple CPU cores to speed up the build. GNU&rsquo;s <code>make</code> does this with the <code>-j</code> argument. Running <code>make</code> with <code>-j10</code> means that you want to run 10 jobs while compiling. This would keep 10 CPU cores busy.</p> <p>Setting the number too high causes more contention from the CPU and can reduce performance. Setting the number too low means that you are spending more time compiling than you would if you used all of your CPU cores.</p> <p>Every once in a while, we adjusted our runners to use a different amount of CPUs or memory and then we had to adjust our pipeline to reflect the new CPU count. This was time consuming and error prone.</p> <p>Many people just use <code>nproc</code> to determine the CPU core count. It works well with make:</p> <div class="highlight"><pre style="color:#93a1a1;background-color:#002b36;-moz-tab-size:4;-o-tab-size:4;tab-size:4">make -j$(nproc)</pre></div> <h2 id="problems-with-containers">Problems with containers</h2> <p>The handy <code>nproc</code> doesn&rsquo;t work well for OpenShift. If you start a pod on OpenShift and limit it to a single CPU core, <code>nproc</code> tells you something very wrong:</p> <div class="highlight"><pre style="color:#93a1a1;background-color:#002b36;-moz-tab-size:4;-o-tab-size:4;tab-size:4">$ nproc 32</pre></div> <p>We applied the single CPU limit with OpenShift, so what&rsquo;s the problem? The issue is how <code>nproc</code> looks for CPUs. Here&rsquo;s a snippet of <code>strace</code> output:</p> <div class="highlight"><pre style="color:#93a1a1;background-color:#002b36;-moz-tab-size:4;-o-tab-size:4;tab-size:4">sched_getaffinity(0, 128, [0, 1, 2, 3, 4, 5]) = 8 fstat(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(0x88, 0x6), ...}) = 0 write(1, &#34;6\n&#34;, 26 ) = 2</pre></div> <p>The <a href="https://linux.die.net/man/2/sched_getaffinity">sched_getaffinity</a> syscall looks to see which CPUs are allowed to run the process and returns a count of those. OpenShift doesn&rsquo;t prevent us from seeing the CPUs of the underlying system (the VM or bare metal host underneath our containers), but it uses cgroups to limit how much CPU time we can use.</p> <h2 id="reading-cgroups">Reading cgroups</h2> <p>Getting cgroup data is easy! Just change into the <code>/sys/fs/cgroup/</code> directory and look around:</p> <div class="highlight"><pre style="color:#93a1a1;background-color:#002b36;-moz-tab-size:4;-o-tab-size:4;tab-size:4">$ cd /sys/fs/cgroup/ $ ls -al cpu/ ls: cannot open directory &#39;cpu/&#39;: Permission denied</pre></div> <p><strong>Ouch.</strong> OpenShift makes this a little more challenging. We&rsquo;re not allowed to wander around in the land of cgroups without a map to exactly what we want.</p> <p>My Fedora workstation shows a bunch of CPU cgroup settings:</p> <div class="highlight"><pre style="color:#93a1a1;background-color:#002b36;-moz-tab-size:4;-o-tab-size:4;tab-size:4">$ ls -al /sys/fs/cgroup/cpu/ total 0 dr-xr-xr-x. 2 root root 0 Apr 5 01:40 . drwxr-xr-x. 14 root root 360 Apr 5 01:40 .. -rw-r--r--. 1 root root 0 Apr 5 13:08 cgroup.clone_children -rw-r--r--. 1 root root 0 Apr 5 01:40 cgroup.procs -r--r--r--. 1 root root 0 Apr 5 13:08 cgroup.sane_behavior -r--r--r--. 1 root root 0 Apr 5 13:08 cpuacct.stat -rw-r--r--. 1 root root 0 Apr 5 13:08 cpuacct.usage -r--r--r--. 1 root root 0 Apr 5 13:08 cpuacct.usage_all -r--r--r--. 1 root root 0 Apr 5 13:08 cpuacct.usage_percpu -r--r--r--. 1 root root 0 Apr 5 13:08 cpuacct.usage_percpu_sys -r--r--r--. 1 root root 0 Apr 5 13:08 cpuacct.usage_percpu_user -r--r--r--. 1 root root 0 Apr 5 13:08 cpuacct.usage_sys -r--r--r--. 1 root root 0 Apr 5 13:08 cpuacct.usage_user -rw-r--r--. 1 root root 0 Apr 5 09:10 cpu.cfs_period_us -rw-r--r--. 1 root root 0 Apr 5 13:08 cpu.cfs_quota_us -rw-r--r--. 1 root root 0 Apr 5 09:10 cpu.shares -r--r--r--. 1 root root 0 Apr 5 13:08 cpu.stat -rw-r--r--. 1 root root 0 Apr 5 13:08 notify_on_release -rw-r--r--. 1 root root 0 Apr 5 13:08 release_agent -rw-r--r--. 1 root root 0 Apr 5 13:08 tasks</pre></div> <p>OpenShift uses the <a href="https://en.wikipedia.org/wiki/Completely_Fair_Scheduler">Completely Fair Scheduler (CFS)</a> to limit CPU time. Here&rsquo;s a quick excerpt from the <a href="https://www.kernel.org/doc/Documentation/scheduler/sched-bwc.txt">kernel documentation</a>:</p> <blockquote> <p>Quota and period are managed within the cpu subsystem via cgroupfs.</p> <p>cpu.cfs_quota_us: the total available run-time within a period (in microseconds) cpu.cfs_period_us: the length of a period (in microseconds) cpu.stat: exports throttling statistics [explained further below]</p> <p>The default values are: cpu.cfs_period_us=100ms cpu.cfs_quota=-1</p> <p>A value of -1 for cpu.cfs_quota_us indicates that the group does not have any bandwidth restriction in place, such a group is described as an unconstrained bandwidth group. This represents the traditional work-conserving behavior for CFS.</p> <p>Writing any (valid) positive value(s) will enact the specified bandwidth limit. The minimum quota allowed for the quota or period is 1ms. There is also an upper bound on the period length of 1s. Additional restrictions exist when bandwidth limits are used in a hierarchical fashion, these are explained in more detail below.</p> <p>Writing any negative value to cpu.cfs_quota_us will remove the bandwidth limit and return the group to an unconstrained state once more.</p> <p>Any updates to a group&rsquo;s bandwidth specification will result in it becoming unthrottled if it is in a constrained state.</p> </blockquote> <p>Let&rsquo;s see if inspecting <code>cpu.cfs_quota_us</code> can help us:</p> <div class="highlight"><pre style="color:#93a1a1;background-color:#002b36;-moz-tab-size:4;-o-tab-size:4;tab-size:4">$ cat /sys/fs/cgroup/cpu/cpu.cfs_quota_us 10000</pre></div> <p>Now we&rsquo;re getting somewhere. But what does <em>10000</em> mean here? OpenShift operates on the concept of <em>millicores</em> of CPU time, or <sup>1</sup>&frasl;<sub>1000</sub> of a CPU. 500 millicores is half a CPU and 1000 millicores is a whole CPU.</p> <p>The pod in this example is assigned 100 millicores. Now we know that we can take the output of <code>/sys/fs/cgroup/cpu/cpu.cfs_quota_us</code>, divide by 100, and get our millicores.</p> <p>We can make a script like this:</p> <div class="highlight"><pre style="color:#93a1a1;background-color:#002b36;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash"><span style="color:#268bd2">CFS_QUOTA</span><span style="color:#719e07">=</span><span style="color:#719e07">$(</span>cat /sys/fs/cgroup/cpu/cpu.cfs_quota_us<span style="color:#719e07">)</span> <span style="color:#719e07">if</span> <span style="color:#719e07">[</span> <span style="color:#268bd2">$CFS_QUOTA</span> -lt <span style="color:#2aa198">100000</span> <span style="color:#719e07">]</span>; <span style="color:#719e07">then</span> <span style="color:#268bd2">CPUS_AVAILABLE</span><span style="color:#719e07">=</span><span style="color:#2aa198">1</span> <span style="color:#719e07">else</span> <span style="color:#268bd2">CPUS_AVAILABLE</span><span style="color:#719e07">=</span><span style="color:#719e07">$(</span>expr <span style="color:#2aa198">${</span><span style="color:#268bd2">CFS_QUOTA</span><span style="color:#2aa198">}</span> / <span style="color:#2aa198">100</span> / <span style="color:#2aa198">1000</span><span style="color:#719e07">)</span> <span style="color:#719e07">fi</span> <span style="color:#b58900">echo</span> <span style="color:#2aa198">&#34;Found </span><span style="color:#2aa198">${</span><span style="color:#268bd2">CPUS_AVAILABLE</span><span style="color:#2aa198">}</span><span style="color:#2aa198"> CPUS&#34;</span> make -j<span style="color:#2aa198">${</span><span style="color:#268bd2">CPUS_AVAILABLE</span><span style="color:#2aa198">}</span> ...</code></pre></div> <p>The script checks for the value of the quota and divides by 100,000 to get the number of cores. If the share is set to something less than 100,000, then a core count of 1 is assigned. <em>(Pro tip: <code>make</code> does not like being told to compile with zero jobs.)</em></p> <h2 id="reading-memory-limits">Reading memory limits</h2> <p>There are other limits you can read and inspect in a pod, including the available RAM. As we found with <code>nproc</code>, <code>free</code> is not very helpful:</p> <div class="highlight"><pre style="color:#93a1a1;background-color:#002b36;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash"><span style="color:#586e75"># An OpenShift pod with 200MB RAM</span> $ free -m total used free shared buff/cache available Mem: <span style="color:#2aa198">32008</span> <span style="color:#2aa198">12322</span> <span style="color:#2aa198">880</span> <span style="color:#2aa198">31</span> <span style="color:#2aa198">18805</span> <span style="color:#2aa198">19246</span> Swap: <span style="color:#2aa198">0</span> <span style="color:#2aa198">0</span> <span style="color:#2aa198">0</span></code></pre></div> <p>But the cgroups tell the truth:</p> <div class="highlight"><pre style="color:#93a1a1;background-color:#002b36;-moz-tab-size:4;-o-tab-size:4;tab-size:4">$ cat /sys/fs/cgroup/memory/memory.limit_in_bytes 209715200</pre></div> <p>If you run Java applications in a container, like Jenkins (or Jenkins slaves), be sure to use the <code>-XX:+UseCGroupMemoryLimitForHeap</code> option. That will cause Java to look at the cgroups to determine its heap size.</p> <p><em>Photo credit: <a href="https://commons.wikimedia.org/wiki/File:Roca_de_la_Ley,_Parque_Nacional_de_%C3%9Eingvellir,_Su%C3%B0urland,_Islandia,_2014-08-16,_DD_022.JPG">Wikipedia</a></em></p> Running Ansible in OpenShift with arbitrary UIDs https://major.io/2019/03/22/running-ansible-in-openshift-with-arbitrary-uids/ Fri, 22 Mar 2019 00:00:00 UTC Major Hayden https://major.io/2019/03/22/running-ansible-in-openshift-with-arbitrary-uids/ <p><img src="../images/2019-03-22-blacksmith-anvil-hammer.jpg" alt="blacksmith_anvil_hammer" /></p> <p>My work at Red Hat involves testing lots and lots of kernels from various sources and we use <a href="https://gitlab.com/gitlab-org/gitlab-ce/">GitLab CE</a> to manage many of our repositories and run our CI jobs. Those jobs run in <em>thousands</em> of <a href="https://www.openshift.com/">OpenShift</a> containers that we spawn every day.</p> <p>OpenShift has some handy security features that we like. First, each container is mounted read-only with some writable temporary space (and any volumes that you mount). Also, OpenShift uses <a href="UIDs">arbitrarily assigned user IDs</a> for each container.</p> <p>Constantly changing UIDs provide some good protection against container engine vulnerabilities, but they can be a pain if you have a script or application that depends on being able to resolve a UID or GID back to a real user or group account.</p> <h2 id="ansible-and-uids">Ansible and UIDs</h2> <p>If you run an Ansible playbook within OpenShift, you will likely run into a problem during the fact gathering process:</p> <div class="highlight"><pre style="color:#93a1a1;background-color:#002b36;-moz-tab-size:4;-o-tab-size:4;tab-size:4">$ ansible-playbook -i hosts playbook.yml PLAY [all] ********************************************************************* TASK [Gathering Facts] ********************************************************* An exception occurred during task execution. To see the full traceback, use -vvv. The error was: KeyError: &#39;getpwuid(): uid not found: 1000220000&#39; fatal: [localhost]: FAILED! =&gt; {&#34;msg&#34;: &#34;Unexpected failure during module execution.&#34;, &#34;stdout&#34;: &#34;&#34;} to retry, use: --limit @/major-ansible-messaround/playbook.retry PLAY RECAP ********************************************************************* localhost : ok=0 changed=0 unreachable=0 failed=1</pre></div> <p>This exception is telling us that <a href="https://linux.die.net/man/3/getpwuid">getpwuid()</a> was not able to find an entry in <code>/etc/passwd</code> for our UID (<code>1000220000</code> in this container).</p> <p>One option would be to adjust the playbook so that we skip the fact gathering process:</p> <div class="highlight"><pre style="color:#93a1a1;background-color:#002b36;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-yaml" data-lang="yaml">- hosts: all gather_facts: no tasks: - name: Run tests command: ./run_tests.sh</code></pre></div> <p>However, this might not be helpful if you need facts to be gathered for your playbook to run. In that case, you need to make some adjustments to your container image first.</p> <h2 id="updating-the-container">Updating the container</h2> <p>Nothing in the container image is writable within OpenShift, but we can change certain files to be group writable for the root user since every OpenShift user has an effective GID of <code>0</code>.</p> <p>When you build your container, add a line to your Dockerfile to allow the container user to have group write access to <code>/etc/passwd</code> and <code>/etc/group</code>:</p> <div class="highlight"><pre style="color:#93a1a1;background-color:#002b36;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-dockerfile" data-lang="dockerfile"><span style="color:#586e75"># Make Ansible happy with arbitrary UID/GID in OpenShift.</span> <span style="color:#719e07">RUN</span> chmod <span style="color:#268bd2">g</span><span style="color:#719e07">=</span>u /etc/passwd /etc/group</code></pre></div> <p>Once your container has finished building, the permissions on both files should look like this:</p> <div class="highlight"><pre style="color:#93a1a1;background-color:#002b36;-moz-tab-size:4;-o-tab-size:4;tab-size:4">$ ls -al /etc/passwd /etc/group -rw-rw-r--. 1 root root 514 Mar 20 18:12 /etc/group -rw-rw-r--. 1 root root 993 Mar 20 18:12 /etc/passwd</pre></div> <h2 id="make-a-user-account">Make a user account</h2> <p>Now that we&rsquo;ve made these files writable for our user in OpenShift, it&rsquo;s time to change how we run our GitLab CI job. My job YAML currently looks like this:</p> <div class="highlight"><pre style="color:#93a1a1;background-color:#002b36;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-yaml" data-lang="yaml">ansible_test: image: docker.io/major/ansible:fedora29 script: - ansible-playbook -i hosts playbook.yml</code></pre></div> <p>We can add two lines that allow us to make a temporary user and group account for our OpenShift user:</p> <div class="highlight"><pre style="color:#93a1a1;background-color:#002b36;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-yaml" data-lang="yaml">ansible_test: image: docker.io/major/ansible:fedora29 script: - echo <span style="color:#2aa198">&#34;tempuser:x:$(id -u):$(id -g):,,,:${HOME}:/bin/bash&#34;</span> &gt;&gt; /etc/passwd - echo <span style="color:#2aa198">&#34;tempuser:x:$(id -G | cut -d&#39; &#39; -f 2)&#34;</span> &gt;&gt; /etc/group - id - ansible-playbook -i hosts playbook.yml</code></pre></div> <p>Note that we want the second GID returned by <code>id</code> since the first one is <code>0</code>. The <code>id</code> command helps us check our work when the container starts. When the CI job starts, we should see some better output:</p> <div class="highlight"><pre style="color:#93a1a1;background-color:#002b36;-moz-tab-size:4;-o-tab-size:4;tab-size:4">$ echo &#34;tempuser:x:$(id -u):$(id -g):,,,:${HOME}:/bin/bash&#34; &gt;&gt; /etc/passwd $ echo &#34;tempuser:x:$(id -G | cut -d&#39; &#39; -f 2)&#34; &gt;&gt; /etc/group $ id uid=1000220000(tempuser) gid=0(root) groups=0(root),1000220000(tempuser) $ ansible-playbook -i hosts playbook.yml PLAY [all] ********************************************************************* TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [Download kernel source] ************************************************** changed: [localhost] PLAY RECAP ********************************************************************* localhost : ok=2 changed=1 unreachable=0 failed=0</pre></div> <p>Success!</p> Get a /56 from Spectrum using wide-dhcpv6 https://major.io/2019/03/19/get-a-slash-56-from-spectrum-using-wide-dhcpv6/ Tue, 19 Mar 2019 00:00:00 UTC Major Hayden https://major.io/2019/03/19/get-a-slash-56-from-spectrum-using-wide-dhcpv6/ <p>After writing my <a href="../2019/03/04/pixel-3-wifi-drops-constantly/">last post</a> on my IPv6 woes with my Pixel 3, some readers asked how I&rsquo;m handling IPv6 on my router lately. I <a href="../2014/09/11/howto-time-warner-cable-ipv6/">wrote about this previously</a> when Spectrum was Time Warner Cable and I was using Mikrotik network devices.</p> <p>There is a good post from 2015 on the blog and it still works today:</p> <ul> <li><a href="../2015/09/11/time-warner-road-runner-linux-and-large-ipv6-subnets/">Time Warner Road Runner, Linux, and large IPv6 subnets</a></li> </ul> <p>I am still using that same setup today, but some readers found it difficult to find the post since Time Warner Cable has renamed to Spectrum. Don&rsquo;t worry &ndash; the old post still works. :)</p> Pixel 3 Wi-Fi drops constantly https://major.io/2019/03/17/pixel-3-wifi-drops-constantly/ Sun, 17 Mar 2019 00:00:00 UTC Major Hayden https://major.io/2019/03/17/pixel-3-wifi-drops-constantly/ <p><img src="../images/2019-03-17-google-pixel-phones.jpg" alt="pixel_phones" /></p> <p>We have two Google Pixel phones in our house: a Pixel 2 and a Pixel 3. Both of them drop off our home wireless network regularly. It causes lots of problems with various applications on the phones, especially casting video via Chromecast.</p> <p>At the time when I first noticed the drops, I was using a pair of wireless access points (APs) from Engenius:</p> <ul> <li><a href="https://www.engeniustech.com/engenius-products/indoor-wireless-ceiling-ap-eap600/">EAP600</a></li> <li><a href="https://www.engeniustech.com/engenius-products/indoor-wireless-ceiling-ap-eap1200h/">EAP1200H</a></li> </ul> <p>Also, here&rsquo;s what I knew at the time:</p> <ul> <li>Mac and Linux computers had no Wi-Fi issues at all</li> <li>The signal level from both APs was strong</li> <li>Disabling one AP made no improvement</li> <li>Disabling one band (2.4 or 5GHz) on the APs made no improvement</li> <li>Clearing the bluetooth/Wi-Fi data on the Pixel had no effect</li> <li>Assigning a static IP address on the Pixel made no improvement</li> <li>Using unencrypted SSIDs made no improvement</li> </ul> <p>At this point, I felt strongly that the APs had nothing to do with it. I ordered a new <a href="https://www.netgear.com/support/product/RBR50.aspx">NetGear Orbi</a> mesh router and satellite anyway. The Pixels still dropped off the wireless network even with the new Orbi APs.</p> <h2 id="reading-logs">Reading logs</h2> <p>I started reading logs from every source I could find:</p> <ul> <li>dhcpd logs from my router</li> <li>syslogs from my APs (which forwarded into the router)</li> <li>output from tcpdump on my router</li> </ul> <p>Several things became apparent after reading the logs:</p> <ul> <li>The Wi-Fi drop occurred usually every 30-60 seconds</li> <li>The DHCP server received requests for a new IP address after every drop</li> <li>None of the network traffic from the phones was being blocked at the router</li> <li>The logs from the APs showed the phone disconnecting itself from the network; the APs were not forcing the phones off the network</li> </ul> <p>All of the wireless and routing systems in my house seemed to point to a problem in the phones themselves. They were voluntarily dropping from the network without being bumped off by APs or the router.</p> <h2 id="getting-logs-from-the-phone">Getting logs from the phone</h2> <p>It was time to get some logs from the phone itself. That would require connecting the phone via USB to a computer and enabling USB debugging on the phone.</p> <p>First, I downloaded the <a href="https://developer.android.com/studio">Android SDK</a>. The full studio release isn&rsquo;t needed &ndash; scroll down and find the <em>Command line tools only</em> section. Unzip the download and find the <code>tools/bin/sdkmanager</code> executable. Run it like this:</p> <div class="highlight"><pre style="color:#93a1a1;background-color:#002b36;-moz-tab-size:4;-o-tab-size:4;tab-size:4"># Fedora 29 systems may need to choose the older Java version for sdkmanager # to run properly. export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.201.b09-2.fc29.x86_64/jre # Install the android-28 platform tools ./sdkmanager &#34;platform-tools&#34; &#34;platforms;android-28&#34;</pre></div> <p>Now we need to enable USB debugging on the phone itself. <strong>Be sure to disable USB debugging when you are done!</strong> Follow these steps:</p> <ol> <li>Go into the phone&rsquo;s settings and choose <em>About Phone</em> from the bottom of the list.</li> <li>Scroll to the bottom and tap the <em>Build number</em> section repeatedly until a message appears saying that you are now a developer.</li> <li>Go back one screen and tap <em>System</em>.</li> <li>Click <em>Advanced</em> to show the additional options and tap <em>Developer Options</em>.</li> <li>In the <em>Debugging</em> section, tap <em>USB Debugging</em> to enable USB debugging.</li> </ol> <p>Connect the phone to your computer via USB and run:</p> <div class="highlight"><pre style="color:#93a1a1;background-color:#002b36;-moz-tab-size:4;-o-tab-size:4;tab-size:4">sudo platform-tools/adb logcat</pre></div> <p>Your screen will fill with logs from your phone.</p> <h2 id="nuggets-in-the-log">Nuggets in the log</h2> <p>I watched the logs and waited for the Wi-Fi to drop. As soon as it dropped, I saw some interesting log messages:</p> <div class="highlight"><pre style="color:#93a1a1;background-color:#002b36;-moz-tab-size:4;-o-tab-size:4;tab-size:4">I wpa_supplicant: wlan0: CTRL-EVENT-AVOID-FREQ ranges=5785-5825 I chatty : uid=1000(system) IpClient.wlan0 expire 3 lines I chatty : uid=1000 system_server expire 1 line D CommandListener: Setting iface cfg E cnss-daemon: wlan_service_update_sys_param: unable to open /proc/sys/net/ipv4/tcp_use_userconfig I chatty : uid=1000(system) android.fg expire 1 line I wpa_supplicant: wlan0: CTRL-EVENT-DISCONNECTED bssid=88:dc:96:4a:b6:75 reason=3 locally_generated=1 I chatty : uid=10025 com.google.android.gms.persistent expire 7 lines V NativeCrypto: Read error: ssl=0x7b349e2d08: I/O error during system call, Software caused connection abort V NativeCrypto: Write error: ssl=0x7b349e2d08: I/O error during system call, Broken pipe V NativeCrypto: Write error: ssl=0x7b349e2d08: I/O error during system call, Broken pipe V NativeCrypto: SSL shutdown failed: ssl=0x7b349e2d08: I/O error during system call, Success D ConnectivityService: reportNetworkConnectivity(158, false) by 10025</pre></div> <p>The line with <code>CTRL-EVENT-AVOID-FREQ</code> isn&rsquo;t relevant because it&rsquo;s simply a hint to the wireless drivers to avoid certain frequencies not used in the USA. The <code>CTRL-EVENT-DISCONNECTED</code> shows where wpa_supplicant received the disconnection message. The last line with <code>ConnectivityService</code> was very interesting. Something in the phone believes there is a network connectivity issue. That could be why the Pixel is hopping off the wireless network.</p> <p>From there, I decided to examine only the <code>ConnectivityService</code> logs:</p> <div class="highlight"><pre style="color:#93a1a1;background-color:#002b36;-moz-tab-size:4;-o-tab-size:4;tab-size:4">sudo platform-tools/adb logcat &#39;ConnectivityService:* *:S&#39;</pre></div> <p>This logcat line tells adb that I want all logs from all log levels about the <code>ConnectivityService</code>, but all of the other logs should be silenced. I started seeing some interesting details:</p> <div class="highlight"><pre style="color:#93a1a1;background-color:#002b36;-moz-tab-size:4;-o-tab-size:4;tab-size:4">D ConnectivityService: NetworkAgentInfo [WIFI () - 148] validation failed D ConnectivityService: Switching to new default network: NetworkAgentInfo{ ni{[type: MOBILE[LTE]... D ConnectivityService: Sending DISCONNECTED broadcast for type 1 NetworkAgentInfo [WIFI () - 148] isDefaultNetwork=true D ConnectivityService: Sending CONNECTED broadcast for type 0 NetworkAgentInfo [MOBILE (LTE) - 100] isDefaultNetwork=true D ConnectivityService: handleNetworkUnvalidated NetworkAgentInfo [WIFI () - 148] ...</pre></div> <p>Wait, what is this &ldquo;validation failed&rdquo; message? The Pixel was making network connections successfully the entire time as shown by tcpdump. This is part of Android&rsquo;s [network connecivity checks] for various networks.</p> <p>The last few connections just before the disconnect were to <code>connectivitycheck.gstatic.com</code> (based on tcpdump logs) and that&rsquo;s Google&rsquo;s way of verifying that the wireless network is usable and that there are no captive portals. I connected to it from my desktop on IPv4 and IPv6 to verify:</p> <div class="highlight"><pre style="color:#93a1a1;background-color:#002b36;-moz-tab-size:4;-o-tab-size:4;tab-size:4">$ curl -4 -i https://connectivitycheck.gstatic.com/generate_204 HTTP/2 204 date: Sun, 17 Mar 2019 15:00:30 GMT alt-svc: quic=&#34;:443&#34;; ma=2592000; v=&#34;46,44,43,39&#34; $ curl -6 -i https://connectivitycheck.gstatic.com/generate_204 HTTP/2 204 date: Sun, 17 Mar 2019 15:00:30 GMT alt-svc: quic=&#34;:443&#34;; ma=2592000; v=&#34;46,44,43,39&#34;</pre></div> <p>Everything looked fine.</p> <h2 id="heading-to-google">Heading to Google</h2> <p>After a bunch of searching on Google, I kept finding posts talking about disabling IPv6 to fix the Wi-Fi drop issues. I shrugged it off and kept searching. Finally, I decided to disable IPv6 and see if that helped.</p> <p>I stopped <code>radvd</code> on the router, disabled Wi-Fi on the phone, and then re-enabled it. As I watched, the phone stayed on the wireless network for two minutes. Three minutes. Ten minutes. <strong>There were no drops.</strong></p> <p>At this point, this is still an unsolved mystery for me. Disabling IPv6 is a terrible idea, but it keeps my phones online. I plan to put the phones on their own VLAN without IPv6 so I can still keep IPv6 addresses for my other computers, but this is not a good long term fix. If anyone has any input on why this helps and how I can get IPv6 re-enabled, please <a href="mailto:major@mhtx.net">let me know</a>!</p> <h2 id="update-2019-03-18">Update 2019-03-18</h2> <p>Several readers wanted to see what was happening just before the Wi-Fi drop, so here&rsquo;s a small snippet from tcpdump:</p> <div class="highlight"><pre style="color:#93a1a1;background-color:#002b36;-moz-tab-size:4;-o-tab-size:4;tab-size:4">07:26:06.736900 IP6 2607:f8b0:4000:80d::2003.443 &gt; phone.41310: Flags [F.], seq 3863, ack 511, win 114, options [nop,nop,TS val 1288800272 ecr 66501414], length 0 07:26:06.743101 IP6 2607:f8b0:4000:80d::2003.443 &gt; phone.41312: Flags [F.], seq 3864, ack 511, win 114, options [nop,nop,TS val 1778536228 ecr 66501414], length 0 07:26:06.765444 IP6 phone.41312 &gt; 2607:f8b0:4000:80d::2003.443: Flags [R], seq 4183481455, win 0, length 0 07:26:06.765454 IP6 phone.41310 &gt; 2607:f8b0:4000:80d::2003.443: Flags [R], seq 3279990707, win 0, length 0 07:26:07.487180 IP6 2607:f8b0:4000:80d::2003.443 &gt; phone.41316: Flags [F.], seq 3863, ack 511, win 114, options [nop,nop,TS val 4145292968 ecr 66501639], length 0 07:26:07.537080 IP6 phone.41316 &gt; 2607:f8b0:4000:80d::2003.443: Flags [R], seq 4188442452, win 0, length 0</pre></div> <p>That IPv6 address is at a Google PoP in Dallas, TX:</p> <div class="highlight"><pre style="color:#93a1a1;background-color:#002b36;-moz-tab-size:4;-o-tab-size:4;tab-size:4">$ host 2607:f8b0:4000:80d::2003 3.0.0.2.0.0.0.0.0.0.0.0.0.0.0.0.d.0.8.0.0.0.0.4.0.b.8.f.7.0.6.2.ip6.arpa domain name pointer dfw06s49-in-x03.1e100.net.</pre></div> <p>I haven&rsquo;t been able to intercept the traffic via man-in-the-middle since Google&rsquo;s certificate checks are very strict. However, checks from my own computer work without an issue:</p> <div class="highlight"><pre style="color:#93a1a1;background-color:#002b36;-moz-tab-size:4;-o-tab-size:4;tab-size:4">$ curl -ki &#34;https://[2607:f8b0:4000:80d::2003]/generate_204&#34; HTTP/2 204 date: Mon, 18 Mar 2019 12:35:18 GMT alt-svc: quic=&#34;:443&#34;; ma=2592000; v=&#34;46,44,43,39&#34;</pre></div> Stop audio pops on Intel HD Audio https://major.io/2019/03/04/stop-audio-pops-on-intel-hd-audio/ Mon, 04 Mar 2019 00:00:00 UTC Major Hayden https://major.io/2019/03/04/stop-audio-pops-on-intel-hd-audio/ <p><img src="../images/2019-03-04-headphones.jpg" alt="headphones" /></p> <p>I recently picked up a Dell Optiplex 7060 and I&rsquo;m using it as my main workstation now. The Fedora installation was easy, but I noticed a variety of &ldquo;pop&rdquo; or clicking sounds when audio played, especially terminal bells.</p> <p>If everything was quiet and I triggered a terminal bell, I would hear a loud pop just before the terminal bell sound. However, if I played music and then triggered a terminal bell, the pop was gone.</p> <p>A quick Google search told me that the likely culprit was power saving settings on my Intel HD Audio chipset:</p> <div class="highlight"><pre style="color:#93a1a1;background-color:#002b36;-moz-tab-size:4;-o-tab-size:4;tab-size:4">$ lspci | grep Audio 00:1f.3 Audio device: Intel Corporation Cannon Lake PCH cAVS (rev 10)</pre></div> <h2 id="fixing-it">Fixing it</h2> <p>There&rsquo;s a handy power saving tunable available at <code>/sys/module/snd_hda_intel/parameters/power_save</code> that can be usd to adjust the timeout or disable power savings entirely. In my case, the delay was set to one second.</p> <div class="highlight"><pre style="color:#93a1a1;background-color:#002b36;-moz-tab-size:4;-o-tab-size:4;tab-size:4">$ cat /sys/module/snd_hda_intel/parameters/power_save 1</pre></div> <p>That would be good for a laptop use case, but my workstation is always plugged in. I disabled it by setting it to zero:</p> <div class="highlight"><pre style="color:#93a1a1;background-color:#002b36;-moz-tab-size:4;-o-tab-size:4;tab-size:4"># echo 0 &gt; /sys/module/snd_hda_intel/parameters/power_save $ cat /sys/module/snd_hda_intel/parameters/power_save 0</pre></div> <p>And the pops are gone! My Klipsch speakers have a built in amplifier and it was likely the abrupt changes in current that was causing the popping noises.</p> <p>This setting will last until you reboot. You can make it permanent by adding this text to <code>/etc/modprobe.d/audio_disable_powersave.conf</code>:</p> <div class="highlight"><pre style="color:#93a1a1;background-color:#002b36;-moz-tab-size:4;-o-tab-size:4;tab-size:4">options snd_hda_intel power_save=0</pre></div> <p>If you&rsquo;re a laptop user and you want power savings but fewer pops, you could increase the delay to a more acceptable number. For example, setting it to <code>60</code> would mean that the card will power down after 60 seconds of silence. Just remember that you&rsquo;ll get a nice pop when the 60 seconds has passed and a new sound is played.</p> <h2 id="learning-more">Learning more</h2> <p>Diving into the kernel code reveals the tunable in <code>/sound/pci/hda/hda_intel.c</code>:</p> <div class="highlight"><pre style="color:#93a1a1;background-color:#002b36;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-c" data-lang="c"><span style="color:#719e07">static</span> <span style="color:#dc322f">int</span> power_save <span style="color:#719e07">=</span> CONFIG_SND_HDA_POWER_SAVE_DEFAULT; module_param(power_save, xint, <span style="color:#2aa198">0644</span>); MODULE_PARM_DESC(power_save, <span style="color:#2aa198">&#34;Automatic power-saving timeout &#34;</span> <span style="color:#2aa198">&#34;(in second, 0 = disable).&#34;</span>);</code></pre></div> <p>The default comes from a kernel config option: <code>CONFIG_SND_HDA_POWER_SAVE_DEFAULT</code>. Most kernel packages on most distributions provide access to the kernel config file that was used to build the kernel originally. It&rsquo;s often found in <code>/boot</code> (named the same as the kernel version) or it might be available at <code>/proc/config.gz</code>.</p> <p>For Fedora, the kernel config is provided in <code>/boot</code> whenever a new kernel is is installed. I inspected mine and found:</p> <div class="highlight"><pre style="color:#93a1a1;background-color:#002b36;-moz-tab-size:4;-o-tab-size:4;tab-size:4">$ grep HDA_POWER_SAVE_DEFAULT /boot/config-4.20.13-200.fc29.x86_64 CONFIG_SND_HDA_POWER_SAVE_DEFAULT=1</pre></div> <p>The <code>power_save</code> setting is applied in <code>/sound/pci/hda/hda_codec.c</code>:</p> <div class="highlight"><pre style="color:#93a1a1;background-color:#002b36;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-c" data-lang="c"><span style="color:#586e75">/** </span><span style="color:#586e75"> * snd_hda_set_power_save - reprogram autosuspend for the given delay </span><span style="color:#586e75"> * @bus: HD-audio bus </span><span style="color:#586e75"> * @delay: autosuspend delay in msec, 0 = off </span><span style="color:#586e75"> * </span><span style="color:#586e75"> * Synchronize the runtime PM autosuspend state from the power_save option. </span><span style="color:#586e75"> */</span> <span style="color:#dc322f">void</span> <span style="color:#268bd2">snd_hda_set_power_save</span>(<span style="color:#719e07">struct</span> hda_bus <span style="color:#719e07">*</span>bus, <span style="color:#dc322f">int</span> delay) { <span style="color:#719e07">struct</span> hda_codec <span style="color:#719e07">*</span>c; list_for_each_codec(c, bus) codec_set_power_save(c, delay); } EXPORT_SYMBOL_GPL(snd_hda_set_power_save);</code></pre></div> <p>We can look where <code>codec_set_power_save</code> is defined in the same file to learn more:</p> <div class="highlight"><pre style="color:#93a1a1;background-color:#002b36;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-c" data-lang="c"><span style="color:#719e07">#ifdef CONFIG_PM </span><span style="color:#719e07"></span><span style="color:#719e07">static</span> <span style="color:#dc322f">void</span> <span style="color:#268bd2">codec_set_power_save</span>(<span style="color:#719e07">struct</span> hda_codec <span style="color:#719e07">*</span>codec, <span style="color:#dc322f">int</span> delay) { <span style="color:#719e07">struct</span> device <span style="color:#719e07">*</span>dev <span style="color:#719e07">=</span> hda_codec_dev(codec); <span style="color:#719e07">if</span> (delay <span style="color:#719e07">==</span> <span style="color:#2aa198">0</span> <span style="color:#719e07">&amp;&amp;</span> codec<span style="color:#719e07">-&gt;</span>auto_runtime_pm) delay <span style="color:#719e07">=</span> <span style="color:#2aa198">3000</span>; <span style="color:#719e07">if</span> (delay <span style="color:#719e07">&gt;</span> <span style="color:#2aa198">0</span>) { pm_runtime_set_autosuspend_delay(dev, delay); pm_runtime_use_autosuspend(dev); pm_runtime_allow(dev); <span style="color:#719e07">if</span> (<span style="color:#719e07">!</span>pm_runtime_suspended(dev)) pm_runtime_mark_last_busy(dev); } <span style="color:#719e07">else</span> { pm_runtime_dont_use_autosuspend(dev); pm_runtime_forbid(dev); } }</code></pre></div> <p>This logic looks to see if <code>CONFIG_PM</code> is set to know if power management is desired at all. From there, it checks if we disabled power saving but there&rsquo;s a discrete graphics card involved (<code>codec-&gt;auto_runtime_pm</code>). This check is important because the discrete graphics card cannot power down unless the HDA card suspends at the same time.</p> <p>Next, there&rsquo;s a check to see if the delay is greater than 0. This would be the case if <code>CONFIG_SND_HDA_POWER_SAVE_DEFAULT</code> was set to <code>1</code> (Fedora&rsquo;s default). If so, the proper auto suspend delays are set.</p> <p>If the delay is 0, then autosuspend is disabled and removed from power management entirely. This is the option I chose and it&rsquo;s working great.</p> <p><em>Photo source: <a href="https://www.maxpixel.net/Headphone-Caption-Music-Sound-Listing-Music-2694489">Max Pixel</a></em></p> Automatic floating windows in i3 https://major.io/2019/02/08/automatic-floating-windows-in-i3/ Fri, 08 Feb 2019 00:00:00 UTC Major Hayden https://major.io/2019/02/08/automatic-floating-windows-in-i3/ <p><img src="../images/2019-02-08-floating-leaf.jpg" alt="floating leaf" /></p> <p>The <a href="https://i3wm.org/">i3 window manager</a> is a fast window manager that helps you keep all of your applications in the right place. It automatically tiles windows and can manage those tiles across multiple virtual desktops.</p> <p>However, there are certain applications that I really prefer in a floating window. Floating windows do not get tiled and they can easily be dragged around with your mouse. They&rsquo;re the type of windows you expect to see on other non-tiling desktops such as GNOME or KDE.</p> <h2 id="convert-a-window-to-floating-temporarily">Convert a window to floating temporarily</h2> <p>If you have an existing window that you prefer to float, select that window and press <em>Mod + Shift + Space bar</em>. The window will pop up in front of the tiled windows and you can easily move it with your mouse.</p> <p>Depending on your configuration, you may be able to resize it by grabbing a corner of the window with your mouse. You can also assign a key combination for resizing in your i3 configuration file (usually <code>~/.config/i3/config</code>):</p> <div class="highlight"><pre style="color:#93a1a1;background-color:#002b36;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-text" data-lang="text"># resize window (you can also use the mouse for that) mode &#34;resize&#34; { bindsym Left resize shrink width 10 px or 10 ppt bindsym Down resize grow height 10 px or 10 ppt bindsym Up resize shrink height 10 px or 10 ppt bindsym Right resize grow width 10 px or 10 ppt bindsym Return mode &#34;default&#34; bindsym Escape mode &#34;default&#34; bindsym $mod+r mode &#34;default&#34; } bindsym $mod+r mode &#34;resize&#34;</code></pre></div> <p>With this configuration, simply press <em>Mod + r</em> and use the arrow keys to grow or shrink the window&rsquo;s borders.</p> <h2 id="always-float-certain-windows">Always float certain windows</h2> <p>For those windows that you always want to be floating no matter what, i3 has a solution for that, too. Just tell i3 how to identify your windows and ensure <code>floating enable</code> appears in the i3 config:</p> <div class="highlight"><pre style="color:#93a1a1;background-color:#002b36;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-text" data-lang="text">for_window [window_role=&#34;About&#34;] floating enable for_window [class=&#34;vlc&#34;] floating enable for_window [title=&#34;Authy&#34;] floating enable</code></pre></div> <p>In the example above, I have a few windows always set to be floating:</p> <ul> <li><code>[window_role=&quot;About&quot;]</code> - Any of the &ldquo;About&rdquo; windows in various applications that are normally opened by <em>Help -&gt; About</em>.</li> <li><code>[class=&quot;vlc&quot;]</code> - The VLC media player can be a good one to float if you need to stuff it away in a corner.</li> <li><code>[title=&quot;Authy&quot;]</code> - Authy&rsquo;s chrome extension looks downright silly as a tiled window.</li> </ul> <p>Any time these windows are spawned, they will automatically appear as floating windows. You can always switch them back to tiled manually by pressing <em>Mod + Shift + Space bar</em>.</p> <h2 id="identifying-windows">Identifying windows</h2> <p>Identifying windows in the way that i3 cares about can be challenging. Knowing when to use <code>window_role</code> or <code>class</code> for a window isn&rsquo;t very intuitive. Fortunately, there&rsquo;s a great script from an <a href="https://faq.i3wm.org/question/2172/how-do-i-find-the-criteria-for-use-with-i3-config-commands-like-for_window-eg-to-force-splashscreens-and-dialogs-to-show-in-floating-mode.1.html">archived i3 faq thread</a> that makes this easy:</p> <script src="https://gist.github.com/major/ff11ccdd73941109abb4ac2d35c976eb.js"></script> <p>Download this script to your system, make it executable (<code>chmod +x i3-get-window-criteria</code>), and run it. As soon as you do that, a plus (+) icon will replace your normal mouse cursor. Click on the window you care about and look for the output in your terminal where you ran the <code>i3-get-window-criteria</code> script.</p> <p>On my system, clicking on a terminator terminal window gives me:</p> <div class="highlight"><pre style="color:#93a1a1;background-color:#002b36;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-text" data-lang="text">[class=&#34;Terminator&#34; id=37748743 instance=&#34;terminator&#34; title=&#34;major@indium:~&#34;]</code></pre></div> <p>If I wanted to float all terminator windows, I could add this to my i3 configuration file:</p> <div class="highlight"><pre style="color:#93a1a1;background-color:#002b36;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-text" data-lang="text">for_window [class=&#34;Terminator&#34;] floating enable</code></pre></div> <h2 id="float-in-a-specific-workspace">Float in a specific workspace</h2> <p>Do you need a window to always float on a specific workspace? i3 can do that, too!</p> <p>Let&rsquo;s go back to the example with VLC. Let&rsquo;s consider that we have a really nice 4K display where we always want to watch movies and that&rsquo;s where workspace 2 lives. We can tell i3 to always float the VLC window on workspace 2 with this configuration:</p> <div class="highlight"><pre style="color:#93a1a1;background-color:#002b36;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-text" data-lang="text">set $ws1 &#34;1: main&#34; set $ws2 &#34;2: 4kdisplay&#34; for_window [class=&#34;vlc&#34;] floating enable for_window [class=&#34;vlc&#34;] move to workspace $ws2</code></pre></div> <p>Restart i3 to pick up the new changes (usually <em>Mod + Shift + R</em>) and start VLC. It should appear on workspace 2 as a floating window!</p> <p><em><a href="https://www.maxpixel.net/Floating-Sea-Leaf-Water-Cute-Leaf-Floating-2438419">Photo source</a></em></p> DevConf.CZ 2019 Recap https://major.io/2019/01/31/devconf.cz-2019-recap/ Thu, 31 Jan 2019 00:00:00 UTC Major Hayden https://major.io/2019/01/31/devconf.cz-2019-recap/ <p><a href="https://devconf.info/cz">DevConf.CZ 2019</a> wrapped up last weekend and it was a great event packed with lots of knowledgeable speakers, an engaging hallway track, and delicious food. This was my first trip to any DevConf and it was my second trip to <a href="https://en.wikipedia.org/wiki/Brno">Brno</a>.</p> <p>Lots of snow showed up on the second day and more snow arrived later in the week!</p> <p><a href="../images/20190131-devconf-2019-snow.jpg"><img src="../images/20190131-devconf-2019-snow-720.jpg" alt="devconf-snow-small" /></a></p> <h2 id="first-talk-of-2019">First talk of 2019</h2> <p>I co-presented a talk with one of my teammates, Nikolai, about some of the fun work we&rsquo;ve been doing at Red Hat to improve the quality of the Linux kernel in an automated way. The room was full and we had lots of good questions at the end of the talk. We also received some feedback that we could take back to the team to change how we approached certain parts of the kernel testing.</p> <p><img src="../images/20190131-devconf-major-nikolai.jpg" alt="devconf-major-nikolai" /></p> <p>Our project, called Continuous Kernel Integration (CKI), has a goal of reducing the amount of bugs that are merged into the Linux kernel. This requires lots of infrastructure, automation, and testing capabilities. We shared information about our setup, the problems we&rsquo;ve found, and where we want to go in the future.</p> <p>Feel free to <a href="https://www.slideshare.net/MajorHayden/cookies-for-kernel-developers">view our slides</a> and watch the video (which should be up soon.)</p> <h2 id="great-talks-from-devconf">Great talks from DevConf</h2> <p>My favorite talk of the conference was <a href="https://twitter.com/openlabbott">Laura Abbott&rsquo;s</a> &ldquo;Monsters, Ghosts, and Bugs.&rdquo;</p> <p><img src="../images/20190131-devconf-laura-abbott-720.jpg" alt="devconf-laura-abbott" /></p> <p>It&rsquo;s the most informative, concise, and sane review of how all the Linux kernels on the planet fit together. From the insanity of linux-next to the wild world of being a Linux distribution kernel maintainer, she helped us all understand the process of how kernels are maintained. She also took time to help the audience understand which kernels are most important to them and how they can make the right decisions about the kernel that will suit their needs. There are <a href="https://twitter.com/majorhayden/status/1089507679977046017">plenty of good points in my Twitter thread</a> about her talk.</p> <p>Dan Walsh gave a detailed overview of how to use Podman instead of Docker. He talked about the project&rsquo;s origins and some of the incorrect assumptions that many people have (that running containers means only running Docker). Running containers without root has plenty of benefits. In addition, a significant amount of work has been done to speed up container pulls and pushes in Podman. I took some <a href="https://twitter.com/majorhayden/status/1088715555635220481">notes on Dan&rsquo;s talk</a> in a thread on Twitter.</p> <p>The firewalld package has gained some new features recently and it&rsquo;s poised to fully take advantage of nftables in Fedora 31! Using nftables means that firewall updates are done faster with fewer hiccups in busy environments (think OpenStack and Kubernetes). In addition, nftables can apply rules to IPv4 and IPv6 simultaneously, depeending on your preferences. My <a href="https://twitter.com/majorhayden/status/1088807448817876992">firewalld Twitter thread</a> has more details from the talk.</p> <p>The cgroups v2 subsystem was a popular topic in a few of the talks I visited, including the lightning talks. There are plenty of issues to get it working with Kubernetes and container management systems. It&rsquo;s also missing the <a href="https://www.kernel.org/doc/Documentation/cgroup-v1/freezer-subsystem.txt">freezer capability</a> from the original cgroups implementation. Without that, pausing a container, or using technology like <a href="https://criu.org/Main_Page">CRIU</a>, simply won&rsquo;t work. Nobody could name a Linux distribution that has cgroups v2 enabled at the moment, and that&rsquo;s not helping the effort move forward. Look for more news on this soon.</p> <p><img src="../images/20190131-devconf-openshif-pun-720.jpg" alt="devconf-openshift-pun" /></p> <p>OpenShift is quickly moving towards offering multiple architectures as a first class product feature. That would incluve aarch64, ppc64le, and s390x in addition to the existing x86_64 support. Andy McCrae and Jeff Young had a talk detailing many of the challenges along with lots of punny references to various &ldquo;arches&rdquo;. I made a <a href="https://twitter.com/majorhayden/status/1089470907607928833">Twitter thread of the main points</a> from the OpenShift talk.</p> <p>Some of the other news included:</p> <ul> <li><a href="https://wiki.linuxfoundation.org/realtime/start">real-time linux patches</a> are likely going to be merged into mainline. (only 15 years in the making!)</li> <li>Fedora, CentOS, RHEL and EPEL communities are eager to bring more of their processes together and make it easier for contributors to join in.</li> <li>Linux 5.0 is no more exciting than 4.20. It would have been 4.21 if Linus had an <a href="https://www.theregister.co.uk/2019/01/07/linux_reaches_the_big_five_point_oh/">extra finger or toe</a>.</li> </ul> <h2 id="devconf-us-boston-2019">DevConf.US Boston 2019</h2> <p>The next DevConf.US is in Boston, USA this summer. I hope to see you there!</p> Using the pressure stall information interface in kernel 4.20 https://major.io/2019/01/27/using-the-pressure-stall-information-interface-in-kernel-4.20/ Sun, 27 Jan 2019 00:00:00 UTC Major Hayden https://major.io/2019/01/27/using-the-pressure-stall-information-interface-in-kernel-4.20/ <p><img src="../images/pressure_cooker.jpg" alt="pressure_cooker" /></p> <p>Fedora 29 now has kernel 4.20 available and it has <a href="https://kernelnewbies.org/Linux_4.20">lots of new features</a>. One of the more interesting and easy to use features is the <a href="https://lwn.net/Articles/759658/">pressure stall information</a> interface.</p> <h2 id="load-average">Load average</h2> <p>We&rsquo;re all familiar with the <a href="https://en.wikipedia.org/wiki/Load_(computing)">load average</a> measurement on Linux machines, even if the numbers do seem a bit cryptic:</p> <div class="highlight"><pre style="color:#93a1a1;background-color:#002b36;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-text" data-lang="text">$ w 10:55:46 up 11 min, 1 user, load average: 0.42, 0.39, 0.26</code></pre></div> <p>The numbers denote how many processes were active over the last one, five and 15 minutes. In my case, I have a system with four cores. My numbers above show that less than one process was active in the last set of intervals. That means that my system isn&rsquo;t doing very much and processes are not waiting in the queue.</p> <p>However, if I begin compiling a kernel with eight threads (double my core count), the numbers change dramatically:</p> <div class="highlight"><pre style="color:#93a1a1;background-color:#002b36;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-text" data-lang="text">$ w 11:00:28 up 16 min, 1 user, load average: 4.15, 1.89, 0.86</code></pre></div> <p>The one minute load average is now over four, which means some processes are waiting to be served on the system. This makes sense because I am using eight threads to compile a kernel on a system with four cores.</p> <h2 id="more-detail">More detail</h2> <p>We assume that the CPU is the limiting factor in the system since we know that compiling a kernel takes lots of CPU time. We can verify (and quantify) that with the pressure stall information available in 4.20.</p> <p>We start by taking a look in <code>/proc/pressure</code>:</p> <div class="highlight"><pre style="color:#93a1a1;background-color:#002b36;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-text" data-lang="text">$ head /proc/pressure/* ==&gt; /proc/pressure/cpu &lt;== some avg10=71.37 avg60=57.25 avg300=23.83 total=100354487 ==&gt; /proc/pressure/io &lt;== some avg10=0.17 avg60=0.13 avg300=0.24 total=8101378 full avg10=0.00 avg60=0.01 avg300=0.16 total=5866706 ==&gt; /proc/pressure/memory &lt;== some avg10=0.00 avg60=0.00 avg300=0.00 total=0 full avg10=0.00 avg60=0.00 avg300=0.00 total=0</code></pre></div> <p>But what do these numbers mean? The shortest explanation is in the patch itself:</p> <blockquote> <p>PSI aggregates and reports the overall wallclock time in which the tasks in a system (or cgroup) wait for contended hardware resources.</p> </blockquote> <p>The numbers here are percentages, not time itself:</p> <blockquote> <p>The averages give the percentage of walltime in which one or more tasks are delayed on the runqueue while another task has the CPU. They&rsquo;re recent averages over 10s, 1m, 5m windows, so you can tell short term trends from long term ones, similarly to the load average.</p> </blockquote> <p>We can try to apply some I/O pressure by making a big tarball of a kernel source tree:</p> <div class="highlight"><pre style="color:#93a1a1;background-color:#002b36;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-text" data-lang="text">$ head /proc/pressure/* ==&gt; /proc/pressure/cpu &lt;== some avg10=1.33 avg60=10.07 avg300=26.83 total=262389574 ==&gt; /proc/pressure/io &lt;== some avg10=40.53 avg60=13.27 avg300=3.46 total=20451978 full avg10=37.44 avg60=12.40 avg300=3.21 total=16659637 ==&gt; /proc/pressure/memory &lt;== some avg10=0.00 avg60=0.00 avg300=0.00 total=0 full avg10=0.00 avg60=0.00 avg300=0.00 total=0</code></pre></div> <p>The CPU is still under some stress here, but the I/O is now the limiting factor.</p> <p>The output also shows a <code>total=</code> number, and that is explained in the patch as well:</p> <blockquote> <p>The total= value gives the absolute stall time in microseconds. This allows detecting latency spikes that might be too short to sway the running averages. It also allows custom time averaging in case the 10s/1m/5m windows aren&rsquo;t adequate for the usecase (or are too coarse with future hardware).</p> </blockquote> <p>The total number can be helpful for machines that run for a long time, especially when you graph them and you monitor them for trends.</p> Running Home Assistant in a Docker container with a Z-Wave USB stick https://major.io/2019/01/14/running-home-assistant-in-a-docker-container-with-zwave-usb-stick/ Mon, 14 Jan 2019 00:00:00 UTC Major Hayden https://major.io/2019/01/14/running-home-assistant-in-a-docker-container-with-zwave-usb-stick/ <p>The <a href="https://www.home-assistant.io/">Home Assistant</a> project provides a great open source way to get started with home automtion that can be entirely self-contained within your home. It already has plenty of <a href="https://www.home-assistant.io/components/">integrations</a> with external services, but it can also monitor <a href="https://en.wikipedia.org/wiki/Z-Wave">Z-Wave</a> devices at your home or office.</p> <p>Here are my devices:</p> <ul> <li><a href="https://www.amazon.com/Monoprice-Z-Wave-Garage-Door-Sensor/dp/B00V5IQ8E8">Monoprice Z-Wave Garade Door Sensor</a></li> <li><a href="https://aeotec.com/z-wave-usb-stick">Aeotec Z-Stick Gen5 (ZW090)</a></li> <li>Fedora Linux server with Docker installed</li> </ul> <h2 id="install-the-z-wave-stick">Install the Z-Wave stick</h2> <p>Start by plugging the Z-Stick into your Linux server. Run <code>lsusb</code> and it should appear in the list:</p> <div class="highlight"><pre style="color:#93a1a1;background-color:#002b36;-moz-tab-size:4;-o-tab-size:4;tab-size:4"># lsusb | grep Z-Stick Bus 003 Device 006: ID 0658:0200 Sigma Designs, Inc. Aeotec Z-Stick Gen5 (ZW090) - UZB</pre></div> <p>The system journal should also tell you which TTY is assigned to the USB stick (run <code>journalctl --boot</code> and search for <code>ACM</code>):</p> <div class="highlight"><pre style="color:#93a1a1;background-color:#002b36;-moz-tab-size:4;-o-tab-size:4;tab-size:4">kernel: usb 3-3.2: USB disconnect, device number 4 kernel: usb 3-1: new full-speed USB device number 6 using xhci_hcd kernel: usb 3-1: New USB device found, idVendor=0658, idProduct=0200, bcdDevice= 0.00 kernel: usb 3-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0 kernel: cdc_acm 3-1:1.0: ttyACM0: USB ACM device kernel: usbcore: registered new interface driver cdc_acm kernel: cdc_acm: USB Abstract Control Model driver for USB modems and ISDN adapters</pre></div> <p>In my case, my device is <code>/dev/ttyACM0</code>. If you have other serial devices attached to your system, your Z-Stick may show up as <code>ttyACM1</code> or <code>ttyACM2</code>.</p> <h2 id="using-z-wave-in-the-docker-container">Using Z-Wave in the Docker container</h2> <p>If you use <code>docker-compose</code>, simply add a <code>devices</code> section to your existing YAML file:</p> <div class="highlight"><pre style="color:#93a1a1;background-color:#002b36;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-yaml" data-lang="yaml">version: <span style="color:#2aa198">&#39;2&#39;</span> services: home-assistant: ports: - <span style="color:#2aa198">&#34;8123:8123/tcp&#34;</span> network_mode: <span style="color:#2aa198">&#34;host&#34;</span> devices: - /dev/ttyACM0 volumes: - /etc/localtime:/etc/localtime:ro - /mnt/raid/hass/:/config:Z image: homeassistant/home-assistant restart: always</code></pre></div> <p>You can add the device to manual <code>docker run</code> commands by adding <code>--device /dev/ttyACM0</code> to your existing command line.</p> <h2 id="pairing">Pairing</h2> <p>For this step, always refer to the instructions that came with your Z-Wave device since some require different pairing steps. In my case, I installed the battery, pressed the button inside the sensor, and paired the device:</p> <ul> <li>Go to the Home Assistant web interface</li> <li>Click <strong>Configuration</strong> on the left</li> <li>Click <strong>Z-Wave</strong> on the right</li> <li>Click <strong>Add Node</strong> and follow the steps on screen</li> </ul> <h2 id="understanding-how-the-sensor-works">Understanding how the sensor works</h2> <p>Now that the sensor has been added, we need to understand how it works. One of the entities the sensor provides is an <code>alarm_level</code>. It has two possible values:</p> <ul> <li><code>0</code>: the sensor is tilted vertically (garage door is closed)</li> <li><code>255</code>: the sensor is tilted horizontally (garage door is open)</li> </ul> <p>If the sensor changes from <code>0</code> to <code>255</code>, then someone opened the garage door. Closing the door would result in the sensor changing from <code>255</code> to <code>0</code>.</p> <h2 id="adding-automation">Adding automation</h2> <p>Let&rsquo;s add automation to let us know when the door is open:</p> <ul> <li>Click <strong>Configuration</strong> on the left</li> <li>Click <strong>Automation</strong> on the right</li> <li>Click the plus (+) at the bottom right</li> <li>Set a good name (like &ldquo;Garage door open&rdquo;)</li> <li>Under triggers, look for <code>Vision ZG8101 Garage Door Detector Alarm Level</code> and select it</li> <li>Set <strong>From</strong> to <code>0</code></li> <li>Set <strong>To</strong> to <code>255</code></li> <li>Leave the <strong>For</strong> spot empty</li> </ul> <p>Now that we can detect the garage door being open, we need a notification action. I love <a href="https://www.pushbullet.com/">PushBullet</a> and I have an action set up for PushBullet notifications already. Here&rsquo;s how to use an action:</p> <ul> <li>Select <strong>Call Service</strong> for <strong>Action Type</strong> in the <strong>Actions</strong> section</li> <li>Select a service to call when the trigger occurs</li> <li><strong>Service data</strong> should contain the json that contains the notification message and title</li> </ul> <p>Here&rsquo;s an example of my service data:</p> <div class="highlight"><pre style="color:#93a1a1;background-color:#002b36;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-json" data-lang="json">{ <span style="color:#268bd2">&#34;message&#34;</span>: <span style="color:#2aa198">&#34;Someone opened the garage door at home.&#34;</span>, <span style="color:#268bd2">&#34;title&#34;</span>: <span style="color:#2aa198">&#34;Garage door opened&#34;</span> }</code></pre></div> <p>Press the orange and white save icon at the bottom right and you are ready to go! You can tilt the sensor in your hand to test it or attach it to your garage door and test it there.</p> <p>If you want to know when the garage door is closed, follow the same steps above, but use <code>255</code> for <strong>From</strong> and <code>0</code> for <strong>To</strong>.</p> Allow a port range with firewalld https://major.io/2019/01/04/allow-port-range-with-firewalld/ Fri, 04 Jan 2019 00:00:00 UTC Major Hayden https://major.io/2019/01/04/allow-port-range-with-firewalld/ <p>Managing iptables gets a lot easier with <a href="https://firewalld.org/">firewalld</a>. You can manage rules for the IPv4 and IPv6 stacks using the same commands and it provides fine-grained controls for various &ldquo;zones&rdquo; of network sources and destinations.</p> <h2 id="quick-example">Quick example</h2> <p>Here&rsquo;s an example of allowing an arbitrary port (for <a href="https://github.com/netdata/netdata">netdata</a>) through the firewall with iptables and firewalld on Fedora:</p> <div class="highlight"><pre style="color:#93a1a1;background-color:#002b36;-moz-tab-size:4;-o-tab-size:4;tab-size:4">## iptables iptables -A INPUT -j ACCEPT -p tcp --dport 19999 ip6tables -A INPUT -j ACCEPT -p tcp --dport 19999 service iptables save service ip6tables save ## firewalld firewall-cmd --add-port=19999/tcp --permanent</pre></div> <p>In this example, <code>firewall-cmd</code> allows us to allow a TCP port through the firewall with a much simpler interface and the change is made permanent with the <code>--permanent</code> argument.</p> <p>You can always test a change with firewalld without making it permanent:</p> <div class="highlight"><pre style="color:#93a1a1;background-color:#002b36;-moz-tab-size:4;-o-tab-size:4;tab-size:4">firewall-cmd --add-port=19999/tcp ## Do your testing to make sure everything works. firewall-cmd --runtime-to-permanent</pre></div> <p>The <code>--runtime-to-permanent</code> argument tells firewalld to write the currently active firewall configuration to disk.</p> <h2 id="adding-a-port-range">Adding a port range</h2> <p>I use <a href="https://mosh.org/">mosh</a> with most of my servers since it allows me to reconnect to an existing session from anywhere in the world and it makes higher latency connections less painful. Mosh requires a range of UDP ports (60000 to 61000) to be opened.</p> <p>We can do that easily in firewalld:</p> <div class="highlight"><pre style="color:#93a1a1;background-color:#002b36;-moz-tab-size:4;-o-tab-size:4;tab-size:4">firewall-cmd --add-port=60000-61000/udp --permanent</pre></div> <p>We can also see the rule it added to the firewall:</p> <div class="highlight"><pre style="color:#93a1a1;background-color:#002b36;-moz-tab-size:4;-o-tab-size:4;tab-size:4"># iptables-save | grep 61000 -A IN_public_allow -p udp -m udp --dport 60000:61000 -m conntrack --ctstate NEW,UNTRACKED -j ACCEPT # ip6tables-save | grep 61000 -A IN_public_allow -p udp -m udp --dport 60000:61000 -m conntrack --ctstate NEW,UNTRACKED -j ACCEPT</pre></div> <p>If you haven&rsquo;t used firewalld yet, give it a try! There&rsquo;s a lot more documentation on common use cases in the <a href="https://docs.fedoraproject.org/en-US/quick-docs/firewalld/">Fedora firewalld documentation</a>.</p> Disable autoplay for videos in Firefox 65 https://major.io/2018/12/18/disable-autoplay-for-videos-in-firefox-65/ Tue, 18 Dec 2018 00:00:00 UTC Major Hayden https://major.io/2018/12/18/disable-autoplay-for-videos-in-firefox-65/ <p>Firefox has some great features, but one of my favorites is the ability to disable autoplay for videos. We&rsquo;ve all had one of those moments: your speakers are turned up and you browse to a website with an annoying advertisement that plays immediately.</p> <p><img src='../gifs/2018-12-18-just-want-it-to-stop.gif' alt="GIF: I just want it to stop." style="display: block; margin: auto;"></p> <p>This feature stopped working for me somewhere in the Firefox 65 beta releases. Also, the usual setting in the preference page (under <em>Privacy &amp; Security</em>) seems to be missing.</p> <p>Luckily we can edit Firefox&rsquo;s configuration directly to get this feature working again. Open up a new browser tab, go to <code>about:config</code>, and adjust these settings:</p> <ul> <li><p>Set <code>media.autoplay.default</code> to <code>1</code> to disable video autoplay for all sites</p></li> <li><p>Set <code>media.autoplay.allow-muted</code> to <code>false</code> to disable video autoplay <em>even for muted videos</em></p></li> </ul> <p>Those changes take effect for any new pages that you open after making the change.</p> Getting started with ham radio repeaters https://major.io/2018/12/13/getting-started-with-ham-radio-repeaters/ Thu, 13 Dec 2018 00:00:00 UTC Major Hayden https://major.io/2018/12/13/getting-started-with-ham-radio-repeaters/ <p><a href="https://en.wikipedia.org/wiki/Amateur_radio">Amateur radio</a> is a fun way to mess around with technology, meet new people, and communicate off the grid. Talking directly to another radio on a single frequency (also called <em>simplex</em>) is the easiest way to get started. However, it can be difficult to communicate over longer distances without amplifiers, proper wiring, and antennas. This is where a radio repeater can help.</p> <h2 id="what-s-in-scope">What&rsquo;s in scope</h2> <p>This post is focused on fairly local communication on VHF/UHF bands. The most common frequencies for local communication in these bands are:</p> <ul> <li>2 meters (~144-148MHz)*</li> <li>70 centimeters (~420-450MHz)*</li> </ul> <p><em>* NOTE: Always consult the <a href="http://www.arrl.org/band-plan">band plan</a> for your area to see which part of the frequency band you could and should use.</em></p> <p>Of course, you can do some amazing things with weak signal VHF (which can be used to commuinicate over <strong>great</strong> distances), but we&rsquo;re not talking about that here. The <a href="http://www.144200.net/about.html">HAMSter Amateur Radio Group</a> is a great place to get started with that.</p> <p>We&rsquo;re also not talking about radio bands longer than 2 meters (which includes high frequency (HF) bands). Some of those bands require advanced FCC licensing that takes additional studying and practice.</p> <h2 id="keeping-it-simple-x">Keeping it simple(x)</h2> <p><a href="https://en.wikipedia.org/wiki/Simplex_communication">Simplex radio</a> involves communication where radios are tuned to a single frequency and only one radio can transmit at a time. This is like a simple walkie-talkie. If one person is transmitting, everyone else listens. If someone else tries to transmit at the same time, then the waves will be garbled and nobody will be able to hear either person. This is often called &ldquo;doubling up&rdquo;.</p> <p>This method works well when radios are in range of each other without a bunch of objects in between. However, it&rsquo;s difficult to talk via simplex over great distances or around big obstables, such as mountains or hills.</p> <h2 id="repeaters">Repeaters</h2> <p><a href="https://en.wikipedia.org/wiki/Repeater">Repeaters</a> are a little more complex to use, but they provide some great benefits. A repeater usually consists of one or two radios, one or two antennas, duplexers, and some other basic equipment. They receive a signel on one frequency and broadcast that same signal on another frequency. They often are mounted high on towers and this gives them a much better reach than antennas on your car or home.</p> <p>I enjoy using a repeater here in San Antonio called <a href="https://repeaterbook.com/repeaters/details.php?state_id=48&amp;ID=11397">KE5HBB</a>. The repeater has this configuration:</p> <ul> <li>Downlink: 145.370</li> <li>Uplink: 144.770</li> <li>Offset: -0.6 MHz</li> <li>Uplink Tone: 114.8</li> <li>Downlink Tone: 114.8</li> </ul> <p>Let&rsquo;s make sense of this data:</p> <ul> <li><p>Downlink: This is the frequency that the repeater uses to <em>transmit</em>. In other words, when people talk on this repeater, this is the frequency you use to hear them.</p></li> <li><p>Uplink: The receiver <em>listens</em> on this frequency. If you want to talk to people who are listening to this repeater, you need to transmit on this frequency.</p></li> <li><p>Offset: This tells you how to calculate the uplink frequency if it is not shown. This repeater has a negative 0.6 offset, so we can calculate the uplink frequency if it was not provided:</p></li> </ul> <div class="highlight"><pre style="color:#93a1a1;background-color:#002b36;-moz-tab-size:4;-o-tab-size:4;tab-size:4">145.370 - 0.600 = 144.770</pre></div> <ul> <li>Uplink/Downlink Tones: Your radio must transmit this tone to <em>open the squelch</em> on the repeater (more on this in a moment). The repeater will use the same tone to transmit, so we can configure our radio to listen for that tone and only open our squelch when it is detected.</li> </ul> <h2 id="opening-the-squelch">Opening the squelch</h2> <p>Transmitting radio waves uses a lot of power and it creates a lot of heat. There are parts of a radio that will wear out much more quickly if a radio is transmitting constantly. This is why receivers have a <em>squelch</em>. This means that a radio must transmit something strong enough on the frequency (or use a tone) to let the repeater know that it needs to repeat something.</p> <p>You may come across repeaters with no tones listed (sometimes shown as <em>PL</em>). This means that you can just transmit on the uplin frequency and the repeater will repeat your signal. These repeaters are easy to use, but they can create problems.</p> <p>Imagine if you&rsquo;re traveling through an area and you&rsquo;re using a frequency to talk to a friend in another car. As you&rsquo;re driving, you move in range of a repeater that is listening on that frequeny. Suddenly your conversation is now being broadcasted through the repeater and everyone listening to that repeater must listen to you. This isn&rsquo;t what you expected and it could be annoying to other listeners.</p> <p>Also, in crowded urban areas, there&rsquo;s always a chance that signals might end up on the repeater&rsquo;s listening frequency unintentionally. That would cause the repeater to start transmitting and it would increase wear.</p> <p>Two repeaters might be relatively close (or just out of range) and the tone helps each repeater identify its own valid radio traffic.</p> <h2 id="tuning-the-tones">Tuning the tones</h2> <p>Most repeaters have a <em>tone squelch</em>. That means you can blast them with 100 watts of radio waves and they won&rsquo;t repeat a thing until you transmit an inaudible tone at the beginning of your transmission.</p> <p>As an example, in the case of KE5HBB, this tone is 114.8. You must configure a <a href="https://en.wikipedia.org/wiki/Continuous_Tone-Coded_Squelch_System">CTCSS</a> tone on your radio so that the tone is transmitted as soon as you begin transmitting. That signals the repeater that it&rsquo;s time to repeat. These signals aren&rsquo;t audible to humans.</p> <p>If you know you&rsquo;re tuned to the right frequency to transmit (the uplink frequency), but the repeater won&rsquo;t repeat your traffic, then you are most likely missing a tone. There&rsquo;s also a chance that you programmed the uplink and downlink tones into your radio in reverse, so check that, too.</p> <h2 id="repeater-transmit-tone">Repeater transmit tone</h2> <p>Some receivers will transmit a tone when they broadcast back to you, but some won&rsquo;t. If you can transmit but you can&rsquo;t hear anyone else when they talk, double check your radio&rsquo;s settings for a tone squelch on the receiving side. Your radio can also listen for these tones and only open its squelch when it hears them.</p> <p>I usually disable receiver squelch for tones on my radio since the repeater operator could disable that feature at any time and I wouldn&rsquo;t be able to hear any transmissions since my radio would be waiting for the tone.</p> <h2 id="testing-a-repeater">Testing a repeater</h2> <p>First off, please don&rsquo;t test a repeater unless you have a proper amateur radio license in your jurisdiction. In the United States, that&rsquo;s the FCC. Don&rsquo;t skip this step.</p> <p>Once you get your repeater&rsquo;s frequencies programmed into your radio properly and you&rsquo;ve double checked the settings for sending tones, you can try &ldquo;breaking the squelch.&rdquo;</p> <p>Press the transmit button on your radio briefly for about half second and release. You should hear something when you do this. For some repeaters, you may hear a <em>KERRRCHUNK</em> noise. That&rsquo;s the sound of the repeater squelch closing the transmission now that you&rsquo;re done with your transmission. On other repeaters, you may hear some audible tones or beeps as soon as you release the transmit button.</p> <p>Once you have it working properly, stop breakng the squelch and introduce yourself! For example, when I&rsquo;m in my car, I might say: <em>&ldquo;W5WUT mobile and monitoring.&rdquo;</em> That lets people on the repeater know that I&rsquo;m there and that I&rsquo;m moving (so I might not be on for a very long time).</p> <p>Good luck on the radio waves! 73&rsquo;s from W5WUT.</p> Use a secret as an environment variable in OpenShift deployments https://major.io/2018/12/06/use-secret-as-environment-variable-in-openshift-deployments/ Thu, 06 Dec 2018 00:00:00 UTC Major Hayden https://major.io/2018/12/06/use-secret-as-environment-variable-in-openshift-deployments/ <p>OpenShift <a href="https://docs.openshift.com/container-platform/3.9/dev_guide/deployments/how_deployments_work.html">deployments</a> allow you to take a container image and run it within a cluster. You can easily add extra items to the deployment, such as environment variables or volumes.</p> <p>The best practice for sensitive environment variables is to place them into a <a href="https://docs.openshift.com/container-platform/3.9/dev_guide/secrets.html">secret object</a> rather than directly in the deployment configuration itself. Although this keeps the secret data out of the deployment, the environment variable is still exposed to the running application inside the container.</p> <h2 id="creating-a-secret">Creating a secret</h2> <p>Let&rsquo;s start with a snippet of a <code>deploymentConfig</code> that has a sensitive environment variable in plain text:</p> <div class="highlight"><pre style="color:#93a1a1;background-color:#002b36;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-yml" data-lang="yml">spec: containers: - env: - name: MYAPP_SECRET_TOKEN value: vPWps5E7KO8KPlljaD3eXED3f6jmLsV5mQ image: <span style="color:#2aa198">&#34;fedora:29&#34;</span> name: my_app</code></pre></div> <p>The first step is to create a secret object that contains our sensitive environment variable:</p> <div class="highlight"><pre style="color:#93a1a1;background-color:#002b36;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-yml" data-lang="yml">apiVersion: v1 kind: Secret metadata: name: secret-token-for-my-app stringData: SECRET_TOKEN: vPWps5E7KO8KPlljaD3eXED3f6jmLsV5mQ</code></pre></div> <p>Save this file as <code>secret-token.yml</code> and deploy it to OpenShift:</p> <div class="highlight"><pre style="color:#93a1a1;background-color:#002b36;-moz-tab-size:4;-o-tab-size:4;tab-size:4">oc apply -f secret-token.yml</pre></div> <p>Query OpenShift to ensure the secret is ready to use:</p> <div class="highlight"><pre style="color:#93a1a1;background-color:#002b36;-moz-tab-size:4;-o-tab-size:4;tab-size:4">$ oc get secret/secret-token-for-my-app NAME TYPE DATA AGE secret-token-for-my-app Opaque 1 1h</pre></div> <h2 id="using-the-secret">Using the secret</h2> <p>We can adjust the deployment configuration to use this new secret:</p> <div class="highlight"><pre style="color:#93a1a1;background-color:#002b36;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-yml" data-lang="yml">spec: containers: - env: - name: MYAPP_SECRET_TOKEN valueFrom: secretKeyRef: key: SECRET_TOKEN name: secret-token-for-my-app image: <span style="color:#2aa198">&#34;fedora:29&#34;</span> name: my_app</code></pre></div> <p>This configuration tells OpenShift to look inside the secret object called <code>secret-token-for-my-app</code> for a key matching <code>SECRET_TOKEN</code>. The value will be passed into the <code>MYAPP_SECRET_TOKEN</code> environment variable and it will be available to the application running in the container.</p> <p><strong>Security note:</strong> If someone has access to change the deployment configuration object, they could get access to the value of the secret without having direct access to the secret object itself. It would be trivial to change the startup command in the container to print all of the environment variables in the container (using the <code>env</code> command) and view them in the container logs.</p> Make alt-arrow keys work with terminator and weechat https://major.io/2018/09/05/make-alt-arrow-keys-work-with-terminator-and-weechat/ Thu, 06 Sep 2018 03:43:30 UTC Major Hayden https://major.io/2018/09/05/make-alt-arrow-keys-work-with-terminator-and-weechat/ <p>As I make the move from the world of GNOME to i3, I found myself digging deeper into the <a href="https://terminator-gtk3.readthedocs.io/en/latest/">terminator</a> preferences to make it work more like <a href="https://help.gnome.org/users/gnome-terminal/stable/">gnome-terminal</a>.</p> <p>I kept running into an issue where I couldn&rsquo;t move up and down between buffers using alt and arrow keys. My workaround was to call the buffer directly with alt-8 (for buffer #8) or alt-j 18 (buffer #18). However, that became tedious. Sometimes I just wanted to quickly hop up or down one or two buffers.</p> <p>To fix this problem, right click anywhere inside the terminal and choose <em>Preferences</em>. Click on the <em>Keybindings</em> tab and look for <code>go_up</code> and <code>go_down</code>. These are almost always set to <em>Alt-Up</em> and <em>Alt-Down</em> by default. That&rsquo;s the root of the problem: terminator is grabbing those keystrokes before they can make it down into weechat.</p> <p>Unfortunately, it&rsquo;s not possible to clear a keybinding within the preferences dialog. Close the window and open <code>~/.config/terminator/config</code> in a terminal.</p> <p>If you&rsquo;re new to terminator, you might not have a <code>[keybindings]</code> section in your configuration file. If that&rsquo;s the case, add the whole section below the <code>[global_config]</code> section. Otherwise, just ensure your <code>[keybindings]</code> section contains these lines:</p> <div class="highlight"><pre style="color:#93a1a1;background-color:#002b36;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-ini" data-lang="ini"><span style="color:#719e07">[keybindings]</span> go_down <span style="color:#719e07">=</span> <span style="color:#2aa198">None </span><span style="color:#2aa198"> go_up = None</span></code></pre></div> <p>Close <em>all</em> of the terminator windows (on all of your workspaces). <strong>This is a critical step!</strong> Terminator only loads the config file when it is first started, not when additional terminals are opened.</p> <p>Open a terminator terminal, start weechat, and test your alt-arrow keys! You should be moving up and down between buffers easily. If that doesn&rsquo;t work, check your window manager&rsquo;s settings to ensure that another application hasn&rsquo;t stolen that keybinding from your terminals.</p> How to thrive at a technical conference https://major.io/2018/05/09/how-to-thrive-at-a-tech-conference/ Wed, 09 May 2018 23:54:28 UTC Major Hayden https://major.io/2018/05/09/how-to-thrive-at-a-tech-conference/ <p><img src="../wp-content/uploads/2018/05/IMG_20180508_202748.jpg" alt="1" /></p> <p>I&rsquo;m at the <a href="https://www.redhat.com/es/summit/2018">2018 Red Hat Summit</a> this week in San Francisco and I am enjoying the interactions between developers, executives, vendors, and engineers. It&rsquo;s my seventh Summit (but my first as a Red Hat employee!), but I regularly meet people who are attending their first technical conference.</p> <p>The question inevitably comes up: &ldquo;I&rsquo;m so tired. How do you survive these events?&rdquo;</p> <p>One attendee asked me to write a blog post on my tips and tricks. This is the post that explains how to thrive, not just survive, at conferences. Beware - these tips are based on my experiences and your mileage may vary depending on your personality, the event itself, and your caffeine intake.</p> <h2 id="discover-the-area">Discover the area</h2> <p>Traveling to a conference is awesome way to experience more of the world! Take time to enjoy the tourist sites but also find out where the locals like to go. Any hotel concierge should be able to give you advice on where to go to truly experience the location.</p> <p>Take some time to learn the area around your hotel and the venue. Be sure you can navigate between the two and find some important spots nearby, like pharmacies and coffee shops.</p> <h2 id="food-water-and-sleep">Food, water, and sleep</h2> <p>These conferences can often feel overwhelming and you may find yourself forgetting to eat the right foods, stay hydrated, and get some rest.</p> <p>Take every opportunity to eat healthier foods during the week that will give you energy without weighing you down. All the stuff that your Mom told you to eat is a good idea. My rule of thumb is to eat a heavy breakfast, a medium sized lunch, and then whatever I want for dinner. Evening events often have free food (more on those events next), and that fits my travel budget well. It also allows me to splurge a bit on foods that I might not eat back home.</p> <p>Take along a water container when you travel. You can&rsquo;t always depend on the conference for making water available and you&rsquo;ll often need more than they offer anyway. I&rsquo;m a big fan of <a href="https://www.nalgene.com/">Nalgene&rsquo;s</a> products since they take a beating and they have really good seals.</p> <p>Sleeping is a real challenge. Early morning keynotes and late night events put a strain on anyone&rsquo;s sleep schedule. Lots of people have trouble sleeping in hotels or in cities where the noise level remains high all night long. The best remedy is to be choosy about the events you attend and the time you spend there. Think about what is more valuable: more time listening to blasting music at a party or more time with your head on the pillow.</p> <p>Consider using an application on your phone that provides various types of noises, such as <a href="https://en.wikipedia.org/wiki/White_noise">white noise</a>. I love the <a href="https://play.google.com/store/apps/details?id=com.tmsoft.whitenoise.full">White Noise</a> app on Android since it has tons of options for various sounds. In my experience, brown noise works best for sleeping. Pink noise can help in extremely noisy environments (like downtown San Francisco) but it&rsquo;s often too loud for me.</p> <h2 id="keep-your-devices-charged">Keep your devices charged</h2> <p>Find a way to keep your devices charged, especially your phone. I use <a href="https://www.anker.com/">Anker</a> battery packs to keep my phone topped up during the day when I can&rsquo;t get to a plug. A dead phone disconnects you from your friends, maps, and conference details.</p> <h2 id="dress-for-success">Dress for success</h2> <p>Your clothing selection really depends on the type of conference and the company you represent. If you need to dress formally each day, then your choices are already made for you.</p> <p>Pack layers of clothing so you can add or remove layers as needed. The walk to the conference center may be warm, but the keynote auditorium could feel like a freezer. This also prepares you for evening events which might be outdoors.</p> <p>Wear clothing that makes you feel comfortable. You&rsquo;ll find a wide range of outfits at most tech conferences and you&rsquo;ll find that nobody really cares how formal or informal you are. If you&rsquo;re there to listen, learn, and contribute, then dress casually. If you&rsquo;re looking for a new job, doing a talk, or if you&rsquo;ll be on camera, choose something a little more formal.</p> <h2 id="the-hallway-track">The hallway track</h2> <p>You won&rsquo;t find the hallway track on any agenda, but it is often the most valuable part of any gathering. The hallway track encompasses those brief encounters you have with other people at the event. Turn those mundane events, such as waiting in line, eating lunch, or between talks, into opportunities to meet other people.</p> <p>Yes, this does mean that you must do something to come out of your shell and start a conversation. This is still difficult for me. Here are some good ways to start a conversation with someone around you:</p> <ul> <li>&ldquo;Hello, my name is Major&rdquo; (put out your hand for a handshake)</li> <li>&ldquo;Where do you work?&rdquo;</li> <li>&ldquo;What do you work on?&rdquo;</li> <li>&ldquo;Man, this line is really long.&rdquo;</li> <li>&ldquo;vim or emacs?&rdquo; <em>(just kidding)</em></li> </ul> <p>The secret is to find something that makes you and the other person feel comfortable. There are situations where you might be met with a cold shoulder, and that&rsquo;s okay. I&rsquo;ve found that sometimes people need some space or the issue could be a language barrier. Making the attempt is what matters.</p> <p>These are excellent opportunities for learning, for listening, and for sharing. These new contacts will show up again and again at the event (more on parties/networking next), and you can talk to them again when you feel the tendency to become a <a href="https://en.wikipedia.org/wiki/Wallflower_(people)">wallflower</a> again.</p> <h2 id="parties-and-networking-events">Parties and networking events</h2> <p>Evening events at conferences are a great way to keep the hallway track going while taking some time to relax as well. Some of the best conversations I&rsquo;ve had at conferences were during evening events or vendor parties. People are more candid since the conference demands are often reduced.</p> <p>However, it&rsquo;s incredibly easy to make some spectacularly bad decisions at these events. This list should help you navigate these events and get value from them:</p> <h3 id="enjoy-an-open-bar-responsibly">Enjoy an open bar responsibly</h3> <p>Early in my career, I looked at an open bar as a magical oasis. Free drinks! As many as I want! This is heaven! (Narrator: <em>It was not heaven. It was something else.</em>)</p> <p>I think about open bars much like I think about a trip to Las Vegas. Before I go, I think about how much money I feel like losing, and I only bet that much. <strong>Once the money is gone, I&rsquo;m done.</strong></p> <p>Go into the event knowing how much or how little you want to consume. <strong>Zero is an entirely valid answer.</strong> Keep in mind that the answer to &ldquo;Why aren&rsquo;t you drinking anything?&rdquo; does not have to be &ldquo;I guess I&rsquo;ll get something.&rdquo; <strong>Nobody needs to know why you&rsquo;re not drinking and you shouldn&rsquo;t feel pressured to do something you don&rsquo;t want to do.</strong></p> <p>Think about how you want to feel in the morning. Is a massive hangover worth another round of shots? Is it worth it to ruin your talk the next day? Is it worth it to get belligerent and say something that may be difficult to take back? Think about these things ahead of time and make a plan before you begin drinking.</p> <h3 id="leave-when-you-want">Leave when you want</h3> <p>Some evening events can last much too late and this could derail your plans for the morning. If the party runs from 7-10PM, don&rsquo;t feel obligated to stay until 10PM. If you&rsquo;re not meeting the right people or if you&rsquo;re not having a good time: leave. It&rsquo;s better to abandon an event early than suffer through it and crawl through the next morning.</p> <h3 id="turn-down-an-uninteresting-invitation">Turn down an uninteresting invitation</h3> <p>The conference may host various events or a vendor may invite you to an event. These are just invitations and your attendance is not required (unless you work for the vendor throwing the party). Feel free to do something else with your time if the event or the venue seem uninteresting or unsafe. (More on safety next.)</p> <h3 id="get-a-party-buddy">Get a party buddy</h3> <p>Remember those people you talked to in the hallway and during lunch? Find those people at the event and tell them you enjoyed the conversation from earlier. I&rsquo;ve been to conferences before where I&rsquo;ve been the only one from my company and after letting the other person know that, they invited me to hang out with them or their group at the event.</p> <p>This is a good idea for two reasons. First, it gives you someone to talk to. More importantly, it helps you stay safe.</p> <h2 id="dealing-with-harassment">Dealing with harassment</h2> <p>This gets its own section. It has happened to me and it will likely happen to you.</p> <p>Nobody ever wants it to happen, but people are often harassed in one way or another at these events. It&rsquo;s inevitable: there are drinks, people are away from home, and they&rsquo;re enjoying time away from work. For some people, this is a combination of factors that leads them to make bad choices at these events.</p> <p>Harassment comes in many forms, but nobody should put up with it. If you see someone being treated badly, step in. If you&rsquo;re being treated badly, get help. If you&rsquo;re treating someone badly, apologize and remove yourself from the situation. This is where a party buddy can be extremely helpful.</p> <p>Harassment is not a women-only or men-only problem. I have been touched in unwelcome ways and verbally harassed at evening events. It is not fun. In my experience, telling the other person to &ldquo;Please stop&rdquo; or &ldquo;That is not okay&rdquo; is usually enough to diffuse the situation.</p> <p>This may not always work. Grab your buddy and get help from conference staffers or a security guard if a situation continues to escalate.</p> <h2 id="more-ideas">More ideas</h2> <p>These are some ideas that help me thrive at conferences and make the most of my time traveling. Feel free to leave some of your ideas below in the comments section!</p> Reaching the fork in the road https://major.io/2018/03/07/reaching-the-fork-in-the-road/ Wed, 07 Mar 2018 16:18:51 UTC Major Hayden https://major.io/2018/03/07/reaching-the-fork-in-the-road/ <p><img src="../wp-content/uploads/2018/03/1024px-Mountain_trail_in_Panachaiko_mountains_Greece.jpg" alt="1" /></p> <p>Walt Disney said it best:</p> <blockquote> <p>We keep moving forward, opening new doors, and doing new things, because we&rsquo;re curious and curiosity keeps leading us down new paths.</p> </blockquote> <p>The world of technology is all about change. We tear down the old things that get in our way and we build new technology that takes us to new heights. Tearing down these old things can often be difficult and that forces us to make difficult choices.</p> <p>Rackspace has been a great home for me for over 11 years. I&rsquo;ve made the incredibly difficult choice to leave Rackspace on March 9th to pursue new challenges.</p> <h2 id="humble-beginnings">Humble beginnings</h2> <p>I came to Rackspace as an entry-level Linux administrator and was amazed by the culture generated by Rackers. The dedication to customers, technology, and quality was palpable from the first few minutes I spent with my team. Although I didn&rsquo;t know it at the time, I had landed at the epicenter of a sink-or-swim technology learning experience. My team had some very demanding customers with complex infrastructures and it forced me to take plenty of notes (and hard knocks). My manager and teammates supported me through it all.</p> <p>From there, I served in several different roles. I was a manager of technicians on a support team and had the opportunity to learn how to mentor. One of my favorite leaders said that &ldquo;good managers know when to put their arm around to people and when to put a boot in their rear.&rdquo; I reluctantly learned how to do both and I watched my people grow into senior engineers and great leaders.</p> <p><img src="../wp-content/uploads/2018/03/6519121761_ab65bab3c1_b.jpg" alt="/wp-content/uploads/2018/03/6519121761_ab65bab3c1_b.jpg" /> Datapoint office closing in 2011</p> <p>I was pulled to Mosso, Rackspace&rsquo;s first cloud offering, shortly after that and discovered an entirely new world. Rackers force-fed me &ldquo;Why&rsquo;s (Poignant) Guide to Ruby&rdquo; and I started building scripts and web front-ends for various services. Rackspace acquired Slicehost after that and I jumped at the chance to work as an operations engineer on the new infrastructure. That led to a lot of late nights diagnosing problems with Xen hypervisors and rails applications. I met some amazing people and began to realize that St. Louis has some pretty good barbecue (but Texas still has them beat).</p> <p><img src="../wp-content/uploads/2018/03/4171091103_7150ded95f_b.jpg" alt="/wp-content/uploads/2018/03/4171091103_7150ded95f_b.jpg" /> Slicehost humor in 2009</p> <p>Not long after that, I found myself managing an operations team that cared for Slicehost&rsquo;s infrastructure and Rackspace&rsquo;s growing Cloud Servers infrastructure. OpenStack appeared later and I jumped at the chance to do operations there. It was an extremely rough experience in the Diablo release, but it taught me a lot. My start with OpenStack involved fixing lots of broken Keystone tests that didn&rsquo;t run on Python 2.6.</p> <p><img src="../wp-content/uploads/2018/03/7730840100_01257c5fa4_b.jpg" alt="/wp-content/uploads/2018/03/7730840100_01257c5fa4_b.jpg" /> Working on OpenStack in 2012</p> <p>If you&rsquo;ve attended some of my talks on impostor syndrome, you may know what came next. We had a security issue and I sent some direct feedback to our CSO about how it was handled. I expected to be told to &ldquo;pack a box&rdquo; after that, but I was actually asked to lead a security architecture team in the corporate security group. It was definitely a surprise. I accepted and joined the team as Chief Security Architect. My coworkers called it &ldquo;joining the dark side&rdquo;, but I did my best to build bridges between security teams and the rest of the company.</p> <p><img src="../wp-content/uploads/2018/03/24142777780_5196ca622b_h.jpg" alt="/wp-content/uploads/2018/03/24142777780_5196ca622b_h.jpg" /> Talking at Rackspace::Solve in 2015</p> <p>This role really challenged me. I had never operated at the Director level before and our team had a ton of work to do. I found myself stumbling (and floundering) fairly often and I leaned on other leaders in the business for advice. This led me to take some courses on critical thinking, accounting, finance, and tough conversations. I&rsquo;ve never had a role as difficult as this one.</p> <p>Our cloud team came calling and asked me to come back and help with some critical projects in the public cloud. We worked on some awesome skunkworks projects that could really change the business. Although they didn&rsquo;t get deployed in one piece, we found ways to take chunks of the work and optimize different areas of our work. An opportunity came up to bring public cloud experience to the private cloud team and I jumped on that one. I discovered the awesome OpenStack-Ansible project and a strong set of Rackers who were dedicated to bringing high-touch service to customers who wanted OpenStack in their own datacenter.</p> <p><img src="../wp-content/uploads/2018/03/imposter-syndrome_hayden.jpg" alt="/wp-content/uploads/2018/03/imposter-syndrome_hayden.jpg" /> Impostor syndrome talk at the Boston OpenStack Summit in 2017</p> <p>During this time, I had the opportunity to deliver several conference talks about OpenStack, Fedora, security, and Ansible. My favorite topic was impostor syndrome and I set out on a mission to help people understand it. My first big talk was at the Fedora Flock conference in Rochester in 2015. This led to deep conversations with technical people in conference hallways, evening events, and even airport terminals about how impostor syndrome affects them. I took those conversations and refined my message several times over.</p> <p><img src="../wp-content/uploads/2018/03/DSCF0425.jpg" alt="/wp-content/uploads/2018/03/DSCF0425.jpg" /> Talking about impostor syndrome at Fedora Flock 2015 (Photo credit: Kushal Das)</p> <h2 id="gratitude">Gratitude</h2> <p>I couldn&rsquo;t even begin to name a list of Rackers who have helped me along the way. I wouldn&rsquo;t be where I am now without the help of hundreds of Rackers. They&rsquo;ve taught me how to build technology, how to navigate a business, and how to be a better human. They have made me who I am today and I&rsquo;m eternally grateful. I&rsquo;ve had an incredible amount of hugs this week at the office and I&rsquo;ve tried my best not to get a face full of tears in the process.</p> <p>I&rsquo;d also like to thank all of the people who have allowed me to mentor them and teach them something along the way. One of the best ways to understand something is to teach it to someone else. I relish any opportunity to help someone avoid a mistake I made, or at least be able to throw something soft under them to catch their fall. These people put up with my thick Texas accent, my erratic whiteboard diagrams, and worse of all, my dad jokes.</p> <p>Another big &ldquo;thank you&rdquo; goes out to all of the members of the open source communities who have mentored me and dealt with my patches.</p> <p>The first big community I joined was the Fedora Linux community. I&rsquo;ve been fortunate to serve on the board and participate in different working groups. Everyone has been helpful and accommodating, even when I pushed broken package builds. I plan to keep working in the community as long as they will have me!</p> <p>The OpenStack community has been like family. Everyone - from developers to foundation leaders - has truly been a treat to work with over several years. My work on Rackspace&rsquo;s public and private clouds has pushed me into various projects within the OpenStack ecosystem and I&rsquo;ve found everyone to be responsive. OpenStack events are truly inspiring and it is incredible to see so many people from so many places who dedicate themselves to the software and the people that make cloud infrastructure work.</p> <h2 id="the-next-adventure">The next adventure</h2> <p>I plan to talk more on this later, but I will be working from home on some projects that are entirely different from what I&rsquo;m working on now. That adventure starts on March 19th after a week of &ldquo;funemployment.&rdquo; I&rsquo;m incredibly excited about the new opportunity and I&rsquo;ll share more details when I can.</p> <p><em>Top photo credit: <a href="https://commons.wikimedia.org/wiki/File:Mountain_trail_in_Panachaiko_mountains,_Greece.jpg">Wikipedia</a></em></p> Install testing kernels in Fedora https://major.io/2018/02/28/install-testing-kernels-in-fedora/ Wed, 28 Feb 2018 13:53:48 UTC Major Hayden https://major.io/2018/02/28/install-testing-kernels-in-fedora/ <p><img src="../wp-content/uploads/2018/02/120928-F-YV474-917.jpg" alt="1" /></p> <p>If you&rsquo;re on the latest Fedora release, you&rsquo;re already running lots of modern packages. However, there are those times when you may want to help with testing efforts or try out a new feature in a newer package.</p> <p>Most of my systems have the <code>updates-testing</code> repository enabled in one way or another. This repository contains packages that package maintainers have submitted to become the next stable package in Fedora. For example, if there is a bug fix for nginx, the package maintainer submits the changes and publish a release. That release goes into the testing repositories and must sit for a waiting period or receive sufficient karma (&ldquo;works for me&rdquo; responses) to move into stable repositories.</p> <h2 id="getting-started">Getting started</h2> <p>One of the easiest ways to get started is to allow a small amount of packages to be installed from the testing repository on a regular basis. Fully enabling the testing repository for all packages can lead to trouble on occasion, especially if a package maintainer discovers a problem and submits a new testing package.</p> <p>To get started, open <code>/etc/yum.repos.d/fedora-updates-testing.repo</code> in your favorite text editor (using <code>sudo</code>). This file tells yum and dnf where it should look for packages. The stock testing repository configuration looks like this:</p> <div class="highlight"><pre style="color:#93a1a1;background-color:#002b36;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-ini" data-lang="ini"><span style="color:#719e07">[updates-testing]</span> name<span style="color:#719e07">=</span><span style="color:#2aa198">Fedora $releasever - $basearch - Test Updates</span> failovermethod<span style="color:#719e07">=</span><span style="color:#2aa198">priority</span> <span style="color:#586e75">#baseurl=http://download.fedoraproject.org/pub/fedora/linux/updates/testing/$releasever/$basearch/</span> metalink<span style="color:#719e07">=</span><span style="color:#2aa198">https://mirrors.fedoraproject.org/metalink?repo=updates-testing-f$releasever&amp;arch=$basearch</span> enabled<span style="color:#719e07">=</span><span style="color:#2aa198">0</span> repo_gpgcheck<span style="color:#719e07">=</span><span style="color:#2aa198">0</span> type<span style="color:#719e07">=</span><span style="color:#2aa198">rpm</span> gpgcheck<span style="color:#719e07">=</span><span style="color:#2aa198">1</span> metadata_expire<span style="color:#719e07">=</span><span style="color:#2aa198">6h</span> gpgkey<span style="color:#719e07">=</span><span style="color:#2aa198">file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-$releasever-$basearch</span> skip_if_unavailable<span style="color:#719e07">=</span><span style="color:#2aa198">False</span></code></pre></div> <p>By default, the repository is not enabled (<code>enabled=0</code>).</p> <p>In this example, let&rsquo;s consider a situation where you want to test the latest kernel packages as soon as they reach the testing repository. We need to make two edits to the repository configuration:</p> <ul> <li><code>enabled=1</code> - Allow yum/dnf to use the repository</li> <li><code>includepkgs=kernel*</code> - Only allow packages matching <code>kernel*</code> to be installed from the testing repository</li> </ul> <p>The repository configuration should now look like this:</p> <div class="highlight"><pre style="color:#93a1a1;background-color:#002b36;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-ini" data-lang="ini"><span style="color:#719e07">[updates-testing]</span> name<span style="color:#719e07">=</span><span style="color:#2aa198">Fedora $releasever - $basearch - Test Updates</span> failovermethod<span style="color:#719e07">=</span><span style="color:#2aa198">priority</span> <span style="color:#586e75">#baseurl=http://download.fedoraproject.org/pub/fedora/linux/updates/testing/$releasever/$basearch/</span> metalink<span style="color:#719e07">=</span><span style="color:#2aa198">https://mirrors.fedoraproject.org/metalink?repo=updates-testing-f$releasever&amp;arch=$basearch</span> enabled<span style="color:#719e07">=</span><span style="color:#2aa198">1</span> repo_gpgcheck<span style="color:#719e07">=</span><span style="color:#2aa198">0</span> type<span style="color:#719e07">=</span><span style="color:#2aa198">rpm</span> gpgcheck<span style="color:#719e07">=</span><span style="color:#2aa198">1</span> metadata_expire<span style="color:#719e07">=</span><span style="color:#2aa198">6h</span> gpgkey<span style="color:#719e07">=</span><span style="color:#2aa198">file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-$releasever-$basearch</span> skip_if_unavailable<span style="color:#719e07">=</span><span style="color:#2aa198">False</span> includepkgs<span style="color:#719e07">=</span><span style="color:#2aa198">kernel*</span></code></pre></div> <h2 id="getting-testing-packages">Getting testing packages</h2> <p>Running <code>dnf upgrade kernel*</code> should now pull a kernel from the <code>updates-testing</code> repository. You can verify this by checking the <code>Repository</code> column in the dnf output.</p> <p>If you feel more adventurous later, you can add additional packages (separated by spaces) to the <code>includepkgs</code> line. The truly adventurous users can leave the repo enabled but remove <code>includepkgs</code> altogether. This will pull all available packages from the testing repository as soon as they are available.</p> <h2 id="package-maintainers-need-feedback">Package maintainers need feedback!</h2> <p>One final note: <strong>package maintainers need your feedback</strong> on packages. Positive or negative feedback is very helpful. You can search for the package on <a href="https://bodhi.fedoraproject.org/">Bodhi</a> and submit feedback there, or use the <code>fedora-easy-karma</code> script via the <code>fedora-easy-karma</code> package. The script will look through your installed package list and query you for feedback on each one.</p> <p>Submitting lots of feedback can earn you some <a href="https://badges.fedoraproject.org/badge/in-search-of-the-bull-tester-i">awesome Fedora Badges</a>!</p> <p><em>Photo credit: <a href="http://www.arpc.afrc.af.mil/News/Article-Display/Article/365815/a-wish-come-true-colorado-native-becomes-cadet-for-a-day/">US Air Force</a></em></p> Takeaways from my foray into amateur radio https://major.io/2018/01/06/takeaways-from-my-foray-into-amateur-radio/ Sat, 06 Jan 2018 19:26:53 UTC Major Hayden https://major.io/2018/01/06/takeaways-from-my-foray-into-amateur-radio/ <p><img src="../wp-content/uploads/2018/01/kenwood_mobile_radio.jpg" alt="1" /></p> <p>The <a href="https://www.overlandexpo.com/">Overland Expo in Asheville</a> last year was a great event, and one of my favorite sessions covered the basics about radio communications while overlanding. The instructors shared their radios with us and taught us some tips and tricks for how to save power and communicate effectively on the trail.</p> <p>Back at the office, I was surprised to discover how many of my coworkers had an FCC license already. They gave me tips on getting started and how to learn the material for the exam. I took some of my questions to Twitter and had plenty of help pouring in quickly.</p> <p>This post covers how I studied, what the exam was like, and what I&rsquo;ve learned after getting on the air.</p> <h2 id="the-basics">The basics</h2> <p>FCC licenses in the US for amateur radio operators have multiple levels. Everything starts with the Technician level and you get the most basic access to radio frequencies. From there, you can upgrade (with another exam) to General, and Extra. Each license upgrade opens up more frequencies and privileges.</p> <h2 id="studying">Studying</h2> <p>A coworker recommended the <a href="http://www.arrl.org/ham-radio-license-manual">official ARRL book</a> for the Technician exam and I picked up a paper copy. The content is extremely dry. It was difficult to remain focused for long periods.</p> <p>The entire exam is available in the public domain, so you can actually go straight to the questions that you&rsquo;ll see on the exam and study those. I flipped to the question section in the ARRL book and found the questions I could answer easily (mostly about circuits and electrical parts). For each one that was new or difficult, I flipped back in the ARRL book to the discussion in each chapter and learned the material.</p> <p>I also used <a href="https://hamstudy.org/">HamStudy.org</a> to quickly practice and keep track of my progress. The site has some handy graphs that show you how many questions you&rsquo;ve seen and what your knowledge level of different topics really is. I kept working through questions on the site until I was regularly getting 90% or higher on the practice tests.</p> <h2 id="testing">Testing</h2> <p>Before you test, be sure to <a href="https://www.fcc.gov/help/getting-fcc-registration-number-frn-universal-licensing-system-uls">get a FCC Registration Number</a> (commonly called a FRN). They are free to get and it ensures that you get your license (often called your &lsquo;ticket&rsquo;) as soon as possible. I was told that some examiners won&rsquo;t offer you a test if you don&rsquo;t have your FRN already.</p> <p>The next step is to <a href="http://www.arrl.org/find-an-amateur-radio-license-exam-session">find an amateur radio exam</a> in your area. Exams are available in the San Antonio area every weekend and they are held by different groups. I took mine with the <a href="http://www.w5ros.org/">Radio Operators of South Texas</a> and the examiners were great! Some examiners require you to check in with them so they know you are coming to test, but it&rsquo;s a good idea to do this anyway. Ask how they want to be paid (cash, check, etc), too.</p> <p>Be sure to take a couple of pencils, a basic calculator, your government issued ID, your payment, and your FRN to the exam. I forgot the calculator but the examiners had a few extras. The examiners complete some paperwork before your exam, and you select one of the available test versions. Each test contains a randomly selected set of 35 questions from the pool of 350.</p> <p>Go through the test, carefully read each question, and fill in the answer sheet. Three examiners will grade it when you turn it in, and they will fill out your Certificate of Successful Completion of Examination (CSCE). Hold onto this paper just in case something happens with your FCC paperwork.</p> <p>The examiners will send your paperwork to the FCC and you should receive a license within two weeks. Mine took about 11-12 business days, but I took it just before Thanksgiving. The FCC will send you a generic email stating that there is a new license available and you can download it directly from the FCC&rsquo;s website.</p> <h2 id="lessons-learned-on-the-air">Lessons learned on the air</h2> <p>Once I passed the exam and keyed up for the first transmission, I feared a procedural misstep more than anything. What if I say my callsign incorrectly? What if I&rsquo;m transmitting at a power level that is too high? What power level is too high? What am I doing?!</p> <p>Everyone has to start somewhere and you&rsquo;re going to make mistakes. Almost 99.9% of my radio contacts so far have been friendly, forgiving, and patient. I&rsquo;ve learned a lot from listening to other people and from the feedback I get from radio contacts. Nobody will yell at you for using a repeater when simplex should work. Nobody will yell at you if you blast a repeater with 50 watts when 5 would be fine.</p> <p>I&rsquo;m on VHF most often and I&rsquo;ve found many local repeaters on <a href="https://www.repeaterbook.com/">RepeaterBook</a>. Most of the repeaters in the San Antonio area are busiest during commute times (morning and afternoon) as well as lunchtime. I&rsquo;ve announced my callsign when the repeater has been quiet for a while and often another radio operator will call back. It&rsquo;s a good idea to mention that you&rsquo;re new to amateur radio since that will make it easier for others to accept your mistakes and provide feedback.</p> <p>when I&rsquo;m traveling long distances, I monitor the national simplex calling frequency (146.520). That&rsquo;s the CB equivalent of channel 19 where you can announce yourself and have conversations. In busy urban areas, it&rsquo;s best to work out another frequency with your contact to keep the calling frequency clear.</p> <h2 id="my-equipment">My equipment</h2> <p>My first purchase was a (cheap) <a href="https://baofengtech.com/uv-5x3">BTECH UV-5X3</a>. The price is fantastic, but the interface is rough to use. Editing saved channels is nearly impossible and navigating the menus requires a good manual to decipher the options. The manual that comes with it is surprisingly brief. There are some helpful how-to guides from other radio operators on various blogs that can help.</p> <p>I picked up a <a href="http://www.kenwood.com/usa/com/amateur/tm-d710ga/">Kenwood TM-D710G</a> mobile radio from a coworker and mounted it in the car. I wired it up with <a href="https://powerwerx.com/anderson-power-powerpole-sb-connectors">Anderson Powerpole connectors</a> and that makes things incredibly easy (and portable). The interface on the Kenwood is light years ahead of the BTECH, but the price is 10x more.</p> <p>My car has the <a href="http://www.cometantenna.com/amateur-radio/mobile-antennas/ma-dual-band/">Comet SBB-5NMO</a> antenna mounted with a <a href="http://www.cometantenna.com/amateur-radio/no-holes-mobile-mounts/mounts-with-coax/">Comet CP-5NMO</a> lip mount. It fits well on the rear of the 4Runner.</p> <p>Managing a lot of repeater frequencies is challenging with both radios (exponentially more so with the BTECH), but the open source <a href="https://chirp.danplanet.com/projects/chirp/wiki/Home">CHIRP</a> software works well. I installed it on my Fedora laptop and could manage both radios easily. The BTECH radio requires you to download the entire current configuration, edit it, and upload it to the radio. The Kenwood allows you to make adjustments to the radio in real time (which is excellent for testing).</p> <h2 id="more-questions">More questions?</h2> <p>If you have more questions about any part of the process, let me know!</p> Ensuring keepalived starts after the network is ready https://major.io/2017/12/15/ensuring-keepalived-starts-network-ready/ Fri, 15 Dec 2017 21:18:37 UTC Major Hayden https://major.io/2017/12/15/ensuring-keepalived-starts-network-ready/ <p><img src="../wp-content/uploads/2017/12/wait.jpg" alt="1" /></p> <p>After a recent <a href="https://github.com/openstack/openstack-ansible">OpenStack-Ansible (OSA)</a> deployment on CentOS, I found that keepalived was not starting properly at boot time:</p> <div class="highlight"><pre style="color:#93a1a1;background-color:#002b36;-moz-tab-size:4;-o-tab-size:4;tab-size:4">Keepalived_vrrp[801]: Cant find interface br-mgmt for vrrp_instance internal !!! Keepalived_vrrp[801]: Truncating auth_pass to 8 characters Keepalived_vrrp[801]: VRRP is trying to assign ip address 172.29.236.11/32 to unknown br-mgmt interface !!! go out and fix your conf !!! Keepalived_vrrp[801]: Cant find interface br-mgmt for vrrp_instance external !!! Keepalived_vrrp[801]: Truncating auth_pass to 8 characters Keepalived_vrrp[801]: VRRP is trying to assign ip address 192.168.250.11/32 to unknown br-mgmt interface !!! go out and fix your conf !!! Keepalived_vrrp[801]: VRRP_Instance(internal) Unknown interface ! systemd[1]: Started LVS and VRRP High Availability Monitor. Keepalived_vrrp[801]: Stopped Keepalived[799]: Keepalived_vrrp exited with permanent error CONFIG. Terminating</pre></div> <p>OSA deployments have a management bridge for traffic between containers. These containers run the OpenStack APIs and other support services. By default, this bridge is called <code>br-mgmt</code>.</p> <p>The keepalived daemon is starting before NetworkManager can bring up the <code>br-mgmt</code> bridge and that is causing keepalived to fail. We need a way to tell systemd to wait on the network before bringing up keepalived.</p> <h2 id="waiting-on-networkmanager">Waiting on NetworkManager</h2> <p>There is a special systemd target, <code>network-online.target</code>, that is not reached until all networking is properly configured. NetworkManager comes with a handy service called <code>NetworkManager-wait-online.service</code> that must be complete before the <code>network-online</code> target can be reached:</p> <div class="highlight"><pre style="color:#93a1a1;background-color:#002b36;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash"><span style="color:#586e75"># rpm -ql NetworkManager | grep network-online</span> /usr/lib/systemd/system/network-online.target.wants /usr/lib/systemd/system/network-online.target.wants/NetworkManager-wait-online.service</code></pre></div> <p>Start by ensuring that the <code>NetworkManager-wait-online</code> service starts at boot time:</p> <div class="highlight"><pre style="color:#93a1a1;background-color:#002b36;-moz-tab-size:4;-o-tab-size:4;tab-size:4">systemctl enable NetworkManager-wait-online.service</pre></div> <h2 id="using-network-online-target">Using network-online.target</h2> <p>Next, we tell the keepalived service to wait on <code>network-online.target</code>. Bring up an editor for overriding the <code>keepalived.service</code> unit:</p> <div class="highlight"><pre style="color:#93a1a1;background-color:#002b36;-moz-tab-size:4;-o-tab-size:4;tab-size:4">systemctl edit keepalived.service</pre></div> <p>Once the editor appears, add the following text:</p> <div class="highlight"><pre style="color:#93a1a1;background-color:#002b36;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-ini" data-lang="ini"><span style="color:#719e07">[Unit]</span> Wants<span style="color:#719e07">=</span><span style="color:#2aa198">network-online.target</span> After<span style="color:#719e07">=</span><span style="color:#2aa198">network-online.target</span></code></pre></div> <p>Save the file in the editor and reboot the server. The keepalived service should come up successfully after NetworkManager signals that all of the network devices are online.</p> <p>Learn more by reading the upstream <a href="https://www.freedesktop.org/wiki/Software/systemd/NetworkTarget/">NetworkTarget</a> documentation.</p> Changes in RHEL 7 Security Technical Implementation Guide Version 1, Release 3 https://major.io/2017/11/02/changes-in-rhel-7-security-technical-implementation-guide-version-1-release-3/ Thu, 02 Nov 2017 15:00:25 UTC Major Hayden https://major.io/2017/11/02/changes-in-rhel-7-security-technical-implementation-guide-version-1-release-3/ <p><a href="../wp-content/uploads/2017/06/2.jpg"><img src="../wp-content/uploads/2017/06/2-300x91.jpg" alt="ansible-hardening logo" width="300" height="91" class="alignright size-medium wp-image-6744" srcset="../wp-content/uploads/2017/06/2-300x91.jpg 300w, ../wp-content/uploads/2017/06/2-768x233.jpg 768w, ../wp-content/uploads/2017/06/2-1024x311.jpg 1024w" sizes="(max-width: 300px) 100vw, 300px" /></a>The latest release of the Red Hat Enterprise Linux Security Technical Implementation Guide (STIG) <a href="https://iase.disa.mil/stigs/os/unix-linux/Pages/index.aspx">was published last week</a>. This release is Version 1, Release 3, and it contains four main changes:</p> <ul> <li>V-77819 - Multifactor authentication is required for graphical logins</li> <li>V-77821 - Datagram Congestion Control Protocol (DCCP) kernel module must be disabled</li> <li>V-77823 - Single user mode must require user authentication</li> <li>V-77825 - Address space layout randomization (ASLR) must be enabled</li> </ul> <h2 id="deep-dive">Deep dive</h2> <p>Let&rsquo;s break down this list to understand what each one means.</p> <h3 id="v-77819-multifactor-authentication-is-required-for-graphical-logins">V-77819 - Multifactor authentication is required for graphical logins</h3> <p>This requirement improves security for graphical logins and extends the existing requirements for multifactor authentication for logins (see V-71965, V-72417, and V-72427). The STIG recommends smartcards (since the US Government often uses <a href="https://en.wikipedia.org/wiki/Common_Access_Card">CAC cards</a> for multifactor authentication), and this is a good idea for high security systems.</p> <p>I use <a href="https://www.yubico.com/products/yubikey-hardware/yubikey4/">Yubikey 4&rsquo;s</a> as smartcards in most situations and they work anywhere you have available USB slots.</p> <h3 id="v-77821-datagram-congestion-control-protocol-dccp-kernel-module-must-be-disabled">V-77821 - Datagram Congestion Control Protocol (DCCP) kernel module must be disabled</h3> <p><a href="https://en.wikipedia.org/wiki/Datagram_Congestion_Control_Protocol">DCCP</a> is often used as a congestion control mechanism for UDP traffic, but it isn&rsquo;t used that often in modern networks. There have been <a href="https://threatpost.com/impact-of-new-linux-kernel-dccp-vulnerability-limited/123863/">vulnerabilities</a> in the past that are mitigated by disabling DCCP, so it&rsquo;s a good idea to disable it unless you have a strong reason for keeping it enabled.</p> <p>The ansible-hardening role has been updated to <a href="https://docs.openstack.org/ansible-hardening/latest/rhel7/domains/kernel.html#v-77821">disable the DCCP kernel module by default</a>.</p> <h3 id="v-77823-single-user-mode-must-require-user-authentication">V-77823 - Single user mode must require user authentication</h3> <p>Single user mode is often used in emergency situations where the server cannot boot properly or an issue must be repaired without a fully booted server. This mode can only be used at the server&rsquo;s physical console, serial port, or via out-of-band management (DRAC, iLO, and IPMI). Allowing single-user mode access without authentication is a serious security risk.</p> <p>Fortunately, every distribution supported by the ansible-hardening role already has authentication requirements for single user mode in place. The ansible-hardening role does not make any adjustments to the single user mode unit file since any untested adjustment could cause a system to have problems booting.</p> <h3 id="v-77825-address-space-layout-randomization-aslr-must-be-enabled">V-77825 - Address space layout randomization (ASLR) must be enabled</h3> <p><a href="https://en.wikipedia.org/wiki/Address_space_layout_randomization">ASLR</a> is a handy technology that makes it more difficult for attackers to guess where a particular program is storing data in memory. It&rsquo;s not perfect, but it certainly raises the difficulty for an attacker. There are multiple settings for this variable and the <a href="https://www.kernel.org/doc/Documentation/sysctl/kernel.txt">kernel documentation for sysctl</a> has some brief explanations for each setting (search for <code>randomize_va_space</code> on the page).</p> <p>Every distribution supported by the ansible-hardening role is already setting <code>kernel.randomize_va_space=2</code> by default, which applies randomization for the basic parts of process memory (such as shared libraries and the stack) as well as the heap. The ansible-hardening role will ensure that the default setting is maintained.</p> <h2 id="ansible-hardening-is-already-up-to-date">ansible-hardening is already up to date</h2> <p>If you&rsquo;re already using the ansible-hardening role&rsquo;s master branch, these changes are <a href="https://github.com/openstack/ansible-hardening/commit/782bb48c14c03aedaefcaf421fd5935ef5f561b8">already in place</a>! Try out the new updates and <a href="https://bugs.launchpad.net/openstack-ansible/+filebug">open a bug report</a> if you find any problems.</p> Import RPM repository GPG keys from other keyservers temporarily https://major.io/2017/09/20/import-rpm-repository-keys-from-other-keyservers-temporarily/ Wed, 20 Sep 2017 15:24:13 UTC Major Hayden https://major.io/2017/09/20/import-rpm-repository-keys-from-other-keyservers-temporarily/ <p><a href="../wp-content/uploads/2017/09/Close-up_of_keys-e1505920978611.jpg"><img src="../wp-content/uploads/2017/09/Close-up_of_keys-e1505920978611.jpg" alt="Keys, but not gpg keys" width="1024" height="399" class="aligncenter size-full wp-image-6815" srcset="../wp-content/uploads/2017/09/Close-up_of_keys-e1505920978611.jpg 1024w, ../wp-content/uploads/2017/09/Close-up_of_keys-e1505920978611-300x117.jpg 300w, ../wp-content/uploads/2017/09/Close-up_of_keys-e1505920978611-768x299.jpg 768w" sizes="(max-width: 1024px) 100vw, 1024px" /></a>I&rsquo;ve been working through some patches to <a href="https://github.com/openstack/openstack-ansible">OpenStack-Ansible</a> lately to optimize how we configure yum repositories in our deployments. During that work, I ran into some issues where pgp.mit.edu was returning 500 errors for some requests to retrieve GPG keys.</p> <p>Ansible was returning this error:</p> <div class="highlight"><pre style="color:#93a1a1;background-color:#002b36;-moz-tab-size:4;-o-tab-size:4;tab-size:4">curl: (22) The requested URL returned error: 502 Proxy Error error: http://pgp.mit.edu:11371/pks/lookup?op=get&amp;search=0x61E8806C: import read failed(2)</pre></div> <p>How does the <code>rpm</code> command know which keyserver to use? Let&rsquo;s use the <code>--showrc</code> argument to show how it is configured:</p> <div class="highlight"><pre style="color:#93a1a1;background-color:#002b36;-moz-tab-size:4;-o-tab-size:4;tab-size:4">$ rpm --showrc | grep hkp -14: _hkp_keyserver http://pgp.mit.edu -14: _hkp_keyserver_query %{_hkp_keyserver}:11371/pks/lookup?op=get&amp;search=0x</pre></div> <p>How do we change this value temporarily to test a GPG key retrieval from a different server? There&rsquo;s an argument for that as well: <code>--define</code>:</p> <div class="highlight"><pre style="color:#93a1a1;background-color:#002b36;-moz-tab-size:4;-o-tab-size:4;tab-size:4">$ rpm --help | grep define -D, --define=&#39;MACRO EXPR&#39; define MACRO with value EXPR</pre></div> <p>We can assemble that on the command line to set a different keyserver temporarily:</p> <div class="highlight"><pre style="color:#93a1a1;background-color:#002b36;-moz-tab-size:4;-o-tab-size:4;tab-size:4"># rpm -vv --define=&#34;%_hkp_keyserver http://pool.sks-keyservers.net&#34; --import 0x61E8806C -- SNIP -- D: adding &#34;63deac79abe7ad80e147d671c2ac5bd1c8b3576e&#34; to Sha1header index. -- SNIP --</pre></div> <p>Let&rsquo;s verify that our new key is in place:</p> <div class="highlight"><pre style="color:#93a1a1;background-color:#002b36;-moz-tab-size:4;-o-tab-size:4;tab-size:4"># rpm -qa | grep -i gpg-pubkey-61E8806C gpg-pubkey-61e8806c-5581df56 # rpm -qi gpg-pubkey-61e8806c-5581df56 Name : gpg-pubkey Version : 61e8806c Release : 5581df56 Architecture: (none) Install Date: Wed 20 Sep 2017 10:17:11 AM CDT Group : Public Keys Size : 0 License : pubkey Signature : (none) Source RPM : (none) Build Date : Wed 17 Jun 2015 03:57:58 PM CDT Build Host : localhost Relocations : (not relocatable) Packager : CentOS Virtualization SIG (http://wiki.centos.org/SpecialInterestGroup/Virtualization) &lt;security@centos.org&gt; Summary : gpg(CentOS Virtualization SIG (http://wiki.centos.org/SpecialInterestGroup/Virtualization) &lt;security@centos.org&gt;) Description : -----BEGIN PGP PUBLIC KEY BLOCK----- Version: rpm-4.11.3 (NSS-3) mQENBFWB31YBCAC4dFmTzBDOcq4R1RbvQXLkyYfF+yXcsMA5kwZy7kjxnFqBoNPv aAjFm3e5huTw2BMZW0viLGJrHZGnsXsE5iNmzom2UgCtrvcG2f65OFGlC1HZ3ajA 8ZIfdgNQkPpor61xqBCLzIsp55A7YuPNDvatk/+MqGdNv8Ug7iVmhQvI0p1bbaZR 0GuavmC5EZ/+mDlZ2kHIQOUoInHqLJaX7iw46iLRUnvJ1vATOzTnKidoFapjhzIt i4ZSIRaalyJ4sT+oX4CoRzerNnUtIe2k9Hw6cEu4YKGCO7nnuXjMKz7Nz5GgP2Ou zIA/fcOmQkSGcn7FoXybWJ8DqBExvkJuDljPABEBAAG0bENlbnRPUyBWaXJ0dWFs aXphdGlvbiBTSUcgKGh0dHA6Ly93aWtpLmNlbnRvcy5vcmcvU3BlY2lhbEludGVy ZXN0R3JvdXAvVmlydHVhbGl6YXRpb24pIDxzZWN1cml0eUBjZW50b3Mub3JnPokB OQQTAQIAIwUCVYHfVgIbAwcLCQgHAwIBBhUIAgkKCwQWAgMBAh4BAheAAAoJEHrr voJh6IBsRd0H/A62i5CqfftuySOCE95xMxZRw8+voWO84QS9zYvDEnzcEQpNnHyo FNZTpKOghIDtETWxzpY2ThLixcZOTubT+6hUL1n+cuLDVMu4OVXBPoUkRy56defc qkWR+UVwQitmlq1ngzwmqVZaB8Hf/mFZiB3B3Jr4dvVgWXRv58jcXFOPb8DdUoAc S3u/FLvri92lCaXu08p8YSpFOfT5T55kFICeneqETNYS2E3iKLipHFOLh7EWGM5b Wsr7o0r+KltI4Ehy/TjvNX16fa/t9p5pUs8rKyG8SZndxJCsk0MW55G9HFvQ0FmP A6vX9WQmbP+ml7jsUxtEJ6MOGJ39jmaUvPc= =ZzP+ -----END PGP PUBLIC KEY BLOCK-----</pre></div> <p>Success!</p> <p>If you want to override the value permanently, create a <code>~/.rpmmacros</code> file and add the following line to it:</p> <div class="highlight"><pre style="color:#93a1a1;background-color:#002b36;-moz-tab-size:4;-o-tab-size:4;tab-size:4">%_hkp_keyserver http://pool.sks-keyservers.net</pre></div> <p><em>Photo credit: <a href="https://commons.wikimedia.org/wiki/File:Close-up_of_keys.jpg">Wikipedia</a></em></p> Thunderbird changes fonts in some messages, not all https://major.io/2017/08/02/thunderbird-changes-fonts-messages-not/ Wed, 02 Aug 2017 12:54:38 UTC Major Hayden https://major.io/2017/08/02/thunderbird-changes-fonts-messages-not/ <p><a href="https://www.mozilla.org/en-US/thunderbird/">Thunderbird</a> is a great choice for a mail client on Linux systems if you prefer a GUI, but I had some problems with fonts in the most recent releases. The monospace font used for plain text messages was difficult to read.</p> <p>I opened <strong>Edit &gt; Preferences &gt; Display</strong> and clicked <strong>Advanced</strong> to the right of <strong>Fonts &amp; Colors</strong>. The default font for monospace text was &ldquo;Monospace&rdquo;, and that one isn&rsquo;t terribly attractive. I chose &ldquo;DejaVu Sans Mono&rdquo; instead, and closed the dialog boxes.</p> <p>The fonts in monospace messages didn&rsquo;t change. I quit Thunderbird, opened it again, and still didn&rsquo;t see a change. The strange part is that a small portion of my monospaced messages were opening with the updated font while the majority were not.</p> <p>I went back into Thunderbird&rsquo;s preferences and took another look:</p> <p><a href="../wp-content/uploads/2017/08/Screenshot-from-2017-08-02-07-48-18.png"><img src="../wp-content/uploads/2017/08/Screenshot-from-2017-08-02-07-48-18.png" alt="thunderbird fonts and colors panel" width="457" height="583" class="aligncenter size-full wp-image-6803" srcset="../wp-content/uploads/2017/08/Screenshot-from-2017-08-02-07-48-18.png 457w, ../wp-content/uploads/2017/08/Screenshot-from-2017-08-02-07-48-18-235x300.png 235w" sizes="(max-width: 457px) 100vw, 457px" /></a></p> <p>Everything was set as I expected. I started with some Google searches and stumbled upon a Mozilla Bug: <a href="https://bug1234567.bugzilla.mozilla.org/show_bug.cgi?id=546877">Changing monospace font doesn&rsquo;t affect all messages</a>. One of the participants in the bug mentioned that any emails received without ISO-8859-1 encoding would be unaffected since Thunderbird allows you set fonts for each encoding.</p> <p>I clicked the dropdown where &ldquo;Latin&rdquo; was selected and I selected &ldquo;Other Writing Systems&rdquo;. After changing the monospace font there, the changes went into effect for all of my monospaced messages!</p> Troubleshooting CyberPower PowerPanel issues in Linux https://major.io/2017/07/25/troubleshooting-cyberpower-powerpanel-issues-in-linux/ Tue, 25 Jul 2017 18:16:11 UTC Major Hayden https://major.io/2017/07/25/troubleshooting-cyberpower-powerpanel-issues-in-linux/ <p><img src="../wp-content/uploads/2017/07/1024px-Sierra_Blanca_and_electricity_pole-e1501006440664.jpg" alt="1" /></p> <p>I have a <a href="https://www.cyberpowersystems.com/product/ups/brg1350avrlcd/">CyberPower BRG1350AVRLCD</a> at home and I&rsquo;ve just connected it to a new device. However, the <code>pwrstat</code> command doesn&rsquo;t retrieve any useful data on the new system:</p> <div class="highlight"><pre style="color:#93a1a1;background-color:#002b36;-moz-tab-size:4;-o-tab-size:4;tab-size:4"># pwrstat -status The UPS information shows as following: Current UPS status: State........................ Normal Power Supply by.............. Utility Power Last Power Event............. None</pre></div> <p>I disconnected the USB cable and ran <code>pwrstat</code> again. <strong>Same output.</strong> I disconnected power from the UPS itself and ran <code>pwrstat</code> again. <strong>Same output.</strong> This can&rsquo;t be right.</p> <h2 id="checking-the-basics">Checking the basics</h2> <p>A quick look at <code>dmesg</code> output shows that the UPS is connected and the kernel recognizes it:</p> <div class="highlight"><pre style="color:#93a1a1;background-color:#002b36;-moz-tab-size:4;-o-tab-size:4;tab-size:4">[ 65.661489] usb 3-1: new full-speed USB device number 7 using xhci_hcd [ 65.830769] usb 3-1: New USB device found, idVendor=0764, idProduct=0501 [ 65.830771] usb 3-1: New USB device strings: Mfr=3, Product=1, SerialNumber=2 [ 65.830772] usb 3-1: Product: BRG1350AVRLCD [ 65.830773] usb 3-1: Manufacturer: CPS [ 65.830773] usb 3-1: SerialNumber: xxxxxxxxx [ 65.837801] hid-generic 0003:0764:0501.0004: hiddev0,hidraw0: USB HID v1.10 Device [CPS BRG1350AVRLCD] on usb-0000:00:14.0-1/input0</pre></div> <p>I checked the <code>/var/log/pwrstatd.log</code> file to see if there were any errors:</p> <div class="highlight"><pre style="color:#93a1a1;background-color:#002b36;-moz-tab-size:4;-o-tab-size:4;tab-size:4">2017/07/25 12:01:17 PM Daemon startups. 2017/07/25 12:01:24 PM Communication is established. 2017/07/25 12:01:27 PM Low Battery capacity is restored. 2017/07/25 12:05:19 PM Daemon stops its service. 2017/07/25 12:05:19 PM Daemon startups. 2017/07/25 12:05:19 PM Communication is established. 2017/07/25 12:05:22 PM Low Battery capacity is restored. 2017/07/25 12:06:27 PM Daemon stops its service.</pre></div> <p>The <code>pwrstatd</code> daemon can see the device and communicate with it. This is unusual.</p> <h2 id="digging-into-the-daemon">Digging into the daemon</h2> <p>If the daemon can truly see the UPS, then what is it talking to? I used <code>lsof</code> to examine what the <code>pwrstatd</code> daemon is doing:</p> <div class="highlight"><pre style="color:#93a1a1;background-color:#002b36;-moz-tab-size:4;-o-tab-size:4;tab-size:4"># lsof -p 3975 COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME pwrstatd 3975 root cwd DIR 8,68 224 96 / pwrstatd 3975 root rtd DIR 8,68 224 96 / pwrstatd 3975 root txt REG 8,68 224175 134439879 /usr/sbin/pwrstatd pwrstatd 3975 root mem REG 8,68 2163104 134218946 /usr/lib64/libc-2.25.so pwrstatd 3975 root mem REG 8,68 1226368 134218952 /usr/lib64/libm-2.25.so pwrstatd 3975 root mem REG 8,68 19496 134218950 /usr/lib64/libdl-2.25.so pwrstatd 3975 root mem REG 8,68 187552 134218939 /usr/lib64/ld-2.25.so pwrstatd 3975 root 0r CHR 1,3 0t0 1028 /dev/null pwrstatd 3975 root 1u unix 0xffff9e395e137400 0t0 37320 type=STREAM pwrstatd 3975 root 2u unix 0xffff9e395e137400 0t0 37320 type=STREAM pwrstatd 3975 root 3u unix 0xffff9e392f0c0c00 0t0 39485 /var/pwrstatd.ipc type=STREAM pwrstatd 3975 root 4u CHR 180,96 0t0 50282 /dev/ttyS1</pre></div> <p><strong>Wait a minute.</strong> The last line of the <code>lsof</code> output shows that <code>pwrstatd</code> is talking to <code>/dev/ttyS1</code>, but the device is supposed to be a <code>hiddev</code> device over USB. If you remember, we had this line in <code>dmesg</code> when the UPS was plugged in:</p> <div class="highlight"><pre style="color:#93a1a1;background-color:#002b36;-moz-tab-size:4;-o-tab-size:4;tab-size:4">hid-generic 0003:0764:0501.0004: hiddev0,hidraw0: USB HID v1.10 Device [CPS BRG1350AVRLCD] on usb-0000:00:14.0-1/input0</pre></div> <p>Things are beginning to make more sense now. I have a USB-to-serial device that allows my server to talk to the console port on my Cisco switch:</p> <div class="highlight"><pre style="color:#93a1a1;background-color:#002b36;-moz-tab-size:4;-o-tab-size:4;tab-size:4">[ 80.389533] usb 3-1: new full-speed USB device number 9 using xhci_hcd [ 80.558025] usb 3-1: New USB device found, idVendor=067b, idProduct=2303 [ 80.558027] usb 3-1: New USB device strings: Mfr=1, Product=2, SerialNumber=0 [ 80.558028] usb 3-1: Product: USB-Serial Controller D [ 80.558029] usb 3-1: Manufacturer: Prolific Technology Inc. [ 80.558308] pl2303 3-1:1.0: pl2303 converter detected [ 80.559937] usb 3-1: pl2303 converter now attached to ttyUSB0</pre></div> <p>It appears that <code>pwrstatd</code> is trying to talk to my Cisco switch (through the USB-to-serial adapter) rather than my UPS! I&rsquo;m sure they could have a great conversation together, but it&rsquo;s hardly productive.</p> <h2 id="fixing-it">Fixing it</h2> <p>The <code>/etc/pwrstatd.conf</code> has a relevant section:</p> <div class="highlight"><pre style="color:#93a1a1;background-color:#002b36;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-ini" data-lang="ini"><span style="color:#586e75"># The pwrstatd accepts four types of device node which includes the &#39;ttyS&#39;,</span> <span style="color:#586e75"># &#39;ttyUSB&#39;, &#39;hiddev&#39;, and &#39;libusb&#39; for communication with UPS. The pwrstatd</span> <span style="color:#586e75"># defaults to enumerate all acceptable device nodes and pick up to use an</span> <span style="color:#586e75"># available device node automatically. But this may cause a disturbance to the</span> <span style="color:#586e75"># device node which is occupied by other software. Therefore, you can restrict</span> <span style="color:#586e75"># this enumerate behave by using allowed-device-nodes option. You can assign</span> <span style="color:#586e75"># the single device node path or multiple device node paths divided by a</span> <span style="color:#586e75"># semicolon at this option. All groups of &#39;ttyS&#39;, &#39;ttyUSB&#39;, &#39;hiddev&#39;, or</span> <span style="color:#586e75"># &#39;libusb&#39; device node are enumerated without a suffix number assignment.</span> <span style="color:#586e75"># Note, the &#39;libusb&#39; does not support suffix number only.</span> <span style="color:#586e75">#</span> <span style="color:#586e75"># For example: restrict to use ttyS1, ttyS2 and hiddev1 device nodes at /dev</span> <span style="color:#586e75"># path only.</span> <span style="color:#586e75"># allowed-device-nodes = /dev/ttyS1;/dev/ttyS2;/dev/hiddev1</span> <span style="color:#586e75">#</span> <span style="color:#586e75"># For example: restrict to use ttyS and ttyUSB two groups of device node at</span> <span style="color:#586e75"># /dev,/dev/usb, and /dev/usb/hid paths(includes ttyS0 to ttySN and ttyUSB0 to</span> <span style="color:#586e75"># ttyUSBN, N is number).</span> <span style="color:#586e75"># allowed-device-nodes = ttyS;ttyUSB</span> <span style="color:#586e75">#</span> <span style="color:#586e75"># For example: restrict to use hiddev group of device node at /dev,/dev/usb,</span> <span style="color:#586e75"># and /dev/usb/hid paths(includes hiddev0 to hiddevN, N is number).</span> <span style="color:#586e75"># allowed-device-nodes = hiddev</span> <span style="color:#586e75">#</span> <span style="color:#586e75"># For example: restrict to use libusb device.</span> <span style="color:#586e75"># allowed-device-nodes = libusb</span> allowed-device-nodes <span style="color:#719e07">=</span></code></pre></div> <p>We need to explicitly tell <code>pwrstatd</code> to talk to the UPS on <code>/dev/hid/hiddev0</code>:</p> <div class="highlight"><pre style="color:#93a1a1;background-color:#002b36;-moz-tab-size:4;-o-tab-size:4;tab-size:4">allowed-device-nodes = /dev/usb/hiddev0</pre></div> <p>Let&rsquo;s restart the <code>pwrstatd</code> daemon and see what we get:</p> <div class="highlight"><pre style="color:#93a1a1;background-color:#002b36;-moz-tab-size:4;-o-tab-size:4;tab-size:4"># systemctl restart pwrstatd # pwrstat -status The UPS information shows as following: Properties: Model Name................... BRG1350AVRLCD Firmware Number.............. Rating Voltage............... 120 V Rating Power................. 810 Watt(1350 VA) Current UPS status: State........................ Normal Power Supply by.............. Utility Power Utility Voltage.............. 121 V Output Voltage............... 121 V Battery Capacity............. 100 % Remaining Runtime............ 133 min. Load......................... 72 Watt(9 %) Line Interaction............. None Test Result.................. Unknown Last Power Event............. None</pre></div> <p>Success!</p> <p><em>Photo credit: <a href="https://commons.wikimedia.org/wiki/File%3ASierra_Blanca_and_electricity_pole.jpg">Wikipedia</a></em></p> Apply the STIG to even more operating systems with ansible-hardening https://major.io/2017/07/21/apply-stig-operating-systems-ansible-hardening/ Fri, 21 Jul 2017 17:38:46 UTC Major Hayden https://major.io/2017/07/21/apply-stig-operating-systems-ansible-hardening/ <p><img src="../wp-content/uploads/2017/07/1024px-Samuils_Fortress_and_Ohrid_Lake.jpg" alt="1" /></p> <p>Tons of improvements made their way into the <a href="https://github.com/openstack/ansible-hardening">ansible-hardening</a> role in preparation for the OpenStack Pike release <a href="https://releases.openstack.org/pike/schedule.html">next month</a>. The role has a <a href="../2017/06/27/old-role-new-name-ansible-hardening/">new name</a>, new <a href="https://docs.openstack.org/ansible-hardening/latest/">documentation</a> and extra tests.</p> <p>The role uses the Security Technical Implementation Guide (STIG) produced by the Defense Information Systems Agency (DISA) and applies the guidelines to Linux hosts using Ansible. Every control is configurable via simple Ansible variables and each control is thoroughly documented.</p> <p>These controls are now applied to an even wider variety of Linux distributions:</p> <ul> <li>CentOS 7</li> <li>Debian 8 Jessie <em>(new for Pike)</em></li> <li>Fedora 25 <em>(new for Pike)</em></li> <li>openSUSE Leap 42.2+ <em>(new for Pike)</em></li> <li>Red Hat Enterprise Linux 7</li> <li>SUSE Linux Enterprise 12 <em>(new for Pike)</em></li> <li>Ubuntu 14.04 Trusty</li> <li>Ubuntu 16.04 Xenial</li> </ul> <p>Any patches to the ansible-hardening role are tested against all of these operating systems (except RHEL 7 and SUSE Linux Enterprise). Support for openSUSE testing <a href="https://github.com/openstack-infra/project-config/commit/0795a7414ca8f06931877919d7ecb0b2d4e5f6e0">landed this week</a>.</p> <p><strong>Work is underway to put the finishing touches on the master branch before the Pike release and we need your help!</strong></p> <p>If you have any of these operating systems deployed, please test the role on your systems! This is pre-release software, so it&rsquo;s best to apply it only to a new server. Read the <a href="https://docs.openstack.org/ansible-hardening/latest/getting-started.html">&ldquo;Getting Started&rdquo;</a> documentation to get started with <code>ansible-galaxy</code> or <code>git</code>.</p> <p><em>Photo credit: <a href="https://commons.wikimedia.org/wiki/File%3ASamuil's_Fortress_and_Ohrid_Lake.JPG">Wikipedia</a></em></p> Customize LDAP autocompletion format in Thunderbird https://major.io/2017/07/18/customize-ldap-autocompletion-format-in-thunderbird/ Tue, 18 Jul 2017 18:08:42 UTC Major Hayden https://major.io/2017/07/18/customize-ldap-autocompletion-format-in-thunderbird/ <p><img src="../wp-content/uploads/2017/07/1280px-Mailbox_USA-e1500401199427.jpg" alt="1" /></p> <p>Thunderbird can connect to an LDAP server and autocomplete email addresses as you type, but it needs some adjustment for some LDAP servers. One of the LDAP servers that I use regularly returns email addresses like this in the thunderbird interface:</p> <div class="highlight"><pre style="color:#93a1a1;background-color:#002b36;-moz-tab-size:4;-o-tab-size:4;tab-size:4">username &lt;firstname.lastname@domain.tld&gt;</pre></div> <p>The email address looks fine, but I&rsquo;d much rather have the person&rsquo;s full name instead of the username. Here&rsquo;s what I&rsquo;m looking for:</p> <div class="highlight"><pre style="color:#93a1a1;background-color:#002b36;-moz-tab-size:4;-o-tab-size:4;tab-size:4">Firstname Lastname &lt;firstname.lastname@domain.tld&gt;</pre></div> <p>In older Thunderbird versions, setting <code>ldap_2.servers.SERVER_NAME.autoComplete.nameFormat</code> to <code>displayName</code> was enough. However, this option isn&rsquo;t used in recent versions of Thunderbird.</p> <h2 id="digging-in">Digging in</h2> <p>After a fair amount of searching the Thunderbird source code with <code>awk</code>, I found a mention of <code>DisplayName</code> in <code>nsAbLDAPAutoCompleteSearch.js</code> that looked promising:</p> <div class="highlight"><pre style="color:#93a1a1;background-color:#002b36;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-javascript" data-lang="javascript"><span style="color:#586e75">// Create a minimal map just for the display name and primary email. </span><span style="color:#586e75"></span> <span style="color:#719e07">this</span>._attributes <span style="color:#719e07">=</span> Components.classes[<span style="color:#2aa198">&#34;@mozilla.org/addressbook/ldap-attribute-map;1&#34;</span>] .createInstance(Components.interfaces.nsIAbLDAPAttributeMap); <span style="color:#719e07">this</span>._attributes.setAttributeList(<span style="color:#2aa198">&#34;DisplayName&#34;</span>, <span style="color:#719e07">this</span>._book.attributeMap.getAttributeList(<span style="color:#2aa198">&#34;DisplayName&#34;</span>, {}), <span style="color:#cb4b16">true</span>); <span style="color:#719e07">this</span>._attributes.setAttributeList(<span style="color:#2aa198">&#34;PrimaryEmail&#34;</span>, <span style="color:#719e07">this</span>._book.attributeMap.getAttributeList(<span style="color:#2aa198">&#34;PrimaryEmail&#34;</span>, {}), <span style="color:#cb4b16">true</span>); } </code></pre></div> <p>Something is unusual here. The LDAP field is called <code>displayName</code>, but this attribute is called <code>DisplayName</code> (note the capitalization of the <em>D</em>). Just before that line, I see a lookup in an attributes map of some sort. There may be a configuration option that is called <code>DisplayName</code>.</p> <p>In Thunderbird, I selected <strong>Edit &gt; Preferences</strong>. I clicked the <strong>Advanced</strong> tab and then <strong>Config Editor</strong>. A quick search for <em>DisplayName</em> revealed an interesting configuration option:</p> <div class="highlight"><pre style="color:#93a1a1;background-color:#002b36;-moz-tab-size:4;-o-tab-size:4;tab-size:4">ldap_2.servers.default.attrmap.DisplayName: cn,commonname</pre></div> <h2 id="fixing-it">Fixing it</h2> <p>That&rsquo;s the problem! This needs to map to <code>displayName</code> in my case, and not <code>cn,commonname</code> (which returns a user&rsquo;s username). There are two different ways to fix this:</p> <div class="highlight"><pre style="color:#93a1a1;background-color:#002b36;-moz-tab-size:4;-o-tab-size:4;tab-size:4"># Change it for just one LDAP server ldap_2.servers.SERVER_NAME.attrmap.DisplayName: displayName # Change it for all LDAP servers by default (careful) ldap_2.servers.default.attrmap.DisplayName: displayName</pre></div> <p>After making the change, quit Thunderbird and relaunch it. Compose a new email and start typing in the email address field. The user&rsquo;s first and last name should appear!</p>