OmniOS Community Edition r151030bi, r151032ai, r151034i OmniOS Community Edition

OmniOS weekly releases for w/c 29th of June 2020 are now available.

  • For all supported OmniOS releases, curl has been updated to fix two vulnerabilities (CVE-2020-8169 and CVE-2020-9177)

For further details, please see https://omniosce.org/releasenotes


Any problems or questions, please get in touch.

OmniOS Community Edition r151030bh, r151032ah, r151034h OmniOS Community Edition

This week’s update for all stable OmniOS versions includes an update to the Intel CPU microcode files to mitigate the new Crosstalk class of data leakage CPU vulnerability.

For r151034, there is also a fix for handling of correctly memory errors and an enhancement to lx-branded zones to better support newer Linux distributions.

For further details, please see https://omniosce.org/releasenotes


Any problems or questions, please get in touch.

Java: trying out String deduplication and the G1 garbage collector The Trouble with Tribbles...

As of 8u20, java supports automatic String deduplication.

-XX:+UseG1GC -XX:+UseStringDeduplication

You need to use the G1 garbage collector, and it will do the dedup as you scan the heap. Essentially, it checks each String and if the backing char[] array is the same as one it's already got, it merges the references.

Obviously, this could save memory if you have a lot of repeated strings.

Consider my illuminate utility. One of the thing it does is parse the old SVR4 packaging contents file. That's a big file, and there's a huge amount of duplication - while the file names are obviously unique, things like the type of file, permissions, owner, group, and names of packages are repeated many times. So, does turning this thing on make a difference?

Here's the head of the class histogram (produced by jcmd pid GC.class_histogram).

First without:

 num     #instances         #bytes  class name
----------------------------------------------
   1:       2950682      133505088  [C
   2:       2950130       70803120  java.lang.String
   3:        862390       27596480  java.util.HashMap$Node
   4:        388539       21758184  org.tribblix.illuminate.pkgview.ContentsFileDetail

and now with deduplication:

 num     #instances         #bytes  class name
----------------------------------------------
   1:       2950165       70803960  java.lang.String
   2:        557004       60568944  [C
   3:        862431       27597792  java.util.HashMap$Node
   4:        388539       21758184  org.tribblix.illuminate.pkgview.ContentsFileDetail

Note that there's the same number of entries in the contents file (there's one ContentsFileDetail for each line), and essentially the same number of String objects. But the [C, which is the char[] backing those Strings, has fallen dramatically. You're saving about a third of the memory used to store all that String data.

This also clearly demonstrates that the deduplication isn't on the String objects, those are unchanged, but on the char[] arrays backing those Strings.

Even more interesting is the performance. This is timing of a parser before:

real        1.730556446
user        7.977604040
sys         0.251854581

and afterwards:

real        1.469453551
user        6.054787878
sys         0.407259095

That's actually a bit of a surprise: G1GC is going to have to do work to do the comparisons to see if the strings are the same, and do some housekeeping if they are. However, with just the G1GC on its own, without deduplication, we get a big performance win:

real        1.217800287
user        3.944160155
sys         0.362586413

Therefore, for this case, G1GC is a huge performance benefit, and the deduplication takes some of that performance gain and trades it for memory efficiency.

For the illuminate GUI, without G1GC:

user       10.363291056
sys         0.393676741

and with G1GC:

user        8.151806315
sys         0.401426176

(elapsed time isn't meaningful here as you're waiting for interaction to shut it down)

The other thing you'll sometime see in this context is interning Strings. I tried that, it didn't help at all.

Next, with a little more understanding of what was going on, I tried some modifications to the code to reduce the cost of storing all those Strings.


I did tweak my contents file reader slightly, to break lines up using a simple String.split() rather than using a StringTokenizer. (The java docs recommend you don't use StringTokenizer any more, so this is also a bit of modernization.) I don't think the change of itself makes any difference, but it's slightly less work to simply ignore fields in an array from String.split() than call nextToken() to skip over the ones you don't want.

Saving the size and mtime as long - primitive types - saves a fair amount of memory too. Each String object is 24 bytes plus the content, so the saving is significant. And given that any uses will be of the numerical value, we may as well convert up front.

The ftype is only a single character. So storing that as a char avoids an object, saving space, and they're automatically interned for us.

That manual work gave me about another 10% speedup. What about memory usage?

Using primitive types rather than String gives us the following class histogram:

 num     #instances         #bytes  class name
----------------------------------------------
   1:       1917289      102919512  [C
   2:       1916938       46006512  java.lang.String
   3:        862981       27615392  java.util.HashMap$Node
   4:        388532       24866048  org.tribblix.illuminate.pkgview.ContentsFileDetail
So, changing the code gives almost the same memory saving as turning on String deduplication, without any performance hit.

There are 3 lessons here:

  1. Don't use Strings to store what could be primitive types if you can help it
  2. Under some (not all) circumstances, the G1 garbage collector can be a huge win
  3. When you're doing optimization occasionally the win you get isn't the one you were looking for

ctags, vim and C Staring at the C

Going to the first matching tag in vim with Control-] can be rather annoying. The exuberant-ctags secondary sort key is the filename, not the tag kind. If you have a struct type that’s also a common member name, you’re forced into using :tselect to find the struct instead of all the members. Most of the time, the struct definition is what you want.

To avoid this issue, I sort the tags file such that any kind == "s" entries come first for that tag. It’s a little annoying due to the format of the file, but it does work:

#!/bin/bash

# ctags, but sub-sorted such that "struct request" comes first, rather than
# members with the same name.

# we can't use "-f -", as that elides the TAG_FILE_SORTED preamble
ctags -R -f tags.$$

awk '
BEGIN {
   FS="\t"
   entry=""
   struct=""
   buf=""
}

$1 != entry {
   if (entry != "") {
           printf("%s%s", struct, buf);
   }
   entry=$1;
   struct="";
   buf="";
}

/^.*"\ts/ {
   struct=struct $0 "\n"
   next
}

$1 == entry {
   buf=buf $0 "\n"
}

END {
   printf("%s%s", struct, buf);
}' <tags.$$ >tags

rm tags.$$

Tracing Kernel Functions: How the illumos AMD64 FBT Provider Intercepts Function Calls Z In ASCII - Writing

A line-by-line breakdown of how the illumos AMD64 FBT provider intercepts function calls.

Installing Debian under FreeBSD's bhyve Josef "Jeff" Sipek

This weekend I tried to move my work dev vm to a slightly beefier vm host. This meant moving the vm from kvm on OmniOS to bhyve on FreeBSD 12.1. After moving the disk images over, I tried to configure vm-bhyve to use them as-is. Unfortunately, grub2-bhyve fails to boot XFS root file systems with CRC support and there was no good way to convert the disk image to get it to boot via UEFI instead (to avoid grub2-bhyve completely). So, I decided to simply reinstall it.

In theory, this shouldn’t have been too difficult since I had the foresight to have /home on a separate virtual disk. In practice, I spent several hours reinstalling Debian Buster over and over, trying to figure out why it’d install and reboot fine, but subsequent boots wouldn’t make it past the EFI firmware.

It turns out that Debian tries to make multi-booting possible and puts its EFI binary into a non-standard location. That combined with bhyve not persisting EFI variables after shutdown results in any boot after the the first poweroff not even trying to look at the Debian-specific path.

This is not a Debian bug, but rather bhyve’s EFI support being incomplete. The easiest way around this is to copy the Debian binary into the standard location immediately after installation. In other words:

# cd /boot/efi/EFI
# mkdir BOOT
# cp debian/grubx64.efi BOOT/bootx64.efi

That’s all that’s needed. The downside to this is that the copy will not get automatically upgraded when grub gets an update.

For completeness, here are the relevant bits of the vm’s config:

loader="uefi"
graphics="yes"
cpu="6"
memory="1G"
network0_type="virtio-net"
network0_switch="public"
zfs_zvol_opts="volblocksize=4096"
disk0_dev="sparse-zvol"
disk0_type="virtio-blk"
disk0_name="disk0"
disk1_dev="sparse-zvol"
disk1_type="virtio-blk"
disk1_name="disk1"

The singular urgency of Ava DuVernay’s 13th The Observation Deck

On Sunday afternoon, I was on the phone with one of my Oxide co-founders, Steve Tuck. He and I were both trying to grapple with the brazen state-sponsored violence that we were witnessing: the murder of George Floyd and the widespread, brutal, and shameless suppression of those who were demonstrating against it.

Specifically, we were struggling with a problem that (bluntly) a lot of white people struggle with: how to say what when. An earlier conversation with Steve had inspired me to publicly say something that I have long believed: that on social media, you should amplify your listening by following voices that you don’t otherwise hear. I had tweeted that out, and Steve and I were talking about it. In particular, Steve was wondering about telling people to see 13th, a 2016 documentary by Ava DuVernay that he had found very moving when he had seen it last year. I had taken my kids to see DuVernay’s superlative Selma years prior, and I had heard about 13th (and I had recalled my wife saying she wanted to watch it), but it was — like many things — “in the queue.” Steve was emphatic though: “you need to watch it.”

If I may, an aside: we — we all, but especially white Americans — run away from difficult subjects. With our viewing habits in particular, there are things that we know we absolutely should watch that we just… don’t. The reasons are often mundane: because the time isn’t right; because we’re tired; because we want to be uplifted; because we’re ashamed. For my wife and me, this was Hotel Rwanda, a Netflix DVD (!) that we had sitting on top of the DVD player for literally years, not having the heart to send it back — and yet never in the mood to watch it.

Fortunately, 13th was not to share Hotel Rwanda‘s fate: not only is Steve persuasive, it felt especially apropos as my wife and I were looking for ways to talk about the George Floyd murder and subsequent uprising with our kids — so I got off the phone with Steve, my wife and I got the family together (pretty easy when living in self-isolation!), and we watched it.

I don’t know that I’ve ever seen a documentary as important. More than anything, it is revealing: it connects dots that I didn’t realize were connected — and it shows, with a stunning diversity of voices (Angela Davis and Newt Gingrich?!), just how deeply racist our criminal justice system is. Not that it isn’t also horrifying in its revelations: my kids gasped several times, including learning of the stunning growth in incarceration — and of the jaw-dropping fraction of inmates that never stood trial for their crime. And there was plenty of horrifying education beyond the figures; I knew a prison/industrial complex was out there, but I never imagined that it could be such a deliberate, pernicious octopus.

I’m not going to spoil any more of it for you; if you haven’t already seen it, you need to watch 13th — and you need to watch it now.

Actually, I’m going to phrase it even more bluntly: it is just 100 minutes (very deliberately not longer, as it turns out; watch the excellent conversation between Ava DuVerney and Oprah Winfrey for details); you more or less can’t leave your home; and America is boiling over on this very issue. So let’s put it another way: if you are unwilling to watch this now, be honest with yourself and realize that you are never going to watch it. And if you are never going to watch it, please spare us all the black squares and the trending hashtags: if you do not have 100 minutes to give to this most important topic at this most critical moment, you are not actually willing to do anything at all.

So you need to watch it, but is watching a documentary really the answer? Well, no, of course not — but that isn’t to say film can’t change the collective mind: The Day After famously informed Ronald Reagan’s views on the dangers of nuclear war, at a(nother) time when humanity felt perilously close to the brink. And I do believe that 13th could have that kind of power — so I would ask you to not only watch it, but use your voice (and, for many of you, your privilege) to get others to do the same.

Once you’ve watched it — and once you’ve gotten your friends and family to watch it — the tough work begins: we need to not merely reform this system, we need to rethink it entirely. And for this, you want to get your resources to the non-profits taking this on (there are a bunch out there; one in particular that I would recommend is The Equal Justice Initiative). Thanks to Ava DuVernay for making such a singularly important film — and thank you in advance for doing her (and us all!) the service of watching it!

OmniOS Community Edition r151030be, r151032ae, r151034e OmniOS Community Edition

This week’s update for all stable OmniOS versions corrects a bug in the pkgdepend command which is important for illumos developers.

For r151034, there is also a fix for cdrom support in kvm zones.

For further details, please see https://omniosce.org/releasenotes


Any problems or questions, please get in touch.

OmniOS Community Edition r151034d OmniOS Community Edition

The first update for the r151034 stable version of OmniOS is now available.

This update requires a reboot

Changes

  • Update Intel CPU microcode to 20200520

  • Update timezone data to 2020a

  • The bhyve zone brand now supports vnc=wait to pause VM execution until a VNC client has connected

  • KVM zone shutdown did not work reliably

  • The bhyve and kvm zone brands now support multiple cdrom entries. These should be configured as properties named cdrom0, cdrom1, etc.

  • The default-recurse pkg property was erroneously affecting the pkg install command; this has been fixed

  • pkg update with a new boot environment could occasionally report pkg: Unable to clone the current boot environment

  • onu to illumos-gate now works with installed zones

  • Fix for a kernel panic when running an NFS client within an lx-branded zone

  • Fix for regression in ftello64() behaviour

  • Fix for (rare) crash in bhyve with some Linux guests

  • Fix for potential lz4 compression failure in vtfontcvt

  • Improve TSO support in vioif driver

For further details, please see https://omniosce.org/releasenotes


Any problems or questions, please get in touch.

Flight Planning My Cruise Power Josef "Jeff" Sipek

When I was working on my private pilot certificate, there was one thing that was never satisfactorily explained to me: how to select the “right” line of the cruise performance table in the POH. Now that I’m a few years older and wiser, I thought I’d write up an explanation for those who, like me six years ago, aren’t getting a good enough answer from their CFIs.

I did my training in a Cessna 172SP, and so the table was relatively simple:

Reading it is trivial. Pick your cruise altitude, then pick the RPM that the instructor told you to use for cruising (e.g., 2200). Now, read across to figure out what your true airspeed and fuel flow will be. That is all there is to it.

When I got checked out in the club’s 182T, things got more confusing. The table itself got split across multiple pages of the POH because of the addition of a new variable: manifold pressure (MP).

The table works much the same way as before. First, select the table based on which altitude you’ll be cruising at, then pick the RPM and manifold pressure, and read across the true airspeed and fuel flow.

On the surface (bad pun intended), this seems like a reasonable explanation. But if you look closely, there are multiple combinations of RPM and MP which give you the same performance. For example, in the above table both 2200/21” and 2400/20” give more or less the same performance. When I asked how to choose between them, all I got was a reminder to “keep the MP at or below the RPM.” It was thoroughly unsatisfying. So, I stuck with something simple like 2300/23”.

Fast forward to today. I fly a fixed gear Cessna Cardinal (177B). Its manual contains a table much like the one above for a 182. Here is a sample for 4000’:

As before, I started with something simple like 2300/23”, but eventually I had a moment of clarity. When flying the 172 and 182, I paid for Wikipedia article: Hobbs time. In other words, it was in my best interest to cruise as fast as possible without much regard for which exact RPM/MP combination I used (all within club and manufacturer limitations, of course).

My bill for the Cardinal is different—it is based on Wikipedia article: tach time. This means that the lower the RPM, the slower I’m spending money. So, like any other optimization problem, I want to find the right spot where my bill, my cruise speed, and my fuel flow (and therefore endurance) are all acceptable.

If the tach timer is calibrated to run at full speed at 2700 RPM, running the engine at only 2300 equates to 85% while using 2400 equates to 88.9%.

So, say I’m flying for two hours. If I use 2400 RPM, I’ll be paying 1.78 hours. On the other hand, if I use 2300 RPM at the same power output, I’ll be paying for 1.70 hours. Not a big difference, but after 24 hours at 2300 instead of 2400, I would have saved a full hour of tach time.

I don’t yet have enough data to verify these figures, but collecting it is on my todo list.

While composing this post, I happened to find an article by Mike Busch about why lower RPM is better. He makes a number of compeling points—reduced noise, better propeller efficiency, and fewer revolutions the engine has to make (which should improve the engine’s lifetime and therefore the overall cost). I have to admit that Mike’s points seems more compeling than the small savings I’ve calculated above.

OmniOS Community Edition r151032ab, r151030bb OmniOS Community Edition

OmniOS weekly releases for w/c 11th of May 2020 are now available.

This update requires a reboot

Security Fixes

  • Fix for a kernel panic when running an NFS client within an lx-branded zone

Other Changes

  • w and whodo produced error messages about processes in non-global zones

  • onu to illumos-gate now works with installed zones

  • pkglint now detects duplicated pkg attributes in legacy actions

For further details, please see https://omniosce.org/releasenotes


Any problems or questions, please get in touch.

A Simple Pibell Staring at the C

With all this free time I finally got around to installing a doorbell at home. I had no interest in Ring or the like: what I really wanted was a simple push doorbell that fit the (Victorian) house but would also somehow notify me if I was downstairs…

There are several documented projects on splicing in a Raspberry Pi into existing powered doorbell systems, but that wasn’t what I wanted either.

Instead, the doorbell is a simple contact switch feeding into the Pi’s GPIO pins. It’s effectively extremely simple but I didn’t find a step by step, so this is what I could have done with reading.

I bought the Pi, a case, a power supply, an SD card, and a USB speaker:

Raspberry Pi 3 A+ Pibow Coupé case Pi power supply NOOBS pre-installed SD Card USB speaker

And the doorbell itself plus wiring:

Brass push doorbell Bell wire Crimping pins Crimp Housing

I bought a pre-installed Raspbian SD card as I don’t have an SD card caddy. After some basic configuration (which required HDMI over to a monitor) I started playing with how to set up the Pi.

Of course the PI is absurdly over-powered for this purpose, but I wanted something simple to play with. And anyway, it’s running Pihole too.

The wiring itself is simple: bell wire over through a hole in the door frame to the back of the doorbell (which is a simple contact push). The other end of the wires are connected to the PI’s GPIO pin 18, and ground. The pin is pulled up and we trigger the event when we see a falling edge.

Actually connecting the wires was a bit fiddly: the bell wire is too thin for the 0.1” connector, and lacking a proper crimping tool I had to bodge it with needle-nose pliers. But once in the pins the housing connection is solid enough.

At first I tried to connect it to Alexa but soon gave up on that idea. There’s no way to “announce” via any API, and it kept disconnecting when used as a Bluetooth speaker. And Alexa has that infuriating “Now playing from…” thing you can’t turn off as well.

During fiddling with this I removed PulseAudio from the Pi as a dead loss.

Nor could I use an Anker Soundcore as a Bluetooth speaker: the stupid thing has some sleep mode that means it misses off the first 3 seconds or so of whatever’s playing.

Instead I have the crappy USB speaker above. It’s not great but is enough to be heard from outside and inside.

Aside from playing whatever through the speaker, the bell emails me in case I can’t hear it. Here’s the somewhat crappy script it’s running:

      #!/usr/bin/python -u

#
# The Pi is wired up such that pin 18 goes through the switch to ground.
# The on-pin pull-up resistor is enabled (so .input() is normally True).
# When the circuit completes, it goes to ground and hence we get a
# falling edge and .input() becomes False.
#
# I get the occasional phantom still so we wait for settle_time before
# thinking it's real.
#

from email.mime.text import MIMEText
from subprocess import Popen, PIPE
from datetime import datetime
import subprocess
import RPi.GPIO as GPIO
import signal
import time
import os

GPIO.setmode(GPIO.BCM)

GPIO.setup(18, GPIO.IN, pull_up_down=GPIO.PUD_UP)

# in seconds
settle_time = 0.1
bounce_time = 1

def notify():
    print('notifying at %s' % time.time())

    msg = MIMEText("At %s" % datetime.now().strftime("%Y-%m-%d %H:%M:%S"))
    msg["From"] = "doorbell <levon@movementarian.org>"
    msg["To"] = "John Levon <john.levon@gmail.com>"
    msg["Cc"] = "John Levon <levon@movementarian.org>"
    msg["Subject"] = "Someone is ringing the doorbell"

    p = Popen(["/usr/sbin/sendmail", "-f", "levon@movementarian.org", "-t", "-oi"], stdin=PIPE)
    p.stdin.write(msg.as_string())
    p.stdin.close()
    while True:
        os.system('aplay -D plughw:1,0 doorbell.wav')
        input_state = GPIO.input(18)
        if input_state:
            break

def settle():
    global settle_time
    time.sleep(settle_time)
    input_state = GPIO.input(18)
    print('input state now %s' % input_state)
    return not input_state

def falling_edge(channel):

    input_state = GPIO.input(18)
    print('got falling edge, input_state %s' % input_state)
    if settle():
        notify()
          
GPIO.add_event_detect(18, GPIO.FALLING, callback=falling_edge, bouncetime=(bounce_time * 1000))

print('started')

signal.pause()

2020-05-06 Josef "Jeff" Sipek

OpenMCT — While I’m not a fan of web-based UIs, this is a rather neat “dashboard” framework by NASA.

Wideband spectrum received in JO32KF — Over 5 years of HF spectrum waterfall in Enschede, NL.

10 Most(ly dead) Influential Programming Languages

Wikipedia article: PACELC theorem — An extension of the Wikipedia article: CAP theorem.

Learn Rust the Dangerous Way — Finally a Rust tutorial that speaks to people comfortable in C.

Interferometry and Synthesis in Radio Astronomy — An open access book.

Aviation Formulary — Great circle math applied to various aviation problems for those too lazy to derive the formulas themselves.

Papírová platidla Československa 1918-1993, České republiky a Slovenské republiky 1993-2016 — Complete list of all bank notes used in Czechoslovakia, Czech Republic, and Slovak Republic.

NOAA GOES Image ViewerWikipedia article: GOES weather satellite imagery.

OpenIndiana Hipster 2020.04 is here openindiana

We have released a new OpenIndiana Hipster snapshot 2020.04. The noticeable changes:

  • All remaining OI-specific applications have been ported from Python 2.7 to 3.5, including Caiman (slim_source) installer.
  • Installation images now don’t ship Python 2.7, however some software can still depend on it.
  • GCC 7 is used as the main system compiler now.
  • Libreoffice 6.4 was added.
  • PKG was updated to use rapidjson instead of simplejson for json processing which reduced memory consumption on operations with large package catalogues.
  • A lot of packages were updated.

More information can be found in 2020.04 Release notes and new medias can be downloaded from http://dlc.openindiana.org.