William D. Revelli, on the Anniversary of His Death

I wasn't even six months into my first job at NRL when I'd heard the legendary director of not only the Michigan Marching Band but all of Michigan's Bands had died at age 92.

William D. Revelli left a lasting impact at Michigan. I can only relay that impact by knowing that it was his name on the building where I showed up days before the rest of the Freshman class at Michigan to begin Band Week. A couple of months later I met the man in person while practicing early in the very building.

I've posted the following text before on Facebook, but it really deserves to live on my own blog as well.

Set the clock back to Homecoming week, 1987. At this point, the Michigan Drumline marched 6 (yes, Six) snares, 4 basses, 4 cymbals and 2... yes TWO tenors, quads to be exact. Being 3rd out of 2, and having class end an hour before Drumline practice started, were all the motivation I needed to get to Revelli Hall early and practice.

Yes, Revelli Hall. There was a large, easily visible, picture of The Man Himself as you walked in. I was an out-of-stater, who didn't really know all of the history of the University of Michigan, or its band. I knew Revelli was a legendary director, had the job for a long time, and commanded a great deal of respect. I'd already learned the first of what I like to think of as Revelli's Three Laws:

1.) To be early is to be on time, to be on time is to be late, and to be late is to be lost.

So I'm practicing. Glad that at least while I couldn't march halftime (never mind pregame, something tenors of my day NEVER got to do), I did have a drum and a uniform. I got to play on the steps, play while marching to & from the stadium, and to play postgame (including Temptation & Warchant).

As I'm practicing a short older man walks in. Yes, it's William D. Revelli, the same guy on the picture. Wow! At this point, I remember that even famous people don't want to be all fawned over (think "Limelight" from Rush, a band every drummer listens to), but I did want to talk with him. I can't remember if he came over to me, but we talked. I can't also remember what exactly we all talked about, but he was very nice, and I thought I was very respectful. He thanked me for talking and I said (trying to be normal), "No problem," and went back to practicing.

He directed us, and I learned the remaining of his three laws as he told the whole band about them:

2.) If you want to know how good a marching band is, close your eyes.

3.) If you have class, you don't need to show it.

Apparently, Dr. Revelli was tickled pink by my usage of "No problem". Alas, when then-director Eric Becher told this story to the band at large the next day, they (we) hissed. Yes, hissed. I don't know if that still happens today with bad news, but when we heard something like, "Oh this writer from the Daily said about us," we'd hiss. My use of "No problem," being cause for hissing, well, scared me a little, and annoyed me equally so. I never meant disrespect to him, I was just not trying to be fawning. A kind graduate assistant, however, confirmed that he was, in fact, tickled pink about it.

A year or two later, while speaking at the MMB Parent's Banquet, Dr. Revelli briefly talked about "No problem" and how he thought it reflected on how kids of our generation (GenX) tackled things in a no-nonsense matter.

So today, on Facebook, a distinguished member of the University of Michigan Band Alumni Association posted this tribute and included a piece of Revelli history I had not known before: his final letter the MMB members got to read on the bus from Ann Arbor to Columbus. Note that Dr. Revelli not only dated, but TIMESTAMPED his letters.

    November 17, 1970 11:45 p.m.

Gentlemen of Michigan:

The evening shadows are long; soon, another day will have passed—gone into
eternity—beyond recall, never to return in our times. Tomorrow’s dawn will
bring another day—a new opportunity to live, to love, to work, to help each
other.

What a great and rewarding season this has been. You have performed nobly; to
each and every one of you, I offer my sincere gratitude and eternal thanks.

Come Saturday, we shall give our very best, our all, for our band, our team,
our University, and ourselves. Once again, we shall uphold our motto—“Non Tam
Pares, Quam Superiores.”

Following is my final message to you. It is not new; you have heard it many
times, yet is as true today as it has been since the birth of mankind and as
it will be for centuries to come. I hope you will make it a part of your
personal credo—for if it becomes so, then I, too, shall be a part of you—“even
if ever so small.”

REVELLI’S CREDO

1. Excellence. Do not be deceived by the cult of the mediocre—the enemy of
excellence, and remember that excellence demands standards. It is revealing to
see how often a man who keeps the highest standards in his professional
calling will tolerate the lowest of standards elsewhere. Would it not be more
desirable to seek in every phase of life the standards that perfection
requires?

2. Adequate Thinking. This means adequate purpose, for purpose, and not
L.S.D. or any other drug is the greatest expander of the mind and stimulant of
the mental faculties.

3. Creativity. Abjure cynicism. Do not be deceived by the false
intellectualism of the cynic. He is often the man who in his heart knows the
truth, and in his mind, has decided not to face it. He devoted his brain to
justifying the stultification of his conscience. Cynicism in you strangles
creativity in others. Give free rein to heart and conscience; creative
imagination will come alive.

4. Commitment. We hear a great deal about commitment in the University
environment, and even in circles where the person has it or not, I think just
about everyone is committed to something. The human being is a committed
animal by nature. The trouble is, he is usually committed first of all to
himself. But, the greatest men and women of history have usually found a
worthy commitment far bigger than themselves. Seek that.

5. Faith. Faith that man’s spirit can carry his abilities far beyond his
ordinary capacity. Faith that man’s closest approach to God lies in his
whole-souled response to the greatest task he feels God has for him. Faith
that man is not a prisoner of circumstances—that he can shape history rather
than be shaped by it.

6. Worthwhile Adventure. Not least, a spirit of a happy worthwhile
adventure. The spirit that comes from the heart and talents mobilized for
urgent worthy ends.

I love each and every one of you, and I pray that God may bless and keep you
healthy and happy always.

Remember—it is not who you are or where you are, but rather what you are that
shall bring you life’s joys and happiness.

Sincerely your devoted friend,

William D. Revelli

Nearly 40 years later, I still feel honored to have met the man. And been able to be (alongside all MMBers) a small part of his legacy.

He always directed Irving Berlin's God Bless America for homecoming. Here's the MMB two years after I graduated playing it under his direction:

Comments? Engage me at...

• BSky
• Fedi
• FB

A little over five and a half years ago, I wrote about retiring Guilt. Well, I have some good news! Earlier this year, I was contacted by Frediano Ziglio about restarting guilt development. After some discussions, we concluded that Frediano should become the maintainer. In other words:

Guilt is maintained again!

Not wasting any time, he collected and committed a number of fixes from the various GitHub forks out there, and tagged the v0.37 release. (There has been v0.37.1 since with a few minor fixes.)

The astute readers noticed that the above link is to GitHub. While I am not a fan, Frediano already maintains open source software on GitHub and therefore is familiar with it. That matters a lot. To make the decision even easier, repo.or.cz is geoblocking all of the UK because of the Online Safety Act 2023 requirements. (I don’t blame repo.or.cz for taking this step.) So as a result, the official repo is now on GitHub:

https://github.com/git-guilt/guilt

The old repo is now configured as a mirror of the GitHub repo, so it will continue to function, but I encourage everyone to update their git remotes.

The move is still a bit of a work in progress. I put a terribly hacky index.html on the old official page and a redirection for the manpages to https://git-guilt.github.io.

The source tarballs on my server (31bits.net) will continue to grow stale until we figure out a good way to redirect to the GitHub releases page.

This is a guest post by my wife, Holly.

In the software industry, some problems don't have Google-able solutions. Just like I can't Google where I left my keys, when I last got an oil change, or why I still haven't filed my taxes, I also can't Google what the race condition is in my code. To answer these questions, I must look inside myself. Usually.

"I hope your trip is going awesome and I hate to bother you, but..."

This tale begins in 2013. Jeff and I had recently gotten married and were on our honeymoon. Our team was rather... small... at the time, so we left the storage system we were responsible for in the hands of Josh, our intern of four months or so, with oversight from the Ops team. Halfway through our trip, Josh reluctantly sent us an email with details of an assertion failure in production that he couldn't debug: sem_post() was supposed to return 0, but it was returning -1. It had happened on a couple servers now, and only started after we left. Jeff and I didn't have the resources to debug it remotely, and restarting the service was a viable workaround, so we left it as a problem for our return.

Over the next four months, we made little progress. We discovered that sem_post() was specifically failing with EINVAL, but none of us could find the race condition in the code. Because that's what it means when your calls to concurrency primitives are failing: you have a race condition because you are bad at writing concurrent code. Four months later, Jeff left the company with the bug still unresolved. Josh (no longer an intern) and I continued poring over the code. The failure was rare and we didn't know how to reproduce it -- it happened every ~500,000 server-hours, and given the size of our fleet, that was on the order of once a month -- but when we saw it, we usually spent a few more hours trying to track it down. No dice. It seemed that we were bad at concurrent programming, and even worse at debugging.

Anger makes you do irrational things

One evening, some 16 months after the first occurrence, Josh messaged me. He said he thought he had finally found our elusive race condition!

Inside, I must confess, I did not react maturely. I was not relieved, proud, nor excited. I was incensed to hear he found it, after I'd sunk so much time into this utterly impossible bug. Blinded by my entirely irrational fury, determined to prove him wrong (did I mention "irrational"?), I rage-Googled the problem.

(Again, for the uninitiated, this is equivalent to Googling "where did I leave my keys". Race conditions are a problem you have to solve yourself. Perhaps you could post all the relevant code on a message board and find someone willing to help you debug your specific problem, because the logic is unique to your program and there's no Google-able "general solution" to your errors with semaphores beyond "be better at writing concurrent code.")

One of the search results caught my eye.

Horses, not zebras

There's a saying in medical schools: "When you hear hoofbeats, think of horses, not zebras." When diagnosing a problem, it is more likely that the issue is a common one rather than a rare one. If you have a runny nose, it's probably a cold and not a cerebrospinal fluid leak. If you're getting errors while using concurrency primitives, you probably wrote the code wrong. After all, concurrency is hard and life isn't an episode of House, M.D.

But zebras do exist.

I had clicked the link for a Bugzilla issue titled sem_post/sem_wait race causing sem_post to return EINVAL and started reading. (Meanwhile, Josh told me "Never mind, I didn't find it after all.") The bug report from someone named Don suggested that there is a race in the implementation of glibc semaphore primitives themselves causing sem_post() to access already-freed memory and fail with EINVAL (we were seeing EINVAL!). He said it's hard to reproduce (we saw it once per hundreds of thousands of server hours!). He could reproduce it by adding a sleep() call to the sem_post() function (which was farther than we'd gotten!). This was our problem! It had to be! Maybe we weren't bad at writing concurrent code!

He does not understand your question because he is assuming you are an idiot

One problem with "think of horses, not zebras" is that other people are well aware of the concept.

Following Don's report of the bug was an exchange with Ulrich (one of the main contributors/maintainers for glibc at the time) in which Ulrich asked "Why would this at all be a bug?" and stated "Your code is wrong in assuming what it does." Don patiently outlined his thinking. Ulrich tersely asserted that Don's usage was illegal. Don made one final plea ("Sorry if it seems I am belaboring this"), and Ulrich never responded.

After four months, someone named Pat replied to Don, and opened his response with a sentence that burned itself into my brain. He said, "Ulrich does not understand your question because he is assuming you are an idiot."

It's not an entirely unreasonable assumption. After all, what is more likely? Don is the first person to identify "a fundamental race rendering semaphores useless" in glibc, a library whose semaphores have been used by a great many people for a great many years seemingly without issue, or Don is kind of bad at concurrent programming and the issue with his code is his own? Ulrich decided he was hearing horses and didn't look for zebras. And I almost can't blame him.

When I was a TA for freshman- and sophomore-level programming classes, my students made semi-regular claims that "the compiler is broken." This was their way of expressing that their code did not do what they expected it to do or, in some cases, that their code did not even compile. Of course the compiler isn't broken! These are freshmen who have been writing C++ for two months and g++ had been compiling it since before they were born. Is it more likely that a student who has just begun to learn programming has a syntax error or a bug in their code, or that they have found a heretofore undiscovered problem with software that has been used by millions of people for decades? While I never assumed they were idiots (my students were wonderful; one was the best man at our aforementioned wedding) I admit I gave approximately zero credence to their claims of having discovered a compiler bug, just as Ulrich gave zero credence to Don's bug report.

Over a decade later, I try to channel my inner Pat. (A mere three days after Pat replied in support of Don's claims, a fix for the race was implemented by Rich Felker.) "Think horses" is the type of heuristic that keeps me from wasting too much time on absurd-sounding claims, but at the same time, I try not to get trampled because I was too busy assuming the guy yelling "Zebras!" is an idiot. I'm not perfect, but I'm trying. And I'm pretty sure none of my freshmen ever found a real compiler bug in class...

In the end, we (Josh) worked around the sem_post() race condition, replacing the fundamentally-broken semaphores with locks and a condition variable, because we were already running an outdated version of glibc and knew it might be a while before Ops could update all the servers. After that change, we never saw the failure again.

History doesn't repeat itself, but it rhymes

Why am I writing this post today?

Recently my team had an issue with a cloud service provider (CSP) that we use. It wasn't related to race conditions, but it was the same type of un-Google-able problem. We assumed the issue was a bug in our code and tried to track it down. After a while with no real leads, and a problem scope that continued widening, we opened a case with the CSP.

(Please pardon my deliberate ambiguity in this section, but I am hesitant to divulge as many details about an issue we haven't yet fully resolved internally as I did for a decade-old bug.)

We suspected a hardware issue, they insisted it was a software issue (first ours, but later conceded it might be theirs). Six weeks of back-and-forth later, the glibc semaphore issue crossed my mind, and I Googled my un-Google-able problem once again (out of frustration, but not out of rage). One of the results was a tweet, which led me to a recent-ish hardware research paper, and with every sentence I read, I became more and more convinced that the issue described in this paper was exactly what we were experiencing.

Once more, I had Googled the impossible and found a plausible explanation that lined up with our problem! This time, however, I found myself in Don's shoes. Because, truly, what is more likely? That we (a not-terribly-significant customer of the large CSP) are repeatedly encountering an issue caused by a hardware issue that is both sufficiently rare and sufficiently interesting that research papers are being written about it? Or that we (again, a rather insignificant customer) are idiots who have written some buggy software or are using some buggy libraries, and that's the source of all of our problems?

After another five weeks of meetings in which we provided all the evidence we had been collecting, including sharing the paper with relevant sections highlighted, and imploring them to look into it, I was finally confident that the CSP had brought the right people -- hardware people -- to our sync-up. Two weeks later, they were able to show us the results of the tests they used to identify the specific hardware that was causing our issue. They're pulling it from production, which should mean the end of the problem for us and any other customers who may have encountered it.

Is there a lesson here? I'm not sure. But if I had a nickel for every time Google led me to the right answer for an un-Google-able problem only to encounter challenges getting anyone to believe the diagnosis because it is vanishingly unlikely, I'd have two nickels, which isn't a lot, but it's weird that it happened twice. Perhaps, once a decade or so, you may find yourself on one side or the other of such an issue, and I hope that you can channel your inner Pat to a speedier resolution than these cases had.

P.S. Early on, a coworker tried diagnosing this issue with the help of AI, which told him with "highest probability" it was a bug with the (JIT) compiler. Call me a hypocrite, but I did not try to understand its explanation because I was assuming that it was an idiot.

OmniOSce v11 r151058 is out!

On the 4th of May 2026, the OmniOSce Association has released a new stable version of OmniOS - The Open Source Enterprise Server OS. The release comes with many tool updates, brand-new features and additional hardware support. For details see the release notes.

Note that LTS r151046 is now end-of-life. You should upgrade to r151054 or r151058 to stay on a supported track. r151054 is a long-term-supported (LTS) release with support until May 2028.

OmniOS is fully Open Source and free. Nevertheless, it takes a lot of time and money to keep maintaining a full-blown operating system distribution. Our statistics show that there are almost 2’000 active installations of OmniOS while fewer than 20 people send regular contributions. If your organisation uses OmniOS based servers, please consider becoming a regular patron or taking out a support contract.

Any problems or questions, please get in touch.

 I've just announced the latest release of Tribblix - m39, now available for download and upgrades.

This follows the "when I feel like it" release model. Which means that (a) I have some potentially breaking changes that need a release boundary to navigate, and (b) enough time has passed that I feel the need to update illumos to pick up any recent changes. There's no hard timescale, but for regular releases I would have thought 3-6 months would be about the right ballpark.

What's new this time around? There's the usual dry list of updates, which could be much longer (anything updated multiple times only gets listed the once, and all the python and perl module updates are missing entirely).

Lets pick some of those updates apart.

The libtiff update was a large one. Not because the update itself was difficult, but because the shared library SONAME got updated. For the time being (and for an indeterminate length of time) I'll ship both the old and new shared libraries, but I've rebuilt (almost) everything against the new version. That's one reason for this being done on an upgrade boundary - to force the breaking change and all the updates associated with it to take place at once.

There's a similar, but much smaller, story associated with OpenEXR.

For OpenSSL, there are a couple of changes. The first is that it's bumped from the 3.0.x series to the 3.5.x series. It's all binary compatible, so doesn't need the world to be rebuilt, but again the reason for pushing it on a release boundary is to ensure that nothing subsequently built against the new version is installed on a system with the old version. The second change is that the surfaced API is now 3.0, rather than the prior 1.1.1.

It wasn't strictly necessary to have an OpenSSH update tied to a release, but that got rolled in. My very first test triggered the post-quantum warning, because one of my build servers is deliberately running a much older version of Tribblix with a much older SSH server.

The underlying illumos gate build also has additional patches for NFSv4.1 - specifically the backchannel fixes in 16390. Hopefully this will make it into regular illumos soon, but it seemed like an excellent feature to get baked in.

I also patch illumos, as I did last time, for a larger range of pids, remove the y2038 clamp in ZFS, and tune the network stack for the 21st century.

There's one Tribblix feature that would be worth talking about in this release - appstack zones, which allow you to build a zone running an application, and do basic configuration of it, all with a simple one-line command. I'll talk about that separately, as it's almost but not quite ready for wider adoption.

I've actually been using IPv6 on and off since the late 1990s - there was an addon for Solaris 2.6 that we installed on a bunch of test machines. It worked great, but wasn't something we ever did properly in production because at the time none of our customers had IPv6.

I've wanted to use IPv6 more, but my home ISP has never had it. Until recently, when I noticed my main machine (running Tribblix, naturally) has an Ipv6 address and was using it. At that point, investigating and testing IPv6 becomes a lot more interesting.

Some of the Tribblix web servers (specifically the pkgs and iso download servers) are hosted in the cloud. I do this for testing and dogfooding, so I know that Tribblix works on those cloud environments.

The iso machine is on Digital Ocean; sadly Digital Ocean don't support IPv6 for custom images, for no good reason that I can see. But the pkgs server is on AWS, and they do support IPv6, but it wasn't enabled.

My first test here was actually to see if I could launch an EC2 instance with an IPv6 address, using the aws cli:

aws ec2 run-instances --ipv6-address-count 1 ...

and this failed, because of the fact that IPv6 wasn't enabled where I was trying to launch it. So I needed to do a few steps to make it work, and this is largely to remind me for when I need to do it again.

First, you need to associate an IPv6 block of addresses with the VPC you're using. Go to the VPC in the console, find something that says Edit CIDRs and, in that, Add a new IPv6 CIDR. Just choose an Amazon provided one and you're good. That should give you a /56 block.

You then need to go into each of the subnets in that VPC and associate an IPv6 CIDR block with it. Find the Edit IPV6 CIDRs button, go into that, and Add IPv6 CIDR. You can add the entire block to one subnet if you want, but normally you can break it down. Under the allocation are some arrows - the up and down arrows change the size of the block, I just went down one to a /60. By default, the starting address of the subnet block is the same as the VPC block - for additional subnets you'll need to use the little right arrow to change the start address, as you can't associate overlapping blocks.

I also went into the subnet settings to Enable auto-assign IPv6 address.

What you then need to do is go to the route table (either from the subnet or the VPC) and add a route for ::/0 with the target being the internet gateway - there should only be the one gateway, the same as used for IPv4. If you don't add the route to the route table you'll get IPv6 addresses but you won't be able to talk to anything outside the VPC over IPv6.

With that, launch a new instance and you'll get IPV6 working nicely.

Nothing to do with EC2, but the one other thing I needed to do was add an additional IPV6 listen directive to each server in my nginx config, as nginx will only listen on IPv4 by default.

It's story time, and still Boxing Day in some parts of the US.

Recent rumblings about treating your customers right, why some folks don't, and the subtle slope between happy customers and captured customers remind me of one of my favorite hole-in-the-wall restaurant employees, Phil. He loves his customers: even the ones who ask for off-menu items, even the ones who can't properly describe what they really want, and especially the ones who like what he cooks and tell their friends about it.

I won't name this hole-in-the-wall, because it's a front for some flavor of organized crime. I'm not sure what crime syndicate it is, and whenever I ask Phil he gets really sheepish about it. Still, the food is good, and Phil is proud of what he and his small staff cook. The most recent time I asked about the owners, however, he told me about how chefs often work at establishments with ties of some sort to organized crime. He then went on a bit longer.

Phil started in by telling me he used to be an excellent pastry chef for a one-Michelin-star restaurant. Its owners used it to launder money from their more nefarious operations, but kept their laundering businesses as clean as possible to avoid IRS or other law-enforcement scrutiny. You don't get a Michelin star without having very happy customers. The restaurant had incredible wait staff, bussing that was simultaneously invisible and highly responsive, and cuisine from world-class chefs. The restaurant management and staff also believed firmly it could get even better, adding a second star perhaps, through an assortment of ways. They were highly motivated to try new foods, new recipes, and new dining experiences. Phil noted that not long after he joined, they doubled the restaurant size with minimal disrutption to the dining environment. He noted he designed his pastry station to fit nicely into either the expanded kitchen, or the original smaller one. They replaced a wall with a glass pane that allowed folks to see the chefs at work.

Phil's chocolate crinkle cookies, listed on the menu as Knockout Cookies, were his most famous pastry. He and his staff developed them to test the pastry station's capabilities, and they were yummy. Soon the other chefs and the staff started eating them. Some of the staff had friends at other restaurants, and so the Knockout Cookies would sometimes be eaten behind closed doors at other restaurants. Any new staff that showed up at Phil's old restaurant would be asked to try a Knockout Cookie, and they usually liked it enough to take some home with them. Phil thought maybe Knockout Cookies could be sold to other restaurants, the way the donut chain AnyDonut did with their donuts.

Many more such projects were in the middle of being planned or tested when the nearly-absentee owners found themselves unwittingly in a turf war against a more dangerous syndicate, which they lost. All of the protection rackets, illicit activities, and money-laundering businesses became part of this new, more dangerous, syndicate.

The new owners of the ONE-MICHELIN-STAR restaurant wanted to cut costs, because more money could be laundered that way. This cost-cutting plan irrtated the staff and more than a few quit immediately or not very long after the turf war ended. Those that stayed behind either fell in with the new syndicate, or naively thought that as long as money was being laundered, they could continue with their new cuisine, recipes, or dining experiences. As if to dispel such notions, the new syndicate even spray-painted the glass pane red!

Another one of the cost-cutting measures was to reduce the in-house pastry selection. The Knockout Cookies, in spite of their devoted fans, were eliminated. Adding insult to injury for Phil, the restaurant started outsourcing chocolate cookies from AnyDonut. Phil was pissed, and scribbled on all of the menus: crossing out Knockout Cookies and writing AnyDonut Chcolate Cookies. This earned him and the pastry staff a meeting with a remaining manager, nose brown with sucking up to the new syndicate. The new manager even threatened a visit from an underboss.

Not long after that half the pastry staff quit, including Phil. It took him way too long to figure out what had happened while he wandered around for any cook or chef job he could find. Turns out AnyDonut was not only a supplier, but they were ALSO a protection-money payer. It was likely more profitable to use AnyDonut since they were paying protection money anyway, and the new syndicate had no interest in self-sabotaging their protection money.

Phil finished, shook his head, and said, "It didn't take long to lose the star." He then slipped me a Knockout Cookie, gratis, and whispered, "I still bake and eat 'em at home."

...

(Inspired by BDHA's parable.)

Comments? Engage me at...

• BSky
• Fedi
• FB

What About Windows Users?

This post has been written in Markdown.

This blog is running on software that I wrote a number of years ago after being inspired by David Graham who threw together his blog using a couple of shell scripts. I was using Wordpress, which was a behemoth, and I thought that writing my own would making blogging a more pleasant experience for me. (It did.)

I quickly learned that I needed to support more than one post format. At first, the formats used raw or raw-ish HTML. Eventually, I added "fmt 3" which is a LaTeX-like format. I still use that for my posts even though it has limitations that I want to address with "fmt 4".

Fast-forward a few years and Dan set up my software to run his blog. Unsurprisingly, he inquired about Markdown support. Since I have no use for it myself, I told him that patches are welcome :)

Fast-forward a few more years and Dan sent me a patch to add "fmt 5"—Markdown support via libmd4c. Thanks, Dan!

So, this post is kind of an announcement for whomever may care that blahgd now supports Markdown-formatted posts.

P.S. I will continue using "fmt 3" because I like it, but I thought it was fitting to make an exception and write this post in Markdown instead.

I wonder if this renders correctly?

I think it does.

I knew about the MD4C project for some time. I finally modified blahg to exploit MD4C so that I can create posts using markdown.

I haven't tested it all out yet, but I hope to soon. It's been a while (eesh, 3.5 years) since my last post, and no surprised, a lot has happened since then.

If I still have my half-dozen readers, hello again.

OmniOSce v11 r151056 is out!

On the 3rd of November 2025, the OmniOSce Association has released a new stable version of OmniOS - The Open Source Enterprise Server OS. The release comes with many tool updates, brand-new features and additional hardware support. For details see the release notes.

Note that r151052 is now end-of-life. You should upgrade to r151054 or r151056 to stay on a supported track. r151054 is a long-term-supported (LTS) release with support until May 2028. Note that upgrading directly from r151052 to r151056 is not supported; you will need to update to r151054 along the way.

For anyone who tracks LTS releases, the previous LTS - r151046 - is now in its last six months. You should plan to upgrade to r151054 for continued LTS support.

OmniOS is fully Open Source and free. Nevertheless, it takes a lot of time and money to keep maintaining a full-blown operating system distribution. Our statistics show that there are almost 2’000 active installations of OmniOS while fewer than 20 people send regular contributions. If your organisation uses OmniOS based servers, please consider becoming a regular patron or taking out a support contract.

Any problems or questions, please get in touch.

So recently I acquired @danmcd on Twitter. It was a long time coming. I was relatively late in early-adopting twitter: late spring 2009. By then someone else had claimed the handle danmcd, to my chagrin.

I was chagrined (in 2009) because I’ve been danmcd at SOMEWHERE since 1988. First .edus, even a .gov and .mil, and of course a series of .coms including my own kebe.com

(Who and/or what is Kebe might be another blog post in and of itself. In the meantime, this answer will suffice:

)

Names are important. Especially in the virtual world, they establish not only presence, but often place as well. I ended up being @kebesays on twitter for a long time. Luckily, Twitter makes handle-swapping relatively easy, so anyone who was following @kebesays got moved over to @danmcd without issue. I still keep /* XXX KEBE SAYS … at the top, because if you see that in my code, it indicates work-in-progress issues; and aren’t we all works-in-progress?

Speaking of names and places: one name and one place that has been associated with Triton and SmartOS - Joyent - will no longer be associated with SmartOS or Triton. Samsung has decided to use other in-house technology for their future, and that work will continue with Joyent. SmartOS and Triton are being spun off to MNX Solutions, where I will be continuing SmartOS development. See the MNX Triton FAQ and my email for more.

Oh and yes, I’ll get to be ‘danmcd‘ at MNX as well.

So what happened?

We’ve introduced a requested feature in SmartOS: the ability to select a platform image from loader(4), aka OS-8231.

To enable this feature, you must (using example bootable pool bootpool):

• Update BOTH the boot bits and the Platform Image to this release. Normally piadm(1M) updates both, so please use either latest or another ISO-using installation.
• Once booted to this PI, utter piadm activate 20210812T031946Z OR install another ISO-using installation (even if you never use it) to have the new piadm(1M) generate the /bootpool/boot/os/ directory the new modifications to loader require.

This represents a minor flag day because an older piadm(1M) will not update an existing /bootpool/boot/os/ directory. The PI-selection menus live in /bootpool/boot/os/, and will remain in an inconsistent state with older PIs using piadm(1M). It is safe to remove /bootpool/boot/os/ if you wish, as the activated (default) PI always boots correctly modulo actual /bootpool/boot/ corruption regardless.

So Tell Me about the Internals and the os/ Directory!

There were two SmartOS repositories that had changes. The first changeset was in illumos-joyent’s loader(5) Forth files. Alongside some additional support routines, the crux of the change is this addition to the main Joyent loader menu:

\
\ If available, load the "Platform Image Selection" option.
\
try-include /os/pi.rc

If the piadm(1M)-generated file /bootpool/os/pi.rc does not exist, the Joyent loader menu appears as it did prior to this fix.

The os/ Directory and illumos Needing platform/

The os/ directory in a bootable pool’s bootpool/boot filesystem contains directories of Platform Image stamps and the aforementioned pi.rc file.

[root@smartos-efi ~]# piadm list
PI STAMP               BOOTABLE FILESYSTEM            BOOT IMAGE NOW  NEXT 
20210715T010227Z       bootpool/boot                  available  no   no  
20210805T161859Z       bootpool/boot                  available  no   no  
20210812T031946Z       bootpool/boot                  next       yes  yes 
[root@smartos-efi ~]# ls /bootpool/boot/os
20210715T010227Z  20210805T161859Z  pi.rc
[root@smartos-efi ~]# 

Each PI stamp directory contains a single platform symbolic link up to the platform-STAMP directory that contains the PI.

[root@smartos-efi ~]# ls -lt /bootpool/boot/os/20210805T161859Z
total 1
lrwxrwxrwx   1 root     root          31 Aug 12 14:41 platform -> ../../platform-20210805T161859Z
[root@smartos-efi ~]#

The Triton Head Node loader menu has a pointer to the “prior Platform Image” has the explicit path of …/os/STAMP/platform contain the platform image. It was a design mistake of the original standalone SmartOS to not lay out platform image in this manner, but given that piadm(1M) must generate the pi.rc file anyway, it is not much more difficult to add symbolic-link construction as well.

The pi.rc File

The pi.rc file includes an additional menu item for the main Joyent loader screen:

It also contains up to three pages of platform images to choose from. Here’s an example of page 1 of 3:

The default PI is on every page, and up to five (5) additional PIs can appear per page. This means 16 PIs (default + 3 * 5) can be offered on a loader screen. Every time a platform image is activated, deleted, or added, the piadm(1M) command regenerates the entire os/ directory, including pi.rc.

So How and Why Do I Use This?

• Temporarily revert to and older Platform Image may be useful to check for regressions or to isolate behavior to a specific release.
• Developers can use *just* platform-image installations (platform-yyyymmddThhmmssZ.tgz to test their new builds without making the bootable pool unusable.

The piadm list output indicates being booted into a non-default PI by its NOW column:

PI STAMP               BOOTABLE FILESYSTEM            BOOT IMAGE NOW  NEXT 
20210114T041228Z       zones/boot                     available  no   no  
20210114T163038Z       zones/boot                     available  no   no  
20210211T055122Z       zones/boot                     none       no   no  
20210211T163919Z       zones/boot                     none       no   no  
20210224T232633Z       zones/boot                     available  no   no  
20210225T124034Z       zones/boot                     none       no   no  
20210226T213821Z       zones/boot                     none       no   no  
20210311T001742Z       zones/boot                     available  no   no  
20210325T002528Z       zones/boot                     available  no   no  
20210422T002312Z       zones/boot                     available  no   no  
20210520T001536Z       zones/boot                     available  no   no  
20210617T001230Z       zones/boot                     available  no   no  
20210701T204427Z       zones/boot                     available  no   no  
20210715T010227Z       zones/boot                     available  no   no  
20210729T002724Z       zones/boot                     available  no   no  
20210804T003855Z       zones/boot                     available  no   no  
20210805T161859Z       zones/boot                     available  yes  no  
20210812T031946Z       zones/boot                     next       no   yes 

In the above example, the SmartOS machine is booted into 20210805T161859Z, but its default is 20210812T031946Z. It would also look this way if piadm activate 20210812T031946Z was just invoked, as the semantics are the same.

That I had to explain MTV's acronym... eeesh.

When Cable TV Was Still Young

Set the wayback machine 40 years plus 6-8 months ago (from the date of this post). Cable TV was rolling out in my suburb of Milwaukee, and it FINALLY arrived at our house. Hurray! We didn't have HBO, but we DID have all of the other fledgling basic cable channels... including Nickelodeon, which was then one of the Warner Amex Satellite Entertainment Company (WASEC) channels. (WASEC, and its progenitor Columbus, Ohio QUBE project, are its own fascinating story.) Nickelodeon mostly had single-digit-aged kids programming, but at night (especially Sunday night) it had a 30-minute show called PopClips, which would play the then mindblowing concept of music videos... or as one friend of mine called them, "Intermissions" (because HBO would play music videos between movies to synch up start times... I didn't have HBO so I trusted him). There is a YouTube narrative video that discusses the show in depth, including its tenuous link to another WASEC channel that was going to start airing 40 years ago today...

I Want My MTV

Anyone sufficiently old knows that MTV stood for Music Television. At midnight US/Eastern time on August 1, 1981, it played its space-program-themed bumper, followed by, "Video Killed the Radio Star" by The Buggles.

Now the local cable company pulled a bit of a dick move with MTV for us. It attached it to HBO. If you didn't have HBO, the cable company scrambled MTV, albeit not as strongly as they did with HBO. They scrambled it by making the picture black-and-white, and cutting out the sound completely. LUCKILY for me, we did have "cable radio" which let us not only get better FM reception, but also the stereo broadcast for MTV. Combine them, and I got to see black-and-white videos with proper sound.

Thanks to people's old videotapes and YouTube, you can watch (modulo a couple of copyright-whiners) the first two hours of MTV here. I'd have embedded this, but I'm guessing the copyright-whiners won that battle too.

There's a lot to unpack about MTV being 40. I'm not going to try too hard in this post, but there are some things that must be acknowledged:

• MTV was a generation-defining phenomenon for Generation X. I suppose late-wave Boomers (the last of whom were graduating high school or already in college) could make a claim to ownership of MTV's first audience, but as MTV matured, it was very much initially for us Xers.
• It was initially narrowly focussed. The only Black people you'd see on MTV initially were JJ Jackson or members of The Specials. That changed a couple of years later, however.
• It spawned at least one knock-off: Friday Night Videos, which unlike MTV didn't require Cable.

Of course MTV doesn't play music videos on it anymore, we have alternatives now: YouTube, DailyMotion, and their ilk. And if you miss your MTV, or want to know what it looked like, you really don't have to look hard; many people have uploaded at least some VHS rips, many alas without music thanks to copyright teardowns. But with artist often putting out their old music on their own YouTube pages, some have taken to curating lists of them. Even NPR has curated the first 100 songs!

Two Decades Ago…

Someone pointed out recently that the famous Internet meme “All your base are belong to us” turned 20 this week. Boy do I feel old. I was still in California, but Wendy and I were plotting our move to Massachusetts.

In AD 2001, S9 Was Beginning

OF COURSE I watched the video back then. The original Shockwave/Flash version on a site that no longer exists. I used my then-prototype Sun Blade 1000 to watch it, on Netscape, on in-development Solaris 9.

I found a bug in the audio driver by watching it. Luckily for me, portions of the Sun bug database were archived and available for your browsing pleasure. Behold bug 4451857. I reported it, and all of the text there is younger me.

The analysis and solution are not in this version of the bug report, which is a shame, because the maintainer (one Brian Botton) was quite responsive, and appreciated the MDB output. He fixed the bug by moving around a not-shown-there am_exit_task() call.

Another thing missing from the bug report is my “Public Summary” which I thought would tie things up nicely. I now present it here:

In A.D. 2001
S9 was beginning.
Brian: What Happen?
Dan: Someone set up us the livelock
Dan: We get signal
Brian: What!
Dan: MDB screen turn on.
Brian: It’s YOU!
4451857: How are you gentleman?
4451857: All your cv_wait() are belong to us.
4451857: You are on the way to livelock.
Brian: What you say?
4451857: You have no chance to kill -9 make your time.
4451857: HA HA HA HA…
Brian: Take off every am_exit_task().
Dan: You know what you doing
Brian: Move am_exit_task().
Brian: For great bugfix!

Pardon my latency

Well, at least I’m staying on track for single-digit blog posts in a year. :)

Okay, seriously, 2020’s pandemic-and-other-chaos tends to distract. Also, I did actually have a few things worth my attention.

RFD 176

The second half of 2020 at work has been primarily about RFD 176 – weaning SmartOS and Triton off of the requirement for a USB key. Phases I (standalone SmartOS) and II (Triton Compute Node) are finished. Phase III (Triton Head Node) is coming along nicely, thanks to real-world testing on Equinix Metal (nee Packet), and I hope to have a dedicated blog post about our work in this space coming in the first quarter 2021.

Follow our progress in the rfd176 branches of smartos-live and sdc-headnode.

Twins & College

My twins are US High School seniors, meaning they’re off to college/university next fall, modulo pandemic-and-other-chaos. This means applications, a little stress, and generally folding in pandemic-and-other-chaos issues into the normal flow of things as well. Out of respect for their privacy and autonomy, I’ll stop here to avoid details each of them can spill on their own terms.

On 2021

Both “distractions” mentioned above will continue into 2021, so I apologize in advance for any lack of content here for my half-dozen readers. You can follow me on any of the socials mentioned on the right, because I’ll post there if the spirit moves me (especially on issues of the moment).

A Gentle Reminder About illumos

A very bad security vulnerability in Solaris was patched-and-announced by Oracle earlier this week. Turns out, we in open-source-descendant illumos had something in the same neighborhood. We can’t confirm it’s the same bug because reverse-engineering Oracle Solaris is off the table.

In general if a vulnerability is an old one in Solaris, there’s a good chance it’s also in illumos. Alex Wilson said it best in this recent tweet:

@anthomsec Hi! re: 2020-14871 (Solaris PAM bug), do you know if this was in the OpenSolaris PAM that went into illumos (https://t.co/Rxs5ct09xs) by any chance? We don't get any info via Oracle, but we do inherit their bugs if they're old enough :(

— Alex Wilson (@arekinath) October 22, 2020

If you want to see the full history, the first 11 minutes of my talk from 2016’s FOSDEM contains WHY a sufficiently old vulnerability in Solaris 10 and even Solaris 11 may also be in illumos.

Remember folks, Solaris is closed-source under Oracle, even though it used to be open-source during the last years of Sun’s existence. illumos is open-source, related, but NOT the same as Solaris anymore. Another suggested talk covers this rather well, especially if you start at the right part.

The Actual Request

Because of this history and shared heritage, if you’re a security researcher, PLEASE make sure you find one of many illumos distributions, install it, and try your proof-of-concept on that as well. If you find the same vulnerability in illumos, please report it to us via the security@illumos.org mailing alias. We have a PGP key too!

Thank you, and please test your Solaris exploits on illumos too (and vice-versa).

Booting from a zpool

The most recent published biweekly release of SmartOS has a new feature I authored: the ability to manage and boot SmartOS-bootable ZFS pools.

A few people read about this feature, and jumped to the conclusion that the SmartOS boot philosophy, enumerated here:

• The "/" filesystem is on a ramdisk
• The "/usr" filesystem is read-only
• All of the useful state is stored on the zones ZFS pool.

were suddenly thrown out the window. Nope.

This change is the first phase in a plan to not depend on ISO images or USB sticks for SmartOS, or Triton, to boot.

The primary thrust of this specific SmartOS change was to allow installation-time enabling of a bootable zones pool. The SmartOS installer now allows one to specify a bootable pool, either one created during the "create my special pools" shell escape, or just by specifying zones.

A secondary thrust of this change was to allow running SmartOS deployments to upgrade their zones pools to be BIOS bootable (if the pool structure allows booting), OR to create a new pool with new devices (and use zpool create -B) to be dedicated to boot. For example:

smartos# zpool create -f -B standalone c3t0d0
smartos# piadm bootable -e standalone
smartos# piadm bootable
standalone                     ==> BIOS and UEFI
zones                          ==> non-bootable
smartos# 

Under the covers

(NOTE: Edited 3 May 2023 to change "1M" man page refs to "8".)

Most of what’s above can be gleaned from the manual page. This section will discuss what the layout of a bootable pool actually looks like, and how the piadm(8) command sets things up, and expects things to BE set up.

Bootable pool basics

The piadm bootable command will indicate if a pool is bootable at all via the setting of the bootfs property on the pool. That gets you the BIOS bootability check, which admittedly is an assumption. The UEFI check happens by finding the disks s0 slice, and seeing if it’s formatted as pcfs, and if the proper EFI System Partition boot file is present.

bootfs layout

For standalone SmartOS booting, bootfs is supposed to be mounted on "/" with the pathname equal to the bootfs name. By convention, we prefer POOL/boot. Let’s take a look:

smartos# piadm bootable
zones                          ==> BIOS and UEFI
smartos# piadm list
PI STAMP           BOOTABLE FILESYSTEM            BOOT BITS?   NOW   NEXT  
20200810T185749Z   zones/boot                     none         yes   yes  
20200813T030805Z   zones/boot                     next         no    no   
smartos# cd /zones/boot
smartos# ls -lt
total 9
lrwxrwxrwx   1 root     root          27 Aug 25 15:58 platform -> ./platform-20200810T185749Z
lrwxrwxrwx   1 root     root          23 Aug 25 15:58 boot -> ./boot-20200813T030805Z
drwxr-xr-x   3 root     root           3 Aug 14 16:10 etc
drwxr-xr-x   4 root     root          15 Aug 13 06:07 boot-20200813T030805Z
drwxr-xr-x   4 root     root           5 Aug 13 06:07 platform-20200813T030805Z
drwxr-xr-x   4 1345     staff          5 Aug 10 20:30 platform-20200810T185749Z
smartos# 

Notice that the Platform Image stamp 20200810T185749Z is currently booted, and will be booted the next time. Notice, however, that there are no “BOOT BITS”, also known as the Boot Image, for 20200810T185749Z, and instead the 20200813T030805Z boot bits are employed? This allows a SmartOS bootable pool to update just the Platform Image (ala Triton) without altering loader. If one utters piadm activate 20200813T030805Z, then things will change:

smartos# piadm activate 20200813T030805Z
smartos# piadm list
PI STAMP           BOOTABLE FILESYSTEM            BOOT BITS?   NOW   NEXT  
20200810T185749Z   zones/boot                     none         yes   no   
20200813T030805Z   zones/boot                     next         no    yes  
smartos# ls -lt
total 9
lrwxrwxrwx   1 root     root          27 Sep  2 00:25 platform -> ./platform-20200813T030805Z
lrwxrwxrwx   1 root     root          23 Sep  2 00:25 boot -> ./boot-20200813T030805Z
drwxr-xr-x   3 root     root           3 Aug 14 16:10 etc
drwxr-xr-x   4 root     root          15 Aug 13 06:07 boot-20200813T030805Z
drwxr-xr-x   4 root     root           5 Aug 13 06:07 platform-20200813T030805Z
drwxr-xr-x   4 1345     staff          5 Aug 10 20:30 platform-20200810T185749Z
smartos# 

piadm(8) manipulates symbolic links in the boot filesystem to set versions of both the Boot Image (i.e. loader) and the Platform Image.

In the prior post, I mentioned a need for four active ethernet ports. These four ports are physical links to four distinct Ethernet networks. Joyent's SmartOS and Triton characterize these with NIC Tags. I just view them as distinct networks. They are all driven by the illumos igb(7d) driver (hmm, that man page needs updating) on HDC 3.0, and I'll specify them now:

• igb0 - My home network.
• igb1 - The external network. This port is directly attached to my FiOS Optical Network Terminal's Gigabit Ethernet port.
• igb2 - My work network. Used for my workstation, and "external" NIC Tag for my work-at-home Triton deployment, Kebecloud.
• igb3 - Mostly unused for now, but connected to Kebecloud's "admin" NIC Tag.

(0)# zoneadm list -cv
  ID NAME             STATUS     PATH                           BRAND    IP    
   0 global           running    /                              ipkg     shared
   1 webserver        running    /zones/webserver               lipkg    excl  
   2 work             running    /zones/work                    lipkg    excl  
   3 router           running    /zones/router                  lipkg    excl  
   4 calendar         running    /zones/calendar                lipkg    excl  
   5 dns              running    /zones/dns                     lipkg    excl  
(0)# 

• global - The illumos global zone is what exists even in the absence of other zones. Some illumos distros, like SmartOS, encourage minimizing what a global zone has for services. HDC's global zone serves NFS and SMB/CIFS to my home network. The global zone has the primary link into the home network. HDC's global zone has no default route, so if any operations that need out-of-the-house networking either go through another zone (e.g. DNS lookups), or a defaut route must be temporarily added (e.g. NTP chimes, `pkg update`).
• webserver - Just like the name says, this zone hosts the web server for kebe.com. For this zone, it uses lofs(7FS), the loopback virtual file system to inherit subdirectories from the global zone. I edit blog entries (like this one) for this zone via NFS from my laptop. The global zone serves NFS, but the files I'm editing are not only available in the global zone, but are also lofs-mounted into the webserver zone as well. The webserver zone has a vnic (see here for details about a vnic, the virtual network interface controller) link to the home network, but has a default route, and the router zone's NAT (more later) forwards ports 80 and 443 to this zone. Additionally, the home network DHCP server lives here, for no other reason than, "it's not the global zone."
• work - The work zone is new in the past six years, and as of recently, eschews lofs(7FS) for delegated ZFS datasets. A delegated ZFS dataset, a proper filesystem in this case, is assigned entirely to the zone. This zone also has the primary (and only) link to the work network, a physical connection (for now unused) to my work Triton's admin network, and an etherstub vnic (see here for details about an etherstub) link to the router zone. The work zone itself is a router for work network machines (as well as serves DNS for the work network), but since I only have one public IP address, I use the etherstub to link it to the router zone. The zone, as of recent illumos builds, can further serve its own NFS. This allows even less global-zone participation with work data, and it means work machines do not need backchannel paths to the global zone for NFS service. The work zone has a full illumos development environment on it, and performs builds of illumos rather quickly. It also has its own Unbound (see the DNS zone below) for the work network.
• router - The router zone does what the name says. It has a vnic link to the home network and the physical link to the external network. It runs ipnat to NAT etherstub work traffic or home network traffic to the Internet, and redirects well-known ports to their respective zones. It does not use a proper firewall, but has IPsec policy in place to drop anything that isn't matched by ipnat, because in a no-policy situation, ipnat lets unmatched packets arrive on the local zone. The router zone also runs the (alas still closed source) IKEv1 daemon to allow me remote access to this server while I'm remote. It uses an old test tool from the pre-Oracle Sun days a few of you half-dozen readers will know by name. We have a larval IKEv2 out in the community, and I'll gladly switch to that once it's available.
• calendar - Blogged about when first deployed, this zone's sole purpose is to serve our calendar both internally and externally. It uses the Radicale server. Many of my complaints from the prior post have been alleviated by subsequent updates. I wish the authors understood interface stability a bit better (jumping from 2.x to 3.0 was far more annoying than it needed to be), but it gets the job done. It has a vnic link to the home network, a default route, and gets calendaring packets shuffled to it by the router zone so my family can access the calendar wherever we are.
• dns - A recent switch to OmniOSce-supported NSD and Unbound encouraged me to bring up a dedicated zone for DNS. I run both daemons here, and have the router zone redirect public kebe.com requests here to NSD. The Unbound server services all networks that can reach HDC. It has a vnic link to the home network, and a default route.

The first picture shows HDC as a single entity, and its physical networks. The second picture shows the zones of HDC as Virtual Network Machines, which should give some insight into why I call my home server a Home Data Center.

Twelve years ago I built my first Home Data Center (HDC). Six years ago I had OmniTI's Supermicro rep put together the second one.

Unlike last time, I'm not going to recap the entirety of HDC 2.0. I will mention briefly that since its 2014 inception, I've only upgraded its mirrored spinning-rust disk drives twice: once from 2TB to 6TB, and last year from 6TB to 14TB. I'll detail the current drives in the parts list.

Like last time, and the time before it, I started with a CPU in mind. AMD has been on a tear with Ryzen and EPYC. I still wanted low-ish power, but since I use some of HDC's resources for work or the illumos community, I figured a core-count bump would be worth the cost of some watts. Lucky me, the AMD Ryzen 7 3700x fit the bill nicely: Double the cores & threads with a 20W TDP increase.

Unlike last time, but like the time before it, I built this one from parts myself. It took a little digging, and I made one small mistake in parts selection, but otherwise it all came together nicely.

Parts

• AMD Ryzen 7 3700x - It supports up to 128GB of ECC RAM, it's double the CPU of the old HDC for only 50% more TDP wattage. It's another good upgrade.
• Noctua NH-U12S (AM4 edition) CPU cooler - I was afraid the stock cooler would cover the RAM slots on the motherboard. Research suggested the NH-U12S would prevent this problem, and the research panned out. Also Noctua's support email, in spite of COVID, has been quite responsive.
• ASRock Rack X470D4U - While only having two Gigabit Ethernet (GigE) ports, this motherboard was the only purpose-built Socket AM4 server motherboard. It has IPMI/BMC on its own Ethernet port (but you'll have to double check it doesn't "failover" to your first GigE port). It has four DIMM slots, and with the current BIOS (mine shipped with it), supports 128GB of RAM. There are variants with Two 10 Gigabit Ethernet (10GigE) ports, but I opted for the less expensive GigE one. If I'd wanted to wait, there's a new, not yet available, X570 version, whose more expensive version has both two 10GigE AND two GigE ports, which would saved me from needing...
• Intel I350 dual-port Gigabit Ethernet card - This old reliable is well supported and tested. It brings me up to the four ethernet ports I need.
• Nemix RAM - 4x32GB PC3200 ECC Unbuffered DIMMS - Yep, like HDC 2.0, I maxxed out my RAM immediately. 6 years ago I'd said 32GB would be enough, and for the most part that's still true, except I sometimes wish to perform multiple concurrent builds, or memory-map large kernel dumps for debugging. The vendor is new-to-me, and did not have a lot of reviews on Newegg. I ran 2.5 passes of memtest86 against the memory, and it held up under those tests. Nightly builds aren't introducing bitflips, which I saw on HDC 1.0 when it ran mixed ECC/non-ECC RAM.
• A pair of 500GB Samsung 860 EVO SATA SSDs - These are slightly used, but they are mirrored, and partitioned as follows:
  • s0 -- 256MB, EFI System partiontion (ESP)
  • s1 -- 100GB, rpool for OmniOSce
  • s2 -- 64GB, ZFS intent log device (slog)
  • s3 -- 64GB, unused, possible future L2ARC
  • s4 -- 2GB, unused
  • The remaining 200-something GB is unassigned, and fodder for the wear-levellers. The motherboard HAS a pair of M.2 connectors for NVMe or SATA SSDs in that form-factor, but these were hand-me-downs, so free.
• A pair of Western Digital Ultrastar (nee HGST Ultrastar) HC530 14TB Hard Drives - These are beasts, and according to Backblaze stats released less than a week ago, its 12TB siblings hold up very well with respect to failure rates.
• Fractal Design Meshify C case - I'd mentioned a small mistake, and this case was it. NOT because the case is bad... the case is quite good, but because I bought the case thinking I needed to optimize for the microATX form factor, and I really didn't need to. The price I paid for this was the inability to ever expand to four 3.5" drives if I so desire. In 12 years of HDC, though, I've never exceeded that. That's why this is only a small mistake. The airflow on this case is amazing, and there's room for more fans if I ever need them.
• Seasonic Focus GX-550 power supply - In HDC 1.0, I had to burn through two power supplies. This one has a 10 year warranty, so I don't think I'll have to stress about it.
• OmniOSce stable releases - Starting with HDC 2.0, I've been running OmniOS, and its community-driven successor, OmniOSce. The every-six-month stable releases strike a good balance between refreshes and stability.

I've given two talks on how I use HDC. Since the last of those was six years ago, I'm going to stop now, and dedicate the next post to how I use HDC 3.0.

Let's throw out the first pitch.

I've moved my blog over from blogspot to here at kebe.com. I've recently upgraded the hardware for my Home Data Center (the subject of a future post), and while running the Blahg software doesn't require ANY sort of hardware upgrade, I figured since I had the patient open I'd make the change now.

Yes it's been almost five years since last I blogged. Let's see, since the release of OmniOS r151016, I've:

• Cut r151018, r151020, and r151022.
• Got RIFfed from OmniTI.
• Watched OmniOS become OmniOSce with great success.
• Got hired at Joyent and made more contributions to illumos via SmartOS.
• Tons more I either wouldn't blog about, or just plain forgot to mention.

While Blahg supports comments, I've disabled them for now. I might re-enabled them down the road, but for now, you can find me on one of the two socials on the right and comment there.

First off, long time no blog!

This is the last post I'm putting on the Blogspot site. In the spirit of eating my own dogfood, I've now set up a self-hosted blog on my HDC. I'm sure it won't be hard for all half-dozen of you readers to move over. I'll have new content over there, at the very least the Hello, World post, a catchup post, and a HDC 3.0 post to match the ones for 1.0 and 2.0.

Today we updated OmniOS to its next stable release: r151016. You can click the link to see its release notes, and you may notice a brief mention the illumos-tools package.

I want to see more people working on illumos. A way to help that is to get people started on actually BUILDING illumos more quickly. To that end, r151016 contains everything to bring up an illumos development environment. You can develop small on it, but this post is going to discuss how we make building all of illumos-gate from scratch easier. (I plan on updating the older post on small/focused compilation after ws(1) and bldenv(1) effectively merge into one tool.)

The first thing you want to do is install OmniOS. The latest release media can be found here, on the Installation page.

After installation, your system is a blank slate. You'll need to set a root password, create a non-root user, and finally add networking parameters. The OmniOS wiki's General Administration Guide covers how to do this.

I've added a new building illumos page to the OmniOS wiki that should detail how straightforward the process is. You should be able to kick off a full nightly(1ONBLD) build quickly enough. If you don't want to edit one of the omnios-illumos-* samples in /opt/onbld/env, just make sure you have a $USER/ws directory, clone one of illumos-gate or illumos-omnios into $USER/ws/testws and use one of the template /opt/onbld/env/omnios-illumos-* files corresponding to illumos-gate or illumos-omnios. For example:

omnios(~)[0]% mkdir ws
omnios(~)[0]% cd ws
omnios(~/ws)[0]% git clone
https://github.com/illumos/illumos-gate/ testws

omnios(~/ws)[0]% /bin/time /opt/onbld/bin/nightly
/opt/onbld/env/omnios-illumos-gate

This is STILL only a test, but with an update. The big question remains: How quickly I can bring over my old Google owned blog entries?

(BTW Jeff, I could get used to this LaTeX-like syntax…)

You’ll start to see stuff trickle in. I’m using the hopefully-pushed-back-soon enhancements to Blahg that allows simple raw HTML for entries. I’ve done some crazy thing to extract, and maybe the hello-world post here will explain that.