Running Tribblix on Digital Ocean The Trouble with Tribbles...

A relatively recent feature offered by Digital Ocean is the ability to deploy your own custom image. So, can I deploy a Tribblix image to Digital Ocean?

Short answer: Yes!

For the process, read on.

I'm using Bhyve to create the image. This is a slight variation on the Installing Tribblix in Bhyve on Tribblix procedure.

The basic process looks like this:

  • Boot Tribblix in Bhyve
  • Install it
  • Tweak the installed image for Digital Ocea
  • Copy the ZFS volume to Digital Ocean

The first variation on the previous install is that I make the ZFS volume a bit smaller. We can resize it when it's deployed, so we don't need to make it too big. A 4G volume is fine; any smaller and there won't be any space for our swap partition. It doesn't actually matter too much, as we compress the image anyway

zap create-zone -t bhyve -z bhyve1 \
-x 192.168.0.236  \
-I /var/tmp/tribblix-0m24.1.iso \
-V 4G

Install as before,

./live_install.sh -G c1t0d0

remove the cdrom and reboot as before.

We now need to set up networking. We need to do this on a temporary basis, as we don't want any of these network configurations to carry over to the installed image. So I temporarily disable nwam, and manually bring up a working network. When the image boots on Digital Ocean, all this configuration will have been forgotten and it will bring up nwam as normal.

So run the following in the newly installed guest:

svcadm disable -t network/physical:nwam
ifconfig vioif0 plumb
ifconfig vioif0 up
ifconfig vioif0 inet 192.168.0.236/24
route add net default 192.168.0.1
echo "nameserver    8.8.8.8"  > /etc/resolv.conf

Now we need to tweak the image. At some later point this will all be integrated into the installer so it will just work. But for now, we'll start by applying any updates:

zap refresh
zap update-overlay -a

Now for the little tweak. I'm going to add a metadata service that will run at boot on Digital Ocean and do the sort of things that cloud-init would do. Fortunately, there's one for illumos, and it's packaged for Tribblix, so install it:

zap install TRIBmetadata-agent

If you look with svcs you'll see that it's offline. That's not a problem (it's because we've got a temporary manual network setup) - once we boot properly on Digital Ocean we'll have nwam running and the metadata service will run just fine.

We can tidy up and save a bit of space:

zap clean-cache -a

and shut down the zone (and the newly installed instance of Tribblix):

zoneadm -z bhyve1 halt

What we want is a raw image. So all we do is dd the ZFS volume to a file.

dd if=/dev/zvol/rdsk/rpool/bhyve1_bhvol0 \
of=/var/tmp/tribblix-do-m24.1.img bs=1048576

That's a 4G file, the size of the volume. As it's stored on ZFS, and ZFS compression is on, it will actually consume a lot less space as the image is mostly empty. But what we don't want to do is upload 4G of empty space. So we can compress it:

gzip -9 /var/tmp/tribblix-do-m24.1.img

(it ends up as about 300M), or you could use bzip2, I think.

There are two options when you upload the image to Digital Ocean - you can either do a direct upload through the browser, or you can give it a URL where the image can be found and get Digital Ocean to pull it from there. I found it much easier to scp the image up to an existing webserver and get Digital Ocean to grab it, as I don't trust browsers to behave well.

Log in to Digital Ocean.

Select 'Images' from the left hand menu
Tab to 'Custom Images'
Import via URL
[Insert the URL where you've stashed the file]

You then get a dialog

Name - tribblix-do-m24.1.img.gz
Distribution - unknown (it's not on the list)
Region - London

Obviously, for me, London is conveniently local. You get warned there will be a charge, and it shows as pending for a few minutes.

Then it pops up 'your image is ready to use'.

To the right of the image in the list is a 'More' dropdown menu from which you can start a droplet. So off we go.

It selects a pretty hefty instance type by default. Reset that to the very cheapest. Choose the ssh key you're going to use, pick a useful (and shorter) hostname, and Create Droplet.

Don't bother with block storage. That's exposed by virtio-scsi, which illumos doesn't yet support.

It'll take a few moments to create the droplet, and once it's ready you'll see the IP address.

At this point, if everything has worked, you should be able to ssh in as root with the ssh key you chose. Chances are that doesn't quite work yet. If it doesn't, simply ssh in as jack, from where you can su to root.

(Remember, jack is on the live ISO, and we haven't deleted it. During experiments I tend not to, to give myself a way in if things don't work right. A proper production image would have the jack user removed and password login for root disabled.)

If the metadata service hasn't run properly (it will resize the ZFS pool, change the hostname, and add the correct key to ssh in as root) then you can restart the metadata service:

svcadm restart metadata

Now ssh to root works, the hostname is set, and the zfs pool has been expanded to the full size.

None of the above is excessively specific to Tribblix, the same general process will work for any of the illumos distributions. (Although you may have to build and install the metadata service yourself.)

Installing Tribblix in Bhyve on Tribblix The Trouble with Tribbles...

One of the big new features recently added to illumos is the Bhyve hypervisor. Rather that the shared-kernel application-level virtualization offered by zones, think of something like VirtualBox, KVM, or Qemu.

One of the things that I am using Bhyve for is to test the Tribblix ISO images and the installer. This allowed me to shrink the installer footprint slightly in recent releases (and showed that one of the tricks I tried wasn't going to work).

Using Bhyve requires a fairly modern system (my own is a little old, and has an Intel Core i7, and works fine). The instructions here also need you to be running current Tribblix, m24.1 or newer.

So, how do I install Tribblix in Bhyve on Tribblix? Most of the following needs to be done as root.

First, make sure that you have the software installed:

zap install-overlay bhyve

I'm doing this on my desktop, so I'm using VNC to connect.

zap install-overlay retro-desktop

You need the current ISO. Again, this has to be m24.1 or newer. Correct the path below to match wherever you've downloaded it to.

While you don't actually have to run bhyve in a zone, that's the standard and most convenient way. So, I use zap to create a zone:

zap create-zone -t bhyve -z bhyve1 \
-x 192.168.0.236  \
-I /var/tmp/tribblix-0m24.1.iso \
-V 8G

Let's run through these. The -t flag says to create a bhyve zone, -z gives it a name, -x gives it an (exclusive) IP address, -I tells it where the ISO image is, and -V tells it to create a ZFS volume of the given size. The other argument that may be of interest is -m, which allows you to set the amount of memory allocated; it defaults to 1024M which is fine for Tribblix.

Then you need to be able to connect to the instance. I'm going to use VNC to connect to the console. We use socat to wire up the bhyve socket in the zone to a network port in the global zone that we can connect to.

socat TCP-LISTEN:5905,reuseaddr,fork UNIX-CONNECT:/export/zones/bhyve1/root/tmp/vm.vnc

You may need to modify the zone name in the path, and you can choose the TCP port (for VNC, it's offset by 5900, so the 5905 is :5 in VNC-speak).



 

 

 

Then, as yourself, you can start a VNC client

vncviewer :5

If that can't connect, then one possibility is that bhyve can't start because it doesn't have enough memory. One little trick to make sure there's enough headroom is to create a file in /tmp and then delete it immediately.

mkfile 1200m /tmp/1200m ; rm /tmp/1200m

You can then log in to the live environment. And do an install like so:

./live_install.sh -G c1t0d0


 

 

 

 

and the install will carry on

 




 

 

 

 

If you want to run this, you have to avoid booting off the CD. The way to do this is to remove the cdrom from the zone specification before you reboot the guest, like so:

zonecfg -z bhyve1 remove attr name=cdrom
zonecfg -z bhyve1 remove fs dir=/var/tmp/tribblix-0m24.1.iso special=/var/tmp/tribblix-0m24.1.iso
zoneadm -z bhyve1 reboot

Obviously, adjusted for the zone name and the ISO file name. I need to teach zap how to do this properly. And then start socat again (if you stopped it) and reconnect using VNC.

You may need to fiddle the networking in the installed guest. The exclusive-ip setting here will force the guest to use the given address. That's unlikely to work terribly well, as the chances of allocating the same address that DHCP would hand out are pretty remote. So to get it on the network you may have to fiddle in the guest by hand.

svcadm disable network/physical:nwam
echo "192.168.0.236/24" > /etc/hostname.vioif0
echo "192.168.0.1" > /etc/defaultrouter
svcadm enable network/physical:default

and populate /etc/resolv.conf with something relevant so that dns works

echo "nameserver 8.8.8.8" > /etc/resolv.conf



 

 

 

 

 

 

Once you're done, you can destroy the zone with

zap destroy-zone -z bhyve1

which will also destroy the ZFS volume it was using.

While the above refers to installing Tribblix inside Bhyve, the same general procedure ought to work to install other illumos distributions.




Compensation as a reflection of values The Observation Deck

Compensation: the word alone is enough to trigger a fight-or-flight reaction in many. But we in technology have the good fortune of being in a well-compensated domain, so why does this issue induce such anxiety when our basic needs are clearly covered? If it needs to be said, it’s because compensation isn’t merely about the currency we redeem in exchange for our labors, but rather it is a proxy for how we are valued in a larger organization. This, in turn, brings us to our largest possible questions for ourselves, around things like meaning and self-worth.

So when we started Oxide — as in any new endeavor — compensation was an issue we had to deal with directly. First, there was the thorny issue of how we founders would compensate ourselves. Then, of course, came the team we wished to hire: hybrid local and remote, largely experienced to start (on account of Oxide’s outrageously ambitious mission), and coming from a diverse set of backgrounds and experiences. How would we pay people in different geographies? How could we responsibly recruit experienced folks, many of whom have families and other financial obligations that can’t be addressed with stock options? How could we avoid bringing people’s compensation history — often a reflection of race, gender, class, and other factors rather than capability — with them?

We decided to do something outlandishly simple: take the salary that Steve, Jess, and I were going to pay ourselves, and pay that to everyone. The three of us live in the San Francisco Bay Area, and Steve and I each have three kids; we knew that the dollar figure that would allow us to live without financial distress — which we put at $175,000 a year — would be at least universally adequate for the team we wanted to build. And we mean everyone literally: as of this writing we have 23 employees, and that’s what we all make.

Now, because compensation is the hottest of all hot buttons, it can be fairly expected that many people will have a reaction to this. Assuming you’ve made it to this sentence it means you are not already lighting us up in your local comments section (thank you!), and I want to promise in return that we know some likely objections, and we’ll address those. But before we do, we want to talk about the benefits of transparent uniform compensation, because they are, in a word, profound.

Broadly, our compensation model embodies our mission, principles, and values. First and foremost, we believe that our compensation model reflects our principles of honesty, integrity, and decency. To flip it around: sadly, we have seen extant comp structures in the industry become breeding grounds for dishonesty, deceit, and indecency. Beyond our principles, our comp model is a tangible expression of several of our values in particular:

  • It has set the tone with respect to teamwork. In my experience, the need to “quantify” one’s performance in exchange for justifying changes to individual compensation are at the root of much of what’s wrong in the tech industry. Instead of incentivizing people to achieve together as a team, they are incentivized to advance themselves — usually with sophisticated-sounding jargon like OKRs or MBOs, or perhaps reasonable-sounding (but ultimately misguided) mantras like “measure everything.” Even at their very best, these individual incentives represent a drag on a team, as their infrequent calibration can prevent a team from a necessary change in its direction. And at worst, they leave individuals perversely incentivized and operating in direct opposition to the team’s best interest. When comp is taken out of the picture, everyone can just focus on what we need to focus on: getting this outlandish thing built, and loving and serving the customers who are taking a chance on it.

  • It is an expression of our empathy. Our approach to compensation reflects our belief in treating other people the way that we ourselves want to be treated. There are several different dimensions for this, but one is particularly visceral: because we have not talked about this publicly, candidates who have applied to Oxide have done so assuming that we have a traditional comp model, and have braced themselves for the combat of a salary negotiation. But we have spoken about it relatively upfront with candidates (before they talk to the team, for example), and (as the one who has often had this discussion) the relief is often palpable. As one recent candidate phrased it to me: “if I had known about this earlier, I wouldn’t have wasted time stressing out about it!”

  • It is (obviously?) proof-positive of our transparency. Transparency is essential for building trust, itself one of the most important elements of doing something bold together. One of the interesting pieces of advice we got early on from someone who has had outsized, repeated success: modulo private personnel meetings, make sure that every meeting is open to everyone. For those accustomed to more opaque environments, our level of transparency can be refreshing: for example, new Oxide employees have been pleasantly surprised that we always go through our board decks with everyone — but we can’t imagine doing it any other way. Transparent compensation takes this to an unusual (but not unprecedented) extreme, and we have found it to underscore how seriously we take transparency in general.

  • It has allowed whole new levels of candor. When everyone can talk about their salary, other things become easier to discuss directly. This candor is in all directions; without comp to worry about, we can all be candid with respect to our own struggles — which in turn allows us to address them directly. And we can be candid too when giving public positive feedback; we don’t need to be afraid that by calling attention to someone’s progress, someone else will feel shorted.

These are (some of!) the overwhelming positives; what about those objections?

  • Some will say that this salary is too low. While cash compensation gets exaggerated all of the time, it’s unquestionable that salaries in our privileged domain have gotten much higher than our $175,000 (and indeed, many at Oxide have taken a cut in pay to work here). But it’s also true that $175,000 per year puts us each in the top 5% of US individual earners — and it certainly puts a roof over our families’ heads and food in their bellies. Put more viscerally: this is enough to not fret when your kids toss the organic raspberries into the shopping cart — or when they devour them before you’ve managed to get the grocery bags out of the car! And speaking of those families: nothing is more anxiety-producing than having a healthcare issue compounded by financial distress due to inadequate insurance; Oxide not only offers the best healthcare plans we could find, but we also pay 100% of monthly premiums — a significant benefit for those with dependents.

  • Some will say that we should be paying people differently based on different geographical locations. I know there are thoughtful people who pay folks differently based on their zip code, but (respectfully), we disagree with this approach. Companies spin this by explaining they are merely paying people based on their cost of living, but this is absurd: do we increase someone’s salary when their spouse loses their job or when their kid goes to college? Do we slash it when they inherit money from their deceased parent or move in with someone? The answer to all of these is no, of course not: we pay people based on their work, not their costs. The truth is that companies pay people less in other geographies for a simple reason: because they can. We at Oxide just don’t agree with this; we pay people the same regardless of where they pick up their mail.

  • Some will say that this doesn’t scale. This is, at some level, surely correct: it’s hard to envision a multi-thousand employee Oxide where everyone makes the same salary — but it has also been (rightly) said that startups should do things that don’t scale. And while it seems true that the uniformity won’t necessarily scale, we believe that the values behind it very much will!

  • Some will say that this makes us unlikely to hire folks just starting out in their career. There is truth to this too, but the nature of our problem at Oxide (namely, technically very broad and very deep), the size of our team (very small), and the stage of our company (still pretty early!) already means that engineers at the earliest stages of their career are unlikely to be a fit for us right now. That said, we don’t think this is impossible; and if we felt that we had someone much earlier in their career who was a fit — that is, if we saw them contributing to the company as much as anyone else — why wouldn’t we reflect that by paying them the same as everyone else?

  • Some will say that this narrows the kind of roles that we can hire for. In particular, different roles can have very different comp models (sales often has a significant commission component in exchange for a lower base, for example). There is truth to this too — but for the moment we’re going to put this in the “but-this-can’t-scale” bucket.

  • Some will say that this doesn’t offer a career ladder. Uniform compensation causes us to ask some deeper questions: namely, what is a career ladder, anyway? To me, the true objective for all of us should be to always be taking on new challenges — to be unafraid to learn and develop. I have found traditional ladders to not serve these ends particularly well, because they focus us on competition rather than collaboration. By eliminating the rung of compensation, we can put the focus on career development where it belongs: on supporting one another in our self-improvement, and working together to do things that are beyond any one of us.

  • Some will say that we should be talking about equity, not cash compensation. While it’s true that startup equity is important, it’s also true that startup equity doesn’t pay the orthodontist’s bill or get the basement repainted. We believe that every employee should have equity to give them a stake in the company’s future (and that an outsized return for investors should also be an outsized return for employees), but we also believe that the presence of equity can’t be used as an excuse for unsustainably low cash compensation. As for how equity is determined, it really deserves its own in-depth treatment, but in short, equity compensates for risk — and in a startup, risk reduces over time: the first employee takes much more risk than the hundredth.

Of these objections, several are of the ilk that this cannot endure at arbitrary scale. This may be true — our compensation may well not be uniform in perpetuity — but we believe wholeheartedly that our values will endure. So if and when the uniformity of our compensation needs to change, we fully expect that it will remain transparent — and that we as a team will discuss it candidly and empathetically. In this regard, we take inspiration from companies that have pioneered transparent compensation. It is very interesting to, for example, look at how Buffer’s compensation has changed over the years. Their approach is different from ours in the specifics, but they are a kindred spirit with respect to underlying values — and their success with transparent compensation gives us confidence that, whatever changes must come with time, we will be able to accommodate them without sacrificing what is important to us!

Finally, a modest correction. The $175,000 isn’t quite true — or at least not anymore. I had forgotten that when we did our initial planning, we had budgeted modest comp increases after the first year, so it turns out, we all got a raise to $180,250 in December! I didn’t know it was coming (and nor did anyone else); Steve just announced it in the All Hands: no three-hundred-and-sixty degree reviews, no stack ranking, no OKRs, no skip-levels, no numerical grades — just a few more organic raspberries in everyone’s shopping basket. Never has a change in compensation felt so universally positive!

OmniOS Community Edition r151030cv, r151034av, r151036v OmniOS Community Edition

OmniOS weekly releases for w/c 29th of March 2021 are now available.

Security Fixes for all releases

For further details, please see https://omniosce.org/releasenotes


Any problems or questions, please get in touch.

Using the 'zadm' utility to create a bhyve VM from an image OmniOS Community Edition

Creating an OmniOS bhyve VM is quick and easy with the provided images. This example shows creating a VM running the rolling bloody release on r151034.


zadm is open source and hosted on Github. Feedback and pull requests are welcome.

Any questions, please get in touch!

OmniOS Community Edition r151030cu, r151034au, r151036u OmniOS Community Edition

OmniOS weekly releases for w/c 22th of March 2021 are now available.

Security Fixes for all releases

Additionally for r151034 and r151036

For further details, please see https://omniosce.org/releasenotes


Any problems or questions, please get in touch.

Setting up a virtual machine in Azure OmniOS Community Edition

With the release of OmniOS r151038, a hard disk image suitable for uploading to Microsoft Azure is available. This image is pre-configured to run in Azure and can be provisioned directly from the command line, using the Azure CLI, as shown below.

This article assumes you already have the Azure CLI installed and working and that you have authenticated to Azure using the az login command.

The image is provided as a compressed virtual hard disk (vhd) file on the OmniOS download site. In the following example, the image related to the rolling “bloody” release is used.

First, download the image to the local machine, and uncompress it:

        $ cd /tmp
$ curl -O https://downloads.omnios.org/media/bloody/azure-151037.vhd.zst
$ zstd -d azure-151037.vhd.zst

Create a resource group to hold the rest of the resources. If you already have a resource group configured in Azure, you can skip this step. In this example, the resource group is called web and it is being created in the ukwest region.

        $ az group create --name web --location ukwest

Create a storage account to hold the disk image - again, if you already have a storage account configured, you can skip this step.

        $ az storage account create --resource-group web --location ukwest \
    --name webstorage --kind Storage --sku Standard_LRS

Obtain the key needed to access this new storage account. This can also be obtained using the Azure web portal. Setting the two environment variables shown makes the following commands shorter as these parameters can then be omitted.

        $ key=`az storage account keys list \
    --resource-group web --account-name webstorage \
    | jq -r 'first | .value'`

$ export AZURE_STORAGE_ACCOUNT=webstorage
$ export AZURE_STORAGE_KEY=$key

Create a container within the storage account. This is where the disk image will be uploaded:

        $ az storage container create --name omniosdisk

Now the image can be uploaded, this will take a few minutes and you will see a progress bar.

        $ az storage blob upload --container-name omniosdisk --type page \
    --file /tmp/azure-151037.vhd --name azure-151037.vhd

Once that is uploaded, it is time to create the virtual machine. In the following example, an admin user is also being created with the username boho. This account will be granted permission to use sudo and will be configured with the provided public SSH key in its authorized_keys file.

        $ az vm create --resource-group web --location ukwest \
    --name web01 --storage-account webstorage --os-type linux \
    --admin-username boho --ssh-key-value ~/.ssh/id_rsa.pub \
    --size Standard_B1s --boot-diagnostics-storage=webstorage \
    --use-unmanaged-disk --image \
    https://webstorage.blob.core.windows.net/omniosdisk/azure-151037.vhd

Once this is complete, a JSON object will be displayed which includes the assigned public IP address for the VM:

        {
  "location": "ukwest",
  "powerState": "VM running",
  "publicIpAddress": "51.140.2xx.1xx"
}

If all went well, you should be able to SSH straight in using the provided admin username and SSH key:

        $ ssh boho@51.140.2xx.1xx
The authenticity of host '51.140.2xx.1xx (51.140.2xx.1xx)' can't be established.
ECDSA key fingerprint is SHA256:...
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
OmniOS r151037  omnios-master-0401360920        March 2021
boho@web01:~$
boho@web01:~$
boho@web01:~$ sudo -s
root@web01:/home/boho#

You can then proceed to add additional data disks to the machine as required using the standard Azure CLI or web interface.

A Four Axis Model of Work Nahum Shalman

In mid December I received (a generous) two month's notice that I would need to find a new job. I had the luxury of being able to reach out to people I'd be interested in working with while wrapping up loose ends at work for one month, and the further luxury of being paid for another month while searching for a new job. With that much time to think about what I want to do next I came up with a framework for how to approach my search, and how to compare different opportunities should I find myself blessed with choices. What I came up with I've taken to calling a four axis model.

Important Caveats

I don't presume to be able to offer advice. I've been incredibly fortunate in so many ways in my life that I find myself standing on top of many different layers of privilege. These are things for which I can take no credit, but have made my life dramatically easier than for the average person. There's an important conversation to be had about how to make what has been possible for me possible for more people, but that is not what this post is about.

The Four Axes

These are my one-word shorthand:

  1. Mission
  2. Technology
  3. Culture
  4. Compensation

For most readers, the use of the word axis probably hints sufficiently, but in case someone else stumbles across this post, the idea here is multidimensional optimization. If all I cared about was money, I could optimize for that 4th axis by interviewing with a bunch of different companies, comparing the compensation, and simply choose the job that pays the most. Since I care about more than just money, I have to be willing to make trade-offs between these different dimensions and try to maximize the overall value that a job offer presents.

Mission

What is the company trying to accomplish in the world? What are they actually accomplishing? How does that mission connect to my values and add meaning to my day-to-day work? What story (real or imagined) will I tell myself each day about why I work where I work? Will ongoing reality re-enforce that story, or undermine it?

Technology

This is where it becomes very clear that my 4 axes are shorthand for more than 4 axes. Technology is shorthand for:

  1. Do they use technologies with which I am already familiar? Will I be able to ramp up in a reasonable amount of time and be productive?
  2. Will I learn and grow as a technologist?
  3. Will I be able to contribute to open source projects?
  4. Do they get bonus points using any of my favorite "nice to have" technologies? (For me, "nice to have" includes things like running illumos instead of Linux, using ZFS, using rust, etc.)

Culture

What are the the company's professed values? What values do their actions reveal? How do people interact within the team, across teams, across larger organizations within the company, etc. Does the company care about diversity? Is the workforce already diverse including if not particularly in upper management? Is the workplace (physical and/or digital) inclusive?

Compensation

Compensation is the value a company provides back to you in exchange for your labor. Some of it is things like an hourly rate or a base salary. Some of it is some form of equity in the company. Some of it is agreed upon explicitly at hiring time, while some of it could be a bonus based on your performance, or the company's performance, or both. Some of it could be contributing some or all of the cost of various things like insurance. Some companies offer fancy offices with catered food and snacks. Some companies will contribute to a retirement account for you. Heck, in some professions they still have pensions! Some companies will match your charitable donations. All of these things are forms of compensation, and you have to figure out how to weight each of the components of the compensation you're being offered.
I'm still relatively inexperienced with how lots of different companies try to balance their compensation packages, and I think I could have done more research during my job search. In the end, I consulted with a financial advisor to help me understand how to value base salary, public company equity, private company equity, etc. I do think it's very important to ensure that your base salary covers your critical expenses, but beyond that, I'm probably the wrong person to provide advice on compensation.

How I use this framework

I will once again repeat the caveat that this post is not advice. I've linked some other resources at the end of this post for further reading (and if you have a favorite resource not listed, let me know about it and I'll add it!)

As a filter

Again, fully recognizing how fortunate I am that simply marking myself as "open to work" on LinkedIn provided me with a steady stream of inquiries from recruiters, this framework allowed me to, for example, politely decline some companies in industries that are just not of interest to me (even though by reputation they pay very very well.)
Similarly, I've come across companies which have pages on the their website showing the leadership team and it's ALL white men.... They might be very nice people, but chances are their workforce isn't particularly diverse.

As a tool for answering questions:

Question: "What are you looking for in your next job?"
Answer: Well, here's this framework I have in my head....

Question: "Why do you want to work at X?"
Answer: Well, here's this framework I have in my head, and here's how I think X maps onto it so far, but also, I'm still wondering about this aspect. Can you tell me more?

As a tool for asking questions:

This is where I don't know if what I'm doing is a good idea, but here it is:
In every interview where I get the chance to ask questions (most of them) I quickly run through a very brief explanation of "I'm looking to optimize across Mission, Technology, and Culture and here's what I mean by those.... Please tell me about your experience with this company along those lines."
The great thing about asking the same question of everyone is that the answers across individuals within the same company are revealing in their similarities and differences and comparing the answers across companies helps with decision making.

As a tool for making a choice:

I was fortunate to get more than one offer at roughly the same time. Having criteria for comparing offers helped me with deciding on which offer to accept.

Further Reading

Tweaking the Tribblix ISO The Trouble with Tribbles...

For the latest release, I've made a couple of tweaks to the Tribblix ISO image.

The first is that it's now a hybrid ISO. So you should be able to write it (simply dd it) to a USB stick and use that to boot. I've never managed to both create and test a usb image, so this should be a win.

The hybrid ISO also boots under bhyve, which the old ISO never did. So that's a second win.

I didn't do any of the work here, simply taking the steps from the OmniOSce kayak build as suggested by tsoome. But the script I use is here.

The second tweak is that the man pages aren't present in their normal form on the live image. Instead, they're shipped as a compressed tarball. This saves a significant amount of space - the manuals are simple text and compress (with bzip2) really well. This doesn't make a great deal of difference to the size of the ISO, but it reduces the size of the ramdisk we boot from by 10%. And that's important because an overlarge ramdisk chews up too much memory if you're trying to install on a small (1GB or less) system.

Of course, the installer now knows to unpack the man pages during installation. This meant I had to add bzip2 to the live image, but that's tiny compared to the space saving you get in return.

I was hoping to use pbzip2, so that if you're on a multiprocessor machine the uncompression is sped up. That turns out not to be a good idea. The problem is that pbzip2 is written in C++, so needs libstdc++. I don't have that, or any of the gcc runtime, on the live image. Adding it would undo all the space savings I was trying to get, but I've not noticed any slowness.

This can be adjusted a bit. The downside is that you have no access to the manual during installation; I could easily have a small subset of the man pages available, and compress away the majority.

The two features came together, thanks to bhyve. Having an image I can boot and install on bhyve is a huge advantage, as it allows me to test the ISO and the installer much more easily. It was that testing that demonstrated that pbzip2 wouldn't work.

All Your Base Are Belong to 20-Somethings, and Solaris 9 Kebe Says: Dan McD's blog

Two Decades Ago…

Someone pointed out recently that the famous Internet meme “All your base are belong to us” turned 20 this week. Boy do I feel old. I was still in California, but Wendy and I were plotting our move to Massachusetts.

In AD 2001, S9 Was Beginning

OF COURSE I watched the video back then. The original Shockwave/Flash version on a site that no longer exists. I used my then-prototype Sun Blade 1000 to watch it, on Netscape, on in-development Solaris 9.

I found a bug in the audio driver by watching it. Luckily for me, portions of the Sun bug database were archived and available for your browsing pleasure. Behold bug 4451857. I reported it, and all of the text there is younger me.

The analysis and solution are not in this version of the bug report, which is a shame, because the maintainer (one Brian Botton) was quite responsive, and appreciated the MDB output. He fixed the bug by moving around a not-shown-there am_exit_task() call.

Another thing missing from the bug report is my “Public Summary” which I thought would tie things up nicely. I now present it here:

In A.D. 2001
S9 was beginning.
Brian: What Happen?
Dan: Someone set up us the livelock
Dan: We get signal
Brian: What!
Dan: MDB screen turn on.
Brian: It’s YOU!
4451857: How are you gentleman?
4451857: All your cv_wait() are belong to us.
4451857: You are on the way to livelock.
Brian: What you say?
4451857: You have no chance to kill -9 make your time.
4451857: HA HA HA HA…
Brian: Take off every am_exit_task().
Dan: You know what you doing
Brian: Move am_exit_task().
Brian: For great bugfix!

OmniOS Community Edition r151030cp, r151034ap, r151036p OmniOS Community Edition

OmniOS weekly releases for w/c 15th of February 2021 are now available.

All supported OmniOS releases have been updated with the following changes:

Security Fixes in all releases

  • OpenSSL updated to 1.1.1j, fixing CVE-2021-23840, CVE-2021-23841.

  • The legacy OpenSSL 1.0.2 has also been patched to mitigate the above CVEs.

  • All shipped Python versions have been updated to address CVE-2021-3177.

Other Changes in all releases

  • Adding a second disk to an Azure instance caused a kernel panic.

Additional changes in r151030cp

  • A memory leak in the SMB client has been addressed.

  • The ZFS I/O pipeline is now able to use the pageout reserve memory pool in order to flush pages to disk under low memory conditions.

  • The hardware database has been updated.

  • Timezone data has been updated.

Additional changes in r151036p

  • The ZFS I/O pipeline is now able to use the pageout reserve memory pool in order to flush pages to disk under low memory conditions.

For further details, please see https://omniosce.org/releasenotes


Any problems or questions, please get in touch.

OmniOS Community Edition r151030co, r151034ao, r151036o OmniOS Community Edition

OmniOS weekly releases for w/c 8th of February 2021 are now available.

All supported OmniOS releases have been updated with the following changes:

Security Fixes

For further details, please see https://omniosce.org/releasenotes


Any problems or questions, please get in touch.

OmniOS Community Edition r151030cm, r151034am, r151036m OmniOS Community Edition

OmniOS weekly releases for w/c 25th of January 2021 are now available.

All supported OmniOS releases have been updated with the following changes:

Features

  • Support for secure RPC for the Netlogon protocol between OmniOS systems and Microsoft Active Directory servers. This is required to fully mitigate CVE-2020-1472, and will shortly be enforced by Windows domain controllers.

    We’re grateful to Matt Barden of Tintri for contributing this feature to illumos.

Security Fixes

  • It was possible for an unprivileged user, including one in a zone, to cause the system to panic.

  • socat updated to fix a potential heap based buffer overflow.

  • sudo updated to fix CVE-2021-3156

Other Changes

  • On r151036 only, OpenJDK 11 now bundles several core fonts and has been updated to 11.0.10+9.

  • OpenJDK 8 has been updated to 1.8.282-b08

  • gcc has been updated so that the -z assert-deflib linker option works correctly for 64-bit objects.

  • ZFS list -t bookmark was not working for zvols.

  • Add support for the IA32_FEATURE_CONTROL MSR in bhyve. This improves support for some guest operating systems.

  • pkg apply-hot-fix could produce error messages during operation.

For further details, please see https://omniosce.org/releasenotes


Any problems or questions, please get in touch.

OmniOS Community Edition r151030cf, r151034af, r151036f OmniOS Community Edition

OmniOS weekly releases for w/c 14th of December 2020 are now available.

All supported OmniOS releases have been updated with the following changes:

Security Fixes

Bug Fixes

  • MacOS Big Sur clients would experience read hangs when accessing OmniOS SMB shares.

  • The pkg apply-hot-fix command, in conjunction with the creation of a new boot environment, would sometimes leave extra origins configured in non-global zones. This caused problems for subsequent package updates.

  • Compiling with gcc -pg did not produce a working profiling binary.

Other Changes

  • The Intel CPU microcode update for some Core (Gen. 11) Mobile processors has been removed as it was reported to cause problems on some platforms.

Additionally in both the r151034 and r151036 releases:

  • In rare cases, a system could crash shortly after boot while maintaining ZFS user quota information (particularly if a zfs recv was running at the same time).

and in r151036 only:

  • The pciutils package tools were unable to enumerate PCI devices.

  • A file’s modification time would change twice when the file was modified from an SMB client. This caused problems for some Windows utilities.

  • Restore the NDMP ZFS backup and restore method which was inadvertently broken in the r151036 release.

For further details, please see https://omniosce.org/releasenotes


Any problems or questions, please get in touch.

OmniOS Community Edition r151036 Stable OmniOS Community Edition

The OmniOS Community Edition Association is proud to announce the general availability of OmniOS - r151036.

OmniOS is published according to a 6-month release cycle; r151036 stable will be supported for a year. The old stable r151032 release is now end-of-life and will receive no further updates. See the release schedule for further details.

Release Notes and Upgrade Instructions

For full release notes including upgrade instructions; review the release notes and find upgrade instructions at omnios.org/upgrade

If you upgrade from r151030 and want to see all new features added since then, make sure to also read the release notes for the intervening r151032 and r151034 releases.

OmniOSce Newsletter

Since the start of OmniOS Community Edition project, we have predominantly announced our new releases via twitter. We are now also offering a newsletter with announcements of updates, bug fixes and new releases. You can subscribe here.

Commercial Support

Have you ever wondered how OmniOS development gets financed? You may have noticed that there is no big company bankrolling it all. The way we keep afloat is by the companies who rely on OmniOS powered servers taking out support contracts for their hardware. How about you? Visit omnios.org/support for more details and to generate a quote. If you aren’t in a position to take a support contract, please consider becoming an OmniOS patron to help secure its future - omnios.org/patron.

About OmniOS Community Edition Association - this Swiss Association is responsible for the ongoing development and maintenance of OmniOS, having been established in Summer 2017 after OmniTI announced their withdrawal from the project.

OmniOSce Association Aarweg 17, 4600 Olten, Switzerland

Running a Zabbix server in an OmniOS zone OmniOS Community Edition

This guide shows how to get Zabbix up and running within a zone on an OmniOS system. Zabbix is an open-source monitoring system and is available from the OmniOS Extra repository.

Zone setup

I’m going to use the lightweight sparse zone brand for this so start by making sure that it is installed:

        % pfexec pkg install brand/sparse
No updates necessary for this image.

If the brand is not already installed, then there will be more output from the above command.

Create a new sparse zone called zabbix. My preference is to configure the IP stack within the zone configuration which results in it being automatically applied to the zone when it’s booted, and enables additional protection against settings such as the IP address from being changed from within the zone. Note that, like all zones, the zone path must be a direct descendant of a ZFS dataset. On my system, /zones is the mount point for such a dataset.

        % pfexec zonecfg -z zabbix
zabbix: No such zone configured
Use 'create' to begin configuring a new zone.
zonecfg:zabbix> create -t sparse
zonecfg:zabbix> set zonepath=/zones/zabbix
zonecfg:zabbix> add net
zonecfg:zabbix:net> set physical=zabbix0
zonecfg:zabbix:net> set global-nic=igb0
zonecfg:zabbix:net> set allowed-address=172.27.10.35/24
zonecfg:zabbix:net> set defrouter=172.27.10.254
zonecfg:zabbix:net> end
zonecfg:zabbix> add attr
zonecfg:zabbix:attr> set name=resolvers; set type=string; set value=1.1.1.1
zonecfg:zabbix:attr> end
zonecfg:zabbix> add attr
zonecfg:zabbix:attr> set name=dns-domain; set type=string; set value=omnios.org
zonecfg:zabbix:attr> end
zonecfg:zabbix:attr> verify; commit; exit

By default, the zone’s boot environment will encompass all of the files and directories within. For an application such as Zabbix, it’s important to create a dedicated area to hold files such as the underlying database which should be consistent across different boot environments.

My system has a ZFS pool called data so I’m going to create a new dataset under that and delegate it to the zone. I’m also going to change the mount point for the dataset to /data so that it appears there within the zone.

        % pfexec zfs create data/zabbix
% pfexec zonecfg -z zabbix 'add dataset; set name=data/zabbix; end'

% pfexec zfs umount data/zabbix
% pfexec zfs set mountpoint=/data data/zabbix
% pfexec zfs set zoned=on data/zabbix

Now it’s time to install the zone. Being a sparse zone, this will be pretty quick - only around 5MiB of files are actually installed.

        % pfexec zoneadm -z zabbix install
A ZFS file system has been created for this zone.

       Image: Preparing at /zones/zabbix/root.
Sanity Check: Looking for 'entire' incorporation.
   Publisher: Using omnios (https://pkg.omnios.org/r151034/core/).
   Publisher: Using extra.omnios (https://pkg.omnios.org/r151034/extra/).
       Cache: Using /var/pkg/publisher.
  Installing: Packages (output follows)
Packages to install: 203
Mediators to change:   5
 Services to change:   6

DOWNLOAD                                PKGS         FILES    XFER (MB)   SPEED
Completed                            203/203     1485/1485      4.9/4.9      --

PHASE                                          ITEMS
Installing new actions                     5927/5927
Updating package state database                 Done
Updating package cache                           0/0
Updating image state                            Done
Creating fast lookup database                   Done
 Postinstall: Copying SMF seed repository ... done.
        Done: Installation completed in 16.942 seconds.

Let’s boot the zone and log in:

        % pfexec zoneadm -z zabbix boot
% pfexec zlogin zabbix
[Connected to zone 'zabbix' pts/17]
OmniOS 5.11 omnios-r151034-831ff8e83b   July 2020
root@zabbix#

Since this is the first boot, it will take a minute for all of the service manifests to be imported. Watch the output of svcs -x until nothing is returned:

        root@zabbix# svcs -x
root@zabbix#

Check Internet connectivity and DNS:

        root@zabbix# ping 1.1.1.1
1.1.1.1 is alive
root@zabbix# ping google.com
google.com is alive

and check the delegated dataset:

        root@zabbix# df -h /data
Filesystem             Size   Used  Available Capacity  Mounted on
data/zabbix           3.51T    42K      1.57T     1%    /data

Install the zabbix server package, which will automatically install the correct version of dependencies:

        root@zabbix# pkg install zabbix-server
           Packages to install:  6
           Mediators to change:  1
            Services to change:  4
       Create boot environment: No
Create backup boot environment: No

DOWNLOAD                                PKGS         FILES    XFER (MB)   SPEED
Completed                                6/6     3082/3082    22.7/22.7  1.2M/s

PHASE                                          ITEMS
Installing new actions                     4013/4013
Updating package state database                 Done
Updating package cache                           0/0
Updating image state                            Done
Creating fast lookup database                   Done
Updating package cache                           2/2

Database

The OmniOS Zabbix package needs a database for storage. The database can be either Postgres or MariaDB, both of which are available as packages from the OmniOS Extra repository. The default supported database is Postgres and that is the database that will be used in the rest of this walk through. To switch zabbix over to using MariaDB, it is necessary to change the zabbix mediator:

        root@zabbix# pkg set-mediator -I mariadb zabbix

Whichever database is used, the underlying database files are one of the things that should be stored on the dedicated dataset that was delegated to the zone.

Create a new ZFS filesystem for the database. For a Postgres database, it’s recommended to set the filesystem recordsize to 8K, and to set the log bias mode to throughput, as shown here. Also for security, executable, setuid and device files are explicitly disabled on the filesystem.

        root@zabbix# zfs create data/zabbix/db
root@zabbix# zfs set recordsize=8k data/zabbix/db
root@zabbix# zfs set logbias=throughput data/zabbix/db
root@zabbix# zfs set exec=off data/zabbix/db
root@zabbix# zfs set devices=off data/zabbix/db
root@zabbix# zfs set setuid=off data/zabbix/db

This new dataset inherits the mountpoint from the filesystem:

        root@zabbix# df -h | grep data/zab
data/zabbix           3.51T    42K      1.57T     1%    /data
data/zabbix/db        3.51T    42K      1.57T     1%    /data/db

Set up the initial database in the dedicated ZFS dataset:

        root@zabbix# chown postgres /data/db
root@zabbix# chmod 0700 /data/db
root@zabbix# svccfg -s postgresql12:default \
        setprop application/datadir = /data/db
root@zabbix# svcadm refresh postgresql12:default
root@zabbix# cd /data/db
root@zabbix:/data/db# sudo -u postgres /opt/ooce/pgsql-12/bin/initdb -D .
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".

Data page checksums are disabled.
...
Success. You can now start the database server using:

Start the database service using svcadm:

        root@zabbix# cd
root@zabbix# svcadm enable postgresql12
root@zabbix# svcs postgresql12
STATE          STIME    FMRI
online         12:32:12 svc:/ooce/database/postgresql12:default

Create the zabbix database user - enter a password to secure the account when prompted.

        root@zabbix# sudo -u postgres createuser --pwprompt zabbix
Enter password for new role:
Enter it again:

Create the database and import initial data:

        root@zabbix# sudo -u postgres createdb -O zabbix \
        -E Unicode -T template0 zabbix
root@zabbix# cd /opt/ooce/zabbix/sql/pgsql
root@zabbix:/opt/ooce/zabbix/sql# cat schema.sql images.sql data.sql \
        | sudo -u zabbix psql zabbix
... lots of output, not shown here ...
COMMIT

Web interface

Zabbix comes with a web interface written in PHP. I’m going to use the nginx web server to serve this over HTTP.

Install packages:

        root@zabbix# pkg install nginx php-74
...
DOWNLOAD                                PKGS         FILES    XFER (MB)   SPEED
Completed                              10/10       517/517    22.9/22.9  2.2M/s
...

Edit the /etc/opt/ooce/nginx/nginx.conf file and replace the example server block in there with the following:

            server {
        listen       80;
        server_name  localhost;
        root /opt/ooce/zabbix/ui;
        index index.php;
        location ~ \.php$ {
                try_files $uri =404;
                fastcgi_pass unix:/var/opt/ooce/php/run/www-7.4.sock;
                fastcgi_index index.php;
                include fastcgi.conf;
        }
    }

A few PHP settings need to be tweaked for proper Zabbix operation. This example sets the time zone to UTC but you can set it to local time if you prefer.

        root@zabbix# cd /etc/opt/ooce/php-7.4/
root@zabbix# sed -i '/post_max_size/s/=.*/= 16M/' php.ini
root@zabbix# sed -i '/execution_time/s/=.*/= 300/' php.ini
root@zabbix# sed -i '/input_time/s/=.*/= 300/' php.ini
root@zabbix# sed -i '/date.timezone/s/.*/date.timezone = UTC/' php.ini

Grant PHP permissions to manage the zabbix UI configuration file:

        root@zabbix# chown php /opt/ooce/zabbix/ui/conf

Enable PHP and the web server:

        root@zabbix# svcadm enable nginx php74
root@zabbix# svcs nginx php74
STATE          STIME    FMRI
online         12:39:30 svc:/network/http:nginx
online         12:39:30 svc:/application/php74:default

Start the Zabbix services:

        root@zabbix# svcadm enable zabbix:server zabbix:agent
root@zabbix# svcs zabbix
STATE          STIME    FMRI
online         12:40:00 svc:/network/zabbix:server
online         12:40:08 svc:/network/zabbix:agent

You should now be able to point a web browser at the server and go through the initial Zabbix setup process:

Zabbix installer

On the database screen, set the type to Postgres via localhost. Enter the password that you set earlier during database creation:

Zabbix database

Once you get back to the login screen, enter Admin with a password of zabbix to get started:

Zabbix login


Any problems or questions, please get in touch.

OmniOS Community Edition r151030bw, r151032aw, r151034w OmniOS Community Edition

OmniOS weekly releases for w/c 5th of October 2020 are now available.

  • For all supported OmniOS releases, pkg has been enhanced to allow for more detailed diagnostic messages via -vv in the event that an upgrade solution can’t be found.

With IPS, it’s possible to get a situation where an upgrade cannot be performed and the error message is not very informative, for example:

r151034% pfexec pkg update -nv
Creating Plan (Running solver): |
pkg update: No solution was found to satisfy constraints
No solution found to update to latest available versions.
This may indicate an overly constrained set of packages are installed.

latest incorporations:

  pkg://omnios/consolidation/osnet/osnet-incorporation@0.5.11,5.11-151036.0:20201002T085228Z
  pkg://omnios/developer/illumos-tools@11,5.11-151036.0:20201002T071229Z
  pkg://omnios/developer/omnios-build-tools@11,5.11-151036.0:20201002T071709Z
  pkg://omnios/entire@11,5.11-151036.0:20201002T072426Z
  pkg://omnios/incorporation/jeos/illumos-gate@11,5.11-151036.0:20201002T085700Z
  pkg://omnios/incorporation/jeos/omnios-userland@11,5.11-151036.0:20201002T072614Z

Dependency analysis is unable to determine the cause.
Try specifying expected versions to obtain more detailed error messages.

This change adds a new behaviour when -v is provided a second time, which shows a better error message that should point directly to the problem

r151034% pfexec pkg update -nvv
Retrieving package list...
Retrieving list of packages to update...
Creating Plan (Solver setup): /
pkg update: Package 'ooce/extra-build-tools' must be uninstalled or upgraded if the requested operation is to be performed.
  Reject:  pkg://extra.omnios/ooce/extra-build-tools@11-151034.0
  Reason:  No version for 'conditional' dependency on ooce/x11/header/xcb-protocols can be found
Package 'ooce/omnios-build-tools' must be uninstalled or upgraded if the requested operation is to be performed.
  Reject:  pkg://extra.omnios/ooce/omnios-build-tools@11-151034.0
  Reason:  No version for 'conditional' dependency on ooce/x11/header/x11-protocols can be found

For further details, please see https://omniosce.org/releasenotes


Any problems or questions, please get in touch.

Using the 'zadm' utility to create an lx zone OmniOS Community Edition

This week’s ascii-cast shows how to use zadm to quickly create an lx-branded zone in OmniOS r151034 and how to configure a basic policy in the zone firewall. The zone firewall works for all zone brands except for KVM and is ideal for securing an lx zone where the native iptables tools don’t work.

nginx is used as an example application here, but in reality one would deploy something like nginx in a native branded zone, most likely with the sparse brand.

This is just a basic zone setup. Configuring more features such as memory and CPU caps will be covered in a future article.


zadm is open source and hosted on Github. Feedback and pull requests are welcome.

Any questions, please get in touch!

OmniOS Community Edition r151030bi, r151032ai, r151034i OmniOS Community Edition

OmniOS weekly releases for w/c 29th of June 2020 are now available.

  • For all supported OmniOS releases, curl has been updated to fix two vulnerabilities (CVE-2020-8169 and CVE-2020-9177)

For further details, please see https://omniosce.org/releasenotes


Any problems or questions, please get in touch.

OmniOS Community Edition r151030bl, r151032al, r151034l OmniOS Community Edition

OmniOS weekly releases for w/c 20th of July 2020 are now available.

This update requires a reboot for r151034.

For r151034 only:

  • Some 64-bit PCI devices were not programmed correctly
  • The mlxcx driver has been updated to fix several problems
  • Panic in imc driver on some systems with broken firmware
  • LX: /proc/<pid>/exe symlink was not always present
  • LX: Would occasionally see defunct processes with busy parents
  • LX: Improve support for networking setup in Void linux zones
  • loader could not read a ZFS pool which had a removed slog device
  • vioblk devices could hang under memory pressure
  • It was not possible to run bhyve in the global zone under a DEBUG kernel (although this is possible for testing, bhyve should always be run within a zone for proper protection)
  • Added a depend.ooceextra facet to illumos packages to allow installation without reference to the omnios-extra repository
  • Updated curl to 7.71.1

Additionally, the following packages have been updated for all supported releases:

  • openjdk to to 1.8.0_262
  • rsync to 3.2.2

For further details, please see https://omniosce.org/releasenotes


Any problems or questions, please get in touch.

Using the 'zadm' utility to create a sparse zone from a template OmniOS Community Edition

This week’s ascii-cast shows how to use zadm to create a sparse-branded zone in OmniOS r151034 using a prepared JSON file, along with a memory cap and CPU shares.


zadm is open source and hosted on Github. Feedback and pull requests are welcome.

Any questions, please get in touch!