Tweaking the Tribblix ISO The Trouble with Tribbles...

For the latest release, I've made a couple of tweaks to the Tribblix ISO image.

The first is that it's now a hybrid ISO. So you should be able to write it (simply dd it) to a USB stick and use that to boot. I've never managed to both create and test a usb image, so this should be a win.

The hybrid ISO also boots under bhyve, which the old ISO never did. So that's a second win.

I didn't do any of the work here, simply taking the steps from the OmniOSce kayak build as suggested by tsoome. But the script I use is here.

The second tweak is that the man pages aren't present in their normal form on the live image. Instead, they're shipped as a compressed tarball. This saves a significant amount of space - the manuals are simple text and compress (with bzip2) really well. This doesn't make a great deal of difference to the size of the ISO, but it reduces the size of the ramdisk we boot from by 10%. And that's important because an overlarge ramdisk chews up too much memory if you're trying to install on a small (1GB or less) system.

Of course, the installer now knows to unpack the man pages during installation. This meant I had to add bzip2 to the live image, but that's tiny compared to the space saving you get in return.

I was hoping to use pbzip2, so that if you're on a multiprocessor machine the uncompression is sped up. That turns out not to be a good idea. The problem is that pbzip2 is written in C++, so needs libstdc++. I don't have that, or any of the gcc runtime, on the live image. Adding it would undo all the space savings I was trying to get, but I've not noticed any slowness.

This can be adjusted a bit. The downside is that you have no access to the manual during installation; I could easily have a small subset of the man pages available, and compress away the majority.

The two features came together, thanks to bhyve. Having an image I can boot and install on bhyve is a huge advantage, as it allows me to test the ISO and the installer much more easily. It was that testing that demonstrated that pbzip2 wouldn't work.

All Your Base Are Belong to 20-Somethings, and Solaris 9 Kebe Says: Dan McD's blog

Two Decades Ago…

Someone pointed out recently that the famous Internet meme “All your base are belong to us” turned 20 this week. Boy do I feel old. I was still in California, but Wendy and I were plotting our move to Massachusetts.

In AD 2001, S9 Was Beginning

OF COURSE I watched the video back then. The original Shockwave/Flash version on a site that no longer exists. I used my then-prototype Sun Blade 1000 to watch it, on Netscape, on in-development Solaris 9.

I found a bug in the audio driver by watching it. Luckily for me, portions of the Sun bug database were archived and available for your browsing pleasure. Behold bug 4451857. I reported it, and all of the text there is younger me.

The analysis and solution are not in this version of the bug report, which is a shame, because the maintainer (one Brian Botton) was quite responsive, and appreciated the MDB output. He fixed the bug by moving around a not-shown-there am_exit_task() call.

Another thing missing from the bug report is my “Public Summary” which I thought would tie things up nicely. I now present it here:

In A.D. 2001
S9 was beginning.
Brian: What Happen?
Dan: Someone set up us the livelock
Dan: We get signal
Brian: What!
Dan: MDB screen turn on.
Brian: It’s YOU!
4451857: How are you gentleman?
4451857: All your cv_wait() are belong to us.
4451857: You are on the way to livelock.
Brian: What you say?
4451857: You have no chance to kill -9 make your time.
4451857: HA HA HA HA…
Brian: Take off every am_exit_task().
Dan: You know what you doing
Brian: Move am_exit_task().
Brian: For great bugfix!

OmniOS Community Edition r151030cp, r151034ap, r151036p OmniOS Community Edition

OmniOS weekly releases for w/c 15th of February 2021 are now available.

All supported OmniOS releases have been updated with the following changes:

Security Fixes in all releases

  • OpenSSL updated to 1.1.1j, fixing CVE-2021-23840, CVE-2021-23841.

  • The legacy OpenSSL 1.0.2 has also been patched to mitigate the above CVEs.

  • All shipped Python versions have been updated to address CVE-2021-3177.

Other Changes in all releases

  • Adding a second disk to an Azure instance caused a kernel panic.

Additional changes in r151030cp

  • A memory leak in the SMB client has been addressed.

  • The ZFS I/O pipeline is now able to use the pageout reserve memory pool in order to flush pages to disk under low memory conditions.

  • The hardware database has been updated.

  • Timezone data has been updated.

Additional changes in r151036p

  • The ZFS I/O pipeline is now able to use the pageout reserve memory pool in order to flush pages to disk under low memory conditions.

For further details, please see https://omniosce.org/releasenotes


Any problems or questions, please get in touch.

OmniOS Community Edition r151030co, r151034ao, r151036o OmniOS Community Edition

OmniOS weekly releases for w/c 8th of February 2021 are now available.

All supported OmniOS releases have been updated with the following changes:

Security Fixes

For further details, please see https://omniosce.org/releasenotes


Any problems or questions, please get in touch.

OmniOS Community Edition r151030cm, r151034am, r151036m OmniOS Community Edition

OmniOS weekly releases for w/c 25th of January 2021 are now available.

All supported OmniOS releases have been updated with the following changes:

Features

  • Support for secure RPC for the Netlogon protocol between OmniOS systems and Microsoft Active Directory servers. This is required to fully mitigate CVE-2020-1472, and will shortly be enforced by Windows domain controllers.

    We’re grateful to Matt Barden of Tintri for contributing this feature to illumos.

Security Fixes

  • It was possible for an unprivileged user, including one in a zone, to cause the system to panic.

  • socat updated to fix a potential heap based buffer overflow.

  • sudo updated to fix CVE-2021-3156

Other Changes

  • On r151036 only, OpenJDK 11 now bundles several core fonts and has been updated to 11.0.10+9.

  • OpenJDK 8 has been updated to 1.8.282-b08

  • gcc has been updated so that the -z assert-deflib linker option works correctly for 64-bit objects.

  • ZFS list -t bookmark was not working for zvols.

  • Add support for the IA32_FEATURE_CONTROL MSR in bhyve. This improves support for some guest operating systems.

  • pkg apply-hot-fix could produce error messages during operation.

For further details, please see https://omniosce.org/releasenotes


Any problems or questions, please get in touch.

OmniOS Community Edition r151030cf, r151034af, r151036f OmniOS Community Edition

OmniOS weekly releases for w/c 14th of December 2020 are now available.

All supported OmniOS releases have been updated with the following changes:

Security Fixes

Bug Fixes

  • MacOS Big Sur clients would experience read hangs when accessing OmniOS SMB shares.

  • The pkg apply-hot-fix command, in conjunction with the creation of a new boot environment, would sometimes leave extra origins configured in non-global zones. This caused problems for subsequent package updates.

  • Compiling with gcc -pg did not produce a working profiling binary.

Other Changes

  • The Intel CPU microcode update for some Core (Gen. 11) Mobile processors has been removed as it was reported to cause problems on some platforms.

Additionally in both the r151034 and r151036 releases:

  • In rare cases, a system could crash shortly after boot while maintaining ZFS user quota information (particularly if a zfs recv was running at the same time).

and in r151036 only:

  • The pciutils package tools were unable to enumerate PCI devices.

  • A file’s modification time would change twice when the file was modified from an SMB client. This caused problems for some Windows utilities.

  • Restore the NDMP ZFS backup and restore method which was inadvertently broken in the r151036 release.

For further details, please see https://omniosce.org/releasenotes


Any problems or questions, please get in touch.

OmniOS Community Edition r151036 Stable OmniOS Community Edition

The OmniOS Community Edition Association is proud to announce the general availability of OmniOS - r151036.

OmniOS is published according to a 6-month release cycle; r151036 stable will be supported for a year. The old stable r151032 release is now end-of-life and will receive no further updates. See the release schedule for further details.

Release Notes and Upgrade Instructions

For full release notes including upgrade instructions; review the release notes and find upgrade instructions at omnios.org/upgrade

If you upgrade from r151030 and want to see all new features added since then, make sure to also read the release notes for the intervening r151032 and r151034 releases.

OmniOSce Newsletter

Since the start of OmniOS Community Edition project, we have predominantly announced our new releases via twitter. We are now also offering a newsletter with announcements of updates, bug fixes and new releases. You can subscribe here.

Commercial Support

Have you ever wondered how OmniOS development gets financed? You may have noticed that there is no big company bankrolling it all. The way we keep afloat is by the companies who rely on OmniOS powered servers taking out support contracts for their hardware. How about you? Visit omnios.org/support for more details and to generate a quote. If you aren’t in a position to take a support contract, please consider becoming an OmniOS patron to help secure its future - omnios.org/patron.

About OmniOS Community Edition Association - this Swiss Association is responsible for the ongoing development and maintenance of OmniOS, having been established in Summer 2017 after OmniTI announced their withdrawal from the project.

OmniOSce Association Aarweg 17, 4600 Olten, Switzerland

OmniOS Community Edition r151030bw, r151032aw, r151034w OmniOS Community Edition

OmniOS weekly releases for w/c 5th of October 2020 are now available.

  • For all supported OmniOS releases, pkg has been enhanced to allow for more detailed diagnostic messages via -vv in the event that an upgrade solution can’t be found.

With IPS, it’s possible to get a situation where an upgrade cannot be performed and the error message is not very informative, for example:

r151034% pfexec pkg update -nv
Creating Plan (Running solver): |
pkg update: No solution was found to satisfy constraints
No solution found to update to latest available versions.
This may indicate an overly constrained set of packages are installed.

latest incorporations:

  pkg://omnios/consolidation/osnet/osnet-incorporation@0.5.11,5.11-151036.0:20201002T085228Z
  pkg://omnios/developer/illumos-tools@11,5.11-151036.0:20201002T071229Z
  pkg://omnios/developer/omnios-build-tools@11,5.11-151036.0:20201002T071709Z
  pkg://omnios/entire@11,5.11-151036.0:20201002T072426Z
  pkg://omnios/incorporation/jeos/illumos-gate@11,5.11-151036.0:20201002T085700Z
  pkg://omnios/incorporation/jeos/omnios-userland@11,5.11-151036.0:20201002T072614Z

Dependency analysis is unable to determine the cause.
Try specifying expected versions to obtain more detailed error messages.

This change adds a new behaviour when -v is provided a second time, which shows a better error message that should point directly to the problem

r151034% pfexec pkg update -nvv
Retrieving package list...
Retrieving list of packages to update...
Creating Plan (Solver setup): /
pkg update: Package 'ooce/extra-build-tools' must be uninstalled or upgraded if the requested operation is to be performed.
  Reject:  pkg://extra.omnios/ooce/extra-build-tools@11-151034.0
  Reason:  No version for 'conditional' dependency on ooce/x11/header/xcb-protocols can be found
Package 'ooce/omnios-build-tools' must be uninstalled or upgraded if the requested operation is to be performed.
  Reject:  pkg://extra.omnios/ooce/omnios-build-tools@11-151034.0
  Reason:  No version for 'conditional' dependency on ooce/x11/header/x11-protocols can be found

For further details, please see https://omniosce.org/releasenotes


Any problems or questions, please get in touch.

Using the 'zadm' utility to create an sparse zone from a template OmniOS Community Edition

This week’s ascii-cast shows how to use zadm to create a sparse-branded zone in OmniOS r151034 using a prepared JSON file, along with a memory cap and CPU shares.


zadm is open source and hosted on Github. Feedback and pull requests are welcome.

Any questions, please get in touch!

Using the 'zadm' utility to create an lx zone OmniOS Community Edition

This week’s ascii-cast shows how to use zadm to quickly create an lx-branded zone in OmniOS r151034 and how to configure a basic policy in the zone firewall. The zone firewall works for all zone brands except for KVM and is ideal for securing an lx zone where the native iptables tools don’t work.

nginx is used as an example application here, but in reality one would deploy something like nginx in a native branded zone, most likely with the sparse brand.

This is just a basic zone setup. Configuring more features such as memory and CPU caps will be covered in a future article.


zadm is open source and hosted on Github. Feedback and pull requests are welcome.

Any questions, please get in touch!

Running a Zabbix server in an OmniOS zone OmniOS Community Edition

This guide shows how to get Zabbix up and running within a zone on an OmniOS system. Zabbix is an open-source monitoring system and is available from the OmniOS Extra repository.

Zone setup

I’m going to use the lightweight sparse zone brand for this so start by making sure that it is installed:

        % pfexec pkg install brand/sparse
No updates necessary for this image.

If the brand is not already installed, then there will be more output from the above command.

Create a new sparse zone called zabbix. My preference is to configure the IP stack within the zone configuration which results in it being automatically applied to the zone when it’s booted, and enables additional protection against settings such as the IP address from being changed from within the zone. Note that, like all zones, the zone path must be a direct descendant of a ZFS dataset. On my system, /zones is the mount point for such a dataset.

        % pfexec zonecfg -z zabbix
zabbix: No such zone configured
Use 'create' to begin configuring a new zone.
zonecfg:zabbix> create -t sparse
zonecfg:zabbix> set zonepath=/zones/zabbix
zonecfg:zabbix> add net
zonecfg:zabbix:net> set physical=zabbix0
zonecfg:zabbix:net> set global-nic=igb0
zonecfg:zabbix:net> set allowed-address=172.27.10.35/24
zonecfg:zabbix:net> set defrouter=172.27.10.254
zonecfg:zabbix:net> end
zonecfg:zabbix> add attr
zonecfg:zabbix:attr> set name=resolvers; set type=string; set value=1.1.1.1
zonecfg:zabbix:attr> end
zonecfg:zabbix> add attr
zonecfg:zabbix:attr> set name=dns-domain; set type=string; set value=omnios.org
zonecfg:zabbix:attr> end
zonecfg:zabbix:attr> verify; commit; exit

By default, the zone’s boot environment will encompass all of the files and directories within. For an application such as Zabbix, it’s important to create a dedicated area to hold files such as the underlying database which should be consistent across different boot environments.

My system has a ZFS pool called data so I’m going to create a new dataset under that and delegate it to the zone. I’m also going to change the mount point for the dataset to /data so that it appears there within the zone.

        % pfexec zfs create data/zabbix
% pfexec zonecfg -z zabbix 'add dataset; set name=data/zabbix; end'

% pfexec zfs umount data/zabbix
% pfexec zfs set mountpoint=/data data/zabbix
% pfexec zfs set zoned=on data/zabbix

Now it’s time to install the zone. Being a sparse zone, this will be pretty quick - only around 5MiB of files are actually installed.

        % pfexec zoneadm -z zabbix install
A ZFS file system has been created for this zone.

       Image: Preparing at /zones/zabbix/root.
Sanity Check: Looking for 'entire' incorporation.
   Publisher: Using omnios (https://pkg.omnios.org/r151034/core/).
   Publisher: Using extra.omnios (https://pkg.omnios.org/r151034/extra/).
       Cache: Using /var/pkg/publisher.
  Installing: Packages (output follows)
Packages to install: 203
Mediators to change:   5
 Services to change:   6

DOWNLOAD                                PKGS         FILES    XFER (MB)   SPEED
Completed                            203/203     1485/1485      4.9/4.9      --

PHASE                                          ITEMS
Installing new actions                     5927/5927
Updating package state database                 Done
Updating package cache                           0/0
Updating image state                            Done
Creating fast lookup database                   Done
 Postinstall: Copying SMF seed repository ... done.
        Done: Installation completed in 16.942 seconds.

Let’s boot the zone and log in:

        % pfexec zoneadm -z zabbix boot
% pfexec zlogin zabbix
[Connected to zone 'zabbix' pts/17]
OmniOS 5.11 omnios-r151034-831ff8e83b   July 2020
root@zabbix#

Since this is the first boot, it will take a minute for all of the service manifests to be imported. Watch the output of svcs -x until nothing is returned:

        root@zabbix# svcs -x
root@zabbix#

Check Internet connectivity and DNS:

        root@zabbix# ping 1.1.1.1
1.1.1.1 is alive
root@zabbix# ping google.com
google.com is alive

and check the delegated dataset:

        root@zabbix# df -h /data
Filesystem             Size   Used  Available Capacity  Mounted on
data/zabbix           3.51T    42K      1.57T     1%    /data

Install the zabbix server package, which will automatically install the correct version of dependencies:

        root@zabbix# pkg install zabbix-server
           Packages to install:  6
           Mediators to change:  1
            Services to change:  4
       Create boot environment: No
Create backup boot environment: No

DOWNLOAD                                PKGS         FILES    XFER (MB)   SPEED
Completed                                6/6     3082/3082    22.7/22.7  1.2M/s

PHASE                                          ITEMS
Installing new actions                     4013/4013
Updating package state database                 Done
Updating package cache                           0/0
Updating image state                            Done
Creating fast lookup database                   Done
Updating package cache                           2/2

Database

The OmniOS Zabbix package needs a database for storage. The database can be either Postgres or MariaDB, both of which are available as packages from the OmniOS Extra repository. The default supported database is Postgres and that is the database that will be used in the rest of this walk through. To switch zabbix over to using MariaDB, it is necessary to change the zabbix mediator:

        root@zabbix# pkg set-mediator -I mariadb zabbix

Whichever database is used, the underlying database files are one of the things that should be stored on the dedicated dataset that was delegated to the zone.

Create a new ZFS filesystem for the database. For a Postgres database, it’s recommended to set the filesystem recordsize to 8K, and to set the log bias mode to throughput, as shown here. Also for security, executable, setuid and device files are explicitly disabled on the filesystem.

        root@zabbix# zfs create data/zabbix/db
root@zabbix# zfs set recordsize=8k data/zabbix/db
root@zabbix# zfs set logbias=throughput data/zabbix/db
root@zabbix# zfs set exec=off data/zabbix/db
root@zabbix# zfs set devices=off data/zabbix/db
root@zabbix# zfs set setuid=off data/zabbix/db

This new dataset inherits the mountpoint from the filesystem:

        root@zabbix# df -h | grep data/zab
data/zabbix           3.51T    42K      1.57T     1%    /data
data/zabbix/db        3.51T    42K      1.57T     1%    /data/db

Set up the initial database in the dedicated ZFS dataset:

        root@zabbix# chown postgres /data/db
root@zabbix# chmod 0700 /data/db
root@zabbix# svccfg -s postgresql12:default \
        setprop application/datadir = /data/db
root@zabbix# svcadm refresh postgresql12:default
root@zabbix# cd /data/db
root@zabbix:/data/db# sudo -u postgres /opt/ooce/pgsql-12/bin/initdb -D .
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".

Data page checksums are disabled.
...
Success. You can now start the database server using:

Start the database service using svcadm:

        root@zabbix# cd
root@zabbix# svcadm enable postgresql12
root@zabbix# svcs postgresql12
STATE          STIME    FMRI
online         12:32:12 svc:/ooce/database/postgresql12:default

Create the zabbix database user - enter a password to secure the account when prompted.

        root@zabbix# sudo -u postgres createuser --pwprompt zabbix
Enter password for new role:
Enter it again:

Create the database and import initial data:

        root@zabbix# sudo -u postgres createdb -O zabbix \
        -E Unicode -T template0 zabbix
root@zabbix# cd /opt/ooce/zabbix/sql/pgsql
root@zabbix:/opt/ooce/zabbix/sql# cat schema.sql images.sql data.sql \
        | sudo -u zabbix psql zabbix
... lots of output, not shown here ...
COMMIT

Web interface

Zabbix comes with a web interface written in PHP. I’m going to use the nginx web server to serve this over HTTP.

Install packages:

        root@zabbix# pkg install nginx php-74
...
DOWNLOAD                                PKGS         FILES    XFER (MB)   SPEED
Completed                              10/10       517/517    22.9/22.9  2.2M/s
...

Edit the /etc/opt/ooce/nginx/nginx.conf file and replace the example server block in there with the following:

            server {
        listen       80;
        server_name  localhost;
        root /opt/ooce/zabbix/ui;
        index index.php;
        location ~ \.php$ {
                try_files $uri =404;
                fastcgi_pass unix:/var/opt/ooce/php/run/www-7.4.sock;
                fastcgi_index index.php;
                include fastcgi.conf;
        }
    }

A few PHP settings need to be tweaked for proper Zabbix operation. This example sets the time zone to UTC but you can set it to local time if you prefer.

        root@zabbix# cd /etc/opt/ooce/php-7.4/
root@zabbix# sed -i '/post_max_size/s/=.*/= 16M/' php.ini
root@zabbix# sed -i '/execution_time/s/=.*/= 300/' php.ini
root@zabbix# sed -i '/input_time/s/=.*/= 300/' php.ini
root@zabbix# sed -i '/date.timezone/s/.*/date.timezone = UTC/' php.ini

Grant PHP permissions to manage the zabbix UI configuration file:

        root@zabbix# chown php /opt/ooce/zabbix/ui/conf

Enable PHP and the web server:

        root@zabbix# svcadm enable nginx php74
root@zabbix# svcs nginx php74
STATE          STIME    FMRI
online         12:39:30 svc:/network/http:nginx
online         12:39:30 svc:/application/php74:default

Start the Zabbix services:

        root@zabbix# svcadm enable zabbix:server zabbix:agent
root@zabbix# svcs zabbix
STATE          STIME    FMRI
online         12:40:00 svc:/network/zabbix:server
online         12:40:08 svc:/network/zabbix:agent

You should now be able to point a web browser at the server and go through the initial Zabbix setup process:

Zabbix installer

On the database screen, set the type to Postgres via localhost. Enter the password that you set earlier during database creation:

Zabbix database

Once you get back to the login screen, enter Admin with a password of zabbix to get started:

Zabbix login


Any problems or questions, please get in touch.

OmniOS Community Edition r151030bi, r151032ai, r151034i OmniOS Community Edition

OmniOS weekly releases for w/c 29th of June 2020 are now available.

  • For all supported OmniOS releases, curl has been updated to fix two vulnerabilities (CVE-2020-8169 and CVE-2020-9177)

For further details, please see https://omniosce.org/releasenotes


Any problems or questions, please get in touch.

OmniOS Community Edition r151030bl, r151032al, r151034l OmniOS Community Edition

OmniOS weekly releases for w/c 20th of July 2020 are now available.

This update requires a reboot for r151034.

For r151034 only:

  • Some 64-bit PCI devices were not programmed correctly
  • The mlxcx driver has been updated to fix several problems
  • Panic in imc driver on some systems with broken firmware
  • LX: /proc/<pid>/exe symlink was not always present
  • LX: Would occasionally see defunct processes with busy parents
  • LX: Improve support for networking setup in Void linux zones
  • loader could not read a ZFS pool which had a removed slog device
  • vioblk devices could hang under memory pressure
  • It was not possible to run bhyve in the global zone under a DEBUG kernel (although this is possible for testing, bhyve should always be run within a zone for proper protection)
  • Added a depend.ooceextra facet to illumos packages to allow installation without reference to the omnios-extra repository
  • Updated curl to 7.71.1

Additionally, the following packages have been updated for all supported releases:

  • openjdk to to 1.8.0_262
  • rsync to 3.2.2

For further details, please see https://omniosce.org/releasenotes


Any problems or questions, please get in touch.

OmniOS Community Edition r151030co, r151034ao, r151036o OmniOS Community Edition

OmniOS weekly releases for w/c 8th of February 2021 are now available.

All supported OmniOS releases have been updated with the following changes:

Security Fixes

For further details, please see https://omniosce.org/releasenotes


Any problems or questions, please get in touch.

Silly SmartOS hack: ipf/ipnat in lx zones Nahum Shalman

Someone on IRC in #smartos was asking about how to turn on NAT in lx branded zones. I was pretty sure it should be possible, and found myself nerd-sniped into figuring out the exact solution.

I don't think I particularly recommend doing this, but figuring out how to do it is a good exercise in figuring out how certain things work in illumos in general, so we'll take a brief excursion into what actually happens in a joyent branded zone when you follow the steps in the wiki page I wrote forever ago. From there we'll figure out a relatively minimal set of commands to run in lx branded zones to accomplish the same thing.

The instructions list three commands to run in the firewall zone, but the third is just to verify that NAT is indeed configured, so let's look closely at the first two which do all the work:

routeadm -u -e ipv4-forwarding
svcadm enable ipfilter

The first one is turning on packet forwarding for ipv4, and the second turns on ipf filtering and NAT. But what do those commands do under the hood and can we replicate that in the lx zone?

If we look at the routeadm man page it helpfully points out that those features are also represented as SMF services which you could manage directly with svcadm and the EXAMPLES section even suggests that we could turn on ipv4 forwarding using svcadm enable ipv4-forwarding.

Once we have an SMF service, we can see how it works by looking at the service manifest. When I was poking around I did this live in a zone by running svccfg export ipv4-forwarding but for the ease of just clicking links, the source for what you'd see there is in /usr/src/cmd/cmd-inet/usr.sbin/routeadm/forwarding.xml.

Narrowing in on the start method which wants to run /lib/svc/method/svc-forwarding %m ipv4 we see that there's a helper script we need to read. Again, I looked at it at that path on my system but the source is in /usr/src/cmd/cmd-inet/usr.sbin/routeadm/svc-forwarding.
In there we find the critical line of /usr/sbin/ipadm set-prop -p forwarding=on $proto

What have we learned so far? routeadm behind the scene enables the ipv4-forwarding SMF service that in turn runs an ipadm command.

So now what about that ipfilter service?
Again, on a live system we can run svccfg export ipfilter to find the start method (source in /usr/src/cmd/ipf/svc/ipfilter.xml) and we find that it calls /lib/svc/method/ipfilter (source: /usr/src/cmd/ipf/svc/ipfilter).

If you read through this script you find some places where it calls out to ipf and ipnat whose manpages can be used to figure out what those various commands do.

Through reading the script and the man page and some experimentation I narrowed down a minimal set of steps of:

ipf -E    # enable IPF
ipnat -CF # Delete all existing rules and active mappings
ipnat -f ${ipnat_config_file:?} # load NAT rules from a file

There's one last wrinkle that tripped me up and wasted some of my time. I blithely copy-and-pasted the ipnat.conf content, ran my commands, and wondered why the packets weren't flowing. The lx brand names network interfaces with a more linux-y eth0, eth1, etc. rather than net0, net1, etc.

So, let's put this all together:

firewall-lx.json:

{
  "alias": "firewall-lx",
  "hostname": "firewall-lx",
  "brand": "lx",
  "kernel_version": "3.10.0",
  "max_physical_memory": 512,
  "image_uuid": "63d6e664-3f1f-11e8-aef6-a3120cf8dd9d",
  "nics": [
    {
      "nic_tag": "admin",
      "ip": "dhcp",
      "allow_ip_spoofing": "1"
    },
    {
      "nic_tag": "stub0",
      "ip": "10.0.0.1",
      "netmask": "255.255.255.0",
      "allow_ip_spoofing": "1",
      "primary": "1"
    }
  ]
}

client.json:

{
  "alias": "client",
  "hostname": "client",
  "brand": "joyent-minimal",
  "max_physical_memory": 256,
  "image_uuid": "cfa9c88e-03f8-11eb-9980-879ff7980a9f",
  "nics": [
    {
      "nic_tag": "stub0",
      "ip": "10.0.0.2",
      "netmask": "255.255.255.0",
      "gateway": "10.0.0.1",
      "primary": "1"
    }
  ]
}

On the host:

nictagadm add -l stub0
vmadm create -f client.json

In a separate window, zlogin into the client zone and start pinging an external ip with ping -ns e.g. ping -ns 8.8.8.8
So far no packets should be flowing, let's fix that.

Back out on the host

vmadm create -f firewall-lx.json

Now zlogin into your lx branded "firewall" zone and let's get packets flowing (note how we invoke the native illumos tools from under /native):

echo "map eth0 10.0.0.2/32 -> 0/32" > /etc/ipnat.conf
/native/usr/sbin/ipadm set-prop -p forwarding=on ipv4
/native/usr/sbin/ipf -E
/native/usr/sbin/ipnat -CF
/native/usr/sbin/ipnat -f /etc/ipnat.conf

At this point, your client zone should start seeing ping replies.
Whether to use this in production and how to get it to survive reboots of the "firewall" zone is left as an exercise for the reader.

Github Pull Requests Staring at the C

Stefan Hajnoczi recently posted about clean commit history.

It’s a controversial viewpoint that not everyone agrees with - there is a sizable population in favour of “never rewrite history”. For me, though, the points he makes there are totally correct: each commit should be a logical change, main (neé master) should stay green, and CI should pass at every single point in main's history. More than just CI though: regardless of whether it passes CI, the main branch should be of good quality at all times, if you want to avoid the Quality Death Spiral.

Unfortunately, Github pull requests make this model a little difficult for a few reasons:

You can’t ever rebase a PR undergoing review

It’s important that a non-draft PR is never rebased, or re-written in any way. Why? Well, aside from making it difficult for a reviewer to see what’s changed since last looking, if you rebase, the commits previously on the PR disappear off into reflog hyperspace.

The View Changes button on review comments is attached to that particular commit hash, which is no longer in the history for that branch, and you get the dreaded:

    We went looking everywhere, but couldn’t find those commits.

Note that if your PR is still a draft, you’re fine to edit the history whichever way you like: in fact, it’s often useful for review purposes to have multiple commits even at the start of a PR review before you move it from draft. Up to you.

The only other safe time to rebase is on final approach. At that point, presuming you are keeping to the “single main commit per PR” approach (see below), you’ll be wanting to squash the entire branch history into a single commit to main. For this, I usually use prr: it’s handy for picking up Reviewed-by automatically, and merging commit comments together for final editing.

Github CI only runs on branch tips

You probably don’t want to have a PR where you’re going to merge more than one commit into main. This is because CI only runs on the top-level commit: if an ancestor commit breaks the build, you’ll never know. Stefan mentions using git rebase --exec for checking commits in a stack, which indeed works great, but unless you’re running exactly the same CI that’s running under Github Actions, you can’t rely on it.

If that’s the case, what if you have one or more changes that depend on another? This is where “stacked PRs” come in, and they’re a bit of a pain…

Stacked PRs are cumbersome

Gerrit has a really useful model for reviewing stacks of changes: instead of the full history, each “patchset” corresponds to the single logical change Stefan talks about above. Every time you push to Gerrit, you’re supposed to have collapsed and rebased additional changes into single commits corresponding to each Gerrit CR. The model has some disadvantages as well (in particular, it’s a bit of a pain to keep a full history locally), but the Gerrit review UI doesn’t suffer from the rebasing issues Github does1.

Presuming - as there is no CI available - gerrithub is a non-starter, the only option available on Github is to use multiple PRs. This is better than it used to be, but is still a little painful.

Essentially, a stacked PR is one that’s opened not against the main branch, but against another PR branch. Say we have changes A and B, where B is dependent on A. You’d create a local branch with A, then push it to Github and open a PR. You’d have another local branch with A and B, then push that branch to Github and open a separate PR.

Now we need to make the B PR be based against the A PR. You can do this via the web UI by clicking Edit, though there is annoying bug here: it doesn’t reset the title and description. You can use gh pr create --base ... to avoid this problem.

Now, in the second PR, you’ll just see the commit for B. Each PR can be reviewed separately, and each PR gets its own CI run.

You also might want to merge additional changes up the stack. Let’s say that you have commit A2 on the A PR, that you want in PR B and C: the best - if rather tedious - way to do this, is to merge A into B, then B into C. That’s a lot of merge commits, but remember we’re squashing a PR every time before merging a PR to main.

You’ll see on the web recommendations to “merge downwards”: you wait for commit approval for the whole stack, then merge the top PR (B) into the PR underneath it (A), and so on, until you merge to main.

I don’t think that’s necessary these days2. Instead, when you have approval for the base PR - and logically, it will make sense that is reviewed first - you can merge it to main. Github will then offer to delete the PR branch. If you do this, the stacked PR gets automatically reset such that its merge base is now main !

There is an annoying thing here though: because of that squash during the merge to main, git, and Github, needs you to merge main back into the commit history of the PR that just changed bases. If you already merged the parent PR, you can always do git merge -Xours master to fix this, since there shouldn’t be any actual diff difference between the PR branch diffs as a whole, and what was merged to master. Or, if you didn’t merge in the parent PR, you’ll need a normal git merge master.

Another bug (as far as I’m concerned) is that if you ask for review on a stacked PR, it doesn’t get tagged with “Review required”, since, technically, you could merge the PR into its parent without approval. And there is no “Review requested” tag.

I would love all this to have some tooling: something that lets me do everything on my local stacked branches, automate merges up, keep track of dependencies, and updating the branches in Github. But I haven’t been able to find anything that can do it.


  1. Gerrit uses Change-ID embedded in the commit message to map commits onto CRs. It’s clumsy but effective. ↩︎

  2. I think it dates from before Github automatically reset a PR when its merge base was deleted ↩︎

2021-01-25 Josef "Jeff" Sipek

OmniOS Community Edition r151030cm, r151034am, r151036m OmniOS Community Edition

OmniOS weekly releases for w/c 25th of January 2021 are now available.

All supported OmniOS releases have been updated with the following changes:

Features

  • Support for secure RPC for the Netlogon protocol between OmniOS systems and Microsoft Active Directory servers. This is required to fully mitigate CVE-2020-1472, and will shortly be enforced by Windows domain controllers.

    We’re grateful to Matt Barden of Tintri for contributing this feature to illumos.

Security Fixes

  • It was possible for an unprivileged user, including one in a zone, to cause the system to panic.

  • socat updated to fix a potential heap based buffer overflow.

  • sudo updated to fix CVE-2021-3156

Other Changes

  • On r151036 only, OpenJDK 11 now bundles several core fonts and has been updated to 11.0.10+9.

  • OpenJDK 8 has been updated to 1.8.282-b08

  • gcc has been updated so that the -z assert-deflib linker option works correctly for 64-bit objects.

  • ZFS list -t bookmark was not working for zvols.

  • Add support for the IA32_FEATURE_CONTROL MSR in bhyve. This improves support for some guest operating systems.

  • pkg apply-hot-fix could produce error messages during operation.

For further details, please see https://omniosce.org/releasenotes


Any problems or questions, please get in touch.

ZFS send/receive Nahum Shalman

A few weeks back I needed to migrate an entire ZFS pool from one machine to another. I used raw send to keep the stream compressed, and I used mbuffer to smooth out the send/receive (see the reference link at the bottom)

First, prepare the receiving end by creating the pool. If this is not a bootable pool, just create it the simple way:

poolname=rpool
disk=/dev/sda
zpool create ${poolname:?} ${disk:?}

Now start up mbuffer to receive the initial send stream ("-s" is for resumable receive in case the receive gets interrupted part way through)
I'm using a block size of 128k, a buffer size of 1G and listening on port 9090:

mbuffer -s 128k -m 1G -I 9090 | zfs receive -Fs ${poolname:?}

Now switch over to the sending machine. Here we will take a snapshot, and then send it over mbuffer to the receving machine. We'll do a raw and resumable send.

poolname=rpool
snapname=migration-snap1
receiver=receiver.example.com
zfs snapshot -r ${poolname:?}@${snapname:?}
zfs send -Rvw ${poolname:?}@${snapname:?} | mbuffer -s 128k -m 1G -O ${receiver:?}:9090

If the sending side has changed since the first snapshot completed sending, do an incremental send/receive to sync up the last changes (repeat as needed)

On receving machine, rerun the receiving command:

mbuffer -s 128k -m 1G -I 9090 | zfs receive -Fs ${poolname:?}

On the sending machine, take another snapshot and send over the incremental stream:

snap2=migration-snap2
zfs snapshot -r ${poolname:?}@${snap2:?}
zfs send -Rvw ${poolname:?}@${snapname:?} ${poolname:?}@${snap2:?} | mbuffer -s 128k -m 1G -O ${receiver:?}:9090

References:

Using mbuffer to speed up slow zfs send | zfs receive - EveryCity
zfs send | zfs receive stalls the sender, resulting in a bursty and slow transfer process. The solution is to deploy mbuffer into the mix. MBuffer