SPARC Tribblix m26 - what's in a number? The Trouble with Tribbles...
I've just released Tribblix m26 for SPARC.
The release history on SPARC looks a little odd - m20, m20.6, m22, m25.1, and now m26. Do these release versions mean anything?
Up to and including m25.1, the illumos commit that the SPARC version was built from matched the corresponding x86 release. This is one reason there might be a gap in the release train - that commit might not build or work on SPARC.
As of m26, the version numbers start to diverge between SPARC and x86. In terms of illumos-gate, this release is closer to m25.2, but the added packages are generally fairly current, closer to m29. So it's a bit of a hybrid.
But the real reason this is a full release rather than an m25 update is to establish a new baseline, which allows me to establish compatibility guarantees and roll over versions of key components, in this case it allows me to upgrade perl.
In the future, the x86 and SPARC releases are likely to diverge further. Clearly SPARC can't track the x86 releases perfectly, as SPARC support is being removed from the mainline source following IPD 19, and many of the recent changes in illumos simply aren't relevant to SPARC anyway. So future SPARC releases are likely to simply increment independently.
How I build the Tribblix AMIs The Trouble with Tribbles...
I run Tribblix on AWS, and make some AMIs available. They're only available in London (eu-west-2) by default, because that's the only place where I use them, and it costs money to have them available in other regions. If you want to run them elsewhere, you can copy the AMI.
It's
not
actually
that
difficult
to
create
the
AMIs,
once
you've
got
the
hang
of
it.
Certainly
some
of
the
instructions
you
might
find
can
seem
a
little
daunting.
So
here's
how
I
do
it.
Some
of
the
details
here
are
very
specific
to
my
own
workflow,
but
the
overall
principles
are
fairly
generic.
The
same
method
would
work
for
any
of
the
illumos
distributions,
and
you
could
customize
the
install
however
you
wish.
The procedure below assumes you're running Tribblix m29 and have bhyve installed.
The
general
process
is
to
boot
and
install
an
instance
into
bhyve,
then
boot
that
and
clean
it
up,
save
that
disk
as
an
image,
upload
to
S3,
and
register
an
AMI
from
that
image.
You need to use the minimal ISO (I actually use a custom, even more minimal ISO, but that's just a convenience for myself). Just launch that as root:
zap
create-zone
-t
bhyve
-z
bhyve1
\
-x
192.168.0.236
\
-I
/var/tmp/tribblix-0m29-minimal.iso
\
-V
8G
Note that this creates an 8G zvol, which is the starting size of the AMI.
Then run socat as root to give you a VNC socket to talk to
socat
TCP-LISTEN:5905,reuseaddr,fork
UNIX-CONNECT:/export/zones/bhyve1/root/tmp/vm.vnc
and as yourself, run the vnc viewer
vncviewer
:5
Once it's finished booting, log in as root and install with the ec2-baseline overlay which is what makes sure it's got the pieces necessary to work on EC2.
./live_install.sh
-G
c1t0d0
ec2-baseline
Back as root on the host, ^C to get out of socat, remove the ISO image and reboot, so it will boot from the newly installed image.
zap
remove-cd
-z
bhyve1
-r
Restart socat and vncviewer, and log in to the guest again.
What
I
then
do
is
to
remove
any
configuration
or
other
data
from
the
guest
that
we
don't
want
in
the
final
system.
(This
is
similar
to
the
old
sys-unconfig
that
many
of
us
used
to
Solaris
will
be
familiar
with.)
zap
unconfigure
-a
I usually also ensure that a functional resolv.conf exists, just in case dhcp doesn't create it correctly.
echo "nameserver 8.8.8.8" > /etc/resolv.conf
Back on the host, shut the instance down by shutting down the bhyve zoned it's running in:
zoneadm
-z
bhyve1
halt
Now the zfs volume you created contains a suitable image. All you have to do is get it to AWS. First copy the image into a plain file:
dd
if=/dev/zvol/rdsk/rpool/bhyve1_bhvol0
of=/var/tmp/tribblix-m29.img
bs=1048576
At this point you don't need the zone any more so you can get rid of it:
zap
destroy-zone
-z
bhyve1
The raw image isn't in a form you can use, and needs converting. There's a useful tool - the VMDK stream converter (there's also a download here) - just untar it and run it on the image:
python2
./VMDK-stream-converter-0.2/VMDKstream.py
/var/tmp/tribblix-m29.img
/var/tmp/tribblix-m29.vmdk
Now copy that vmdk file (and it's also a lot smaller than the raw img file) up to S3, in the following you need to adjust the bucket name from mybucket to something of yours:
aws
s3
cp
--cli-connect-timeout
0
--cli-read-timeout
0
\
/var/tmp/tribblix-m29.vmdk
s3://mybucket/tribblix-m29.vmdk
Now you can import that image into a snapshot:
aws
ec2
import-snapshot
--description
"Tribblix
m29"
\
--disk-container
file://m29-import.json
where
the
file
m29-import.json
looks
like
this:
{
"Description":
"Tribblix
m29
VMDK",
"Format":
"vmdk",
"UserBucket":
{
"S3Bucket":
"mybucket",
"S3Key":
"tribblix-m29.vmdk"
}
}
The command will give you a snapshot id, that looks like import-snap-081c7e42756d7456b, which you can follow the progress of with
aws ec2 describe-import-snapshot-tasks --import-task-ids import-snap-081c7e42756d7456b
When that's finished it will give you the snapshot id itself, such as snap-0e0a87acc60de5394. From that you can register an AMI, with
aws
ec2
register-image
--cli-input-json
file://m29-ami.json
where the m29-ami.json file looks like:
{
"Architecture":
"x86_64",
"Description":
"Tribblix,
the
retro
illumos
distribution,
version
m29",
"EnaSupport":
false,
"Name":
"Tribblix-m29",
"RootDeviceName":
"/dev/xvda",
"BlockDeviceMappings":
[
{
"DeviceName":
"/dev/xvda",
"Ebs":
{
"SnapshotId":
"snap-0e0a87acc60de5394"
}
}
],
"VirtualizationType":
"hvm",
"BootMode":
"legacy-bios"
}
If you want to create a Nitro-enabled AMI, change "EnaSupport" from "false" to "true", and "BootMode" from "legacy-bios" to "uefi".
What, no fsck? The Trouble with Tribbles...
There was a huge amount of resistance early on to the fact that zfs didn't have an fsck. Or, rather, a separate fsck.
I recall being in Sun presentations introducing zfs and question after question was about how to repair zfs when it got corrupted.
People were so used to shoddy file systems that were so badly implemented that a separate utility was needed to repair file system errors caused by fundamental design and implementation errors in the file system itself that the idea that the file system driver itself ought to take responsibility for managing the state of the file system was totally alien.
If you think about ufs, for example, there were a number of known failure modes, and what you did was take the file system offline, run the checker against it, and it would detect the known errors and modify the bits on disk in a way that would hopefully correct the problem. (In reality, if you needed it, there was a decent chance it wouldn't work.) Doing it this way was simple laziness - it would be far better to just fix ufs so it wouldn't corrupt the data in the first place (ufs logging went a long way towards this, eventually). And you were only really protecting against known errors, where you understood exactly the sequence of events that would cause the file system to end up in a corrupted state, so that random corruption was either undetectable or unfixable, or both.
The way zfs thought about this was very different. To start with, eliminate all known behaviour that can cause corruption. The underlying copy on write design goes a long way, and updates are transactional so either complete or not. If you find a new failure mode, fix that in the file system proper. And then, correction is built in rather than separate, which means that it doesn't need manual intervention by an administrator, and all repairs can be done without taking the system offline.
Thankfully we've moved on, and I haven't heard this particular criticism of zfs for a while.