Homebrew social networking The Observation Deck

One of the most persistent cycles in the history of computing is the oscillation between centralization and decentralization. This cycle becomes entrenched because each has distinct advantages and disadvantages: centralized solutions often yield better economies of scale, allowing for higher quality artifacts — but they can also easily become stifled, reluctant to innovate for fear of disrupting the good thing they have going for themselves. Decentralized ones, by contrast, can be a mess but they democratize innovation, where a “worse” solution by some metric is in fact vastly preferred because it is much better by another metric (e.g., cost or convenience).

With the collapse of Twitter, we are seeing this oscillation in spectacular fashion in a new domain: social networking. In our Oxide and Friends discussion with Kris Nóva, Nóva made the observation that the decentralization of the Fediverse allows for people to experiment with different ways of doing things by running their own instance — “the control and freedom to build your own community” — and that she particularly loved the influx of folks with large scale internet experience into the much smaller scale of running a Mastodon instance. This observation was a bit of an aha moment for me, because it made me realize the stunning resemblance that Mastodon bears to a much earlier revolution: the homebrew computing movement of the late 1970s.

Homebrew computing — as epitomized by the Homebrew Computer Club in Silicon Valley — consisted of hobbyists designing and assembling their own computers. But like Nóva and her fellow Hachydermians, these hobbyists were in fact experienced engineers; their “homebrew” was just them building for themselves, for fun. Yes, homebrew computing was fragmented and incoherent; the participants at the time were not possibly thinking that the machines that they developed would form the basis (for better and for ill!) for all future computing. But by the late 1970s, the time for radical decentralization of compute had indisputably arrived — and its spirit would burn so bright that it would endure long after the homebrew became commercial, as viscerally expressed in Apple’s iconic 1984 Superbowl ad.

So it is now with social networking, where centralization is frankly long past its “Disrupt By” date. The progression here now seems painfully clear in hindsight: advertising-based models demand engagement — and algorithms will naturally observe that when we are enraged, we stay engaged. If follows that when algorithms are promoting content that enrages us, the social network will — either implicitly or explicitly — select for those that are most divisive. Not only has this has encouraged us all to be more divisive (viz. “hot take” entering the lexicon), it has embraced people who are divisive to their core, elevating them to heights that frankly would have been impossible without centralized social networking. The net effect of all of this is that while we may find this engaging, we certainly don’t find it to be enjoyable: I (like many, I suspect) have resented the attention of mine that it has held — like an extraordinarily bad reality TV program that I can’t stop myself from bingewatching. So while it is tempting for us to look at the destruction of Twitter as a catalyzing event, in truth I think it’s merely an accelerant: the model has been broken for a long time, it’s just now being speedrun to the logical extreme. (I would also add that GenZ may have already figured this out — and that my own kids view the destruction of Twitter the way I might view a collapse of LinkedIn: with ironic detachment and some sense of disbelief that anyone could really care that much.)

What does homebrew decentralized social networking mean in practice? As he has so many times before, Tim Bray has an excellent piece on this. Part of what makes decentralized social networking so appealing is that there is no algorithm deciding engagement. Rather, the stuff that’s promoted — boosted, in Mastodon’s parlance — comes purely from folks that you follow. That is, you only see a strict timeline. The effects of this are both surprising and refreshing. I have said that it’s harder for things to go viral on Mastodon, but my own experience is in fact more nuanced: things can go viral because boosting is a lighter operation that retweeting — and when they do go, they seem to go further and longer. Take, for example, Tim’s post on his blog entry: as I write this, that has 295 favorites and 595 boosts. Tim has 7.1K followers on Mastodon; you would never see numbers like that on Twitter, where likes will essentially always dwarf retweets.

All of this points to an even larger decentralization: that of the social networks themselves. In the last few years, some began looking at the ratio of their followers to the number that they follow, seeking to maximize it as a show of social dominance. If it needs to be said, there is a really easy way to do this: just unfollow everyone! This act feels innocent (if juvenile), but it is in fact deeply isolating: it is opting to be fed exclusively by the algorithm, losing any tether to genuine social connection. It is perhaps unsurprising that the people that I saw do this rapidly descended into dark, bizarre thinking, where they saw the world as increasingly conspiratorial and contentious. Being stingy with follows also nullifies one of the important values of social networking: hearing entirely new voices. Speaking for myself personally, this was especially important after #MeToo in 2017 and George Floyd in 2020, when new people that I followed elevated yet more voices that needed to be heard, leading to more new follows — and more new voices in a virtuous cycle. In decentralized social networking, this virtue is elevated, if by default: there is no algorithm, so if you insist on following no one, you will see… nothing. (As my mother was fond of saying when I found an empty mailbox as a kid: “If you want to receive a letter, you need to send a letter!”) There is something beautiful in this, and I would like to believe that a world in which people need to simply follow more people — to listen more! — will foster more temperance and less divisiveness. (And perhaps even the occasional apology?)

All of this brings us back to another point that Nóva made: “any large change has been uncomfortable and unexpected.” This was true of the homebrew computing movement where, important as these machines historically were, they were not easy to use! We can expect some discomfort in our future, but we can also expect fruitful experimentation — and surely some important interactions that centralized social networking never would have allowed!

Finally, a programming note: Adam and I were early adopters of Twitter Spaces, turning it into our Oxide and Friends podcast. We, like Tim, are leaving, and will be moving to Discord. Fittingly, for our first Discord recording tomorrow (Monday November 28th, 5p Pacific), we will be joined by Tim to get his perspective; join us!

A decade of Tribblix The Trouble with Tribbles...

I seem to have just missed the anniversary, but it turns out that Tribblix has existed for slightly over a decade.

The initial blog post on Building Tribblix was published on October 24th, 2012. But the ISO image (milestone 0) was October 21st, and it looks like the packages were built on October 4th. So there's a bit of uncertainty about the actual date, and I had been playing around with some of the bits and pieces for a while before that.

There have been a lot of releases. We're now on Milestone 28, but there have been several update releases along the way, so I make it 42 distinct releases in total. That doesn't include the LX-enabled OmniTribblix variant (there have been 20 of those by the way).

The focus (given hardware availability) has been x86, naturally. But the SPARC version has seen occasional bursts of life. Now I have a decent build system, it's catching up. Will there be an ARM version? Who knows...

Over the years there have been some notable highlights. It took a few releases to become fully self-hosting; package management had to be rebuilt; LibreOffice was ported; Xfce and MATE added as fully functional desktop offerings (with a host of others); a whole family of zones, including reimplementing the traditional sparse root; made available on clouds like AWS and Digital Ocean; network install via iPXE; huge numbers of packages (it's never-ending churn); and maintaining Java by default.

And it's been (mostly) fun. Here's to the next 10 years!

TREASURE - The Remote Execution and Access Service Users Really Enjoy The Trouble with Tribbles...

Many, many years ago I worked on a prototype of a software ecosystem I called TREASURE - The Remote Execution and Access Service Users Really Enjoy.

At the time, I was running the infrastructure and application behind an international genomics service. The idea was that we could centrally manage all the software and data for genomic analysis, provide high-end compute and storage capability, and amortize the cost across 20,000 academics so that individual researchers didn't have to maintain it all individually.

Originally, access was via telnet (I did say it was a long time ago). After a while we enabled X11, so that graphical application would work (running X11 directly across the internet was fun).

Then along came the web. One of my interesting projects was to write a web server that would run with the privileges of the authenticated user. (This was before apache came along, by the way!) And clearly a web browser might be able to provide a more user-friendly and universal interface than a telnet prompt.

We added VNC as well (it came out of Cambridge and we were  aware of it well before it became public), so that users could view graphical applications more easily. This had a couple of advantages - all the hard work and complexity was at our end, where we had control, and X11 is quite latency sensitive so performance improved.

But ultimately what I wanted to do was to run the GUI on the user's machine, wit access to the user's files. Remember that the GUI is then not running where the software, genome databases, and all the compute power are located.

Hence the Remote Execution part of TREASURE - what we wanted was a system that would call across to a remote service to do the work, and return the result to the user. And the Access part was about making it accessible and transparent, which would lead to a working environment that people would enjoy using.

Ultimately, the core of TREASURE was originally a local GUI that knew how to run applications. Written in Java, it would therefore run on pretty much any client (and we had users with all sorts of Unix workstations in addition to Windows making inroads). The clever bit was to replace the java Runtime.getRuntime().exec() calls that ran applications locally with some form of remote procedure call. Being of its time, this might involve CORBA, RMI, SOAP, or JAX-WS with data marshalled as XML. In fact, I implemented pretty much every remote call mechanism available (and this did in fact come in useful as other places did make available some services using pretty random protocols). And then of course there's the server side which was effectively a CGI script.

The other key part was to work out which files needed to be sent across. Sometimes it was obvious (it's a GUI, the user has selected a file to analyse), but sometimes we needed to send across auxiliary files as well. And on the server side it ran in a little sandbox so you knew what output files had bee generated so you could return those.

Effectively, this was a production form of serverless computing running over 20 years ago. Only we called it GRID computing back then.

Another interesting feature of the architecture was the TREASURE CHEST, which was a source of applications. There were literally hundreds of possible applications you could run, and many more if you included interfaces to other providers. So rather than write all those into the app, there was a plugin system where you could download a jar file and run it as a plugin, and the TREASURE CHEST was where you could find these application. Effectively an app store, in modern terminology.

Sadly the department got closed down due to political incompetence, so the project never really got beyond the prototype stage. And while I still have bits and pieces of code, I don't appear to have a copy of the whole thing. A lot of the components would need to be replaced, but the overall concept is still sound.

Tribblix for SPARC m25.1 The Trouble with Tribbles...

Following hot on the heels of the Tribblix Milestone 22 ISO for SPARC, it's possible to upgrade that to a newer version. The new version that's available is m25.1.

(If the available versions look a bit random, that's because they are. Not every release on x86 was built for SPARC, and not all of the ones that were actually worked properly. So we have what we have.)

The major jump, aside from the underlying OS, in m25.1 for SPARC is that it brings in gcc7 (for applications, illumos itself is still built with gcc4), and generally there's a bunch of more modern applications available.

To upgrade m22 to m25.1 is a manual process. This is because there are steps that are necessary, and if you don't follow them exactly the system won't boot.

The underlying cause here of the various problems in this process is that it's a big jump from m22 to m25.1 and you will hit bugs in the upgrade process that have been fixed in intermediate releases.

First, take a note of the current BE, eg tribblix. You might need it later if things go bad and you need to reboot into the current (hopefully working) release.

You can manually add available versions for upgrade with the following trick (this is just one line, despite how it might be formatted):

echo "m25.1|http://pkgs.tribblix.org/release-m25.1.sparc/TRIBzap.0.0.25.1.zap|Tribblix m25.1" >> /etc/zap/version.list

and check that's visible with

zap upgrade list

and then start the upgrade with

zap upgrade m25.1

Do not activate or reboot yet!

You MUST do the following:

beadm mount m25.1 /a
zap install -C /a TRIBshell-ksh93
pkgadm sync -q -R /a
beadm umount m25.1

and then you should be safe to reboot:

beadm activate m25.1
init 6

If it doesn't come back, you can boot into the previous release (that you took the name of earlier, remember) from the ok prompt

boot -Z rpool/ROOT/tribblix

Once you're up and running m25.1 it's time to clean up.

zap refresh

and then remove some of the old opensxce packages

zap uninstall \
SUNWfont-xorg-core \
SUNWfont-xorg-iso8859-1 \
SUNWttf-dejavu \
SUNWxorg-clientlibs \
SUNWxorg-xkb \
SUNWxvnc \
SUNWxwcft \
SUNWxwfsw \
SUNWxwice \
SUNWxwinc \
SUNWxwopt \
SUNWxwxft \
SUNWxwrtl \
SUNWxwplr \
SUNWxwplt

and then bring packages up to current

zap update-overlay -a

and this should give you a system that's in a workable state, and roughly matching my active SPARC environment.

Tribblix for SPARC m22 ISO now available The Trouble with Tribbles...

I've made available a newer ISO image for Tribblix on SPARC.

This is an m22 ISO. So it's actually relatively old compared to the mainstream x86 release.

I actually had a number of random SPARC ISO images, but for a while I've had no way of testing any of them. (And many of the problems with the SPARC ISOs in general is because I had no real way of testing them properly.)

Arrive a newish T4-1 (thanks Andy!), and I can now trivially create an LDOM, assign it a zvol for a root disk and a ISO image to boot from, and testing is trivial again. And while some of the ISO images I have are clearly so broken as to not be worth considering, the m22 version looks pretty reasonable.

In terms of available application packages, it exactly matches the old m20 release. I do have newer packages on some of my test systems, but they are built with a newer gcc and so need a proper upgrade path. But that's going to be easier now too.

There is a minor error on the m22 ISO, in that the xz package shipped appears to be wrong. To fix, simply

zap install TRIBcompress-xz

and to update to the latest available applications (the ISO is early 2020, the repo is middle of 2021)

zap refresh
zap update TRIBlib-security-openssl
zap update-overlay -a

The reason for updating openssl on its own is that a number of applications are compiled against openssl 1.1.1, so you need to be sure that gets updated first.

Next step is to push on to something newer.

Twitter, when the wall came down The Observation Deck

I, like many people, have a complicated relationship with Twitter. As Adam and I regaled in a recent Twitter Space, it started when debugging the Twitter fail whale in the offices of Obvious in 2007, where I became thoroughly unimpressed with their self-important skipper, Jack Dorsey. In part because I thought he was such a fool, I refused to join Twitter out of principle.

That changed when after I left Sun, and — acknowledging Twitter’s importance — I passively/aggressively created an account. I quickly learned to appreciate it: for example, at conferences (RIP, #surgecon), live tweets brought an important new element to technical presentations. After giving a talk, you could read feedback in real-time! While there was the occasional nasty remark, I found most of this engagement to be really valuable: because the people who had tweeted feedback on your talk were in the room, they tended to be pretty civil, and if they didn’t have anything nice to say, they didn’t say anything at all. (A reserve seldom summoned for YouTube comments, which turned into a hellhole.) And as an attendee of a talk, it remains fun to tweet the bits that really resonate — which I did as recently as a month ago during Rachel Stephens’s terrific talk at Monktoberfest.

When we started Oxide, Twitter was our marketing department: @oxidecomputer is how we have connected with all sorts of communities — technologists, fans, and future customers alike. It’s been fun to offer up some swag and especially gratifying to watch others discover what we’ve built!

And then came social audio. The Twitter Space that we started as a loony experiment that I talked Adam into has proved really important for Oxide: we have turned it into our Oxide and Friends podcast, which has allowed for some incredible discussions. With Spaces, Twitter felt fresh again: the Spaces team was really thoughtful (especially given that we were so frequently complaining about it!), and it was clear that the product just meant a lot to them — that they saw the same promise that we did. Twitter was flawed, of course, but things seemed to be generally headed in the right direction…

But then, Musk. It’s a tribute to the platform that the first place I wanted to discuss the takeover of Twitter was on the platform itself, and I started an impromptu space on Twitter’s fate. As I write this, that discussion was only a little over a week ago — but it sounds like it’s from another era: we were wildly off the mark about how much damage would be done in just a week!

Perhaps this shouldn’t have been surprising, but Musk has absolutely no idea what he’s doing, having screwed up the most basic element of the business: he doesn’t even know who the customer is! (It’s, um, the ad buyer, stupid.) Instead of doing what any sane new CEO of a troubled entity would do (namely, determining what changes need to be made by spending a bunch of time listening to customers, users, and employees — and then carefully plotting and executing those changes) he seems to be just… making it up as he’s going along. (Who knew that Stephen King is such an effective price negotiator?!) Maybe this would work where customers don’t have a choice or are locked into long contracts, but that isn’t the case here: customers can walk immediately. And advertisers themselves don’t want to be anywhere near controversy, which is why — as Josh Marshall points out — the Drudge Report never had mainstream advertisers, despite having plenty of eyeballs on it. Add to this that Twitter isn’t essential for advertisers and that the macroeconomic environment sucks, and it’s very easy to see how ad buyers would take a wait-and-see approach — reticence which, in the instant world of ad buys, means an immediate decline in revenue.

It’s bad enough that Musk seemed to be surprised by this, but what happened next is truly next-level in terms of executive incompetence: he reacted to the revenue drop by threatening those that are reducing their spend! (Does it need to be said that menacing customers who choose to buy less from you doesn’t really work in a free economy?) And as if this fecal pie were wanting for a topping, there is a fetid cherry on top: deep, bungled layoffs. Even if Musk could plot a path out of this self-inflicted disaster, he will likely lack the team and know-how to do it quickly enough to make a difference for controversy-adverse advertisers.

So it feels like we are at or near a tipping point. I will date myself here, but I am reminded of the extraordinary days leading up to November 9, 1989. Something that had seemed important but small — the opening of the Austrian-Hungarian border in August — led to a chain reaction of change, with each day seeing a larger change than the one preceding it, until it finally reached an unfathomable crescendo: the wall fell. If this predates you this may be hard to appreciate, but no one ever thought the wall would come down, let alone that Germany would reunify. For me personally, it was an early lesson in the dynamics of change: yes, the status quo can maintain itself for a very, very long time — but when change finally arrives, it can happen faster than anyone would have thought possible.

For Twitter, the wall is about to come down: the world is going to change — and it’s not going to change back. I keep wondering about “what is going to replace Twitter”, but I am increasingly of the belief that this is the wrong question, that no single thing is going to replace Twitter. That is, Twitter as an idea — a single social platform catering to all demographics and uses — will become like the evening nightly news or the morning newspaper: a relic from a bygone era. Instead, we will find different venues for different kinds of interaction: where there was one social network, there will be several, if not many — and not all will be open to everyone. Why am I confident about this coming fracturing? Because my kids — ages 18, 15 and 10 — think that this is just stupidly obvious. (As my 10-year-old daughter is fond of saying: “Duh, DAD!”)

So there will be a replacement for social audio (we will gladly pay for a quality service here!), and there will be a replacement for brands building their community, and there will be a replacement for those who want to shitpost or those that just want to engage with folks with similar interests — and they may or may not have overlap. But all of this still leaves me looking for something to replace that personal connection with people I have known over Twitter…

Like others, I created a Mastodon account in early 2017, but (frankly, like others) hadn’t used it really until this past week. And after a week… I like it! Yes, the Fediverse is different: it isn’t trying to glue your eyeballs to the screen, and it’s harder for things to go viral. There is less media, fewer memes, no advertising. And there are humans explicitly in the loop: Mastodon instances are moderated on an instance-by-instance basis — and should an instance descend into a hellscape, it may find itself defederated. But because of all of this, there is also less opportunism, less trolling, less dunking on your enemies, less nastiness. So it also feels more relaxing, more earnest — and easier to put down. It feels a little civic, like BBSs back in the day. It also feels raw, like Twitter did, years ago. And a good reminder that so much about Twitter — the retweet, the quote tweet, the tweetstorm — was invented by its users. So no, Mastodon isn’t the replacement for Twitter, but that’s a good thing, and I intend to be diverting my energies there.

While I won’t be using it as much, I will still be on Twitter, and it will still be around — but I don’t see how this embodiment makes it as a going concern. Musk will surely tire of it, and thanks to the debt that Musk has heaped upon it, it will go bankrupt. And then, like Friendster before it, it will probably be kicked around, bought and sold a few times, before it finally lands as a SlideShare-like ghost town, where you need to watch a 15 second ad before every dril tweet.

In the meantime, you can find me on Mastodon as bcantrill@mastodon.social. See you in the Fediverse — or wherever the new world finds us!

DevOps as a HR problem The Trouble with Tribbles...

I wrote about one way in which HR and IT can operate more closely, but there's another interaction between IT and HR that might not be so benign.

DevOps is ultimately about breaking down silos in IT (indeed, my definition of DevOps is as a cultural structure where teams work together to meet the needs of the business rather than competing against each other to meet the needs of the team).

However, in a business, individuals and teams are actually playing a game in which the rules and criteria for success are set by HR in the shape of the (often annual) review cycle. And all too often promotions, pay rises, even restructuring, are based around individual and team performance in isolation. And who can blame individuals and teams for optimising their behaviour around the performance targets they've been set?

It's similar to Conway's Law, in which the outputs of an organization mirror its organisational structure - here, the outputs of an organisation will mirror the performance targets that have been set. If you want to improve collaboration and remove silos, then make sure that HR are on board and get them to explicitly put those into the annual performance targets.

On the intersection between IT and HR The Trouble with Tribbles...

A while ago I mentioned The three strands of Information Technology, and how this was split into an internal-facing component (IT for the business, IT for the employee) and external-facing (IT for the customer).

In a pure technology company, there's quite a mismatch, with the customer-facing component being dominant and the internal-facing parts being minimised. In this case, do you actually need an IT department, in the traditional sense?

You need a (small) team to do the work, of course. But one possibility is to assign them not to a separate IT organization but to the HR department.

Why would you do this? Well, the primary role of internal IT in a technology company is simply to make sure that new starters get the equipment and capabilities they need on day one, and that they hand stuff back and get their access removed when they leave. And if there's one part of an organisation that knows when staff are arriving and leaving, it's the HR department. Integrating internal IT directly into the rest of the onboarding and offboarding process dramatically simplifies communication.

It helps security and compliance too. One of the problems you often see with the traditional setup where IT is completely separate from HR is that it can take forever to revoke a staff member's access when they leave; integrating the two functions massively shortens that cycle.