I'm so disappointed with the online accounting software options available to me; and I've spent far far too much time in the past couple of days looking for an accounting solution for my new business. The current state of affairs makes me wonder if just using a spreadsheet might be as easy.
I am posting my experiences here for two reasons.
My blahg uses nvlists for logging extra information about its operation. Historically, it used Sun libnvpair. That is, it used its data structures as well as the XDR encoding to serialize the data to disk.
A few months ago, I decided to replace libnvpair with my own nvlist implementation—one that was more flexible and better integrated with my code. (It is still a bit of a work-in-progress, but it is looking good.) The code conversion went smoothly, and since then all the new information was logged in JSON.
Last night, I decided to convert a bunch of the previously accumulated libnvpair data files into the new JSON-based format. After whipping up a quick conversion program, I ran it on the data. The result surprised me—the JSON version was about 55% of the size of the libnvpair encoded input!
This piqued my interest. I re-ran the conversion but with CBOR (RFC 7049) as the output format. The result was even better with the output being 45% of libnvpair’s encoding.
This made me realize just how inefficient libnvpair is when serialized. At least part of it is because XDR (the way libnvpair serializes data) uses a lot of padding, while both JSON and CBOR use a more compact encoding for many data types (e.g., an unsigned number in CBOR uses 1 byte for the type and 0, 1, 2, 4, or 8 additional bytes based on its magnitude, while libnvpair always encodes a uint64_t as 8 bytes plus 4 bytes for the type).
Since CBOR is 79% of JSON’s size (and significantly less underspecified compared to the minefield that is JSON), I am hoping to convert everything that makes sense to CBOR. (CBOR being a binary format makes it harder for people to hand-edit it. If hand-editing is desirable, then it makes sense to stick with JSON or other text-based formats.)
The blahg-generated dataset that I converted consisted of 230866 files, each containing an nvlist. The following byte counts are a simple concatenation of the files. (A more complicated format like tar would add a significant enough overhead to make the encoding efficiency comparison flawed.)
|Format||Size||% of nvpair|
I also took each of the concatenated files and compressed it with gzip, bzip2, and xz. In each case, I used the most aggressive compression by using -9. The percentages in parentheses are comparing the compressed size to the same format’s uncompressed size. The results:
|nvpair||471 MB||37.4 MB (7.9%)||21.0 MB (4.5%)||15.8 MB (3.3%)|
|JSON||257 MB||28.7 MB (11.1%)||17.9 MB (7.0%)||14.5 MB (5.6%)|
|CBOR||203 MB||26.8 MB (13.2%)||16.9 MB (8.3%)||13.7 MB (6.7%)|
(The compression ratios are likely artificially better than normal since each of the 230k files has the same nvlist keys.)
Since tables like this are hard to digest, I turned the same data into a graph:
CBOR does very well uncompressed. Even after compressing it with a general purpose compression algorithm, it outperforms JSON with the same algorithm by about 5%.
I look forward to using CBOR everywhere I can.
Doug Engelbart Institute — Online exhibits, historic videos, texts, archive photos, and stories about Doug Engelbart, the inventor of the mouse, hypertext, and GUIs…all in the 1960s
Completely Painless Programmer’s Guide to XYZ, RGB, ICC, xyY, and TRCs — Brain-hurting amount of information about color profiles, etc.
darktable — A Lightroom-like open source software
World plugs — Info about every electric plug form factor in the world
The client and the server must share knowledge that the connection is
ending in order to avoid a truncation attack. Either party may
initiate the exchange of closing messages.
This message notifies the recipient that the sender will not send
any more messages on this connection. Note that as of TLS 1.1,
failure to properly close a connection no longer requires that a
session not be resumed. This is a change from TLS 1.0 to conform
with widespread implementation practice.
Either party may initiate a close by sending a close_notify alert.
Any data received after a closure alert is ignored.
Unless some other fatal alert has been transmitted, each party is
required to send a close_notify alert before closing the write side
of the connection. The other party MUST respond with a close_notify
alert of its own and close down the connection immediately,
discarding any pending writes. It is not required for the initiator
of the close to wait for the responding close_notify alert before
closing the read side of the connection.
1. The exception here is historical tape devices, which might actually perform operations like rewinding the tape automatically upon close(). I think this semantic is probably lost in the mists of time for most of us.
Several years ago, I wrote about how Google gets traffic information and how to turn off this location reporting on Android phones. Since then I’ve switched to iPhones. While I normally use the built-in Maps app, I keep Google Maps installed as a fallback—just in case.
I upgraded my phone recently and so I spent some time going through all the apps and making sure they worked and didn’t have more access than necessary. This is when I discovered that the Google Maps app for iOS defaults to collecting location information and sharing it with Google. Given my previous post, this isn’t really surprising.
Anyway, as with the Android post, here are the steps to limit Google’s collection of location information.
First of all, in Settings → Privacy → “Location Services”, I changed Google Maps’s permission to “While Using the App”. If I’m not using the app, then it doesn’t need to know where I am.
Second, in the app itself, go to: Settings → “About, terms & privacy” → “Location data collection”. That’s right, this setting is buried in what appears to be a page with the boring legal notices.
And then turn off the the toggle:
That should do it…at least assuming that Google honors the settings in its own app.
On a dream team, there are no “brilliant jerks.” The cost to teamwork is just too high. Our view is that brilliant people are also capable of decent human interactions, and we insist upon that. When highly capable people work together in a collaborative context, they inspire each other to be more creative, more productive and ultimately more successful as a team than they could be as a collection of individuals.This policy isn't some useless feel-good text from a nameless source: it was originally published on CEO Reed Hastings' [slideshare account]. To be effective, such a policy for your company may also need to come from your CEO. All Netflix candidates are told to read the culture deck (memo) when interviewing, and are told that, yes, we take it seriously. While this helps people realize that jerks should not be tolerated, it doesn't necessarily stop jerks from being hired in the first place: Bob is brilliant and charismatic and would probably pass the interview. However, he would then find himself at a company where his colleagues recognize his bad behavior as unacceptable, and are empowered to speak up about it. In over three years at Netflix, I've worked with zero brilliant jerks. The "no brilliant jerks" policy works, and it's been great. If we hired any in that time, they either changed their ways or left the company before I could interact with them. Some brilliant jerks can mend their ways: Alice might be motivated to change if she can be made to understand that her behavior is hurting the company, which she cares about. She should be encouraged to exercise empathy, and to leave others feeling positive and motivated to work harder, rather than demotivated. My colleague, Justin Becker, explores this in detail in his QCon talk, including the topic of emotional intelligence (EQ). In the next section I'll share an example. As for Bob: he should be told that his behavior hurts people and the company, and given the opportunity to change – but the reality is that he probably doesn't care. He firmly believes that "nice guys finish last," and, so far, being a jerk has worked well for him. His managers have the power to change that equation, because they control things that Bob wants: they allow him to work on his pet projects and to speak at events, they give him promotions and bonuses, and ultimately they let him keep his job.
– Netflix culture memo
"I'd rather have a hole in my organization than an asshole."As a colleague/victim/witness, you should report Bob's behavior to management, but you probably shouldn't ask them outright to fire Bob (among other reasons, what if someday you were thought to be a Bob?). Give management information, but let them decide how to act. Actually firing a brilliant jerk is a complicated topic for a separate post, ideally written by a manager who has dealt with this. There's usually a process to follow, which unfortunately Bob may exploit to his advantage, showing improvement when needed to keep his job, but then reverting back to his bad old ways. He may also have convinced management that his technical skills and fame are so important that the company would fail without him. This isn't true, but fear may cause management to hesitate. For management to deal effectively with Bob, they must themselves be convinced that his behavior should not be tolerated, regardless of his brilliance. For some companies, that will require truly understanding the damage that Bob causes (listed above), to justify taking action. For companies like Netflix with an explicit "no brilliant jerks" policy, it's much easier for management to take action, as they don't need to convince anyone that jerks are a problem: that's already covered in company policy. Regular one-on-one meetings with staff, and scheduled skip-level meetings, should also help inform management about the damage jerks are causing. Netflix does this well: I have scheduled one-on-one meetings with my manager once every two weeks, their manager once a month, and their manager at least once a year. That's three levels of management I talk directly and in private with, without even having to ask for a meeting. We're also encouraged to give other employees direct and honest feedback, intervene if we see harassment, and escalate up to and including the CEO. As for public speaking: Bob draws power from being a public face of the company. Speaking events should be shared among staff who want to speak, and training can be made available to improve their skills (various companies offer this), so that Bob isn't the only good speaker. Conference organizers can also adopt a "no brilliant jerks" policy (some already do), and attendees can avoid conferences that host known jerks. If Alice needs to learn empathy, Bob needs to learn both empathy and to stop being selfish, and sharing public speaking or other rewarding projects is an example of the latter. ## When I acted like a jerk Many people sometimes act like Alice, and it can be easy to talk them out of it (Alice herself is harder, since it's her by-default behavior). I'll explain this with a story, this time of a moment when I acted like a jerk. Early in my career, an engineer at my company made a big mistake in my area of expertise, and sent an email that dodged responsibility and showed no path to fix it. I was furious and phoned the engineer: my intent was to make him realize that he'd made a big mistake, and put him on the right path. I was blunt, and told him off. I didn't enjoying doing so, but I felt I was doing a Good Thing for the company, and fixing a problem. A week later, his manager phoned me unexpectedly. He told me that he was aware of my phone call, and didn't think I was technically wrong, but did I know that the engineer has been demotivated and unproductive since I talked to him, and was it my intent to make his staff unproductive? No, of course not. The manager continued: do you think you could have told my engineer what you needed to, in a way that left them feeling positive and motivated to fix it? Sure, I probably can. Good. Always do that in the future, please. I did. The phone call lasted less than two minutes, and was immediately effective. I suspect the manager had done this before. Notice that he did not accuse me of being a jerk, rather, he posed two questions, which were basically: 1) are you intending to hurt the company?, and 2) are you able to act decently? There's only really one right answer to those questions. If he had just said "you are a jerk" I may have just replied "no, I'm not", but by asking questions instead, it put the onus on me to think about the answer, and triggered a moment of self reflection. ## Additional topics There are some additional topics I have not covered in detail here, but should mention: - **What about jerks in open source communities?** A good reference for this is the [no more rock stars] post, where the rock star described is pretty much Bob. See that post for the section on: How do we as a community prevent rock stars?. - **What about Bob as a manager?** Bob may seek and be offered promotions into management, and become even more damaging to the company. Bob the manager exploits and threatens his subordinates. That's a topic for another post. - **Does Bob sexually assault others?** Is Bob more likely to be a harasser due to delusions of grandeur and a sense of entitlement ([Al Capone theory]), or is he smart enough not to go that far, or, is he simply not that kind of jerk? I don't know. That's outside of my firsthand experience, so I didn't include it. - **What about those junior engineers?** The ones Bob exploits for his own gain. That's another big topic. Some related reading here, here, and here (from which I borrowed the words "reflecting glory"). - **Is Bob actually brilliant?** It's a little hard to tell, since Bob takes credit for the work of others. Bob also creates enemies in the industry, and some staff even sabotage Bob's work. In the long run, it hurts Bob's career. Not a brilliant result, really. - **What if Bob is pretending to be brilliant?** (Updated) The consequences for the company can be worse. I didn't explore this topic here, but it would be a character similar to Bob who isn't actually brilliant, but pretends to be, and has most people believing him. To quote from a HN comment: "The myth of "brilliant jerks" is harmful because it lets any jerk pretend he's doing it because he's brilliant, when chances are he's just afraid of being unmasked as mediocre.". - **Does anyone exist who is really as bad as fictional Bob?** Yes. Fortunately they are rare. One reviewer thinks that my post will not be effective unless I name such a real-life Bob as a concrete example. Maybe they are right, but I've avoided that here. This isn't about one Bob, it's about all selfish brilliant jerks. - **Should we publicly call out brilliant jerks in tech?** It's a complex topic. The book [Is Shame Necessary] does make the point that shame and humiliation have a legitimate place in society when they are natural consequences to abusive behavior, which is also discussed in this post. Calling out abusers may save future victims of abuse, so long as the calling out is proportional and not abusive itself. For victims of abuse, I could not make a blanket recommendation: I don't know your specific situation and how safe it is for you to speak up. I've been in this situation myself, and was facing an extreme threat to myself and my family, and I understand how risky it can be. - **How do I know if I'm the jerk?** If you always think that being right is all that matters, and don't consider your impact on teamwork or relationships, you might be an Alice or a Bob. If you think that hurting other people is simply doing what it takes to get ahead, then you might be a Bob. See the earlier sections for more characteristics, and my colleague's QCon presentation: [Am I a Brilliant Jerk?] ## Conclusion Should brilliant jerks be tolerated? To explore this, I described two fictional brilliant jerks: Alice, who is selfless, and Bob, who is selfish. This makes it clear that the behavior of selfish jerks, like Bob, should definitely not be tolerated. Bob can kill companies. When CEOs and VCs sometimes say that brilliant jerks may be worth it, I imagine they are thinking of Alice, a selfless jerk, and not Bob. (Alice is debatable.) Early on in my career, I supported brilliant jerks of any type and thought they were worth it. I was wrong. People had warned me about them, that their behavior was "not ok," but they never went into much detail as to why. I've shared many details here. I didn't figure this all out until seeing the behavior and damage firsthand. (I've not only experienced it, but I may have reached my lifetime dosage of asshole-rads.) Companies can adopt a "no asshole rule", or more politely, a "no brilliant jerks" policy. Colleagues may be genuinely conflicted about how to deal with Bob: on the one hand, he is a real jerk, but on the other he is a "high performer," so isn't it in the company's best interest to tolerate his behavior? A policy helps you decide, and it can be as simple as three words: no brilliant jerks. ## Acknowledgements Picture and editing by Deirdré Straughan. Thanks to review feedback and suggestions from Alice Goldfuss, Baron Schwartz, Ed Hunter, Justin Becker, David Blank-Edelman, Valerie Aurora, and others. References and related reading: - Sutton, R. The No Asshole Rule. Grand Central Publishing, 2010. - Babiak, P., Hare, R. D. Snakes in Suits. Harper Business, 2007. - Jacquet, J. Is Shame Necessary. Random House, 2015. - Sutton, R. The Asshole Survival Guide. Houghton Mifflin Harcourt, 2017. - https://hbr.org/2015/12/its-better-to-avoid-a-toxic-employee-than-hire-a-superstar - https://retrospective.co/brilliant-jerks-cost-more-than-they-are-worth/ - http://boz.com/articles/be-kind.html - https://hypatia.ca/2016/06/21/no-more-rock-stars/ - https://www.youtube.com/watch?v=fJOSX-W0yHA&feature=youtu.be&t=10m53s - https://www.slideshare.net/reed2001/culture-1798664/36-Brilliant_Jerks_Some_companies_tolerate - https://jobs.netflix.com/culture - https://qconsf.com/sf2017/presentation/am-i-brilliant-jerk - https://hypatia.ca/2017/07/18/the-al-capone-theory-of-sexual-harassment/ - The blood bag series: part 1, part 2, part 3 - (Updated) discussion on hackernews [Am I a Brilliant Jerk?]: https://qconsf.com/sf2017/presentation/am-i-brilliant-jerk [test]: http://en.wikipedia.org/wiki/The_No_Asshole_Rule [Be Kind]: http://boz.com/articles/be-kind.html [Code of Conflict]: https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=b0bc65729070b9cbdbb53ff042984a3c545a0e34 [No Brilliant Jerks]: https://www.slideshare.net/reed2001/culture-1798664/36-Brilliant_Jerks_Some_companies_tolerate [Brilliant Jerks Cost More Than They Are Worth]: https://retrospective.co/brilliant-jerks-cost-more-than-they-are-worth/ [abc]: http://www.recode.net/2017/2/22/14700114/is-uber-lost [No Asshole Rule]: http://amzn.to/2zitvVd [Al Capone theory]: https://hypatia.ca/2017/07/18/the-al-capone-theory-of-sexual-harassment/ [no more rock stars]: https://hypatia.ca/2016/06/21/no-more-rock-stars/ [memo]: https://jobs.netflix.com/culture [slideshare account]: https://www.slideshare.net/reed2001/culture-1798664/36-Brilliant_Jerks_Some_companies_tolerate [when is naming abuse itself abusive]: https://blog.valerieaurora.org/2016/10/24/when-is-naming-abuse-itself-abusive/ [Paul Graham]: https://youtu.be/UacbJ72dluU?t=4m18s [CEOs]: http://www.businessinsider.com/dick-costolo-on-why-companies-hire-brilliant-jerks-2017-5 [PCL-R]: https://www.sociopathicstyle.com/psychopathic-traits/ [Is Shame Necessary]: https://www.amazon.com/Shame-Necessary-New-Uses-Tool/dp/0307950131
– Fred Wilson, Velocity NY 2013 keynote.
First off, I'm a developer of open source application libraries, some of which are fairly popular.
TLDR: Library developers should not use ExternalProject_Add, but instead rely on FindPackage, demanding that their downstream developers pre-install their dependencies.
|Bon Apetit, CMake lovers!|
|And Gophers everywhere rejoiced!|
Let me start by saying this... I hate the GPL. Oh yeah, and a heads up, I am just a software engineer, and not a lawyer. Having said that....
I've released software under the GPL, but I never will again. Don't get me wrong, I love open source, but GPL's license terms are unaccountably toxic, creating an island that I am pretty sure that original GPL authors never intended.
A problem I looked at recently involved configuring a system to send (relay) email via a customer's own SMTP servers. There are 2 parts to this:
# /usr/lib/sendmail -bt -d0.1 < /dev/null
Compiled with: DNSMAP LDAPMAP LOG MAP_REGEX MATCHGECOS MILTER MIME7TO8
MIME8TO7 NAMED_BIND NDBM NETINET NETINET6 NETUNIX NEWDB NIS
PIPELINING SASLv2 SCANF STARTTLS TCPWRAPPERS USERDB
250-AUTH GSSAPI DIGEST-MD5 CRAM-MD5
m4 ../m4/cf.m4 /tmp/sendmail.mc > /tmp/sendmail.cf
makemap hash smarttable < smarttable
makemap hash authinfo < authinfo
cp /tmp/sendmail.cf /etc/mail
svcadm restart sendmail
Authinfo:smtp.gmail.com "U:root" "I:email@example.com" "P:mypassword" "M:LOGIN PLAIN"(There are just 2 lines there, starting with Authinfo:, even if the blog shows it wrapped.)
Authinfo:smtp.gmail.com:587 "U:root" "I:firstname.lastname@example.org" "P:mypassword" "M:LOGIN PLAIN"
This is a follow up to my previous post outlining my chromebook setup.
I managed to get IPMItool to compile in my Termux environment. Here's how you can too. (So far I've only been able to get this to work on an x86 Chromebook, but not on ARM phones.)
apt update apt upgrade apt install coreutils pkg upgrade pkg install termux-tools proot util-linux net-tools pkg install openssh tracepath tree git
pkg install automake autoconf libtool sed grep clang openssl-dev readline-dev
git clone https://git.savannah.gnu.org/git/gnulib.git git clone https://github.com/nshalman/ipmitool-source cd ipmitool-source
./bootstrap-gnulib ./bootstrap ./configure make
src/ipmitooland you can proceed to manage your servers with IPMI from your Chromebook.
Having talked about running Tribblix on AWS, one of the things that would be quite neat would be to be able to build illumos-gate.
This is interesting because it's a relatively involved process, and might require proper resources - it's not really possible to build illumos inside VirtualBox, for instance, and many laptops don't run illumos terribly well. So it's hard for the average user to put together a decent - most likely dedicated - rig capable of building or developing illumos, which is clearly a barrier to contribution.
Here's how anyone can build illumos, using Tribblix.
Build yourself an EC2 instance as documented here, with 2 changes:
zpool create storage c2t5d0
zfs set compression=lz4 storage
zfs destroy rpool/export/home
zfs create -o mountpoint=/export/home storage/home
zfs create -o mountpoint=/export/zones storage/zones
zap update-overlay -a
zap install-overlay develop
groupadd -g 10000 it
useradd -g it -u 11730 -c "Peter Tribble" -s /bin/tcsh \
-d /export/home/ptribble ptribble
mkdir -p /export/home/ptribble
chown -hR ptribble:it /export/home/ptribble
zap create-zone -z illumos-build -t whole \
-i 172.xxx.xxx.xxx -o develop \
-O java -O illumos-build -U ptribble
cd /usr/bin ; ln -s ../gnu/bin/xgettext gxgettext
git clone git://github.com/illumos/illumos-gate.git
wget -c \
bzcat ../on-closed-bins.i386.tar.bz2 | tar xf -
bzcat ../on-closed-bins-nd.i386.tar.bz2 | tar xf -
cp usr/src/tools/scripts/nightly.sh .
chmod +x nightly.sh
pfexec zlogin -l ptribble illumos-build
time ./nightly.sh illumos.sh
There's now a public Tribblix AMI available to run on AWS.
This was built according to the notes I gave earlier. And is part of making Tribblix the illumos for everyone.
This is to be considered slightly experimental, and there are a couple of constraints:
First, the AMI is only available in the London region for now (I'm in the UK, so that's where I'm running things). I could make it available elsewhere, but there are costs associated with doing so and, as everything related to Tribblix comes out of my own pocket, I'm not going to incur costs unless there's a demonstrable need. If you want to run in a different region, then you can always copy the AMI.
Second, the size of the image is quite small. Again, there's a constraint on cost. But the idea here is that you wouldn't store any non-trivial data in the image itself - you would create an appropriately sized EBS volume, attach that and create a zfs pool for your data. The Tribblix repo server does just that - the package repo lives on the second pool.
So, how to use this? I'm going to assume some level of AWS familiarity, that you have an account and know basically how to use AWS, and that your account is set up with things like an ssh key pair.
Go to the AWS console, and navigate to the EC2 dashboard. Unless you've copied the AMI to the region of your choice, make sure you're working in London - the dropdown is in the top right:
ssh -i peter1-london.pem \
For my first trip to Paris I gave the closing keynote at [EuroBSDcon 2017] on performance methodologies, using FreeBSD 11.1 as an analysis target. In the past I've shared similar methodologies applied to other operating systems, and finished porting them to BSD for this talk. It was a few days of work, which is really not bad. That's a virtue of these methodologies: once you learn them, you can apply them to anything throughout your career, and it doesn't take too much time to re-apply them. The video is on youtube:
# ./tstates.d Tracing scheduler events... Ctrl-C to end. ^C Time (ms) per state (read script for info): COMM PID CPU RUNQ SLP USL SUS SWP LCK IWT YLD irq15: ata1 12 0 0 0 0 0 0 0 15024 0 [...] sleep 877 0 0 505 0 0 0 0 0 0 bufdaemon 19 0 11 0 15057 0 0 0 0 0 sleep 879 0 0 2614 0 0 0 0 0 0 devd 523 0 0 15024 0 0 0 0 0 0 syncer 21 1 9 0 15055 0 0 0 0 0 fsck_ufs 878 1 0 0 10 0 0 0 0 0 fsck 836 1 0 12 0 0 0 0 0 0 dd 883 2 0 0 0 0 0 0 0 0 bufspacedaemon 20 3 5 0 15019 0 0 0 0 0 dtrace 873 3 23 15980 0 0 0 0 0 0 sh 881 4 0 3 1 0 0 0 0 0 csh 865 5 7 13882 0 0 0 0 0 0 rand_harvestq 6 8 20 0 15846 0 0 0 0 0 kernel 0 29 15 0 0 0 0 0 0 0 cam 4 45 14 0 0 0 0 0 0 0 sshd 863 52 85 13757 0 0 0 0 0 0 intr 12 79 192 0 0 0 0 0 0 0 cksum 876 1591 177 0 234 0 0 0 0 0 idle 11 14114 1902 0 0 0 0 0 0 0This tool breaks down thread time into different states by tracing scheduler events (which can have noticeable overhead: measure in a lab environment before use). The states are: - **CPU**: on-CPU - **RUNQ**: Waiting on a CPU run queue - **SLP**: Interruptible sleep - **USL**: Uninterruptible sleep (eg, disk I/O) - **SUS**: Suspended - **SWP**: Swapped - **LCK**: Waiting for a lock - **IWT**: Waiting for an interrupt - **YLD**: Yield I added USL since the talk to split out disk I/O from the sleep state. The output above includes a ```sleep 0.5``` command, and a ```cksum```. EuroBSDcon was a great conference, and I had a lot of fun catching up with the BSD folk and meeting new people. If you missed my talk, you can see it online above, and I hope you find it useful. [EuroBSDcon 2017]: https://2017.eurobsdcon.org/about/ [thread state analysis]: http://www.brendangregg.com/tsamethod.html [tstates.d]: https://github.com/brendangregg/DTrace-tools/blob/master/sched/tstates.d