Linux Conf Au – 2019

The yearly reminder that we need to question our values as programmers has come and gone again. That’s right – the far left of the programming world has come to remind us of the Four Freedoms and how most of us there like the idea of them, but fundamentally earn a paycheck either not respecting them, or outright disregarding them. Aside from cognitive dissonance, I did learn some very interesting things – some of which are even applicable to my day-to-day job!

The best one-liner of the conference goes to Stewart Smith, who was describing how boot loaders were becoming much more complex, how they have to verify a signed image over a constrained bus before the main memory was brought up, and how that was a difficult problem. Upon being asked why he didn’t just use UEFI as it was a solved problem, he replied “because everyone who holds UEFI finds that it doesn’t spark joy”.

The most applicable talk that I went to, was given by Katie McLaughlin – an Operations Engineer who has learned through experience how to “Be kind to 3am you”. This is, of course, a talk on the lessons learned to help you resolve issues in production when the pager goes off at 3am when few people are at their best.

She explains that where the documentation lives is important – it should be the default place that you go to every day. If your fingers “just know” from muscle memory how to search the documentation when you are both desperately in need of coffee, but stubbornly refusing it in the hope that the issue will be resolved soon and sleep might be an option in the near future, then your life is infinitely better. The instructions also need to be clear for similar reasons.

The theme of the talk was that generally lowering the cognitive load required to address the situation was the main objective. Having commands that you can copy and paste from a document and run without any change is ideal. For example, docker –stats is subtly different to docker –status ”Don’t run with scissors” was the advice – don’t do dangerous things because you feel like you need to get somewhere fast.

The piece of learning that I thought would be useful, was that when we are setting up alerts in our monitoring software, that we should include a link to the documentation that helps to resolve it in the alert itself. If it doesn’t exist, we should create it – we are alerting for a known reason, so we should know ways to mitigate / resolve the issue.

Oh and I almost forgot – we should consider saving a sanitized bash history directly in the post mortems – it gives a clearer idea of what we did (human memory is terribly unreliable) and gives juniors / new employees a starting point for learning how to handle things.

The talk that was the most interesting gedankenexperiment was where Peter Chubb played with the idea of writing a distributed filesystem using an RDBMS as the underlying storage. The idea is relatively simple – configure the database to use raw disk (rather than pay the penalty for a filesystem layered on a filesystem) and then implement the POSIX interfaces with a combination of FUSE and SQL. Simple, right? Well, the performance was fairly poor, but yes, he managed to have it more or less work and be more or less POSIX compatible.

The two talks that encouraged the most self-reflection we “I’m sorry Dave, I can’t do that – ethics in software development” by Dr Morgan Leigh and “Facebook, Dynamite, Uber, Bombs and you” by Lana Brindley. Each were reminding us that we, as software professionals, wield a large amount of power and could we please try to grow our morality glands. Software basically gets away with anything because there is neither a governing organization holding us to specific standards (such as the FAA for aviation) nor the understanding that software can be held accountable for such things. It didn’t take too many bridge collapses before engineers got oversight, so why is it that software is getting away with so much? These are the philosophical questions posed alongside the plea to realise that each of us can make moral choices – walking away from a job is not nearly as much of a burden to us as it might be for other professions.

A special mention should go to “Autopsy of a Browser” by Erin Zimmer, which detailed the history of browsers from the very first one written by Sir Tim Berners-Lee at Cern to the modern day, where everyone seems to be converging on using the <blink> HTML engine from the chromium project (with a couple of notable exceptions) with the warning that we’re repeating the history of 2001-2006 where Internet Explorer 6 dominated and the web stagnated as a consequence. Personally, I think the talk was much closer to a homicide investigation, or possibly a war crimes tribunal. Autopsy just doesn’t have the grand scale that she talked about. Her talk did, however, nicely explain why some of the HTML today is the way it is (most notably the img tag). Market share and stubbornness.

But the two stand-out talks that I didn’t expect to be so well presented or approachable were “The Kernel Report” by Jon Corbet and “The tragedy of Systemd” by Benno Rice. I have seen each of them give conference talks before, so I shouldn’t be surprised that they were skilled presenters, but I was mostly surprised given their topics. With tech that would have been very tempting to wander off into the deep bowels of linux as their subjects, each of them managed to make their talks approachable to the likes of me, who has never contributed a patch to either system, and found a great sense of pride in being able to just compile a kernel in my younger days, when I had the time and energy for such whimsy.

Both talks were approached with similar levels of empathy to the aforementioned philosophical discussions, which feel like they would somehow be easier to approach in that manner. Though I can’t actually explain why, and I wonder if that’s me being biased, given that each of those two were presented by women, whereas these two were presented by men and perhaps they should get some sort of bonus for not acting like unsympathetic techno-philes.

I guess now is the time I wrap up, but before I do, I will mention the organisers – they did a fantastic job. There were some things that I would have done differently, but differently is not necessarily better, and I’m sure they did what they could with the resources they had. Overall it was a well-oiled machine, with the ultimate proof being that all the talks (as far as I’m aware) happened at the time specified, were recorded and began appearing online with less than 24 hour lags. As far as conference metrics go, you can’t do much better than that. So I enjoyed being reminded that I have a morality gland and I should choose to wield the incredible amount of power that I have for good. I am a straight white male in software, after all.