A couple weeks ago I was in Nashville, Tennessee for the first time, I carved out a little tourist time but my real reason for being there was LISA18!
This is one of the handful of conferences I attended this year and didn’t speak at, but I was one the talks co-chair, so I was happy to be able to attend and see the results of the conference I had a hand in putting together, which is where I’ll begin. LISA is a USENIX conference, and I’ve been a member of USENIX for several years. As a systems administrator, I was a member of SAGE (The System Administrators Guild), was on a LISA panel in 2012, and spoke at LISA in 2015. I was asked about my interest in being a talks co-chair this year over the winter, and we started calls about the event in March. The next couple months were spent scouring my contacts lists and recalling my favorite systems talks from the past couple years to invite people to submit proposals. At the end of May, the CFP closed and it was time to review proposals! There were over 325 of them, and I read every single one. We also had a small army of program committee members who each had an assigned list to review and vote on, along with the training co-chairs, and of course the overall program chairs, Rikki Endsley and Brendan Gregg. In early July, the program, talk and tutorial chairs met in Boston to finalize our selections and put together a schedule. With the schedule in place, my work as talks co-chair over the final months before the conference was mostly just helping select from alternates when we had speakers who couldn’t make it.
I’ve participated in a number of conferences at this point, and this was one of my best experiences. Everyone I worked with was friendly and worked hard to put together a solid line-up. The USENIX staff were super helpful every time we had questions and then handled the conference logistics so all the volunteers could focus on our specific roles. I’m honored to have had the opportunity to participate and hope I can participate in more USENIX work in the future.
As for the event itself, I had a great time! It began on Monday, October 29th with an introduction from the program chairs before launching into the first two keynotes. First up was Jon Masters who gave a talk on Meltdown and Spectre. While I haven’t been paid to do systems administration these past two years, I’ve continued to run several servers for the Ubuntu community, and I run a few of my own. As such, the vulnerabilities have impacted me enough that I’ve had beyond just an academic curiosity in both. Still, I appreciated the time he spent getting into the details of how the vulnerabilities worked and why that matters, as well as his call for folks working in software and hardware to be more explicit in their communications. As developers and systems engineers we can’t afford to continue treating hardware as a magical black box, and we need to more carefully consider the security cost of software-driven speed improvements.
The second keynote was from Tameika Reed on The Beginning, Present, and Future of Sysadmins. If the “Systems Administration” job title was ever cool, it’s certainly not now, in spite of the job still very much existing. In this funny and thoughtful talk, Tameika took us through the key skills that made a good systems administrator, and how they’re all still relevant in the world of automation, SREs, and DevOps methodologies. The specific implementations may have changed, but we still need backups, monitoring, security and more. The key takeaway: “Your skillsets still apply today but you have to evolve.”
Talk-wise the first day I went to a mix of culture talks and container talks. Culture was an import category for LISA this year, and I took time to hear from Emily Gladstone Cole on strategies to be your security team’s best friend, Kurt Andersen on steps to reduce operational toil on a team, and Rich Bowen on transferring ownership and leadership of a project. On the containers side, I enjoyed meeting and hearing from Michael Jennings of the Los Alamos National Laboratory in a talk where he explored the use of containers in HPC (High Performance Computing) and the current projects tackling the needs of HPC, including Charliecloud. I also had the pleasure of being session chair for David Morrison’s talk, where he talked about the use of Apache Mesos at Yelp.
The second day of keynotes brought the topic of social consciousness to the forefront. In the first, Dr. Sarah Lewis Cortes took us on a tour of the Darknet and explored recent cases of security vulnerabilities that have cost hundreds of millions in lawsuits to the retailers in question. In these security breaches, initial access that opened the door to attackers was accessed via a third party, so even if you’re not a retailer handling credit cards, we need to be conscious of how the security of our systems is impacting those we interact with. Directly following her talk, we heard from Jeffrey Snover who spoke more directly to social responsibility. It’s easy to get hyper-focused on solving a problem in your small corner of your code base, but stepping back from time to time to understand the broader implications of what you’re writing from an ethics and security perspective is important. With software “eating the world” we’re no longer just writing and running software, we’re writing and running key infrastructure that the world today operates on.
That afternoon I was a session chair again, and of particular note I was able to introduce Daniel J. Walsh of SELinux fame, who has now been working on containers and container security. One of the biggest takeaways from his talk was the tendency to run containers as root on your host system, and stressed that just ceasing this practice could probably be the best possible thing you could do to improve container security. His opensource.com article from the spring gives more details: Just say no to root (in containers).
Tuesday also brought us the LISA social! I’m not exactly a social butterfly, so I was happy to get settled in with a couple folks I knew, and their extended group, for food and interesting conversation. Plus, they had DIY(ish) LISA18 t-shirts featuring a raccoon, the state animal of Tennessee!
The final day of LISA landed on Halloween, and several conference-goers dressed up, which was a lot of fun. Unfortunately I wasn’t one of them, the contents of my suitcase covered both my trip to All Things Open and LISA (did laundry in between) and I had to limit my suitcase size to account for pregnancy-related weight restrictions in what I could haul around. I was delighted to see Vice Admiral Holdo and Rainbow Dash suitably represented among the attendees, as well closing keynote speaker Tom McLaughlin dressed as David S. Pumpkins.
Talk-wise, it was lovely to hear from Sabice Arkenvirr, who used her experience fixing up complicated infrastructure messes to stress the value of using off-the-shelf open source solutions rather than coming up with home-brewed alternatives that no one else in the industry would know how to use, and often end up being immature compared to the common solutions. She also wrote an opensource.com article on the topic, We already have nice things, and other reasons not to write in-house ops tools!
I also enjoyed the talk from Steve Mushero on Taking Over & Managing Large Messy Systems. At these conferences we very appropriately spend a lot of time focused on stories of doing things right, and highlighting the forward-thinking work of major engineering organizations that set the pace for the industry. Unfortunately, the result of this is sometimes losing sight of what folks in operations are seeing in the rest of the industry, which can often be poorly implemented logging and backups, security problems, and poor maintenance. Steve’s talk put a spotlight on the problems that most of the industry still deals with, and talked about strategies to quickly triage the worst of them.
The conference concluded with a pair of keynotes. The first was on serverless from Tom McLaughlin. Tom is great and I enjoyed hanging out with him before the conference, but I’ll say right up front that we’re not on the same page when it comes to serverless. He makes a lot of great points about moving past the need to do system-level infrastructure work (leave that to the experts, who you can pay) so you can focus on just building your application on the platform they provide. Alas, as a strong proponent of open source, I can’t get over the lock-in and proprietary nature of serverless solutions. Sorry, Tom! I will concede that it’s something that companies may consider, but with the knowledge of how dependent they are on a third party with regard to reliability, price, features, and more. Nora Jones of Netflix concluded the conference with a technical and cultural dive into Chaos Engineering at Netflix.
The conclusion of the conference was a bit sad because of how much I enjoyed being a part of it! But I had a lovely evening out to vegan dinner with Brendan and Rikki before a little after-event enjoying of a non-alcoholic beer with others from USENIX and the LISA organizing crew.
More photos from LISA18 can be found here: https://www.flickr.com/photos/pleia2/albums/72157673027760167