I had the pleasure of participating in SeaGL this year. This conference was a special one in my calendar because it’s the last conference on my schedule before our son arrives, my key participation was in the form of giving one of the keynotes, and they were very transparent about their efforts to make a diverse schedule and shared the acceptance results. According to their statistics, 44% of the talks the accepted were from people who self-identified as a member of an under-represented group in tech.

Diversity is important to me, and not just on a feel good level of finally giving voice to people whose views and perspectives have traditionally been marginalized or ignored. I enjoy the energy and welcoming feeling that I find diverse conferences. The diversity of speakers also shows through in the talks themselves, as people with different backgrounds and life experiences will often give very different styles of talks without (and this is important!) compromising on quality or level of technical depth. It makes the experience of spending a day drifting between technical talks much more enjoyable.

So I’ll start by saying thank you to the organizers of SeaGL for the work they put into crafting such an excellent line-up of diverse speakers. Your efforts were noticed and I had a wonderful, educational time as a result.

The keynote I put together for this conference was months in the making. As I reflected on the work I’ve done in open source communities over the past 17 years, I’ve lived through a massive shift in the types of projects, contributors, and companies involved. I also found myself wondering where an individual contributor from the year 2001 would fit into this modern landscape. When I pitched this idea to the conference organizers as a topic for my talk, I only really had my own observations and random chats with other people whose background is similar to mine at conferences over the past couple of years, most of whom were feeling they had lost motivation to contribute outside of their work in tech. In preparation for this talk, I strategically expanded my pool of people to talk to, and also sent out an anonymous survey asking for feedback from anyone who had thoughts to share about their motivations.

Thanks to Brian MacDonald for taking this photo during my keynote (source)

With all this data collected, I had an interesting picture of the state of open source today. The gist of my keynote was that open source has changed since the creation of the term 20 years ago. Some of these ways are uncomfortable for people who have been around that whole time, especially with regard to the influx of money that has caused the environment to change in significant ways. However, there is a lot of good that comes of this too. A lot of us are now paid to work on open source and the opportunity to be paid has opened the doors to folks who may not have traditionally been able to participate. The adoption and support by mainstream firms has made it a more compelling option for non-profits and other local organizations that are rewarding to work with. The investment by for-profit companies in the development tooling landscape is making it easier for contributors to work across projects. I covered these things and more in a hopeful view into the present and future role individuals play in open source today. Slides from my talk can be found here: SeaGL2018-OpenSourceToday-IndividualsNeedNotApply.pdf.

My keynote was followed by one from Molly de Blanc who took us on a very personal glimpse into how technology can be a positive influence mental health, especially when it comes to connecting us instantly to our support networks. On the software freedom side, she explored choices we make with regard to our privacy and security, and trade-offs we may be encountered with as we use different platforms and tools.

With this being my last conference for a while, I went to a couple talks that first day, but ended up spending most of the rest of the day in the “hallway track” as I enjoyed lunch with some folks who worked in and around Debian, and generally catching up with people I’ve known from past jobs and communities.

Day two kicked off with a keynote from Stephen Walli on The Democratization of Software. He looked back at history to demonstrate that freely shared software was more the norm than the exception, and discussed how frictionless it is for anyone to get involved with developing software these days if given the time. Getting access to the code and improving upon it is now essentially a solved problem. Our real challenge today is operating the software securely and reliably at scale, and that’s more difficult. He posited that several of the major vulnerabilities in open source software were not in fact failures in open source, but our methods for testing and deploying the software into production. The problems caused by something like left-pad being removed from NPM shouldn’t have happened on production systems because it should have been tested and the dependency chain thoroughly vetted.

Next we heard from Tameika Reed, Linux systems engineer and founder of Women In Linux. I had the pleasure of hearing from Tameika just a few weeks earlier as she gave one of the keynotes at LISA in Nashville, and it was a pleasure to not just see her again, but sit through one of her keynotes filled with wisdom and humor. In this talk the focus was on making the mental shift from System Administration to System Architect or Team Lead. She stressed the importance of how your view into the work your doing has to change from focusing on a small component of the technical work to stepping back to get the 30,000 foot view. From there you dril down into that to get a view of how the work of your team impacts the organization. She also stressed the human side of a more senior or management role, and that having empathy was key.

After the keynotes I found myself really enjoying a talk by Lucy Wyman about being tactful about what you automate in your infrastructure. Putting pressure on people who want to “automate everything” she outlined some times when you might not want to, like when the complexity of the task means that you have more exceptions to your automation strategy than rules, and would spend a lot more time reconfiguring your automation than you’d actually find value from. What I really loved about this talk though was her invitation of audience participation. In addition to sharing thoughts at key points in her presentation, her findings were validated by real stories from the audience about times that automation went terribly wrong. Plus, they were often really funny, if painful, stories that I think made for a dynamic presentation that took on a life of its own.

I also attended a talk from Duane O’Brien on the risks of heavy handed use of GitHub profiles and activity when evaluating candidates for roles in your organization. There may be a place for it as one piece of criteria that you check, but it excludes a lot of open source projects, disadvantages people who couldn’t work in the open in past roles or don’t have time to do open source work during their personal time. Aricka Flowers does a great job of writing a whole article on the talk over on the GitLab blog: We all love open source, but hiring based on contributions could be harmful.

After Duane, I was thrilled to hear a talk from Carla Schroeder. I met Carla online via LinuxChix well over a decade ago. I pitched in with editing one of her books long before I was known professionally and that gave me an incredible confidence boost at a time I really needed it. As someone who open doors for me so early in my career, I’m eternally grateful and was happy to finally meet her in person. In her talk on documentation, she went through the characteristics of good documentation, several suggestions for documentation layout and linking strategies, and the importance of crafting your pages for good SEO.

The final slot in the schedule went to a very popular talk: The Tragedy of systemd by Benno Rice. Even before the conference I knew this is one of the talks I wanted to see. As a FreeBSD developer, Benno brought an interesting perspective to the discussion of systemd, along with Unix history and the path that Apple’s OSX took with launchd. I am still unconvinced that systemd is the solution to our increasingly complicated world of init systems, but his talk did bring up some interesting points about how we’ve approached them in the past and the value that systemd creator Lennart Poettering brought to the Linux world by introducing systemd. He recommended Poettering’s Rethinking PID 1 which digs into the guts of the philosophy behind systemd and went on to address many of the criticisms. I certainly appreciated the talk a lot and while this delivery wasn’t recorded, his version of the talk given at BSDCan was (Benno Rice: The Tragedy of systemd — BSDCan 2018) and I’ve sent it to a few people already. Thanks to Benno for approaching such a controversial topic with evidence and positivity, I learned a lot.

Sadly, that concluded the conference for me! There was an after party, but I had decided to head home that night so I could sleep in my own bed and avoid the additional hotel night expense.

More photos from SeaGL can be found here: https://www.flickr.com/photos/pleia2/albums/72157675535928818

Thanks again to all the organizers and other volunteers for making my last conference for a while such a great experience!

A couple weeks ago I was in Nashville, Tennessee for the first time, I carved out a little tourist time but my real reason for being there was LISA18!

This is one of the handful of conferences I attended this year and didn’t speak at, but I was one the talks co-chair, so I was happy to be able to attend and see the results of the conference I had a hand in putting together, which is where I’ll begin. LISA is a USENIX conference, and I’ve been a member of USENIX for several years. As a systems administrator, I was a member of SAGE (The System Administrators Guild), was on a LISA panel in 2012, and spoke at LISA in 2015. I was asked about my interest in being a talks co-chair this year over the winter, and we started calls about the event in March. The next couple months were spent scouring my contacts lists and recalling my favorite systems talks from the past couple years to invite people to submit proposals. At the end of May, the CFP closed and it was time to review proposals! There were over 325 of them, and I read every single one. We also had a small army of program committee members who each had an assigned list to review and vote on, along with the training co-chairs, and of course the overall program chairs, Rikki Endsley and Brendan Gregg. In early July, the program, talk and tutorial chairs met in Boston to finalize our selections and put together a schedule. With the schedule in place, my work as talks co-chair over the final months before the conference was mostly just helping select from alternates when we had speakers who couldn’t make it.

I’ve participated in a number of conferences at this point, and this was one of my best experiences. Everyone I worked with was friendly and worked hard to put together a solid line-up. The USENIX staff were super helpful every time we had questions and then handled the conference logistics so all the volunteers could focus on our specific roles. I’m honored to have had the opportunity to participate and hope I can participate in more USENIX work in the future.

As for the event itself, I had a great time! It began on Monday, October 29th with an introduction from the program chairs before launching into the first two keynotes. First up was Jon Masters who gave a talk on Meltdown and Spectre. While I haven’t been paid to do systems administration these past two years, I’ve continued to run several servers for the Ubuntu community, and I run a few of my own. As such, the vulnerabilities have impacted me enough that I’ve had beyond just an academic curiosity in both. Still, I appreciated the time he spent getting into the details of how the vulnerabilities worked and why that matters, as well as his call for folks working in software and hardware to be more explicit in their communications. As developers and systems engineers we can’t afford to continue treating hardware as a magical black box, and we need to more carefully consider the security cost of software-driven speed improvements.

The second keynote was from Tameika Reed on The Beginning, Present, and Future of Sysadmins. If the “Systems Administration” job title was ever cool, it’s certainly not now, in spite of the job still very much existing. In this funny and thoughtful talk, Tameika took us through the key skills that made a good systems administrator, and how they’re all still relevant in the world of automation, SREs, and DevOps methodologies. The specific implementations may have changed, but we still need backups, monitoring, security and more. The key takeaway: “Your skillsets still apply today but you have to evolve.”

Talk-wise the first day I went to a mix of culture talks and container talks. Culture was an import category for LISA this year, and I took time to hear from Emily Gladstone Cole on strategies to be your security team’s best friend, Kurt Andersen on steps to reduce operational toil on a team, and Rich Bowen on transferring ownership and leadership of a project. On the containers side, I enjoyed meeting and hearing from Michael Jennings of the Los Alamos National Laboratory in a talk where he explored the use of containers in HPC (High Performance Computing) and the current projects tackling the needs of HPC, including Charliecloud. I also had the pleasure of being session chair for David Morrison’s talk, where he talked about the use of Apache Mesos at Yelp.

The second day of keynotes brought the topic of social consciousness to the forefront. In the first, Dr. Sarah Lewis Cortes took us on a tour of the Darknet and explored recent cases of security vulnerabilities that have cost hundreds of millions in lawsuits to the retailers in question. In these security breaches, initial access that opened the door to attackers was accessed via a third party, so even if you’re not a retailer handling credit cards, we need to be conscious of how the security of our systems is impacting those we interact with. Directly following her talk, we heard from Jeffrey Snover who spoke more directly to social responsibility. It’s easy to get hyper-focused on solving a problem in your small corner of your code base, but stepping back from time to time to understand the broader implications of what you’re writing from an ethics and security perspective is important. With software “eating the world” we’re no longer just writing and running software, we’re writing and running key infrastructure that the world today operates on.

That afternoon I was a session chair again, and of particular note I was able to introduce Daniel J. Walsh of SELinux fame, who has now been working on containers and container security. One of the biggest takeaways from his talk was the tendency to run containers as root on your host system, and stressed that just ceasing this practice could probably be the best possible thing you could do to improve container security. His opensource.com article from the spring gives more details: Just say no to root (in containers).

Tuesday also brought us the LISA social! I’m not exactly a social butterfly, so I was happy to get settled in with a couple folks I knew, and their extended group, for food and interesting conversation. Plus, they had DIY(ish) LISA18 t-shirts featuring a raccoon, the state animal of Tennessee!

The final day of LISA landed on Halloween, and several conference-goers dressed up, which was a lot of fun. Unfortunately I wasn’t one of them, the contents of my suitcase covered both my trip to All Things Open and LISA (did laundry in between) and I had to limit my suitcase size to account for pregnancy-related weight restrictions in what I could haul around. I was delighted to see Vice Admiral Holdo and Rainbow Dash suitably represented among the attendees, as well closing keynote speaker Tom McLaughlin dressed as David S. Pumpkins.

Talk-wise, it was lovely to hear from Sabice Arkenvirr, who used her experience fixing up complicated infrastructure messes to stress the value of using off-the-shelf open source solutions rather than coming up with home-brewed alternatives that no one else in the industry would know how to use, and often end up being immature compared to the common solutions. She also wrote an opensource.com article on the topic, We already have nice things, and other reasons not to write in-house ops tools!

I also enjoyed the talk from Steve Mushero on Taking Over & Managing Large Messy Systems. At these conferences we very appropriately spend a lot of time focused on stories of doing things right, and highlighting the forward-thinking work of major engineering organizations that set the pace for the industry. Unfortunately, the result of this is sometimes losing sight of what folks in operations are seeing in the rest of the industry, which can often be poorly implemented logging and backups, security problems, and poor maintenance. Steve’s talk put a spotlight on the problems that most of the industry still deals with, and talked about strategies to quickly triage the worst of them.

The conference concluded with a pair of keynotes. The first was on serverless from Tom McLaughlin. Tom is great and I enjoyed hanging out with him before the conference, but I’ll say right up front that we’re not on the same page when it comes to serverless. He makes a lot of great points about moving past the need to do system-level infrastructure work (leave that to the experts, who you can pay) so you can focus on just building your application on the platform they provide. Alas, as a strong proponent of open source, I can’t get over the lock-in and proprietary nature of serverless solutions. Sorry, Tom! I will concede that it’s something that companies may consider, but with the knowledge of how dependent they are on a third party with regard to reliability, price, features, and more. Nora Jones of Netflix concluded the conference with a technical and cultural dive into Chaos Engineering at Netflix.

The conclusion of the conference was a bit sad because of how much I enjoyed being a part of it! But I had a lovely evening out to vegan dinner with Brendan and Rikki before a little after-event enjoying of a non-alcoholic beer with others from USENIX and the LISA organizing crew.

More photos from LISA18 can be found here: https://www.flickr.com/photos/pleia2/albums/72157673027760167