I have already written about the UbuCon Summit and Ubuntu booth at SCALE14x (14th annual Southern California Linux Expo), but the conference went far beyond Ubuntu for me!
First of all, I love this new venue. SCALE had previously been held at hotels near LAX, with all the ones I’d attended being at the Hilton LAX. It was a fine venue itself, but the conference was clearly outgrowing it even when I last attended in 2014 and there weren’t many food options around, particularly if you wanted a more formal meal. The Pasadena Convention Center was the opposite of this. Lots of space, lots of great food of all kinds and price ranges within walking distance! A whole plaza across from the venue made a quick lunch at a nice place quite doable.
It’s also worth mentioning that with over 3000 attendees this year, the conference has matured well. My first SCALE was 9x back in 2011, and with every year the growth and professionalism has continued, but without losing the feel of a community-run, regional conference that I love so much. Even the expo hall has continued to show a strong contingent of open source project and organization booths among the flashy company-driven booths, but even the company booths weren’t over done. Kudos to the SCALE crew for their work and efforts that make SCALE continue to be one of my favorite open source conferences.
As for the conference itself, MJ and I were both able to attend for work, which was a nice change for us. Plus, given how much conference travel I’ve done on my own, it’s nice to travel and enjoy an event together.
Thursday was taken up pretty much exclusively by the UbuCon Summit, but Friday we started to transition into more general conference activities. The first conference-wide keynote was on Friday morning with Cory Doctorow presenting No Matter Who’s Winning the War on General Purpose Computing, You’re Losing where he explored security and Digital rights management (DRM) in the exploding field of the Internet of Things. His premise was that we did largely win the open source vs. proprietary battle, but now we’re in a whole different space where DRM are now threatening our safety and stifling innovation. Security vulnerabilities in devices are going undisclosed when discovered by third parties under threat of prosecution for violating DRM-focused laws which have popped up worldwide. Depending on the device, this fear of disclosure could actually result in vulnerabilities causing physical harm to someone if compromised in a malicious way. He also dove into more dystopian future where smart devices are given away for free/cheap but then are phoning home and can be controlled remotely by an entity that doesn’t have your personal best interest in mind. The talk certainly gave me a lot to think about. He concluded by presenting the Apollo 1201 Project “a mission to eradicate DRM in our lifetime” that he’s working on at the EFF, article here.
Later that morning I made my way over to the DevOpsDayLA track to present on Open Source tools for distributed systems administration. Unfortunately, the projectors in the room weren’t working. Thankfully my slides were not essential to the talk, so even though I did feel a bit unsettled to present without slides, I made it through. People even said nice things afterwards, so I think it went pretty well in spite of the technology snafu. The slides that should have been seen during the talk are available here (PDF) and since I am always asked, I do maintain a list of other open source infras. Thanks to @scalexphotos for capturing a photo during my talk.
In the afternoon I spent some time in the expo hall, where I was able to see many more familiar faces! Again, the community booths are the major draw for me, so it was great visiting with participants of projects and groups there. It was nice to swing by the Ubuntu booth to see how polished everything was looking. I also got to see Emma of System76, who I hadn’t seen in quite some time.
Friday evening had a series of Birds of a Feather (BoF) sessions. I was able to make my way over to one on OpenStack before wrapping up my evening.
Saturday morning began with a welcome from Pasadena City Council member Andy Wilson who was enthusiastic about SCALE14x coming to Pasadena and quickly dove into his technical projects and the work being done in Pasadena around tech. I love this trend of city officials welcoming open source conferences to their area, it means a lot that the work we’re doing is being taken seriously by the cities we’re in. Then it moved into a keynote by Mark Shuttleworth on Open Source in the World of App Stores which had many similarities to his talk at the UbuCon Summit, but was targeted more generally about how distributions can help keep pace today’s computing that deploys “at the speed of git.”
I then went to Akkana Peck’s talk on Stupid GIMP tricks (and smart ones, too). It was a very visual talk, so I’m struggling to do it justice in written form, but she demonstrated various tools for photo editing in GIMP that I knew nothing about, I learned a lot. She concluded by talking about the features that came out in the 2.8 release and then the features planned and being worked on in the upcoming 2.9 release. Video of the talk here In the afternoon I attended a Kubernetes talk, noting quickly that the containers track was pretty packed throughout the conference.
Akkana Peck on GIMP
Between “hallway track” chats about everything from the Ubuntu project to the OpenStack project infrastructure tooling, Saturday afternoon also gave me the opportunity to do a bit more wandering through the expo hall. I visited my colleagues at the HPE booth and was able to see their cloud in a box. It was amusing to see the suitcase version and the Ubuntu booth with an Orange box. Putting OpenStack clouds in a single demonstration deployment for a conference is a popular thing!
My last talk of the day was by OpenStack Magnum Project Technical Lead Adrian Otto on Docker, Kubernetes, and Mesos: Compared. He walked us through some of the basics of Magnum first, then dove into each technology. Docker Swarm is good for simple tooling that you’re comfortable with and doing exactly what you tell it (imperative) and have 100s-1000s machines in the cluster. Kubernetes is more declarative (you tell it what you want, it figures out how to do it) and currently has some scaling concerns that make it better suited for a cluster of up to 200 nodes. Mesos is a more complicated system that he recommended using if you have a dedicated infrastructure team and can effectively scale to over 10k nodes. Video of the talk here
Sunday began with a keynote by Sarah Sharp on Improving Diversity with Maslow’s Hierarchy of Needs. She spoke about diversity across various angles, from income and internet bandwidth restrictions to gender and race, and the intersection of these things. There are many things that open source projects assume: unlimited ability to download software, ability for contributors to have uninterrupted “deep hack mode” time, access to fast systems to do development on. These assumptions fall apart when a contributor is paying for the bandwidth they use, are a caretaker who doesn’t have long periods without interruptions or a new system that they have access to. Additionally, there are opportunities that are simply denied to many genders, as studies have show that mothers and daughters don’t have as many opportunities or as much access to technology as the fathers and sons in their household. She also explored safety in a community, demonstrating how even a single sexist or racist contributor can single-handedly destroy diversity for your project by driving away potential contributors. Having a well-written code of conduct with a clear enforcement plan is also important and cited resources for organizations and people who could help you with that, warning that you shouldn’t roll your own. She concluded by asking audience members to recognize the problem and take action in their communities to help improve diversity. Her excellent slides (with notes) are here and a video of the talk here.
I then made my way to the Sysadmin track to see Jonah Horowitz and Albert Tobey on From Sys Admin to Netflix SRE. First off, their slides were hilarious. Lots of 80s references to things that were out-dated as they made their way through how they’re doing Site Reliability Engineering (SRE) at Netflix and inside their CORE (Cloud Operations Reliability Engineering) team. In their work, they’ve moved past configuration management, preferring to deploy baked AMIs (essentially, golden images). They also don’t see themselves as “running applications for the developers” and instead empower developers to do their own releases and application-level monitoring. In this new world of managing fleets of servers rather than individual systems, they’ve worked to develop a blameless culture where they do postmortems so that anything that is found to be done manually or otherwise error-prone can be fixed so the issue doesn’t happen again. The also shared the open source tooling that they use to bypass traditional monitoring systems and provide SREs with a high level view of how their system is working, noting that no one in the organization “knows everything” about the infrastructure. This tooling includes Spinnaker, Atlas and Vector, along with their well-known Simian Army which services within Netflix must run (unless they have a good reason not to) to test tolerance of random instance failures. Video of the talk can be found here and slide here.
After lunch I made my way to A fresh look at SELinux… by Daniel Walsh. I’d seen him speak on SELinux before, and found his talk valuable then too. This time I was particularly interested in how it’s progressed in RHEL7/Centos7, like the new rules for a file type, such as knowing what permissions /home/user/.ssh should have and having an semanage command to set those permissions to that default instead of doing it manually. I also learned about semanage -e (equivalency) to copy permissions from one place to another and the new mv -Z which moves things while retaining the SELinux properties. Finally, I somehow didn’t have a good grasp on improvements to the man pages, doing things like `man httpd_selinux` works and is very helpful! I was also amused to learn a bout stopdisablingselinux.com (especially since our team does not turn it off, and that took some work on my part!). In closing, there’s also an SELinux Coloring Book (which I’ve written about before), and though I didn’t get to the session in time to get one, MJ picked me up on in the expo hall. Video of the talk here
With that, we were at the last talk of the conference. I went over to Dustin Kirkland’s talk on “adapt install [anything]” on your Ubuntu LTS server/desktop! Adapt is a wrapper around LXD containers that allows you to locally (unprivileged user) install versions of Ubuntu software from various versions and run it locally on your system. The script handles provisioning the container, many default settings and keeping it updated automatically, so you really can “adapt install” and run a series of adapt commands to interact with it as if it were prepared locally. It all reminded me of the pile of chroot-building scripts I had back when I was doing Debian packaging, but more polished than mine ever were! He wrote a blog post following up his talk here: adapt install [anything] which includes a link to his slides. Video from the talk here (link at 4 hours 42 minutes).
With the conference complete, it was sad to leave, but I had an evening flight out of Burbank. Amusingly, even my flight was full of SCALE folks, so there were some fun chats in the boarding area before our departure.
Huge thanks to everyone who made SCALE possible, I’m looking forward to next year!
More photos from SCALE14x here: https://www.flickr.com/photos/pleia2/albums/72157663821501532