I was in Salt Lake City for OpenStack Days Mountain West back in December, it was the first time I’d ever been to SLC and I certainly didn’t expect to return so quickly. Still, back in early March one of the organizers for Salt Lake City DevOps Days reached out to me and asked if I’d be interested in giving a keynote for the event. After some brain-racking as to an appropriate topic, I happily agreed to join them to talk about “The Open Sourcing of Infrastructure” which is part history lesson, and part learn-from-history lesson.

But before I talk about that, let me say a few words about Salt Lake City. When I was there in December I didn’t have a great opportunity to really take in how beautiful it was there. I walked around Temple Square and admired the Christmas lights and buildings, acknowledged the mountains, but my heart was elsewhere as I worked through a difficult time. This time I was in a better place. As I rode past the city and into South Jordan, UT where the conference was being held, I really got to check out the scenery. The whole area is surrounded by mountains, which were snow-capped even in May. It’s really something to wake up to, and be reminded of every time you look out the window. Beautiful mountains, right there!

The conference itself was held at Noah’s Event Venue, a great space that easily accommodated the 400 attendees, with a large auditorium on the ground floor, and several rooms throughout the space for open spaces and workshops in the afternoon. The sponsor room could have been bigger, it was a bit overwhelming crowd-wise when I ventured in a couple times and the sponsors were squished in pretty close to each other. Everything else went well though, the lunch lines moved quickly, the outdoor-ish space where we ate gave us a lovely view of the mountains (and was even better when they brought in some heaters the evening of the second day!).

This is the second year of this conference, and last year they established a tradition of having a stuffed animal mascot. Last year it was a unicorn and this year it was a Yak (a la yak shaving). Obviously I had to get my picture taken with the both of them. They also sat up there on the lectern during my talk, hooray!

Talk-wise, there were a few that stood out for me. The first was the opening keynote for the event. They brought in Ross Clanton, formerly of Target, but now at Verizon. I had the opportunity to meet and chat with him and the closing keynote speaker, Gwen Dobson, at the speaker dinner prior to the event. As we figuratively compared prep-for-our-keynote notes prior to the conference, I was certainly eager to hear from both of them.

Ross began his talk by giving some DevOps methodology background, but the meat of what was interesting to me was the strategies used inside of Target and Verizon to really drive the DevOps model. Executive buy-in was essential, but from there you also need management to take training seriously, in several forms. You don’t teach an organization to adopt DevOps by reading a book and expecting and over-night transformation. Instead, you need varied methods of moving the organization forward and celebrating wins, he suggests:

• Encourage collaborative learning environments where peers teach peers as much as instructors do, and it’s OK to fail and ask questions
• Run internal DevOps days, bring in a couple outside speakers but also internal folks who have expertise and stories to share
• Host six-week immersion engagements (“Learning Dojos”) where teams work on their actual backlog using DevOps strategies and have the freedom to learn and ask questions, while solving real problems, not examples created by instructors
• Gamification of team progress, where teams get points for various DevOps skills and capabilities they’ve started incorporating into their work and are rewarded (Verizon has the DevOps Cup, like the Stanley Cup, awarded each year!)
• Even if you aren’t winning a DevOps cup, make sure management knows how important it is that they acknowledge and celebrate any positive progress made toward the adoption of DevOps principles
• Don’t fight people who resist change in your organization, instead do awesome things with your allies, make progress, and most of the nay-sayers will join you eventually

Later that morning we heard from Rob Richardson on “CI/CD on the Microsoft Stack.” This was interesting to me because in spite of my own aversion to proprietary software, I do understand that CI is important for the entire software industry and had been remiss in ever looking into what is available for developers on Windows doing .NET programming. His talk walked the audience through setting up a CI/CD pipeline using TeamCity for CI and hooked into Octopus Deploy for CD (note: both proprietary) that are available and have support for Microsoft-focused environments, and specifically .NET in the case of Octopus Deploy.

Now, I won’t say that this is immediately valuable to me in a practical sense, since I don’t use any of these tools and am uncomfortable building infrastructure tooling around proprietary solutions anyway, but I was appreciative for the broadening of horizons. I learned that there are easy CI/CD options for folks working in the Microsoft world, and adoption of them by people outside of my open source bubble will make the software world better for all of us.

That evening I met up with a couple colleagues from Mesosphere who were attending the conference! Sam Agnew works in sales and joined us from his home base of Denver to meet with folks at the conference and Tim Harper works in engineering on Marathon remotely from a city just south of SLC. After the evening social at the event venue, we all went out to enjoy a nice meal of Mexican food and some drinks. They’re both super friendly and easy to talk to, so it was fun to get to know them a bit. I also found great value in chatting with them about Mesosphere and DC/OS, they believe in the company and products as much as I do, but don’t have the Silicon Valley slant on their opinions and observations about where we’re going.

The second day I gave the opening keynote. During this talk I guided the audience through the past couple decades of infrastructure with an eye on the shift from proprietary to open source software. From there I focused on what we’re open sourcing on the operations side today, and things to consider as we once again become dependent on proprietary technologies, even if they are “in the cloud” this time.

I stopped short of flat out telling people not to use proprietary tooling, or to never consider building their applications into proprietary, hosted APIs and tool kits. A lot of companies successfully do this and a lot of the sponsors at the event make their money by providing hosted products that make sense for them. Instead I implored them to think about their choices carefully, and provided a list of things to think about, including the risk of vendor lock-in, price increases, security and reliability concerns and understanding if/how your (and your customer’s!) data will be used by the vendor. Looking back, these were the same things we asked ourselves a decade ago when we shifted to using Linux as the infrastructure platform of choice. Slides from my talk are up here (PDF).

Thanks to Sam Agnew for taking a picture during my talk!

The final talk that really stood out for me came from Rob Treat who spoke on “Production Testing through Monitoring.” There is a lot of focus in the DevOps world around CI with testing, but the truth is you’ll never find all bugs through testing. He shared a handful of funny (but serious!) examples where once in production, users did things that the developers never thought of that caused serious production issues. This wasn’t because they weren’t testing, but instead because our imagination will simply never come up with every potential use, or misuse, of the software we’re building. This is where monitoring and metrics become essential.

As someone with an operations background who really likes monitoring (I run Nagios at home), this seemed obvious to me, but he took it one step further to make it something worth noting: You don’t just monitor basic things like CPU heat, processes running and return codes (in fact, you might be tracking too much of this kind of stuff), you also track things that make sense for your particular business. This returned me to the talk by Jeffery Smith at the DevOps Days in Seattle where he stressed the importance of IT actually understanding the business.

Rob demonstrated by walking us through an example of using metrics as they tried to figure out why traffic and sales were lower than normal for a couple days. After looking through a bunch of technical reasons, they finally overlaid email bounce statistics over the data and learned that for a couple days, bounces were higher than normal. Since much of the company’s sales traffic is driven by these emails, that caused a clear problem on those days. Having the data to draw that conclusion was vital, but they wouldn’t have known to collect that data if they hadn’t been tuned into how the company drives sales and the fact that tracking something like email bounces would be valuable.

Huge thanks to the organizers of this event. They did a great job making us feel welcome and making sure we had everything we needed. As speakers we also got amazing Utah-themed gift baskets which they graciously offered to ship to us (couldn’t bring it on the plane due to liquids involved, and I didn’t check a bag). The attendees were great too, everyone I spoke to was very friendly, even after they found out what strong feelings I have about using open source and open standards, hah!

More photos from this event here: https://www.flickr.com/photos/pleia2/albums/72157681808549041