openstack – pleia2's blog Elizabeth Krumbach Joseph's public journal about Linux, sysadmining, beer, travel, pink gadgets and her life in the city where little cable cars climb halfway to the stars. Sat, 25 Feb 2017 02:57:27 +0000 en-US hourly 1 OpenStack Days Mountain West 2016 Tue, 27 Dec 2016 15:02:19 +0000 A couple weeks ago I attended my last conference of the year, OpenStack Days Mountain West. After much flight shuffling following a seriously delayed flight, I arrived late on the evening prior to the conference with plenty of time to get settled in and feel refreshed for the conference in the morning.

The event kicked off with a keynote from OpenStack Foundation COO Mark Collier who spoke on the growth and success of OpenStack. His talk strongly echoed topics he touched upon at the recent OpenStack Summit back in October as he cited several major companies who are successfully using OpenStack in massive, production deployments including Walmart, AT&T and China Mobile. In keeping with the “future” theme of the conference he also talked about organizations who are already pushing the future potential of OpenStack by betting on the technology for projects that will easily exceed the capacity of what OpenStack can handle today.

Also that morning, Lisa-Marie Namphy moderated a panel on the future of OpenStack with John Dickinson, K Rain Leander, Bruce Mathews and Robert Starmer. She dove right in with the tough questions by having panelists speculate as to why the three major cloud providers don’t run OpenStack. There was also discussion about who the actual users of OpenStack were (consensus was: infrastructure operators), which got into the question of whether app developers were OpenStack users today (perhaps not, app developers don’t want a full Linux environment, they want a place for their app to live). They also discussed the expansion of other languages beyond Python in the project.

That afternoon I saw a talk by Mike Wilson of Mirantis on “OpenStack in the post Moore’s Law World” where he reflected on the current status of Moore’s Law and how it relates to cloud technologies, and the projects that are part of OpenStack. He talked about how the major cloud players outside of OpenStack are helping drive innovation for their own platforms by working directly with chip manufacturers to create hardware specifically tuned to their needs. There’s a question of whether anyone in the OpenStack community is doing similar, and it seems that perhaps they should so that OpenStack can have a competitive edge.

My talk was next, speaking on “The OpenStack Project Continuous Integration System” where I gave a tour of our CI system and explained how we’ve been tracking project growth and steps we’ve taken with regard to scaling it to handle it going into the future. Slides from the talk are available here (PDF). At the end of my talk I gave away several copies of Common OpenStack Deployments which I also took the chance to sign. I’m delighted that one of the copies will be going to the San Diego OpenStack Meetup and another to one right there in Salt Lake City.

Later I attended Christopher Aedo’s “Transforming Organizations with OpenStack” where he walked the audience through hands on training his team did about the OpenStack project’s development process and tooling for IBM teams around the world. The lessons learned from working with these teams and getting them to love open processes once they could explain them in person was inspiring. Tassoula Kokkoris wrote a great summary of the talk here: Collaborative Culture Spotlight: OpenStack Days Mountain West. I rounded off the day by going to David Medberry’s “Private Cloud Cattle and Pet Wrangling” talk where he drew experience from the private cloud at Charter Communications to discuss the move from treating servers like pets to treating them like cattle and how that works in a large organization with departments that have varying needs.

The next day began with a talk by OpenStack veteran, and now VP of Solutions at SUSE, Joseph George. He gave a talk on the state of OpenStack, with a strong message about staying on the path we set forth, which he compared to his own personal transformation to lose a significant amount of weight. In this talk, he outlined three main points that we must keep in mind in order to succeed:

  1. Clarity on the Goal and the Motivation
  2. Staying Focused During the “Middle” of the Journey
  3. Constantly Learning and Adapting

He wrote a more extensive blog post about it here which fleshes out how each of these related to himself and how they map to OpenStack: OpenStack, Now and Moving Ahead: Lessons from My Own Personal Transformation.

The next talk was a fun one from Lisa-Marie Namphy and Monty Taylor with the theme of being a naughty or nice list for the OpenStack community. They walked through various decisions, aspects of the project, and more to paint a picture of where the successes and pain points of the project are. They did a great job, managing to pull it off with humor, wit, and charm, all while also being actually informative. The morning concluded with a panel titled “OpenStack: Preferred Platform For PaaS Solutions” which had some interesting views. The panelists brought their expertise to the table to discuss what developers seeking to write to a platform wanted, and where OpenStack was weak and strong. It certainly seems to me that OpenStack is strongest as IaaS rather than PaaS, and it makes sense for OpenStack to continue focusing on being what they’ve called an “integration engine” to tie components together rather than focus on writing a PaaS solution directly. There was some talk about this on the panel, where some stressed that they did want to see OpenStack hooking into existing PaaS software offerings.

Great photo of Lisa and Monty by Gary Kevorkian, source

Lunch followed the morning talks, and I haven’t mentioned it, but the food at this event was quite good. In fact, I’d go as far as to say it was some of the best conference-supplied meals I’ve had. Nice job, folks!

Huge thanks to the OpenStack Days Mountain West crew for putting on the event. Lots of great talks and I enjoyed connecting with folks I knew, as well as meeting members of the community who haven’t managed to make it to one of the global events I’ve attended. It’s inspiring to meet with such passionate members of local groups like I found there.

More photos from the event here:

]]> 0
OpenStack book and Infra team at the Ocata Summit Fri, 02 Dec 2016 14:58:06 +0000 At the end of October I attended the OpenStack Ocata Summit in beautiful Barcelona. My participation in this was a bittersweet one for me. It was the first summit following the release of our Common OpenStack Deployments book and OpenStack Infrastructure tooling was featured in a short keynote on Wednesday morning, making for quite the exciting summit. Unfortunately it also marked my last week with HPE and an uncertain future with regard to my continued full time participation with the OpenStack Infrastructure team. It was also the last OpenStack Summit where the conference and design summit are being hosted together, so the next several months will be worth keeping an eye on community-wise. Still, I largely took the position of assuming I’d continue to be able to work on the team, just with more caution in regards to work I was signing up for.

The first thing that I discovered during this summit was how amazing Barcelona is. The end of October presented us with some amazing weather for walking around and the city doesn’t go to sleep early, so we had plenty of time in the evenings to catch up with each other over drinks and scrumptious tapas. It worked out well since there were fewer sponsored parties in the evenings at this summit and attendance seemed limited at the ones that existed.

The high point for me at the summit was having the OpenStack Infrastructure tooling for handling our fleet of compute instances featured in a keynote! Given my speaking history, I was picked from the team to be up on the big stage with Jonathan Bryce to walk through a demonstration where we removed one of our US cloud providers and added three more in Europe. While the change was landing and tests started queuing up we also took time to talk about how tests are done against OpenStack patch sets across our various cloud providers.

Thanks to Johanna Koester for taking this picture (source)

It wasn’t just me presenting though. Clark Boylan and Jeremy Stanley were sitting in the front row making sure the changes landed and everything went according to plan during the brief window that this demonstration took up during the keynote. I’m thrilled to say that this live demonstration was actually the best run we had of all the testing, seeing all the tests start running on our new providers live on stage in front of such a large audience was pretty exciting. The team has built something really special here, and I’m glad I had the opportunity to help highlight that in the community with a keynote.

Mike Perez and David F. Flanders sitting next to Jeremy and Clark as they monitor demonstration progress. Photo credit for this one goes to Chris Hoge (source)

The full video of the keynote is available here: Demoing the World’s Largest Multi-Cloud CI Application

A couple of conference talks were presented by members of the Infrastructure team as well. On Tuesday Colleen Murphy, Paul Belanger and Ricardo Carrillo Cruz presented on the team’s Infra-Cloud. As I’ve written about before, the team has built a fully open source OpenStack cloud using the community Puppet modules and donated hardware and data center space from Hewlett Packard Enterprise. This talk outlined the architecture of that cloud, some of the challenges they’ve encountered, statistics from how it’s doing now and future plans. Video from their talk is here: InfraCloud, a Community Cloud Managed by the Project Infrastructure Team.

James E. Blair also gave a talk during the conference, this time on Zuul version 3. This version of Zuul has been under development for some time, so this was a good opportunity to update the community on the history of the Zuul project in general and why it exists, status of ongoing efforts with an eye on v3 and problems it’s trying to solve. I’m also in love with his slide deck, it was all text-based (including some “animations”!) and all with an Art Deco theme. Video from his talk is here: Zuul v3: OpenStack and Ansible Native CI/CD.

As usual, the Infrastructure team also had a series of sessions related to ongoing work. As a quick rundown, we have Etherpads for all the sessions (read-only links provided):

Friday concluded with a Contributors Meetup for the Infrastructure team in the afternoon where folks split off into small groups to tackle a series of ongoing projects together. I was also able to spend some time with the Internationalization (i18n) team that Friday afternoon. I dragged along Clark so someone else on the team could pick up where I left off in case I have less time in the future. We talked about the pending upgrade of Zanata and plans for a translations checksite, making progress on both fronts, especially when we realized that there’s a chance we could get away with just running a development version of Horizon itself, with a more stable back end.

With the i18n team!

Finally, the book! It was the first time I was able to see Matt Fischer, my contributing author, since the book came out. Catching up with him and signing a book together was fun. Thanks to my publisher I was also thrilled to donate the signed copies I brought along to the Women of OpenStack Speed Mentoring event on Tuesday morning. I wasn’t able to attend the event, but they were given out on my behalf, thanks to Nithya Ruff for handling the giveaway.

Thanks to Nithya Ruff for taking a picture of me with my book at the Women of OpenStack area of the expo hall (source) and Brent Haley for getting the picture of Lisa-Marie and I (source).

I was also invited to sit down with Lisa-Marie Namphy to chat about the book and changes to the OpenStack Infrastructure team in the Newton cycle. The increase in capacity to over 2000 test instances this past cycle was quite the milestone so I enjoyed talking about that. The full video is up on YouTube: OpenStack® Project Infra: Elizabeth K. Joseph shares how test capacity doubled in Newton

In all, it was an interesting summit with a lot of change happening in the community and with partner companies. The people that make the community are still there though and it’s always enjoyable spending time together. My next OpenStack event is coming up quickly, next week I’ll be speaking at OpenStack Days Mountain West on the The OpenStack Project Continuous Integration System. I’ll also have a pile of books to give away at that event!

]]> 0
Ohio LinuxFest 2016 Wed, 30 Nov 2016 18:29:44 +0000 Last month I had the pleasure of finally attending an Ohio LinuxFest. The conference has been on my radar for years, but every year I seemed to have some kind of conflict. When my Tour of OpenStack Deployment Scenarios was accepted I was thrilled to finally be able to attend. My employer at the time also pitched in to the conference as a Bronze sponsor and by sending along a banner that showcased my talk, and my OpenStack book!

The event kicked off on Friday and the first talk I attended was by Jeff Gehlbach on What’s Happening with OpenNMS. I’ve been to several OpenNMS talks over the years and played with it some, so I knew the background of the project. This talk covered several of the latest improvements. Of particular note were some of their UI improvements, including both a website refresh and some stunning improvements to the WebUI. It was also interesting to learn about Newts, the time-series data store they’ve been developing to replace RRDtool, which they struggled to scale with their tooling. Newts is decoupled from the visualization tooling so you can hook in your own, like if you wanted to use Grafana instead.

I then went to Rob Kinyon’s Devs are from Mars, Ops are from Venus. He had some great points about communication between ops, dev and QA, starting with being aware and understanding of the fact that you all have different goals, which sometimes conflict. Pausing to make sure you know why different teams behave the way they do and knowing that they aren’t just doing it to make your life difficult, or because they’re incompetent, makes all the difference. He implored the audience to assume that we’re all smart, hard-working people trying to get our jobs done. He also touched upon improvements to communication, making sure you repeat requests in your own words so misunderstandings don’t occur due to differing vocabularies. Finally, he suggested that some cross-training happen between roles. A developer may never be able to take over full time for an operator, or vice versa, but walking a mile in someone else’s shoes helps build the awareness and understanding that he stresses is important.

The afternoon keynote was given by Catherine Devlin on Hacking Bureaucracy with 18F. She works for the government in the 18F digital services agency. Their mandate is to work with other federal agencies to improve their digital content, from websites to data delivery. Modeled after a startup, she explained that they try not to over-plan, like many government organizations do and can lead to failure, they want to fail fast and keep iterating. She also said their team has a focus on hiring good people and understanding the needs of the people they serve, rather than focusing on raw technical talent and the tools. Their practices center around an open by default philosophy (see: 18F: Open source policy), so much of their work is open source and can be adopted by other agencies. They also make sure they understand the culture of organizations they work with so that the tools they develop together will actually be used, as well as respecting the domain knowledge of teams they’re working with. Slides from her talk here, and include lots of great links to agency tooling they’ve worked on:

Catherine Devlin on 18F

That evening folks gathered in the expo hall to meet and eat! That’s where I caught up with my friends from Computer Reach. This is the non-profit I went to Ghana with back in 2012 to deploy Ubuntu-based desktops. I spent a couple weeks there with Dave, Beth Lynn and Nancy (alas, unable to come to OLF) so it was great to see them again. I learned more about the work they’re continuing to do, having switched to using mostly Xubuntu on new installs which was written about here. On a personal level it was a lot of fun connecting with them too, we really bonded during our adventures over there.

Tyler Lamb, Dave Sevick, Elizabeth K. Joseph, Beth Lynn Eicher

Saturday morning began with a keynote from Ethan Galstad on Becoming the Next Tech Entrepreneur. Ethan is the founder of Nagios, and in his talk he traced some of the history of his work on getting Nagios off the ground as a proper project and company and his belief in why technologists make good founders. In his work he drew from his industry and market expertise from being a technologist and was able to play to the niche he was focused on. He also suggested that folks look to what other founders have done that has been successful, and recommended some books (notably Founders at Work and Work the System). Finaly, he walked through some of what can be done to get started, including the stages of idea development, basic business plan (don’t go crazy), a rough 1.0 release that you can have some early customers test and get feedback from, and then into marketing, documenting and focused product development. He concluded by stressing that open source project leaders are already entrepreneurs and the free users of your software are your initial market.

Next up was Robert Foreman’s Mixed Metaphors: Using Hiera with Foreman where he sketched out the work they’ve done that preserves usage of Hiera’s key-value store system but leverages Foreman for the actual orchestration. The mixing of provisioning and orchestration technologies is becoming more common, but I hadn’t seen this particular mashup.

My talk was A Tour of OpenStack Deployment Scenarios. This is the same talk I gave at FOSSCON back in August, walking the audience through a series of ways that OpenStack could be configured to provide compute instances, metering and two types of storage. For each I gave a live demo using DevStack. I also talked about several other popular components that could be added to a deployment. Slides from my talk are here (PDF), which also link to a text document with instructions for how to run the DevStack demos yourself.

Thanks to Vitaliy Matiyash for taking a picture during my talk! (source)

At lunch I met up with my Ubuntu friends to catch up. We later met at the booth where they had a few Ubuntu phones and tablets that gained a bunch of attention throughout the event. This event was also my first opportunity to meet Unit193 and Svetlana Belkin in person, both of whom I’ve worked with on Ubuntu for years.

Unit193, Svetlana Belkin, José Antonio Rey, Elizabeth K. Joseph and Nathan Handler

After lunch I went over to see David Griggs of Dell give us “A Look Under the Hood of Ohio Supercomputer Center’s Newest Linux Cluster.” Supercomputers are cool and it was interesting to learn about the system it was replacing, the planning that went into the replacement and workload cut-over and see in-progress photos of the installation. From there I saw Ryan Saunders speak on Automating Monitoring with Puppet and Shinken. I wasn’t super familiar with the Shinken monitoring framework, so this talk was an interesting and very applicable demonstration of the benefits.

The last talk I went to before the closing keynotes was from my Computer Reach friends Dave Sevick and Tyler Lamb. They presented their “Island Server” imaging server that’s now being used to image all of the machines that they re-purpose and deploy around the world. With this new imaging server they’re able to image both Mac and Linux PCs from one Macbook Pro rather than having a different imaging server for each. They were also able to do a live demo of a Mac and Linux PC being imaged from the same Island Server at once.

Tyler and Dave with the Island Server in action

The event concluded with a closing keynote by a father and daughter duo, Joe and Lily Born, on The Democratization of Invention. Joe Born first found fame in the 90s when he invented the SkipDoctor CD repair device, and is now the CEO of Aiwa which produces highly rated Bluetooth speakers. Lily Born invented the tip-proof Kangaroo Cup. The pair reflected on their work and how the idea to product in the hands of customers has changed in the past twenty years. While the path to selling SkipDoctor had a very high barrier to entry, globalization, crowd-funding, 3D printers and internet-driven word of mouth and greater access to the press all played a part in the success of Lily’s Kangaroo cup and the new Aiwa Bluetooth speakers. While I have no plans to invent anything any time soon (so much else to do!) it was inspiring to hear how the barriers have been lowered and inventors today have a lot more options. Also, I just bought an Aiwa Exos-9 Bluetooth Speaker, it’s pretty sweet.

My conference adventures concluded with a dinner with my friends José, Nathan and David, all three of whom I also spent time with at FOSSCON in Philadelphia the month before. It was fun getting together again, and we wandered around downtown Columbus until we found a nice little pizzeria. Good times.

More photos from the Ohio LinuxFest here:

]]> 1
Seeking a new role Mon, 17 Oct 2016 23:23:18 +0000 Today I was notified that I am being laid off from the upstream OpenStack Infrastructure job I have through HPE. It’s a workforce reduction and our whole team at HPE was hit. I love this job. I work with a great team on the OpenStack Infrastructure team. HPE has treated me very well, supporting travel to conferences I’m speaking at, helping to promote my books (Common OpenStack Deployments and The Official Ubuntu Book, 9th and 8th editions) and other work. I spent almost four years there and I’m grateful for what they did for my career.

But now I have to move on.

I’ve worked as a Linux Systems Administrator for the past decade and I’d love to continue that. I live in San Francisco so there are a lot of ops positions around here that I can look at, but I really want to find a place where my expertise with open source, writing and public speaking can will be used and appreciated. I’d also be open to a more Community or Developer Evangelist role that leverages my systems and cloud background.

Whatever I end up doing next the tl;dr (too long; didn’t read) version of what I need in my next role are as follows:

  • Most of my job to be focused on open source
  • Support for travel to conferences where I speak at (6-12 per year)
  • Work from home
  • Competitive pay

My resume is over here:

Now the long version, and a quick note about what I do today.

OpenStack project Infrastructure Team

I’ve spent nearly four years working full time on the OpenStack project Infrastructure Team. We run all the services that developers on the OpenStack project interact with on a daily basis, from our massive Continuous Integration system to translations and the Etherpads. I love it there. I also just wrote a book about OpenStack.

HPE has paid me to do this upstream OpenStack project Infrastructure work full time, but we have team members from various companies. I’d love to find a company in the OpenStack ecosystem willing to pay for me to continue this and support me like HPE did. All the companies who use and contribute to OpenStack rely upon the infrastructure our team provides, and as a root/core member of this team I have an important role to play. It would be a shame for me to have to leave.

However, I am willing to move on from this team and this work for something new. During my career thus far I’ve spent time working on both the Ubuntu and Debian projects, so I do have experience with other large open source projects, and reducing my involvement in them as my life dictates.

Most of my job to be focused on open source

This is extremely important to me. I’ve spent the past 15 years working intensively in open source communities, from Linux Users Groups to small and large open source projects. Today I work on a team where everything we do is open source. All system configs, Puppet modules, everything but the obvious private data that needs to be private for the integrity of the infrastructure (SSH keys, SSL certificates, passwords, etc). While I’d love a role where this is also the case, I realize how unrealistic it is for a company to have such an open infrastructure.

An alternative would be a position where I’m one of the ops people who understands the tooling (probably from gaining an understanding of it internally) and then going on to help manage the projects that have been open sourced by the team. I’d make sure best practices are followed for the open sourcing of things, that projects are paid attention to and contributors outside the organization are well-supported. I’d also go to conferences to present on this work, write about it on a blog somewhere (company blog? and be encouraging and helping other team members do the same.

Support for travel to conferences where I speak at (to chat at 6-12 per year)

I speak a lot and I’m good at it. I’ve given keynotes at conferences in Europe, South America and right here in the US. Any company I go to work for will need to support me in this by giving me the time to prepare and give talks, and by compensating me for travel for conferences where I’m speaking.

Work from home

I’ve been doing this for the past ten years and I’d really struggle to go back into an office. Since operations, open source and travel doesn’t need me to be in an office, I’d prefer to stick with the flexibility and time working from home gives me.

For the right job I may be willing to consider going into an office or visiting client/customer sites (SF Bay Area is GREAT for this!) once a week, or some kind of arrangement where I travel to a home office for a week here and there. I can’t relocate for a position at this time.

Competitive pay

It should go without saying, but I do live in one of the most expensive places in the world and need to be compensated accordingly. I love my work, I love open source, but I have bills to pay and I’m not willing to compromise on this at this point in my life.

Contact me

If you think your organization would be interested in someone like me and can help me meet my requirements, please reach out via email at

I’m pretty sad today about the passing of what’s been such a great journey for me at HPE and in the OpenStack community, but I’m eager to learn more about the doors this change is opening up for me.

]]> 4
OpenStack QA/Infrastructure Meetup in Walldorf Sat, 24 Sep 2016 15:30:37 +0000 I spent this week in the lovely town of Walldorf, Germany with about 25 of my OpenStack Quality Assurance and Infrastructure colleagues. We were there for a late-cycle sprint, where we all huddled in a couple of rooms for three days to talk, script and code our way through some challenges that are much easier to tackle when all the key players are in a room together. QA and Infra have always been a good match for an event like this since we’re so tightly linked as things QA works on are supported by and tested in the Continuous Integration system we run.

Our venue this time around were the SAP offices in Walldorf. They graciously donated the space to us for this event, and kept us blessedly fed, hydrated and caffeinated throughout the day.

Each day we enjoyed a lovely walk from and to the hotel many of us stayed at. We lucked out and there wasn’t any rain while we were there so we got to take in the best of late summer weather in Germany. Our walk took us through a corn field, past flowers, gave us a nice glimpse at the town of Walldorf on the other side of the highway and then began in on the approach to the SAP buildings of which there are many.

The first day began with an opening from our host at the SAP offices, Marc Koderer and by the QA project lead Ken’ichi Ohmichi. From there we went through the etherpad for the event to figure out where to begin. A big chunk of the Infrastructure team went to their own room to chat about Zuulv3 and some of the work on Ansible, and a couple of us hung back with the QA team to move some of their work along.

Spending time with the QA folks I learned about future plans for a more useful series of bugday graphs. I also worked with Spencer Krum and Matt Treinish to land a few patches related to the new Firehose service. Firehose is a MQTT-based unified message bus that seeks to encompass all the developer-facing infra alerts and updates in a single stream. This includes job results from Gerrit, updates on bugs from Launchpad, specific logs that are processed by logstash and more. At the beginning of the sprint only Gerrit was feeding into it using germqtt, but by the end of Monday we had Launchpad bugs submitting events over email via lpmqtt. The work was mostly centered around setting up Cyrus with Exim and then configuring the accounts and MX records, and trying to do this all in a way that the rest of the team would be happy with. All seems to have worked out, and at the end of the day Matt sent out an email announcing it: Announcing

That evening we gathered in the little town of Walldorf to have a couple beers, dinner, and relax in a lovely beer garden for a few hours as the sun went down. It was really nice to catch up with some of my colleagues that I have less day to day contact with. I especially enjoyed catching up with Yolanda and Gema, both of whom I’ve known for years through their past work at Canonical on Ubuntu. The three of us also were walk buddies back to the hotel, before which I demanded a quick photo together.

Tuesday morning we started off by inviting Khai Do over to give a quick demo of the Gerrit verify plugin. Now, Khai is one of us, so what do I mean by “come over”? Of all the places and times in the world, Khai was also at the SAP offices in Walldorf, Germany, but he was there for a Gerrit Hackathon. He brought along another Gerrit contributor and showed us how the verify plugin would replace our somewhat hacked into place Javascript that we currently have on our review pages to give a quick view into the test results. It also offers the ability in the web UI to run rechecks on tests, and will provide a page including history of all results through all the patchsets and queues. They’ve done a great job on it, and I was thrilled to see upstream Gerrit working with us to solve some of our problems.

Khai demos the Gerrit verify plugin

After Khai’s little presentation, I plugged my laptop into the projector and brought up the etherpad so we could spend a few minutes going over work that was done on Monday. A Zuulv3 etherpad had been worked on to capture a lot of the work from the Infrastructure team on Monday. Updates were added to our main etherpad about things other people worked on and reviews that were now pending to complete the work.

Groups then split off again, this time I followed most of the rest of the Infrastructure team into a room where we worked on infra-cloud, our infra-spun, fully open source OpenStack deployment that we started running a chunk of our CI tests on a few weeks ago. The key folks working on it gave a quick introduction and then we dove right into debugging some performance problems that were causing failed initial launches. This took us through poking at the Glance image service, rules in Neutron and defaults in the Puppet modules. A fair amount of multi-player (using screen) debugging was done up on the projector as we shifted around options, took the cloud out of the pool of servers for some time, and spent some time debugging individual compute nodes and instances as we watched what they did when they came up for the first time. In addition to our “vanilla” region, Ricardo Carrillo Cruz also made progress that day on getting our “chocolate” region working (next up: strawberry!).

I also was able to take some time on Tuesday to finally get notice and alert notifications going to our new @openstackinfra Twitter account. Monty Taylor had added support for this months ago, but I had just set up the account and written the patches to land it a few days before. We ran into one snafu, but a quick patch (thanks Andreas Jaeger!) got us on our way to automatically sending out our first Tweet. This will be fun, and I can stop being the unofficial Twitter status bot.

That evening we all piled into cars to head over to the nearby city of Heidelberg for dinner and drinks at Zum Weissen Schwanen (The White Swan). This ended up being our big team dinner. Lots of beers, great conversation and catching up on some discussions we didn’t have during the day. I had a really nice time and during our walk back to the car I got to see Heidelberg Castle light up at night as it looms over the city.

Friday kicked off once again at 9AM. For me this day was a lot of talking and chasing down loose ends while I had key people in the room. I also worked on some more Firehose stuff, this time working our way down the path to get logstash also sending data to Firehose. In the midst of which, we embarrassingly brought down our cluster due to failure to quote strings in the config file, but we did get it back online and then more progress was made after everyone got home on Friday. Still, it was good to get part of the way there during the sprint, and we all learned about the amount of logging (in this case, not much!) our tooling for all this MQTT stuff was providing for us to debug. Never hurts to get a bit more familiar with logstash either.

The final evening was spent once again in Walldorf, this time at the restaurant just across the road from the one we went to on Monday. We weren’t there long enough to grow tired of the limited selection, so we all had a lovely time. My early morning to catch a train meant I stuck to a single beer and left shortly after 8PM with a colleague, but that was plenty late for me.

Photo courtesy of Chris Hoge (source)

Huge thanks to Marc and SAP for hosting us. The spaces worked out really well for everything we needed to get done. I also have to say I really enjoyed my time. I work with some amazing people, and come Thursday morning all I could think was “What a great week! But I better get home so I can get back to work.” Hey! This all was work! Also thanks to Jeremy Stanley, our fearless Infrastructure Project Team Leader who sat this sprint out and kept things going on the home front while we were all focused on the sprint.

A few more photos from our sprint here:

]]> 0
Common OpenStack Deployments released! Tue, 13 Sep 2016 18:53:18 +0000 Back in the fall of 2014 I signed a contract with Prentice Hall that began my work on my second book, Common OpenStack Deployments. This was the first book I was writing from scratch and the first where I was the lead author (the first books I was co-author on were the 8th and 9th editions of The Official Ubuntu Book). That contract started me on a nearly two year journey to write a this book about OpenStack, which I talk a lot about here: How the book came to be.

Along the way I recruited my excellent contributing author Matt Fischer, who in addition to his Puppet and OpenStack expertise, shares a history with me in the Ubuntu community and Mystery Science Theater 3000 fandom (he had a letter read on the show once!). In short, he’s pretty awesome.

A lot of work and a lot of people went into making this book a reality, so I’m excited and happy to announce that the book has been officially released as of last week, and yesterday I got my first copy direct from the printer!

As I was putting the finishing touches on it in the spring, the dedication came up. I decided to dedicate the book to the OpenStack community, with a special nod to the Puppet OpenStack team.


This book is dedicated to the OpenStack community. Of the community, I’d also like to specifically call out the help and support received from the Puppet OpenStack Team, whose work directly laid the foundation for the deployment scenarios in this book.

Huge thanks to everyone who participated in making this book a reality, whether they were diligently testing all of our Puppet manifests, lent their OpenStack or systems administration experience to reviewing or gave me support as I worked my way through the tough parts of the book (my husband was particularly supportive during some of the really grim moments). This is a really major thing for me and I couldn’t have done it without all of you.

I’ll be continuing to write about updates to the book over on the blog that lives on the book’s website: (RSS). You can also follow updates on Twitter via @deploymentsbook, if that’s your thing.

If you’re interested in getting your hands on a copy, it’s sold by all the usual book sellers and available on Safari. The publisher’s website also routinely has sales and deals, especially if you buy the paper and digital copies together, so keep an eye out. I’ll also be speaking at conferences over the next few months and will be giving out signed copies. Check out my current speaking engagements here to see where I’ll be and I will have a few copies at the upcoming OpenStack Summit in Barcelona.

]]> 0
FOSSCON 2016 Tue, 23 Aug 2016 21:01:44 +0000 Last week I was in Philadelphia, which was fun and I got to do some Ubuntu stuff but I was actually there to speak at FOSSCON. It’s not the largest open source conference, but it is in my adopted home city of Philadelphia and I have piles of friends, mentors and family there. I love attending FOSSCON because I get to catch up with so many people, making it a very hug-heavy conference. I sadly missed it last year, but I made sure to come out this year.

They also invited me to give a closing keynote. After some back and forth about topics, I ended up with a talk on “Listening to the Needs of Your Global Open Source Community” but more on that later.

I kicked off my morning by visiting my friends at the Ubuntu booth, and meeting up with my OpenStack and HPE colleague Ma Dong who had flown in from Beijing to join us. I made sure we got our picture taken by the beautiful Philadelphia-themed banner that the HPE open source office designed and sent for the event.

At 11AM I gave my regular track talk, “A Tour Of OpenStack Deployment Scenarios.” My goal here was to provide a gentle introduction, with examples, of the basics of OpenStack and how it may be used by organizations. My hope is that the live demos of launching instances from the Horizon web UI and OpenStack client were particularly valuable in making the connection between the concepts of building a cloud the actual tooling you might use. The talk was well-attended and I had some interesting chats later in the day. I learned that a number of the attendees are currently using proprietary cloud offerings and looking for options to in-house some of that.

The demos were very similar to the tutorial I gave at SANOG earlier this month, but the talk format was different. Notes from demos here and slides (219K).

Thanks to Ma Dong for taking a picture during my talk! (source)

For lunch I joined other sponsors at the sponsor lunch over at the wonderful White Dog Cafe just a couple blocks from the venue. Then it was a quick dash back to the venue for Ma Dong’s talk on “Continuous Integration And Delivery For Open Source Development.”

He outlined some of the common mechanisms for CI/CD in open source projects, and how the OpenStack project has solved them for a project that eclipses most others in size, scale and development pace. Obviously it’s a topic I’m incredibly familiar with, but I appreciated his perspective as a contributor who comes from an open source CI background and has now joined us doing QA in OpenStack.

Ma Dong on Open Source CI/CD

After his talk it was also nice to sit down for a bit to chat about some of the latest changes in the OpenStack Infrastructure. We were able to catch up about the status of our Zuul tooling and general direction of some of our other projects and services. The day continued with some chats about Jenkins, Nodepool and how we’ve played around with infrastructure tooling to cover some interesting side cases. It was really fun to meet up with some new folks doing CI things to swap tips and stories.

Just before my keynote I attended the lightning talks for a few minutes, but had to depart early to get set up in the big room.

They keynote on “Listening to the Needs of Your Global Open Source Community” was a completely new talk for me. I wrote the abstract for it a few weeks ago for another conference CFP after the suggestion from my boss. The talk walked through eight tips for facilitating the collection of feedback from your community as one of the project leaders or infrastructure representatives.

  • Provide a simple way for contributors to contact project owners
  • Acknowledge every piece of feedback
  • Stay calm
  • Communicate potential changes and ask for feedback
  • Check in with teams
  • Document your processes
  • Read between the lines
  • Stick to your principles

With each of these, I gave some examples from my work mostly in the Ubuntu and OpenStack communities. Some of the examples were pretty funny, and likely very familiar with any systems folks who are interfacing with users. The Q&A at the end of the presentation was particularly interesting, I was very focused on open source projects since that’s where my expertise lies, but members of the audience felt that my suggestions were more broadly applicable. In those moments after my talk I was invited to speak on a podcast and encouraged to write a series of articles related to my talk. Now I’m aiming for writing some content on over the next couple weeks.

Slides from the talk are here (7.3M pdf).

And thanks to Josh, José, Vincent and Nathan for snapping some photos of the talk too!

The conference wound down and following the keynote with a raffle and we then went our separate ways. For me, it was time for spending time with friends over a martini.

A handful of other photos from the conference here:

]]> 0
SANOG 28 Sun, 07 Aug 2016 06:36:30 +0000 This week I traveled to Mumbai, India to participate in SANOG 28, (South Asian Network Operators Group). This was an unusual conference for me. My husband is the networking guru and he routinely attends NANOG meetings, for the North American group. I even had dinner here at SANOG with a woman who knows him. The closest I’ve gotten to NANOG is tagging along when the conference brings him to interesting of useful place (San Juan, Philadelphia) and doing some dinners with attendees who I know when I happen to be around. Plus, I usually go to open source or systems operations conferences. This was the first time I’d been to a conference focused on networking operations.

So, what brought me to the other side of the world to this uncharacteristic-for-me conference? I was encouraged to submit a proposal to do an OpenStack tutorial, and it was accepted! I’m really grateful to my friend Devdas Bhagat who encouraged me to submit. He has kept me in the loop all week with social activities and generally being around for me as I started interacting with a community that’s so new to me.

As the conference began, I learned that there have been nine SANOGs in India, and that this was the third time they’ve come to Mumbai. SANOG itself covers Afghanistan, Bangladesh, Bhutan, India, Maldives, Nepal, Pakistan and Sri-Lanka, but given the venue the first speaker spoke on some of the challenges confronting India specifically.

I enjoyed a keynote by Joe Abley of Dyn, where he spoke on treating your technical teams well and making sure you’re doing everything you can to support them in their work and goals. He also mentioned the splitting of technical from managerial tracks. This is becoming increasingly common in the bay area, they learned some time ago that engineering and manager skills are very different and people should be leveling up on their own tracks. It’s a message that I’m glad is being spread more widely, as an engineer myself I can confidently say that I’ll be a happy person if I can continue moving up in my career to conquer more interesting technical problems, and without ever having to manage other people.

Speaking directly to the technical talks we had Paul Wilson, the director of APNIC, give a keynote on the transition of IANA stewardship from the US Government to ICANN. Speaking as an operations person who is aware of the broader internet governance work because that’s where my servers live, I knew this transition had been in the works for several years but I didn’t know much about the actual plans or status. This presentation was the clearest, most concise summary of the plans, progress and status of the work they had been doing, and how close they are to finishing!

The most surprising part about this conference for me was the status of IPv6 in APAC, a view into which was presented by Byron Ellacott of APNIC. I had been under the naive assumption that given the explosive growth of network infrastructure in the regions over the past several years, it would go without saying that these green fields be IPv6 capable. I was wrong. While IPv6 adoption in the US and a few countries in Europe has continued to grow, it remains very low, to non-existent in most APAC regions. At a speaker dinner later in the week I asked about this, and the consensus was a chicken and egg problem. A considerable amount of content is still IPv4, so until that moves to IPv6, providing capability for it doesn’t make sense. As long as adoption remains low (estimated 6.5% worldwide) and IPv4 is still supported, organizations don’t have incentive to offer their content over IPv6. Instead, they’re taking extensive advantage of NAT and keep trying to find ways to get more IPv4 addresses (even if the math is against them). The whole discussion gave me some pause about the push for IPv6. Having a husband in the industry and working on a team that is eager to see strong IPv6 support in our infrastructure, I was an early adopter (I’ve had a AAAA record for this blog for years!). I thought we were all moving in the direction of adoption, but are we really?

The second day began with a talk about the status of Root DNS anycast in South Asia and how that impacts users by Anurag Bhatia of Hurricane Electric. It continued with an update from Champika Wijayatunga if ICANN on the rotation of and changes to ICANN’s Root Zone Keysigning key (KSK) and related Verisign Zonesigning Key (ZKZ), which I didn’t now a lot about but you can by checking out ICANN’s site on the topic. It definitely was surprising to learn that a rotation plan for the KSK wasn’t previously in place and that it’s remained the same since 2010.

These first morning talks concluded with a pair that were amusingly juxtaposed: The first was by Matthew Jackson on how geo-restrictions in New Zealand lead to the development of technologies to get around the limitations and subsequent policy changes. As a native of the US, I’ve only rarely been impacted by region-blocking, but it has always been troublesome to me. As he said in his talk: “The internet we built wasn’t meant to be geo-restricted.” Indeed. The talk that came after it was about ISP/network-level content filtering technologies. Hah!

As the day wound down, so did the conference. The closing event was held at the nearby Mumbai Cricket Association Indoor Cricket Academy and Recreation Center. It’s the off-season, so no Cricket was happening and the field was dark, but the inside of the building was beautiful. Though I’m not much of a party type, it was nice to meet a few folks and have some snacks before concluding my evening.

Hanging out with Devdas at the closing party!

The week continued with tutorials. On Thursday I presented mine: An Introduction to OpenStack. When my presentation was being evaluated by the committee in early July, I worked with them to tune the description to make an allowance for familiarity with Linux. Following acceptance, they strategically scheduled my tutorial the day after an Introduction to Linux hosted by Devdas.

As I wrote about in this interview, the tutorial was divided up into three parts:

  1. Introduction to some OpenStack deployments
  2. Demonstration
  3. Building your own cloud

Since the audience was very networking focused (less open source, systems), what I sought to communicate was the basic concepts around OpenStack and some of the services it could provide. Then, by giving a demonstration of using different components through a DevStack install, give people a practical view into launching instances, adding block storage, metering and object storage. The talk concluded by doing a section very similar to my CodeConf talk back in June, where I explored the next steps as they begin their journey into OpenStack territory.

The tutorial was 90 minutes long, and I had a few very engaged members of the audience. Afterwards I was able to talk to a couple of folks who previously had trouble separating all the Open* named projects, and were glad to learn more about OpenStack so at least that one stood out. My publisher also gave me some coupons for the digital version of Common OpenStack Deployments so I was able to give those out to three participants, and pre-order discounts for the rest of the audience.

Slides from the here, which include a link out to the DevStack demonstration instructions: sanog_2016_intro_to_openstack.pdf

I think what I enjoyed most about this conference was simply being exposed to a new community, it was a real plasure to be able to sit down at dinner with some of the brilliant people solving problems with these expanding networks. Beyond our discussions about the expansion of (or lack of) IPv6, I was able to chat with a DNS engineer at RIPE about infrastructure they use for the root server they run. I was specifically interested in how much organizational sharing happens between operators of the root DNS servers. His answer? Very little, intentionally. As a champion of open source infrastructures, it took some time for me to come around, but I conceded that in this it does make sense. By using different tooling and methodologies, the heart of the internet is kept safe against inevitable vulnerabilities that arise in one tool or another.

Huge thanks to the organizers of this conference and everyone who made me feel so welcome during my first visit to India. These past few nights I’ve had some great food and very friendly company of some great people from organizations whose work I admire.

More photos from the event here:

]]> 0
Newton OpenStack Summit Days 3-5 Tue, 10 May 2016 19:39:24 +0000 On Monday and Tuesday I was pretty focused on the conference side of the OpenStack Summit, but with all the keynotes behind us, when Wednesday rolled around I found myself much more focused on the Design Summit side.

Our first session of the day was on Community Task Tracking, which we jokingly called the “task tracking bake-off.” As background, couple years ago the OpenStack Infrastructure team placed our bets on an in-project developed task tracker called StoryBoard. The hope had been that the intention to move off of Launchpad and onto this new platform would bring support from companies looking to help with development. Unfortunately this didn’t pan out. Development landed on the shoulders of a single poor, overworked soul. At this point we started looking at the Maniphest component of Phabricator. Simultaneously we ended up with a contributor putting together configuration management for Maniphest and had a team pop up to continue support of StoryBoard for a downstream that had begun using it. A few weeks ago I organized a bug day where the team got together to do a serious once through of outstanding bugs and provide feedback to the StoryBoard team about what we need to use it, we went from 571 active bugs down to 414.

This set the stage for our session. We could stand up a Maniphest server or place our bets with StoryBoard again. We had a lot to consider.

  Pros Cons
Storyboard Strong influence over direction, already running and being used in our infra, good API We need to invest in development ourselves, little support for design/UI folks (though we could run a standalone Pholio)
Maniphest Investment is made by a large exiting development team, feature rich with pluggable components like Pholio for design folks Little influence over direction (like with Gerrit), still have to stand up and migrate to, weak API

Both had a few things lacking that we’d need before we go full steam into use by all of OpenStack, so there seemed to be consensus that they were similar in terms of work and time needed to get to that point. Of all the considerations, the need to develop our own vs. depending on upstream is the one that weighed most heavily upon me. Will companies really step up and help with development once we move everyone into production? What happens if our excellent current StoryBoard developers are reassigned to other projects? Having an active upstream certainly is a benefit. The session didn’t end with a formal selection, but we will be discussing it more over the next couple weeks so we can move toward making a recommendation to the Technical Committee (TC). Read-only session etherpad here.

The next session I attended was in the QA track, for the DevStack Roadmap. The session centered around finally making DevStack use Neutron by default. It’s been some time since nova-networking was deprecated, so this switch was a long time in coming. In addition to the technical components of this switch, there documentation needs to be updated around the networking decisions. Since I’ve just recently done some deep dives into OpenStack networking, somehow I ended up volunteering to help with this bit! Read-only session etherpad here.

Before the very busy lunch I had coming up, there was one more morning session, on Landing Page for Contributors. The current pages we have on the wiki, like the Main page on the wiki itself and the How To Contribute wiki aren’t the most welcoming of pages, they’re more walls of text that a new contributor has to sift through. This session talked through a lot of the tooling that could be used to make a more inviting, approachable page, drawing from other projects who have forged this path in the past. Of course it is also important that the content is reviewed and maintainable from the project perspective too, so something that can be held in revision control is key. Read-only session etherpad here.

As lunch rolled around I rushed upstairs to assist with the Git and Gerrit – Lunch and Learn. The event began by expecting and separating out about 1/3 of the folks in the room who hadn’t completed the prerequisites. It was the job of myself and the other helpers to start working with these folks to get their accounts set up and git-review installed. This wasn’t a trivial task, in spite of my intimate knowledge of how our system works and years of using it, almost all the attendees used Windows or Mac. I use Linux full time and we don’t maintain good (or any) documentation for in our development workflow for OpenStack development for these other operating systems.

A lot of folks did make it through configuration, and it was nice to be reminded about how our community is growing and that our tools need to do as well. A patch was submitted several months back to add a video of how to set things up on Windows, but that’s inconsistent with the rest of our documentation and has not been accepted. It would be great to see some folks using these other operating systems help us get the written documentation into better shape. Beyond the prerequisites, session leaders Amy Marrich and Tamara Johnston walked folks through setting up their environment, submitting a patch to the sandbox repo, submitting a test bug, reviewing a change and more. The slide deck they used has been uploaded to Amy’s AustinSummit GitHub project. I also took a few minutes to explain the Zuul Status page and a bit about each of the pipelines that a change may go through on the way to being merged.

Git and Gerrit – Lunch and Learn

Directly after lunch I was in another infrastructure session, this time to talk about Launch-Node, Ansible and Puppet. Launching new, long-lived servers in our infrastructure is one of those tasks that has remained frustratingly hands on. This manual work has been a time sink and a lot of it can be automated, so we as a team consider this situation a bug. Our Launch-Node script has been developed to start tackling this and the session went through some of the things we need to be careful of, including handling of DNS and duplicate hostnames (what if we’re spinning up a replacement server?), when do we unmount and disassociate cinder volumes and trove databases with the old server and bring them up on the new? Lots of great discussion around all of this was had. Fixes were already coming in by the end of this session and we have a good path moving forward. Read-only session etherpad here.

The next infrastructure session focused on Wiki upgrades. We’ve been struggling with spam problems for a several months. We need to do an upgrade to get some of the latest anti-spam tooling, which also requires upgrading the operating system in order to get a newer version of PHP. The people-power we have for this is limited, as we all have a lot of other projects on our plates. The session began with outlining what we need to do to get this done, and wound down with the proposal to shut down the wiki in a year. The OpenStack project has great, collaborative tooling for publishing documentation and things, we also use etherpads a lot for notes and to do lists, is there really still an active need for a wiki? Thierry Carrez sent an email today that started work on socializing our options, whether to carry on with the wiki or not. As the discussions continue on list, I hope to help in finding tooling for teams that need it and the current tools don’t satisfy. While we do that over the next year, Paul Belanger has bravely stepped forward to lead up the ongoing maintenance of the wiki until the possible retirement. Read-only session etherpad here.

Thursday morning kicked off bright and early with a session on Proposal jobs. As some quick background, proposal jobs are run on a privileged server in the OpenStack infrastructure that has the credentials to publish to a few places, like translations files up to Zanata. With this in mind, and as general good policy, we like to keep jobs we’re running here down to a minimum, using non-privileged servers as much as possible to complete tasks. The session walked through several of the existing jobs and news ones that were being proposed to sort through how they could be done differently, and make sure we’re all on the same page as a team when it comes to approving new jobs on these servers. Read-only session etherpad here.

It was then on to a session to “Robustify” Ansible-Puppet. Several months back we switched over to a system of triggering Puppet runs with Ansible instead of using the Puppetmaster software. This process quickly became complicated, so much so that even I struggled to trace the whole path of how everything works. Thankfully Monty Taylor and Spencer Krum started off the session by whiteboarding how everything works together, or doesn’t, as the case may be. It was a huge help to see it sketched out so that the pain points could be identified, one of those rare times when it was super valuable to be together in a room as a team rather than trying to explain things over IRC. We learned that inventory creation for Ansible is one of our pain points, but the complexity of the whole system has made fixing problems tricky, you pull one thread and something else gets undone! We also discussed the status of logging, and how we can better prepare for edge cases where things Really Go Wrong and we can’t access to the server to see the logs to find out what happened. There’s also some Puppetboard debugging to do, as folks rely on the data from that and it hasn’t been entirely accurate in reporting failures lately. In all, a great session, read-only session etherpad here.

Monty and Spencer explain our Ansible-Puppet setup

Next up for Infrastructure was a fishbowl session about OpenID/SSO for Community Systems. The OpenStack Foundation invested in the development of OpenStackID when few other options that fit our need were mature in this space. Today we have the option of using ipsilon, which has a bigger development community and is already in use by another major open source project (Fedora). The session outlined the benefits of consuming an upstream tool instead, including their development model, security considerations and general resources that have been spent to roll our own solution. The session also outlined exactly what our needs are to move all of our authentication away from Launchpad hosted by Canonical. I think it was a good session with some healthy discussion about where we are with our tooling, read-only session etherpad here.

I spent my time after lunch with the translations/internationalization (i18n) folks in a 90 minute work session on Translation Processes and Tools (read-only session etherpad here). My role in this session, along with Steve Kowalik and Andreas Jaeger was to represent the infrastructure team and the tooling we could provide to help the i18n team get their work done. Of particular focus were the translations check site that we need to work toward bringing online and our plan to upgrade Zanata, and the underlying operating system it’s running on. We also discussed some of the other requirements of the team, like automated polling of Active Technical Contributor (ATC) status for translators and improved statistics on Stackalytics for translations. Andreas was also able to take time to show off the new translations procedure for reno-driven release notes, which allows for translations throughout the cycle as they’re committed to the repositories rather than a mad rush to complete them at the end. It was also nice to catch up with Alex Eng from the Zanata team and former i18n PTL Daisy (Ying Chun Guo) who I had such a great time with in Tokyo, I wish I’d had more time to grab a meal with them.

In our final Infrastructure-focused session of the day, we met to discuss operating system upgrades. With the release of the latest Ubuntu LTS (16.04) the week prior to the summit, we find ourselves in a world of three Ubuntu LTS releases in the mix. We decided to first carve out some time to get all of our 12.04 systems upgraded to 14.04. From there we’ll work to get our Puppet modules updated and services prepared for running on 16.04. Of particular interest to me is getting the Zanata server on 16.04 soon so we can upgrade the version of Zanata that it’s running and requires a newer version of Java than 14.04 provides. We also spent a little time splitting out the easier servers to upgrade from the more difficult ones, especially since some systems have very little data and don’t actually need an in place upgrade, we can simply redeploy workers. We will do a more thorough evaluation when we’re closer to upgrade time, which we’re scheduling for some time this month. Read-only session etherpad here.

Thursday evening meant it was time for our Infrastructure Team dinner! Over 20 self-proclaimed infrastructure team members piled into cars to make it across town to Freedmans to enjoy piles of BBQ. I had to pass on all things bready (including beer) but later in the evening we made our way inside to the bar where we found agave tequila that was not forbidden for me. The rest was history. Lots of fun and great chats with everyone, including a bunch of non-infra people who had been clued into our late night shenanigans and decided to join us.

Infra evening gathering, photo by Monty Taylor

Friday was our day for team work session gatherings. Infrastructure ended up in room 404 (which, in fact, was difficult to find). Jeremy Stanley (fungi) kicked the day off by outlining topics for Infra and QA that we may find valuable to work on together while we were in the room. I worked on a few things with folks for about an hour before switching tracks to join my translations friends again over in their work session.

Steve, Andreas and I made our way over to the i18n session to chat with them about the ability to translate more things (like DevStack documentation) and to give them an update from our upgrades session for an idea of when they could expect the Zanata upgrade. Perhaps the most exciting part of the morning was their request for us to finally shut down the OpenStack Transifex project. We switched to Zanata when Transifex went closed source, but our hosted account has lingered around for a year since we’ve used it “just in case” we needed something from it. With two OpenStack cycles on Zanata behind us, it was time to shut it down. We were all delighted when we saw the email: [Transifex] The organization OpenStack has been deleted by the user jaegerandi.

Cheerful crowd of i18n contributors!

After one more lunch at Cooper’s BBQ, I made it back to the Infrastructure room for more afternoon work, but I could feel the cloud of exhaustion hitting me by then. Most of what I managed was informally chatting with my fellow contributors and sketching out work to be done rather than actually getting much done. There’d be plenty of time for that once I returned home!

I concluded my time in Austin with a few colleagues with a visit to the Austin Toy Museum, some leisurely time at the Blue Cat Cafe (my first cat cafe!) and a quiet sushi dinner. With that, another great OpenStack Summit was behind me. My flight home left at 6AM Saturday morning.

Edit: Infrastructure PTL Jeremy Stanley has also written summaries of sessions here: Newton Summit Infra Sessions Recap

]]> 0
Newton OpenStack Summit Days 1-2 Sun, 01 May 2016 16:28:47 +0000 This past week I attended my sixth OpenStack Summit. This one took us to Austin, Texas. I was last in Austin in 2014 when I quickly stopped by to give a talk at the Texas LinuxFest, but I wasn’t able to stay long during that trip. This trip gave me a chance (well, several) to finally have some local BBQ!

I arrived Sunday afternoon and took the opportunity to meet up with Chris Aedo and Paul Belanger, who I’d be on the stage with on Monday morning. We were able to do our first meetup together and do a final once through of our slides to make sure they had all the updates we wanted and we were clear on where the transitions were. Gathering at the convention center also allowed to pick up our badges before the mad rush that would come the opening of the conference itself on Monday morning.

With Austin being the Live Music Capital of the World, we were greeted in the morning by live music from the band Soul Track Mind. I really enjoyed the vibe it brought to the morning, and we had a show to watch as we settled in and waited for the keynotes.

Jonathan Bryce and Lauren Sell of the OpenStack Foundation opened the conference and gave us a tour of numbers. The first OpenStack summit was held in Austin just under six years ago with 75 people and they were proud to announce that this summit had over 7,500. It’s been quite the ride that I’m proud to have been part of since the beginning of 2013. In Jonathan’s keynote we were able to get a glimpse into the real users of OpenStack, with highlights including the fact that 65% of respondents to the recent OpenStack User Survey are using OpenStack in production and that half of the Fortune 100 companies are using OpenStack in some capacity. It was also interesting to learn how important the standard APIs for interacting with clouds was for companies, a fact that I always hoped would shine through as this open source cloud was being adopted. The video from his keynote is here: Embracing Datacenter Diversity.

As the keynotes continued the ones that really stood out for me were by AT&T (video: AT&T’s Cloud Journey with OpenStack) and Volkswagen Group (Driving the Future of IT Infrastructure at Volkswagen Group.

The AT&T keynote was interesting from a technical perspective. It’s clear that the rise of mobile devices and the internet of things has put pressure on telecoms to grow much more quickly than they have in the past to handle this new mobile infrastructure. Their keynote shared that they expected this to grow an additional ten times by 2020. To meet this need, the networking aspects of technologies like OpenStack are important to their strategy as they move away from “black box” hardware from networking vendors and to more software-driven infrastructure that could grow more quickly to fit their needs. We learned that they’re currently using 10 OpenStack projects in their infrastructure, with plans to add 3 more in the near future, and learned about their in house AT&T Integrated Cloud (AIC) tooling for managing OpenStack. When the morning concluded, all their work was rewarded with a Super User award, they wrote about here.

The Volkswagen Group keynote was a lot of fun. As the world of electric and automated cars quickly approaches they have recognized the need to innovate more quickly and use technology to get there. They still seem to be in the early days of OpenStack deployments, but have committed a portion of one of their new data centers to just OpenStack. Ultimately they see a hybrid cloud future, leveraging both public and private hosting.

The keynote sessions concluded with the announcement of the 2017 OpenStack Summit locations: Boston and Sydney!

Directly after the keynote I had to meet Paul and Chris for our talk on OpenStack Infrastructure for Beginners (video, slides). We had a packed room. I lead off the presentation by covering an overview of our work and by giving a high level tour of the OpenStack project infrastructure. Chris picked up by speaking to how things worked from a developer perspective, tying that back into how and why we set things up the way we did. Paul rounded out the presentation by diving into more of the specifics around Zuul and Jenkins, including how our testing jobs are defined and run. I think the talk went well, we certainly had a lot of fun as we went into lunch chatting with folks about specific components that they were looking either to get involved with or replicate in their own continuous integration systems.

Chris Aedo presenting, photo by Donnie Ham (source)

After a delicious lunch at Cooper’s BBQ, I went over to a talk on “OpenStack Stable: What It Actually Means to Maintain Stable Branches” by Matt Riedemann, Matthew Treinish and Ihar Hrachyshka in the Upstream Development track of the conference. This was a new track for this summit, and it was great to see how well-attended the sessions ended up being. The goal of this talk was to inform members of the community what exactly is involved in management of stable releases, which has a lot more moving pieces than most people tend to expect. Video from the session up here. It was then over to “From Upstream Documentation To Downstream Product Knowledge Base” by Stefano Maffulli and Caleb Boylan of DreamHost. They’ve been taking OpenStack documentation and adjusting it for easier and more targeted for consumption by their customers. They talked about their toolchain that gets it from raw source from the OpenStack upstream into the proprietary knowledge base at DreamHost. It’ll be interesting to see how this scales long term through releases and documentations changes, video here.

My day concluded by participating in a series of Lightning Talks. My talk was first, during which I spent 5 minutes giving a tour of I was inspired to give this talk after realizing that even though the links are right there, most people are completely unaware of what things like Reviewday (“Reviews” link) are. It also gave me the opportunity to take a closer, current look at OpenStack Health prior to my presentation, I had intended to go to “OpenStack-Health Dashboard and Dealing with Data from the Gate” (video) but it conflicted with the talk we were giving in the morning. The lightning talks continued with talks by Paul Belanger on Grafyaml, James E. Blair on Gertty and Andreas Jaeger on the steps for adding a project to OpenStack. The lightning talks from there drifted away from Infrastructure and into more general upstream development. Video of all the lightning talks here.

Day two of the summit began with live music again! It was nice to see that it wasn’t a single day event. This time Mark Collier of the OpenStack Foundation kicked things off by talking about the explosion of growth in infrastructure needed to support the growing Internet of Things. Of particular interest was learning about how operators are particularly seeking seamless integration of virtual machines, containers and bare metal, and how OpenStack meets that need today as a sort of integration engine, video here.

The highlights of the morning for me included a presentation from tcp cloud in the Czech Republic. They’re developing a Smart City in the small Czech city of Písek. He did an overview of the devices they were using and presented a diagram demonstrating how all the data they collect from around the city gets piped into an OpenStack cloud that they run. He concluded his presentation by revealing that they’d turned the summit itself into a mini city by placing devices around the venue to track temperature and CO2 levels throughout the rooms, very cool. Video of the presentation here.

tcp cloud presentation

I also enjoyed seeing Dean Troyer on stage to talk about improving user experience (UX) with OpenStackClient (OSC). As someone who has put a lot of work into converting documented commands in my book in an effort to use OSC rather than the individual project clients I certainly appreciate his dedication to this project. The video from the talk is here. It was also great to hear from OVH, an ISP and cloud hosting provider who currently donates OpenStack instances to our infrastructure team for running CI testing.

Tuesday also marked the beginning of the Design Summit. This is when I split off from the user conference and then spend the rest of my time in development space. This time the Design Summit was held across the street from the convention center in the Hilton where I was staying. This area of the summit takes us away from presentation-style sessions and into discussions and work sessions. This first day focused on cross-project sessions.

This was the lightest day of the week for me, having a much stronger commitment to the infrastructure sessions happening later in the week. Still, went to several sessions, starting off with a session led by Doug Hellmann to talk about how to improve the situation around global requirements. The session actually seemed to be an attempt to define the issues around requirements and get more contributors to help with requirements project review and to chat about improvements to tests. We’d really like to see requirements changes have a lower chance of breaking things, so trying to find folks to sign up to do this test writing work is really important.

I had lunch with my book writing co-conspirator Matt Fischer to chat about some of the final touches we’re working on before it’s all turned in. Ended up with a meaty lunch again at Moonshine Grill just across the street from the convention center, after which I went into a “Stable Branch End of Life Policy” session led by Thierry Carrez and Matt Riedemann. The stable situation is a tough one. Many operators want stable releases with longer lifespans, but the commitment from companies to put engineers on it is extremely limited. This session explored the resources required to continue supporting releases for longer (infra, QA, etc) and there were musings around extending the support period for projects meeting certain requirements for up to 24 months (from 18). Ultimately by the end of the summit it does seem that 18 months continues to be the release lifespan of them all.

I then went over to the Textile building across from the conference center where my employer, HPE, had set up their headquarters. I had a great on-camera chat with Stephen Spector about how open source has evolved from hobbyist to corporate since I became involved in 2001. I then followed some of the marketing folks outside to shoot some snippits for video later.

The day of sessions continued with a “Brainstorm format for design summit split event” session that talked a lot about dates. As a starting point, Thierry Carrez wrote a couple blog posts about the proposal to split the design summit from the user summit:

With these insightful blog posts in mind, the discussion moved forward on the assumption that the events would be split and how to handle that timing-wise. When in the cycle would each event happen for maximum benefit for our entire community? In the first blog post he had a graphic that had a proposed timeline, which the discussions mostly stuck to, but dove deeper into discussing what is going on during each release cycle week and what the best time would be for developers to gather together to start planning the next release. While there was good discussion on the topic, it was clear that there continues to be apprehension around travel for some contributors. There are fears that they would struggle to attend multiple events funding-wise, especially when questions arose around whether mid-cycle events would still be needed. Change is tough, but I’m on board with the plan to split out these events. Even as I write this blog post, I notice the themes and feel for the different parts of our current summit are very different.

My session day concluded with a session about cross-projects specifications for work lead by Shamail Tahir and Carol Barrett from the Product Working Group. I didn’t know much about OpenStack user stories, so this session was informative for seeing how those should be used in specs. In general, planning work in a collaborative way, especially across different projects that have diverse communities is tricky. Having some standards in place for these specs so teams are on the same page and have the same expectations for format seems like a good idea.

Tuesday evening meant it was time for the StackCity Community Party. Instead of individual companies throwing big, expensive parties, a street was rented out and companies were able to sponsor the bars and eateries in order to throw their branded events in them. Given my dietary restrictions this week, I wasn’t able to partake in much of the food being offered, so I only spent about an hour there before joining a similarly restricted diet friend over at Iron Works BBQ. But not before I picked up a dinosaur with a succulent in it from Canonical.

I called it an early night after dinner, and I’m glad I did. Wednesday through Friday were some busy days! But those days are for another post.

More photos from the summit here:

]]> 0