Wednesday morning started off with a Women of OpenStack breakfast where I happened to sit down next to Leslie Hawthorn without realizing it until we introduced ourselves, so that was a pleasant surprise! We’d both been intending to meet for some time. The table we were at also had a couple engineers working on OpenStack at Yahoo! and a UX engineer from Red Hat, a tech reporter and more, so it was quite the interesting mix and we all swapped business cards. It’s been really cool to meet the GNOME Outreach Program for Women interns from this past cycle too, they’ve been doing really impressive work and writing about it in blogs I’ve enjoyed reading. I was also able to work with Anita Kuno directly while she was helping out with the infrastructure (-infra) team.

It was then over to the conference for more inspiring keynotes. The OpenStack Foundation brought in Randall Sobie to talk about clouds in high energy physics, including a 5000 core OpenStack cloud CERN. Predictably dealing with massive amounts of data is one of the challenges of such scientific organizations that collect it to solve some of the most interesting problems of and in the universe.

Next up was a representative from the NSA who, while not able to tell us much at all about their private cloud deployment of OpenStack, did let slip their numbers had reached the 1000s and at a certain point in the process and that their staff was seeking to collaborate with other government organizations to deploy similar infrastructures elsewhere. Again, predictably, an organization like the NSA needs to handle not only big data, but analysis and manipulation of data. Next up was HP’s keynote, where they dove into some of the projects HP supports, including the one I’m employed with them to work on, the continuous integration team! They also talked about OpenStack on Openstack (TripleO), Heat and related tools. It was also interesting to learn that HP ships a Linux server every minute.

Off to sessions! The Adding new integrated projects to Tempest session shared some best practices from longer-term OpenStack projects as the newly integrated Ceilometer and Heat projects are now seeking to be included in Tempest (also check out these slides by Sean Dague) testing. The testr / testtools feedback/next-steps
session began with a description of exactly what testr and Test Repository are and are for and then beyond that how more tests can migrate to it. Tempest Best Practice Guide where they discussed some ideas for what the QA team should develop in terms of best practices and then steps the team can take to help with organization and productivity of the team, including leveraging milestones, hosting regular bug triage days and improving documentation with regular documentation weeks.

I went to Devananda van der Veen’s Provisioning Bare Metal with OpenStack talk (slides here). Although I am quite familiar with it now, it’s cool to see it in context and learn about some of the future plans. The Tempest – Gap Analysis – Identify new tests session took a look at some of the gaps in testing that some of the projects have, there are a number of them and they currently lack manpower to write them, a few new blueprints will be created to help track progress.

One of my favorite sessions of the day was Vulnerability management: infra needs, scoring…. The topic was handling of security patches and how our team could facilitate that work — preferably without completely duplicating our entire development infrastructure in private exclusively for the security patches. Ideas tossed around included pointing security testers in the direction of the git repositories for tests we use so they could run them, plus offering to run a stand-alone Gerrit-only server (rather than the full infrastructure so they could use a familiar code review system when handling the changes with key team members. Discussion then moved into vulnerability handling in general and it was really interesting to learn about the teams involved. Final session of the day was Technical Committee membership evolution. Given my work with Ubuntu this governance discussion regarding the mechanisms for handling the growth of the TC was an interesting one, even if they didn’t come to any formal conclusions.

The evening wrapped up for me by heading out for an enjoyable and geeky evening with some of my -infra colleagues. I work with some really fun, open source committed folks and it’s really cool to be able to meet up in person during these summits.

Thursday was the final day of the conference! Moving on from QA, it was a day of -infra discussions. First session of the day was Continuous-deployment for upstream Openstack where the team discussed some of the goals and intentions for such a deployment model (alongside the standard 6 month release cycle) and sought to acknowledge that there were organizations who were doing this already in various ways. Next up was Dependency Management, where most of the discussion centered around discussion of Python modules and that the -infra team now maintains requirements project for the new pipy mirror. The Sorting out test runners, wrappers and venvs session focused on the fact that a lot of people are confused about how to properly run the same tests that are run in the CI infrastructure, so are left puzzled when their commits fail. The last session before lunch was another -infra meeting centered around Failure management in the gate. The concern was the few days in the past cycle where it could take several hours for a change to merge due to a couple of issues with the various moving parts in the gating infrastructure. There was some discussion around how to optimize these jobs (and failures) and also improvements to the notification system for such slowness or failures beyond the IRC bot in the #openstack-infra channel, added this past cycle.

After lunch it was off to the Bare Metal Testing session. We had already talked about the virtualized testing earlier in the week (essentially used to quickly test the logic of the process), so the focus with this session was how to handle bare metal testing on actual bare metal. Some of the challenges and risks encountered with testing on bare metal were identified and the session ended with some action items to put together a document to give to any potential donor of this hardware and data center space to explain the requirements and expectations. OpenStack CI logging Review of non-infra-managed tooling was an interesting session to review the OpenStack core tooling that isn’t yet under control of the -infra team and plans to move that over. I took a couple action items in this session to help draft up some puppet modules to pull some things over. The OpenStack project has had to cease using the code name for the networking project due to a trademark issue (details here) which led to the Projects (re)naming session. It turns out the code names for the project aren’t protected under OpenStack trademarks and it would be very costly to do so, particularly as the project grows, and they feel it would be inevitable for a situation like the networking software’s to arise again. The consensus tended to be to try and reduce the dependence upon code names in official and beyond development public spaces, but it was acknowledged that it’s a hard human problem to get around. The last session I attended of the day was on the Havana release schedule where Thierry Carrez presented the proposed release cycle for the next 6 months and took some feedback about it and then the summit in general.

Thanks to Thierry Carrez for this photo

Right after that last session I headed back to the hotel to grab my suitcase and then it was off on the uneventful train + plane trip home.

Since my new job is working with the OpenStack infrastructure team, I am able to attend the OpenStack Design Summit this week in Portland, Oregon!

I woke up just before 4AM on Monday to take a 6:15AM flight to Portland, putting me in Portland shortly after 8AM and to the conference center around 9AM, at the start of the event. I learned quickly that the unfortunate part about arriving at that time is the huge registration lines:

ODS is very similar to the UDS (Ubuntu Developer Summit) that used to be held twice a year by Canonical, indeed it was somewhat modeled after it. It’s nice to have the UDS experience under my belt so everything felt familiar and I was able to dive into sessions without feeling lost. It’s also been delightful to run into a lot of people who I know from the Ubuntu community who are also now working on OpenStack in some capacity.

There are a few differences though. First is the attendees, while UDS really was all about people developing the pieces of Ubuntu (software, packaging, documentation, design), ODS is a summit for both developers and users. Then there is the size of the summit. I don’t believe UDS ever broke 1000, but the expanded scope of ODS means that there are a lot more people, I believe the number floating around is somewhere around 2500. The format is also slightly different, instead of every session being workshop-like where you all sit around working on a blueprint for something, there are also presentations. The topics for presentations do range a bit, many being user-focused but also developer-focused ones to entice development and feedback on new initiatives. It has made for a really interesting mix that I’m quite enjoying.

The first two sessions I attended were OpenStack-on-OpenStack Overview (or “TripleO”), which was a workshop session going through some of the general ideas and blueprints for TripleO plans this cycle and sessions at the summit. I was then off to Robert Collins’ standing-room-only presentation of OpenStack on OpenStack: deploying OpenStack via bare-metal.

Robert Collins on TripleO

Now I am quite familiar with the TripleO project, having worked on testing baremetal stuff in their incubator these past several weeks to work out how we’ll be eventually including it in the OpenStack continuous integration testing infrastructure. It was however nice to take a step back and look at the more broadly with the more long-term view.

In the afternoon I attended a couple documentation-related sessions, starting off with Translation management enhancement. The project has translations for documentation that have been chugging along, but it seems like these past several months and into this next cycle they’ll be working to expand upon this and make it more organized and streamlined for contributors. Then there was a session for Documentation for Newly Integrated Projects, where questions around when to integrate were raised (while they are in incubator? after? upon release where they are official?). I was happy to learn from this session that the docs in general were going through a bit of a reorganization that had been discussed early in the day, while I have found the existing ones to be an invaluable resource and am delighted that OpenStack has placed such a commitment on docs from an early point, there is always room for improvement and making things easier for everyone!

Toward the end of the day I got to my first testing-related session (which, strictly-speaking is most applicable to my job as someone on the team who runs the infrastructure for the CI-related testing), Reviewing OpenStack Networking Unit Tests. In this session they reviewed the current tests they are doing with OpenStack Networking, looking over types of tests, their effectiveness and current speed at which they run.

I also made it down to the expo hall. It was full of OpenStack!

It was pretty cool to see all these companies so invested in the project already.

And thanks to Benjamin Kerensa for taking my photo at the Ubuntu booth:

Monday evening I went out with a bunch of my colleagues to Montage for some fun conversation and delicious Louisiana-inspired food (I had spicy, cheesy macaroni with alligator).

Tuesday was a busy day. The opening of the day was an exciting one, Jonathan Bryce, Executive Director of the OpenStack Foundation, gave a talk where he discussed the progress of OpenStack and then introduced three major companies who are all using it in some capacity: Bloomberg, Best Buy and Comcast. It’s always super motivating to see major companies like these making this kind of serious investment in a technology you’re working on.

As far as sessions go, I started off by going to the Heat RefStack – A reference implementation of OpenStack talk by Monty Taylor of HP and Rob Hirschfeld of Dell (amusingly the computer they presented from was a Thinkpad). The core of their talk was really focused around interoperability between the public clouds running OpenStack so customers truly have options when it comes to selecting providers. To meet this they introduced RefStack, a plan for an OpenStack reference implementation.

I was then in testing mode for pretty much the rest of the day! Started off with Strategies for Gating in a growing project as discussions centered around how to best handle the number of tests increasing as more projects are added and existing ones get larger. After lunch it was in to Multi-node Openstack Testing, which began with an Openstack in Openstack (not to be confused with on) presentation by a group who has been working on their own OpenStack running on Openstack implementation for testing and more. Then it was on to discussing how the CI team may be using TripleO style testing for the baremetal testing, and since I’ve been working on this it was a chance for me to take a more active role in participating in a session. I even sat up front. From there it was on to FITS testing of public clouds, where some of the mechanisms and ideas for a testing mechanism for OpenStack deployments could be tested for inclusion of certain features, perhaps even implementing an eventual “certification” type program that tracks public clouds that are using enough OpenStack to be considered OpenStack-powered clouds.

Gating/Validation of OpenStack Deployments was next up. The QA team has many types of tests that they do, and there was some discussion about what should go into the CI infrastructure vs. what should be tested by developers and beyond, not necessarily being things that code commits depend upon. A fair amount of talk was around things like stress testing, which can be sometimes difficult to do in a virtual environment where results depend upon the cloud it was running on, but more on that in a moment. I then attended Upgrade testing and Grenade, Grenade is a tool which, using devstack, tests an upgrade from one OpenStack release to another and was deployed on the CI infrastructure this past cycle so this discussion was to discuss changes and improvements. Last session of the day was Beyond the API – End to End Testing of OpenStack, which is where the topic of resource performance testing came up again. The was consensus that while it’s tricky to do on hosted cloud VMs due to variability of environment, these more qualitative tests could be tracked based on averaging out tests through metrics that the CI team already produces (or could work toward producing more/better of). This way they don’t need specific details about how each individual test runs for some of this, but instead could rely upon over time metrics to show if there are any kinds of trends.

Tonight! Each night there are parties put on by sponsors, and tonight was the one by HP. Of all the conference parties I’ve been to, I have to say this was one of my favorites. They had people going around serving drinks and food, the food selection was enjoyable to me, the venue was spacious and it wasn’t too noisy – you could easily have conversations! And conversations I did have, mostly with the awesome Portland Debian folks who came by and various Ubuntu folks who I know through my Ubuntu work.

It was a great start to the week, looking forward to the next two days!