• Archives

  • Categories

  • Other profiles


  • wallaceadngromit.net

  • Partimus

  • Xubuntu

  • DC/OS

CubaConf 2017

I spent the second week of November in Havana, Cuba for CubaConf 2017.

It’s an open source conference, and so much, more as their website explained:

CubaConf, is an international conference about free software and open technologies that will have its second edition in La Habana from November 7th to 9th, 2017. CubaConf is a meeting from the international community of enthusiasts of the free technologie’s world, peer to peer focused and collective development. It seeks the participation of people with different profiles and diverse geographical locations.

From the organization of the conference, it is aimed at the diversity of both topics and projects, as of its participants, so the participation of women, minority groups, in these activities is crucial to build new inclusive spaces of knowledge and dialogue about technology. The main idea is to know people experience and organizations who work with and in free software, with old hardware technologies and little or slow bandwidth, to talk about how free software can help developing countries.


Conference Registration

I learned about the conference from and then tagged along with the OpenNMS crew, Tarus Balog and Alejandro Galue, which I was grateful for. Last week Tarus wrote a great blog post about the conference here: 2017 Cubaconf. It was nice having folks to walk with as we wandered through the streets of Old Havana to find the venue on Tuesday morning. The format of the conference was a day of planned conference talks, one day of unconference and then a workshop day.

I joined them to give my Open Sourcing of Infrastructure talk, but this is one conference where I’m certain I learned more from the attendees and environment they work in than they ever did from me. The day began with a keynote from Ismael Olea who came to the conference from Spain where he participates in HackLab Almería. His talk was in Spanish, and he spoke quickly, but thanks to verbose slides (available here) I was able to make out the gist of his talk. Almería is not a major global city, but they’ve worked to build a thriving community through events and online built around giving people the autonomy to self-organize and do whatever they want, as long as it aligns with their ethos of technological, social and creative experimentation. The broad, generalist approach echoed much of what I heard from Josh Simmons at Linux.conf.au earlier this year: when you live in a less densely populated area, multidisciplinary groups are key to success. The added element of self-organization definitely appealed to the Cuban audience, which was a constant theme throughout the conference.

The first talk after the keynote I attended was by Valessio Brito who presented a career path on how you can make money doing open source software. He outlined opportunities for working in support, consulting, customization and development, and stressed the importance of building an open source portfolio. Even once you’re hired, he expressed the value to your career of staying involved with the open source community instead of being overtaken by the tasks internal to the company where you work. Of course as a developer advocate I’m thrilled when our internal engineers express an interest in being a part of our community, so it was nice to hear from a third party too.

My talk on The Open Sourcing of Infrastructure was next. This is a talk I’ve given at a few different places now, and I think I got the most interesting response yet from this Cuban audience. The talk stresses the importance of controlling your data and resources instead of trusting a third party company to do it, and this really resonated. For the Cubans I met, the ethos of free software (not just open source) was incredibly important. I learned quickly while there in Havana that they have a very resourceful, DYI culture that values the ability to have control of your resources. While reading up on this phenomena upon returning home, I found this article from PBS News Hour in 2015, How communism turned Cuba into an island of hackers and DIY engineers. Had I been more aware going in, I could have shifted my talk away from convincing them to use open source infrastructure tooling, they were already convinced! If I were to do it over, I think I would have focused on how they could do it, instead of the specific technologies (OpenStack, DC/OS, Kubernetes, etc) being a single slide of recommendations.

Unfortunately my talk slot overlapped with my friends from OpenNMS! But I’m sure they did a fabulous job.

At lunchtime we walked over to the nearby Casa de África where they had outdoor event space for us to enjoy sandwiches. It was there that I met an engineer who was deploying DC/OS for a bank in Mexico and several folks from the US and Europe whose attendance at the conference was deeply tied to the spirit and message that free software brought, not just the open source aspect of it. It was interesting to learn their perspectives and what brought them there to Cuba.

After lunch I went to a talk by Christian Weilbach on “Free data and the infrastructure of the commons.” Companies control vast amounts of data today, most obviously by companies like Google and Facebook, but generally by most companies who have a technical presence and customers. He explained the risks here, and his plea was to work to adopt open source methodologies to the free access of data. He didn’t have the answers, but was keen on seeing us get there because of how important data is in today’s world. Indeed, a quote I took from the talk centered around the future money being in data, not in the technology stacks which are being commoditized.

The final talk of the day I attended was by Molly de Blanc on “Freedom Embedded: Certifying fully free hardware.” We’ve met at a few conference, so it was a wonderful surprise to see her at this conference and get the opportunity to catch up. Her talk began with some background of the Free Software Foundation (FSF), where she works. I find their perspective on free software to be on the extreme side, but her talk was about their “Respects Your Freedom” hardware product certification, which is actually really cool. They’re seeking to document hardware that runs with 100% on free software, documenting their criteria for certification here. She also mentioned h-node.org during her talk, which I wasn’t familiar with.

Then we had the conference group photo! Not everyone made it in the picture, but it was a decent chunk of the conference attendees.


CubaConf 2017 Group Photo! Credit: Gabriela Fernández (source)

That evening we made our way to the evening social held at a gallery in Old Havana. They served light appetizers and offered an open bar, where I got to enjoy some rum on the rocks. Rum isn’t usually my drink, but I was in Cuba! That event was where I was able to speak with a developer from cuban.engineer where I learned a bit about how they do software development in Cuba. At his company they rely heavily upon their local LAN instead of having internet access in the office, and collaborate on a local GitLab instance. Collaboration and syncing upstream and online is done, but it’s simply not a part of their constant daily activity. This is in sharp contrast with every software project I’ve ever worked on, being a remote employee for over a decade and involved with online software communities since 2001, internet access is essential to my ability to collaborate. It’s a very different environment than I’m used to, and it was fascinating to learn how they make it work.


Snacks and socializing at the first conference night social event

Wednesday began with a keynote by Dr. Mixael S. Laufer, who I had the pleasure of meeting the day before. The organization he belongs to seeks to equalize access to healthcare, and drawing from similar tenets of open source and Maker movements, encourages a DYI approach to healing. Again, giving the culture in Cuba this was a fantastic message and I’m sure one they already work towards, but it was a fascinating topic for me to learn more about.

The rest of the day was reserved for the unconference, which began by people writing down and pitching their ideas, and attendees voting on sessions which then landed on a schedule put together there on site.

The first discussion I ended up in for the unconference was from Zak Rogoff, who told us a tale of how the Free Software Foundation tried, and failed, to fight back against DRM entering web standards. He recounted the grass-roots campaigns they attempted. I have to admit that I continue to be put off by the campaigns that the FSF launches because I do find them to be on the extreme or childish side. Still, with this campaign they had a very firm foundation in something I believed in, and it’s a shame they failed to get the attention of me and others like me. One of the great things they did do was partner with the EFF and Cory Doctorow which led to An open letter to the W3C Director, CEO, team and membership. I believe they would have had more success if more of these strategic partnerships with organizations I hold in high regard had been made, along ones which could have spoken more directly to the serious challenges DRM presents to security and accessibility. His conclusion was the same. I hope the FSF takes this lesson he presented to heart, for the most part they are fighting the good fight, I’d just like to see a better approach.

Lunch was next! I was able to spend more time chatting with attendees, but unfortunately by the time we were wrapping up in the mid afternoon I wasn’t feeling my best. I had to depart and head back to the AirBnB for the rest of the day, with the exception of a stop at the pharmacy and a futile attempt to eat some chicken soup for dinner. I’m even more disappointed to report that being ill took me completely out of the game on Thursday. That means I missed the women group photo! But it was nice to see a decent representation of women at the conference, I wish I had made time to meet with more of them.

Huge thanks to the organizers of this event who made me feel so welcome. In spite of not being able to attend all of them, I appreciated the existence of social events each night and had a nice time chatting with people throughout the event.

Mi visita a Cuba

I was recently honored to have a speaking engagement accepted for CubaConf 2017. The conference lasted Tuesday through Thursday, so I flew in on Monday and left on Friday, giving me some buffer on each end of the trip to get settled in and explore the city a little bit while I was there. I’ll write about the conference itself soon, but I want to first talk about my experience in Havana.

I learned about the conference from Tarus BALOG in blog post and he was who I reached out to as soon as I knew my talk was accepted. Turns out he had a room in an AirBnB that I could stay in, perfect! The trip quickly turned into a gathering of minds with him and a colleague from OpenNMS giving presentations as well. I was picked up at the Havana airport mid-day on Monday by a cab sent by the AirBnB. But before we get to the AirBnB, let’s talk about cars.

Whenever I saw anything about Cuba, the cars from the 1950’s played a role. They’re on the posters, post cards, advertisements, image searches for Havana, everything. This is caused by the trade embargo which limited imports after that decade. But things have loosened in recent years, how true is the rolling museum of cars in Havana today? There are definitely some modern cars, but wow, there really are classic cars everywhere. From what I’ve read, they’ve also got all kinds of crazy things under the hoods as the resourceful Cubans did whatever they had to parts-wise to keep these old cars running. This is a great article about it all, with tons of pictures: Here’s What Cuba’s Car Scene Looks Like In 2017.

We stayed on the third floor of Gallery III – In the Heart of Old Havana, a building which contained three condo units rented out as “homestays” via AirBnB. Our host Yaima met me upon arrival and kindly shared some cake the staff was having to celebrate her birthday. I wasn’t sure what to expect with this lodging, but was generally pleasantly surprised. The WiFi was not working the first couple of days we were there, and it was only on by-request in the common area downstairs anyway, more on internet access later though. I took the bedroom with the smaller (but still Queen!) sized bed and a bathroom for myself right next to the kitchen which included a stall-less shower and drain in the middle of the room. My roommates for the week took the larger rooms and more modern bathroom, though I did borrow their shower one day. We had a little trouble the first day with hot water, which meant some quick cold showers, but they were able to fix that on subsequent days, even if we did have to strategically time our showers around others in the building.

Everything else was wonderful. The condo itself was beautifully decorated and well-lit. They stocked the refrigerator with bottled water (you can’t drink the tap water), sodas and beers that they billed you for at the end of your stay. In spite of the 80+ degree heat, simply opening the doors on both side of the condo and running a fan made for an incredibly comfortable place to chat, or read, both of which I did a lot of during my stay. The bedrooms had individual units for air-conditioning, which was valuable to a good night sleep, and essential later in the week when I wasn’t feeling well. They also provided breakfast each morning for what amounts to $3/day and there was always at least one staff member there to keep an eye on the cameras, watch for us having trouble getting inside, provide recommendations for food (or pharmacy!) or to turn on WiFi when we needed it. They also had staff who cleaned the condo and made the beds every day, a nice touch I wasn’t expecting.

After settling in a bit I was able to take some time to walk around Old Havana. I walked over to Castillo De Los Tres Reyes Del Morro, another one of the Havana icons. Then made my way past expansive, beautiful hotels, sadly many of which landed on the restricted list for US travelers on November 8th, while I was there.

The rest of my afternoon was spent at the AirBnB condo reading and relaxing following what has been a crazy month of travel for me. My roommates arrived in the evening and we made our way out to the first cocktail stop: Floridita. You see, my friend Tarus is not only is an open source enthusiast, but also runs a Vintage Spirits and Forgotten Cocktails blog! He was the guy to know for cocktails in Havana. Now, the big claim to fame that Floridita has is as the birthplace of the Daiquiri. We ordered a trio of them. Unfortunately they had run out of grapefruit juice so we couldn’t enjoy the papa doble (Hemingway daiquiri). Ernest Hemingway was a daiquiri fan and had them at that very bar! I got my selfie with Hemmingway while I was there and we enjoyed live music. As the evening wore on we decided to also eat at the restaurant there. Specializing in fish dishes, it was a perfect ending to the evening.

Throughout this lively Monday I was tweeting, but that’s only thanks to a very expensive Digicel SIM that MJ bought for me for my trip. $25 for the first 100MB, then $50 for each subsequent 300MB. I’ve grown accustom to my Project Fi SIM working in most of the world, so it was a surprise to my wallet when I had to pay such a premium. Still, I quickly learned why. All the internet I encountered on our trip was mobile internet. Old Havana is dotted with public access points that people buy access to by the hour via codes they buy at local shops. This is what my traveling companions did, and this is also how the AirBnB worked, since they were near (or owned?) one of these public access points and they just covered the costs as part of business, and this is why we needed their help when we wanted to get online. This created a fascinating side-effect: you could always tell when you were near a public access point because everyone was on their phones. Elsewhere? Not so much! It did come as a surprise to me that this is also how many organizations do internet access as well, the university where the conference was held had a LAN for us to upload our slides and share photos on, but no outside access, they advised using a nearby WiFi park with the cards if we needed access. Even people working at tech companies there in Havana told us that they did most of their work locally on their LAN (yay GitLab!) and only synced up to the actual internet at specific intervals. I’m convinced that they must be much better technologists if they don’t have the ability to Google answers to everything, hah!

Tuesday and Wednesday were pure conference days. Tuesday night found us at a conference event, which they put on every evening during the conference at various locations throughout Old Havana. Walking through Old Havana to and from the conference venue and events was a exploration unto itself. There were a few streets shut down to car traffic and filled with tourists and shops. I was able to pick up some post cards and stamps to send to my regular recipients. I skipped the rum since I’m not much of a rum drinker and didn’t want to check my bag on my way home, but I did pick up a couple cigars, though I’ll have to think about who to give them to. No one smokes anymore …but they’re Cuban cigars! The infrastructure there leaves something to be desired, the humid climate wreaks havoc on permanent structures, and there’s evidence of that throughout Old Havana. There are also a lot of pot holes and it’s often difficult to figure out just where you want to walk. Still, walking at night with a group felt pretty safe, and I got to see lots of cats, who appear to come out after dark!

Unfortunately my stomach became the ruler of all my plans starting Wednesday afternoon. I was struck with a brutal bout of travelers diarrhea and forced to remain at the AirBnB for the third day of the conference and the social events, one of which was at a brewery! So sad! I also missed further cocktail adventures with Tarus and Alejandro, no cocktail at the Hotel Nacional for me. In addition to bio-breaks, Thursday was essentially spent napping and listening to audiobooks since I wasn’t even well enough to read or look at a screen. I did get to visit a local pharmacy though, and that was a nice adventure. I was sick for several days though, but thankfully by Friday I was well enough to at least tolerate getting on a plane.

I shall conclude by talking about cars again. A real treat awaited us when we came downstairs to catch our cab to the airport, an orange 1953 Pontiac!

It was my only opportunity on this trip to actually ride in a classic car, and I was really pleased with how well-maintained and improved it was. It lacked seat belts that I instinctively reached for upon getting into it, but it had top notch air-conditioning. The round side mirrors had blinker lights built into them. And everything about the car was shiny and beautiful, like many of the classic cars I’d seen. It’s clear that the owner spent a lot of time maintaining it and it was a beautiful way to conclude my Cuban trip.

Meetups in Germany

Earlier this month I spent several days in Germany. I made some time to do some tourist stuff, but I was actually there for work. I visited and worked from the Mesosphere office for a couple days and host a Meetup. I also made my way over to Berlin for a day to host a Meetup. Throughout my journey I was able to enjoy good German food and sights, and meet lots of friendly people!

The Mesosphere office itself is on the top floor of an office building, at the 7th floor that makes it a reasonably tall building for the city so there are some nice views of the surrounding area.

But Meetups! The first Meetup of the week was over in Berlin. The topic of both the Meetups was “MesosCon Recap” which consisted of me presenting a short slide deck (slides for Berlin) that gave an overview of what MesosCon was, some of the key themes, and what came of events like the new Town Hall on DC/OS. After my introduction, for each location we invited speakers who were local to that region of Germany so we had a different line-up for each evening. Each speaker was given a half hour slot to give a shortened version of a MesosCon EU talk. It was a new format for me, but it was a lot of fun, and I enjoyed bringing a piece of MesosCon to folks who couldn’t make it out to Prague.

The speaker lineup in Berlin consisted of:

  • Till Rohrmann of Data Artisans on “Apache Flink Meets Apache Mesos and DC/OS” (slides)
  • Kevin Klues of Mesosphere on “Running Distributed TensorFlow on DC/OS” (slides)
  • Tim Nolet from magnetic.io on “How to extend Marathon-LB with Canary releasing features using Vamp” (slides)

Huge thanks to all of them for coming out, both Kevin and Tim gave great talks at MesosCon EU, and it was great to have Till join us to present using a slide deck that was not his own, but as a leader in the subject matter he did a fabulous job.

The staff at betahaus, the venue, were also incredibly helpful. They got us all sorted with the projector, pizza delivery and with drinks throughout the evening.

Thursday night I was joined by my colleague Matt Jarvis for the Meetup in Hamburg! At this one Matt and I ran through the introduction slide deck and made some adjustments (slides for Hamburg). This introduction was longer than the one I did in Berlin, partially because the speaker lineup was shorter, and because I had the help of Matt who took time to flesh out some of the impressive statistics about cluster and workload sizes from the keynotes, and he could cover the tracks he was track lead on, so it wasn’t just my perspective from the operations tracks.

Following our introduction, we were joined by our pair of MesosCon EU speakers from Mesosphere for slightly shortened renditions of their talks:

  • Johannes Unterstein on “Marathon and Jobs – Today and Tomorrow” (slides)
  • Adam Bordelon on “Mesos Security Exposed!” (slides)

Logistics were a bit easier since this Meetup was held at the Mesosphere office. They tell me that we don’t normally have a skeleton joining us in the large meeting space, but it was just after Halloween so the decorations were lingering. It was a fun night and the crowd was a bit bigger than the one we had in Berlin, but I admit that I skipped having Meetup pizza for a second night in a row and instead joined Matt for a later dinner at the restaurant in our hotel.

Wall, Egyptians and a friend in Berlin

On November 1st I hopped on a Deutsche Bahn ICE train in Hamburg. A couple hours later stepped off in Berlin. It was drizzling out, but not enough to put me off the plans for the day, which included visiting the remains of the Berlin wall, the Egyptian Collection in the Neues Museum and a late lunch with my friend and Berlin local, Daniel Holbach.

The wall was first. The fall of the Berlin Wall happened in 1989 when I was just eight years old. At the time I was too young to understand what had occurred, but it and the cold war were recent enough that I learned a considerable amount about both at school in subsequent years. Seeing it in person brought up feelings, it was an important stop on my journey through Berlin. Still, I skipped the crowded visitor center and was satisfied with my self-guided walk around the grounds.

From the wall I walked south to the museum. Seeing The Egyptian collection in Berlin, now housed at the Neues Museum, has been on my list for years. It includes the famous Nefertiti Bust, which at over three thousand years old, has been remarkably preserved. I had seen photos of it and it was featured in documentaries, and it recently grabbed headlines through the controversial Nefertiti Hack. I’ll admit right away that the long-standing claims the Egyptian people have to the bust did cause mixed feelings when I went to see it, but my fascination with ancient Egypt won out, I really wanted to see it. It’s hard to tell from flattering photos and angles just how impressive the preservation really is, so seeing it in person was a wonderful experience. No photos were allowed in the exhibit area, but there are plenty of them already.

Stepping back to take in the whole collection is a bit overwhelming. They have a huge room dedicated to their collection of wall segments from ancient tombs and buildings, some are just etched but many of them have period paint still attached to them. I nearly skipped the basement level in the interest of time and energy, but I’m glad I didn’t, that’s where they had some of the more intricate sarcophagi and their collections of animal statues, charms and figures. I’ve seen the collections at the British Museum in London (technically a larger collection) and the Met in NYC, but the presentation at the Neues Museum really stood out for me.

I think part of what made it so impressive for me was the building itself. Built in the mid 19th century, even the rooms in this museum were worth taking pictures of. I’m sure when I took this picture it was all about the room, not the artifacts:

And while I remember that this room was the one housing papyrus, the only reason I remember that was because you can just barely make out the Nefertiti bust at the end of the room, incidentally included in my photo as I was admiring the room.

Once I had concluded my time in ancient Egypt, I did quickly make my way through the other collections in the museum. Of particular note was the Berlin Gold Hat. It’s a tall, conical hat that estimates of similar artifacts has tended to put at around 3000 years old, though unfortunately this artifact itself lacks verified provenance. It’s housed in its own dark room, surrounded by related artifacts. At almost two and a half feet high, it’s definitely something you should see if you visit the museum.

The afternoon was creeping on in as I hopped in a taxi to head south to my hotel. I was able to check in, drop off some of the contents of my increasingly heavy backpack and head out to meet up with my friend Daniel for lunch. We ended up at a Mexican place for a pair of the largest burritos I’ve ever seen. They were pretty good too.

I’ve probably known Daniel for a decade. He was a community manager and developer advocate in the Ubuntu community before there was a name for such roles. His guidance and gentle leadership helped me and so many others be successful members of the Ubuntu community and he’s one of the kindest people I know. He took a break at the end of last year when he left Canonical and I hadn’t really been in touch, so I was thrilled when he replied to my email inviting him to meet up in Berlin.

We had a great afternoon of catching up, wandering through a park, down streets, stopping for coffee. He also know where the event space was for the Meetup I was hosting that evening, so we walked over there as the afternoon wound down and I was able to get settled in before he departed. The Ubuntu community was a very special place for me at a formative time in my life, and I’m so pleased that I have these relationships with people around the world.

I spent that evening hosting the Meetup there in Berlin before walking back to my hotel. I’d say my only mild regret is that I didn’t wake up early enough the next morning to visit the Berlin Zoo before taking the train back to Hamburg. I was starting to get a bit travel weary and I did enjoy the relaxed morning and time it gave me to catch up on some email. I’ll be back some day!

More photos from my visit to Berlin here: https://www.flickr.com/photos/pleia2/albums/72157665780629679

Tiny things in Hamburg

I spent a week in Hamburg, Germany recently to meet with colleagues at our Hamburg HQ and host a couple Meetups. Booking my visit on the heels of MesosCon EU meant that the week I was there coincided with a national holiday. I learned that on October 31, 1517 Martin Luther nailed his Ninety-five Theses on the door of a church, a key point in the Protestant Reformation. Germany was observing the 500th anniversary of this act. I spent this Reformation Day exploring Hamburg.

My exploring had nothing to do with religion, so my first stop on this holiday from work was to the Miniatur Wunderland.

The largest model railway in the world, and one of the most successful permanent exhibitions in Northern Germany.

Now, I have been getting into model railroads, but calling this a model railroad doesn’t quite do it justice. It’s really a whole miniature world, including a model airport that cost 4 million Euros to build. The place takes up three stories of a historic warehouse building, with dozens of people employed to run it all, and along with an impressive control center, there are three racks of hardware at the entrance.

Stats and train nerd stuff aside, it’s a fun place to be. I bought tickets late, so the only ones available online for the day were in the 7-8AM window right as they opened for the day. This timing was not easy with my jet lag situation, but it was worth it. Since it was a holiday, the place was packed with people by 9AM, so getting in early was a good call, I was able to take lots of great pictures before it got too crowded to get close to a lot of the exhibits. Throughout the morning I walked my way through various countries in Europe (various German towns, Switzerland, Austria, Italy) and even a small section representing the United States. The scaled down version of Hamburg itself was pretty cool, you’re able to find the museum itself and I saw many familiar spots even having just been there a couple days. You could spend days exploring the details that were included throughout, I spent about two and a half hours there and feel like I only got an overview of the layout. I’ll have to go back next time I’m in town.

I may have gotten a bit carried away with the picture taking, over 400 more photos from Miniatur Wunderland here: https://www.flickr.com/photos/pleia2/albums/72157665685279559

October 31st is also Halloween. It’s a much bigger holiday in the United States than most other places, and over the past few years I’ve been traveling on Halloween more often than not so I often don’t get to properly celebrate. I wasn’t going to let this year get away from me though, so I booked morning tickets at The Hamburg Dungeon, conveniently located right next to Miniatur Wunderland. These dungeons are a chain, there’s even one in San Francisco, and the idea is that they hire actors to act out dramatized versions of some of the more gruesome history in their respective cities. It’s silly, but a lovely haunted house-like way to get a taste of Halloween as a solo tourist.

I then took a walk to find some food and ended up at Kartoffelkeller, potato cellar! So many potato dishes. It was good and I finally had a beer. Well-fed, I spent the rest of my late afternoon back at my hotel, 25hours Hotel Hamburg HafenCity, where they have ship cabin themed rooms and a whole floor dedicated to games and socializing, but for me made a great spot to get some writing done as I enjoyed the herbal tea they had available to guests. Yum. It was back to work and a detour off to Berlin, which I’ll write about soon, before I returned to Hamburg on Thursday evening for a Meetup.

Now, the other thing I learned about Hamburg during my visit was how water-centric the city is. I knew Hamburg was in the north of Germany, but I had never zoomed in on a map and noticed that it was connected to the North Sea by the huge Elbe River. This river not only winds its way through Hamburg (so many bridges!) but also creates a huge harbor that industry in the city appears to revolve around. Friday at lunchtime I went out with my colleague Jörg who took me up to the Elbphilharmonie, a concert hall just a few blocks from the office. Every day you can get free timed tickets to take some very long escalators up into the concert hall to walk the perimeter and see some amazing views of Hamburg and the harbor.


Lots of water in Hamburg!

As work wound down on Friday I was able to make time to visit the International Maritime Museum, and even better, since they close at 6PM I learned that after 4:30PM the tickets are only €6! An hour and a half isn’t a lot of time to explore that museum, it’s several stories and takes you through about three thousand years of maritime history, but my visit was long enough for me to enjoy it. I will say that one of the most interesting things about this museum was how many ship models there were. The top floor was entirely made up of models, but every floor had models of varying scales and types, from boats and ships, to engines. I was starting to get the impression that Germans, at least those local to Hamburg, really like models.

The highlight of the museum for me was their display of Enigma machines. I’d seen an Engima machine before, but this was the first time I’d seen a pair in a museum in Germany. The visit to the museum was worth it just to see these.

It was a bit of a whirlwind visit, and I really didn’t get to see much of the city or explore the local cuisine much. Hopefully my work will bring me here again in the near future.

More photos from wandering around Hamburg here: https://www.flickr.com/photos/pleia2/albums/72157665634578459

MesosCon EU 2017

I attended my first MesosCon was back in September in Los Angeles. In October I head the pleasure of participating in my second, this time the European version in Prague. As I’ve mentioned previously, this was a busy week for me. My participation in All Things Open kicked off the week, and I landed in Prague on Wednesday evening with just enough time to check into my room, say hello to some folks I knew who were in town for the Open Source Summit and then head off to a planned dinner with work folks at Restaurant Mlýnec.

The next morning arrived quickly as I was up and available for a 7:30AM breakfast with some of the MesosCon Europe keynote speakers. Part of the breakfast was spent chatting with the panelists who would join me on stage that morning to participate in a panel on “SMACK in the Enterprise” that I was moderating. At 8:30 it was time to go down to the keynote room and see that everything was on schedule for our 9AM start. The folks at the Linux Foundation do a great job coordinating these events, it was a pleasure working with them as a speaker and track lead throughout the event.

At 9AM Ben Hindman opened the conference with a talk on the current state of Apache Mesos, reflecting on past MesosCons and the past year of developments. Improvements in the platform have included things like nesting of containers, and the creation of the Container Storage Interface (CSI). Fault domains and the promise of multi-cloud also made an appearance in his keynote.

Directly following Ben’s talk was one that, as a pure open source enthusiast, made me really happy to attend. Rich Bowen of the Apache Software Foundation who came to talk to us about The Apache Way. Given my background and current role, I’m familiar with the history of the foundation and loosely keep tabs on their current work. Still, seeing a history presented with a message is always an enjoyable way to consume it, and Rich did a masterful job weaving the history in with where we are today, and explaining what “Apache” means in the Apache Mesos project. From bottom-up leadership to collaborative decision-making and they way they approach conflict-resolution, there’s a lot to admire. He also drove home the importance of transparency in a project, with no decisions being made privately or in ways that are difficult to document for future participants of the project. He also touched upon where the project was today, with over 100 projects within the Apache Software Foundation and the Apache License remaining strong in the industry.

The SMACK (Spark, Mesos, Akka, Cassandra, Kafka) in the Enterprise panel was next, and as moderator I was thrilled to be joined by representatives from Audi, Deutsche Telecom and ASML. From connected cars to how they’re making innovations on mobile networks, learning how companies are using Mesos and the entire stack to do fast data processing really brings all the work we do into focus. Innovation is happening across various industries as we all become more familiar with what we’ll need to succeed in a world with so much data coming at us, and I’m proud to be a part of it.


Thanks to the Apache Community for taking this photo (source)

The keynotes concluded with one from Netflix Engineering Director, Katharina Probst. Internally at Netflix they’ve built a massive infrastructure that is, at a high level, managed day to day by a relatively small team of Site Reliability Engineers. This has been accomplished through the tooling they built called Mantis, that allows not only for the essential autoscaling that a company like Netflix requires, with viewing numbers dropping considerably when each region has their work day, and picking up in the evening hours, but also real time testing and metrics from their platform. The key for them has been not only monitoring to detect that there is a problem somewhere, but finding out exactly where it is and reporting to the engineers what the problem may be, so a fix can be developed without a long session of troubleshooting first. As someone who used to work in operations, this was something I could really appreciate, I’ve spent many long evenings chasing down problems, evenings that could have been made much shorter with the addition of some automation to rule out the usual suspects quickly. The scale of their operations also never ceases to amaze me, and she was able to share some statistics around that as well.

This first day of the conference landed me as the track lead for the DevOps and Ops track. The day began with a talk from David vonThenen around external storage. In this talk he gave a cost/benefit evaluation of local versus external storage, and then, not keeping to just files, he ran the same evaluation when looking at storage for databases, both traditional like Postgres and MySQL and the newer distributed databases. Tying this in to Mesos, he touched upon CSI work and mentioned that REX-Ray is currently being used by DC/OS to handle the attachment of external volumes. These were interesting considerations in what has been a quickly growing part of our ecosystem as demands upon reliable storage solutions for containers quickly increase. His talk was followed by one from Ádám Sándor titled “Knee Deep in Microservices” and which had a Doom theme to demonstrate the new “demons” we have let loose as we’ve also taken all the benefits of migrating to microservices. He cited the key elements of DevOps, resilience, elasticity, resource abstraction, and tooling that helps us monitor containers as key improvements in the microservices platforms that are helping us tame the demons we have unleashed. It was also nice for me to hear these things, these massive distributed systems are complicated, we need to be continually improving our toolkit (weapons!) to effectively manage them.

After lunch Adam Bordelon and Alexander Rojas joined the track to give a popular talk titled “Mesos Security Exposed!” I won’t enumerate them all here, but by digging into how Mesos works, they were able to pull back the curtain and explain what needed securing in your cluster, from API endpoints to use of TLS and the handling of private data (secrets). Julien Stroheker then spoke on “Doing Real DevOps with DC/OS” where he gave a live demonstration of a continuous integration pipeline with Jenkins, Docker, Gradle and Vamp. The sessions for the day concluded with Zain Malik giving us a tour of mesos2iam, “IAM Credentials for Containers Running Inside a Mesos Cluster”.

But the day was not over! Appetizers and drinks at the attendee reception were directly followed by Town Halls, where attendees could casually gather in groups to talk about Mesos, Marathon, DC/OS or Kubernetes. My colleague Matt led the DC/OS session, with me taking notes and pitching in here and there. Our Town Hall began with introductions, with 15-20 people in attendence as the 90 minute evening session progressed we had a nice mix of people from a variety of unrelated industries. With the ice broken, it was easy to get people talking about pain points they had with DC/OS and we had Adam Bordelon in the room to give history and insight into specific features and concerns that people were bringing up. At the end of the session we had a nice list of shared struggles, but the attendees also were able to swap knowledge with each other. Empathy goes a long way, and in my own work I know how valuable it is to know that concerns you have are being shared and solved by others too.


Attendees file in for the DC/OS town hall Thursday evening

The Town Halls wrapped up at 8:30 and I headed straight to bed. Between the jet lag and the 13+ hour day, I was exhausted.

On Friday the keynotes also began at 9AM, but the keynote slot was shorter in order to squeeze more track sessions in during the rest of the day.

After an welcome and opening remarks from Jörg Schad, the first keynote came from Yrieix Garnier who gave a more Enterprise-focused take on “The Future of Apache Mesos and DC/OS” than Ben had explored in his keynote the day before. He was able to pull up statistics about the data-processing power of the platforms, sharing that 50% of DC/OS clusters were running some form of the SMACK or ELK stacks. The big news from his talk was unveiling of TensorFlow support in DC/OS. We then had Pierre Cheynier join us to share a talk that he had originally proposed during our CFP, but we upgraded to a keynote, “Operating 600+ Mesos Servers on 7 Datacenters @Criteo”. The ability to scale is a key feature of Mesos, so it was fascinating to learn about the scale of their work and just how much data they were storing (171PB on Hadoop!). He also shared a series of tips for other organizations looking to operate at this scale, including effectively automating everything (configuration management, scaling, CI system), defensive configuration (things will go wrong, be prepared), visibility to operations as to what is going on (metrics, alerts, tracing), and the importance of doing networking right, and addressing problems like QoS and “noisy neighbors” during design. He also covered some incidents which gave further insight into the sorts of things they were able to effectively prevent, or not.

In a slight shift from Thursday, I spent Friday as the track lead for the Users and Ops track. The first talk came from the oldest company in the lineup, the publisher Houghton Mifflin Harcourt. I was looking forward to this talk, and as Robert Allen got into the details of “How HMH Went from Months to Minutes for Infrastructure Delivery” I was not disappointed. In a common theme for the day, he brought up the slow, inconsistent, old technology team was not keeping up with either the industry or their own product lines, as the publishing industry is rapidly changing to serve a more tech savvy customer base. He walked us through the creation of their “Bedrock tech services” team, and their DevOps focused goals, including comprehensive developer involvement from idea to production, a continuous delivery approach that encouraged small, frequent updates, and a change in culture that made them shift from feeling like they must prevent failure, and instead acting as if failure will occur and planning accordingly. He then dove into the technologies used, beyond Apache Mesos, they’ve also been using Apache Aurora, Terraform for orchestration, Vault for secrets, and a Jenkins plus Artifactory CI/CD pipeline. He also stressed the importance of metrics and logging, all things close to my own interests as well!

Tim Nolet then joined us to give a talk on “Advanced Deployment Strategies and Workflows for Containerized Apps on DC/OS” where he too walked us down the painful memory lane of the massive, error-prone deployments of the 2000s, the rise of DevOps and today the productization of a lot of the DevOps tooling, including Vamp, which he works on the development of. As a product, Vamp seeks to simplify and tame the ecosystem of microservices you have running by simplifying the process of deployments. He then showed a demo of load being distributed across different versions of a deployed application, as well as different versions being served up to customers using different clients.

In our next user story, we heard from Jay Chin on “Optimising Mesos Utilization at Opentable”. He began his talk with a quick production history of the infrastructure at OpenTable, sharing that they made the move to microservices in 2013. During this time they were orchestrating their microservices via standard configuration management tooling, a process that turned out to not only difficult to maintain at scale, but was actively disliked by the engineers who had to work on it. In 2014 they switched to Mesos, and through resource abstraction and running a consolidated cluster, they were able to simplify application-level operations. Additionally, it allowed them to easily create centralized metrics and logging. It’s a story I’d heard before from other companies, and one I was thrilled to hear again, but where OpenTable really led the way here was by open sourcing some interesting tools they were using, including the following mentioned during his talk:

The day continued with Ilya Dmitrichenko on “Time Traveling in the Universe of Microservices and Orchestration” where he used his own career as a baseline for the changes he was seeing in the industry with regard to the rise of microservices. His dabbling with old Sun boxes and awareness of things like openvz is consistent with some of my own hobbiest and day job work early in my career. Indeed, containers have always been with us. He went on to track the rise of Docker simplifying the space and tooling coming together to make management of a microservices infrastructure feasible for smaller organizations. The culmination of this story was his current work at Weaveworks which, just like Vamp improves deployments, improves and simplifies your network policy.

Jorge Salamero of Sysdig joined us next with the amusingly titled talk, “WTF, My Container Just Spawned a Shell”. I was immediately fond of this talk because after a sea of Macs, Jorge uses Xubuntu. He began his presentation by talking about how we consume container images, and the rise in static scanning of these images for libraries with vulnerabilities and more, but that this only goes so far. There are things that these scans miss because they need constant updating, and which behavioral analysis while the container is running will catch. He introduced how Sysdig’s metrics tooling used with Falco can give you a comprehensive view inside the containers you’re running. Suddenly you have access to security tracking that can show you every command taking place inside of your containers, and from there you can train it to be aware of problem behavior. He also talked about SysDig Inspect, the open source project that they have built their Sysdig Secure product with.

Julien Stroheker then joined us again to talk about his DC/OS autoscaler. This is a talk I’d already seen twice, so I won’t go into it, but it is a cool project that he’s always looking for help on! The final talk of the day came to us from Fredrik Lindner of Tunstall Nordic AB, who shared “There and Back Again: How Tunstall Healthcare Built an IoT Platform for Health Monitoring Using Mesos Cluster on Azure.” Just like so many other industries, elder care is undergoing a transformation with the help of IoT technology. He shared details of their “Evity” platform they were developing on top of DC/OS to help manage the data coming in from these devices so they can effectively and quickly meet the needs of the families they work with. It was a good story to end with, as with much of the technology we saw showcased throughout this MesosCon, as consumers we just assume it will all work. My FitBit will reliably track my steps and share them with my friends, my car will show me traffic, and when something goes wrong it’s unexpected and unsettling. This is even more pronounced with you get into the space of technologies that help with things like healthcare, where reliability, accuracy and speed have even more urgency. We’re building the platform that people are using to make sure these all run well, and that’s pretty exciting.

When I finally stepped out of that track, all the booths had been taken down and most of the conference attendees had gone their own ways. I was able to finally leave the hotel with Matt and Jörg to explore some of the old city in what turned into a lovely night that thankfully didn’t keep me out too late. The conference was great, but predictably exhausting, especially coming on the heels of All Things Open on another continent.

Huge thanks to everyone involved in organizing, and to all the speakers in my tracks who made the event really interesting by sharing their stories, tools and expertise. This event was a little smaller than the one in Los Angeles, but it didn’t feel like it, and the quality of the event was top notch.

More of my photos from the event can be found here: https://www.flickr.com/photos/pleia2/albums/72157689885991786 and more photos, slides and videos are hosted by the Linuxd Foundation at http://events.linuxfoundation.org/events/mesoscon-europe

Tourist in Prague

I’m in the midst of a bunch of travel for work, with a trip to Prague directly following the fun I had in Raleigh at All Things Open. I took a flight out of Raleigh on Tuesday evening, making connections in Charlotte and London before finally landing in Prague on schedule on Wednesday evening. I was there for MesosCon EU, but I’ll write about that later, first I want to talk about what I got to see Friday evening and on the Saturday I spent there to do some tourist things.

The Open Source Summit was happening when I got into town, so I knew a whole lot of people who were there for that event. I was thrilled to meet up with Gina Feichtinger, who I’d long known online through LinuxChix, but never had the opportunity to meet. Unfortunately our schedules didn’t work out to spend more than a few minutes chatting before I had to run off to dinner on Wednesday night, but that was enough time to get a picture!

The conference concluded on Friday night and I was able to get my first real taste of the city as I went out with some colleagues for drinks at Lokal and then a nice dinner together over at U Sádlů. Prague is beautiful at night, and I enjoyed wandering around Old Town all evening.


Looking over at Church of Our Lady before Týn (not a castle!)

Saturday was my big tourist day. I met up with my colleagues Matt and Jörg to head up to Prague Castle. Prague Castle does not look like a castle, I’d call it a palace. There were tours but with just a couple hours of morning on our hands we decided to just explore the outside. We did make our way inside St. Vitus Cathedral, which is inside the Prague Castle complex. It’s a beautiful cathedral.

After visiting inside, we walked around it and then through the village area that surrounds the castle and took in some beautiful views. We then slowly made our way down the steep hill and back to the river. We parted ways as we got back to old town and they had flights to catch. This is when I made my way to a nearby hotel to meet up with my second Matt of the day, this time a friend and former colleague from OpenStack work.

Our first stop was for some lunch at U Fleků. It’s was founded in 1499 and claims to be the oldest brewery in Prague. They’ve really capitalized on this, it’s a prime tourist spot. They wander around with trays of beer, mead and becherovka. The food was nothing to write home about and it was a bit touristy for me, especially once the accordion began to play. Still, I’m glad we stopped by and I certainly enjoyed the beers, mead and the sip I stole of Matt’s becherovka. From there we took the long walk out to Vyšehrad. This is essentially the other castle site in the city, but it’s now a public park instead of a residence, though the cathedral and famous cemetery are in use for their intended purposes. We stopped at a cafe while wandering around to address some beer/mead sleepiness from lunch.

The views from here were beautiful as well, allowing for selfie opportunity!

While in Prague I also enjoyed tram watching. Had I bothered to figure out how to ride them I would have saved myself an incredible amount of walking. Maybe next time. Our journey back to our respective hotels did bring us back to the old town, this time in full weekend swing. It was packed with people. I was pretty eager to be on my way, though stopping for a filled Trdelnik was tempting, the whole neighborhood smelled like them.

My evening was spent with yet another friend who was in town for the Open Source Summit, Nigel! Nigel and I met through our involvement in the Ubuntu community. We’ve both gone our own ways at this point, but we’ve kept in touch and as we’ve both followed open infrastructure paths there’s frequent cause for us to chat. We met at my hotel and went to a nearby restaurant where I enjoyed both the dark and the light lager with lots of fun conversation.

That last night in Prague was an early one, as I had a 7AM flight Sunday morning to head off to Hamburg for my next adventure.

Lots more photos from Prague here: https://www.flickr.com/photos/pleia2/sets/72157687755704821/

All Things Open 2017

The goal of the trip down to Raleigh last week was to attend All Things Open. I attended back in 2014 and was eager to attend again, but was hampered by conflicting events. This year I committed to going, in spite of an unfortunate overlap with the Open Source Summit in Prague, which is important to me because MesosCon Europe happens on the Thursday and Friday of that week. This means I had to get on a plane immediately at the conclusion of All Things Open on Tuesday in order to make it to Prague by Wednesday night to begin my MesosCon EU activities. It was a long week.

But the week began in the most relaxing way possible as I stepped off the train in Raleigh Sunday night. Conference badge was picked up and I was off to a pre-conference social to say my first hellos to fellow conference-goers. I was quickly able to find some friends, but the social venue was quite busy and I was eager to find some dinner.

On the way out of the social I collected some folks and five of us ended up at an empanada and tequila restaurant not too far from the conference venue. Highlight of the night besides good company? They had pumpkin pie emapandas as a dessert. Heavenly.

The next morning the conference began with a series of keynotes. Todd Lewis, the grand architect of All Things Open and super nice guy, kicked everything off with welcomes and piles of gratitude. His personal kindness is one of the main reasons I love this conference so much. It’s also the largest open source conference on the east coast, this year drawing 3000 attendees, which is a new record for the event. That’s how I convince my employers to support the event year after year.

The first keynote came from Tim Yeaton of Red Hat who began his talk with a fun walk through a typical day that brought him to Starbucks, Chili’s, Target and then on to a United flight, all companies who are major users of open source and helping to drive the incredible growth throughout our industry. Jake Flomenberg of Accel joined us next to talk about the rise of open innovation, with companies like Intel and Goldman Sachs choosing open source software first and proprietary only as a last resort, a distinct change from what we were seeing a decade ago. He shared that there’s also a rise in venture capital backing of open source-driven companies who are driving innovation, it’s no longer just Red Hat making money in the world of open source business, others are following the path of Project, Product and Profit.

Danese Cooper, who has worn many hats in her lengthy open source career but is currently the Head of Open Source Software at Paypal, then graced the keynote stage. She began her talk by walking us thorough a history of why many people got involved with open source in the early days, recounting a story from Mark Shuttleworth. He explained his motivation behind investing his wealth in an open source project and company was because he cared and wanted to build something that mattered. The rest of us in open source software weren’t much different, even if we didn’t have millions to invest, my own passion drove my involvement through years of unpaid work. She shared that while many people today are getting involved for the very valid reason of being paid for their work, we can’t forget the lessons we learned along the way. To this end, we need to document our history and motivations so that future generations will know how we got to where we are, as well as mentor new people, serve as board members in open source organizations, and encourage good practices within our communities and company. She concluded by also stressing that we all must continue to follow our moral compass as we contribute, and that money to pay our mortgages will follow. That’s certainly true in my case.

Perhaps the most exciting keynote for me of the morning was from Sara Chipps, the CEO of Jewelbots. Jewelbots are tech-enabled friendship bracelets that have default modes to let you know when friends are nearby, but you can also extend them by writing C++ programs. She gave some history of the company, sharing that she wanted to build something for girls and noticed that they already had traditional friendship bracelets. By creating a tech-enabled one she discovered that the drive to extend the devices functionality-wise would “trick” the girls into having to write code. It worked, of the 10,000 units shipped, 44% of the users have extended them by writing C++ code, an incredible conversion rate. Finally, all the code for the devices has been made open source and is available on GitHub. Super inspiring keynote, it makes me so happy to see things like GoldieBlox and Jewelbots in the market for girls these days.

The final keynote of the day came from Kelsey Hightower of Google and Kubernetes fame. His talk was entirely demo-driven as he stressed that containers were not hype, they do simplify infrastructure by putting applications in nice little boxes. But that they aren’t a panacea, and while they help, we haven’t gotten to our future of flawless automation with them or any infrastructure technology. Much to our delight, he then was able to use Google-driven voice commands to show off not just a Kubernetes deployment on Google’s cloud, but an on the fly upgrade which he sees as the next big evolution in maintainability of large clusters.

With the keynotes behind us, the first talk of the day I attended was by Alicia Carr on Home Automation. The talk itself was a tour of the home automation devices she used in her day to day life, exploring how they interacted with each other (or not!) and her favorites on the market. She addressed the topic of security, admitting that we’re still in the early days of home automation and it’s not where it needs to be to be entirely safe and fool-proof for full, mainstream adoption, but that the companies in the space were aware of these limitations and working on it. Unfortunately there wasn’t much open source to this talk, some of the devices did allow you to code against them, and trigger tooling like IFTTT can allow you do do interesting things. I’d really love to see a talk about the home automation ecosystem of free software hackers that’s emerged, and the open source considerations therein. What devices are the most friendly to hacking? Which can be independently secured most easily?

The next talk I went to was John Mertic on “Accelerating Big Data Implementations for the Connected World” where he focused on the ODPi which seeks to strengthen the engagement model between upstream projects and end users. Five years ago the industry started adopting tooling like Hadoop to do big data processing at a large scale, but even then we weren’t at a place where we are today with the emergence of the Internet of Things and vastly more data coming in than we ever imagined. He shared that that the major challenge now is standardization. The options today for companies seeking to wrangle their data tend to be relying upon raw open source projects, which are frequently not productized enough for simple use, or being led by a vendor who may not have a standardization strategy in mind, instead steering you towards their version of productized piece of open source software. Companies need a resource to help fill in the gap there, and this is were the ODPi comes in.

After lunch I went to Corey Quinn’s talk on AWS cost control. He’s a fun and engaging speaker so I always enjoy going to his talks, but it was also incredibly informative. Though I’m not personally seeking to reduce AWS costs, it was fascinating to learn from his experience about what finance departments, C-level executives and company boards actually care about. Many consulting companies around AWS cost reduction focus on a series of technical strategies that can reduce cost, but his talk wasn’t about those, instead he focused on understanding what you’re running from a business perspective. It’s great to be efficient, but at the end of the day a large bill is actually fine if you can account for what you’re spending it on. Tracking workloads and having an itemized, defendable listings of costs goes much further than a painful cost reduction that knocks a few percents off of what is still an expensive black box as far as finance is concerned. Even better, when you do a project to start tracking your workloads, you’ll inevitably find services that aren’t being used and other extra fat that can be trimmed from your bill, and you do end up with the cost reduction too. Given the audience, he also shared the Open Guide to AWS and his own Last Week in AWS newsletter, both valuable resources to folks administrating AWS.

At 2:30 I gave my talk of the conference! I spoke again on The Open Sourcing of Infrastructure (slides here), taking the audience through the work we’ve done to choose open source as our infrastructure platform and how we can use those lessons moving forward in a world of proprietary cloud services. I gave a version of this talk back at FOSSCON in Philadelphia, but it’s one of those talks that’s always evolving based on audience feedback. The talk went well and afterwards I had a chat that will lead to some additional slides about security. Most specifically, there have been high profile security vulnerabilities in open source software and the major open source tooling that’s using it tends to have patches out within hours of disclosure. We don’t have that with proprietary tooling, in fact, the latest WPA vulnerability has left millions of proprietary devices without patches, and no indication to the customers about when patches will be available, assuming they will be and the devices and software are able to be upgraded at all.

The next talk I went to was from Elsie Phillips of CoreOS in the business track about building an selling an open core product. She walked through business models around training, support and consulting, sharing that these are still valuable avenues of making money, but that they generally weren’t enough and had very slim profit margins. There’s also a balancing act that must be made when you have open core software, making sure that the open source version is valuable and that the “Enterprise” version simply adds features, not that the open source version is restricted in some way. She stressed that the proprietary version should add polish like automation and other tooling that is not impossible to do on your own with the open source version, but saves the company engineering investment if it’s done for them out of the box. I happen to work for a company that uses this model, so while my open source heart bristles at some of the decisions made about what goes into the open source product, we do all have bills to pay and I often find myself in agreement as to where the line is drawn.

The final talk I attended of this first dway was by my Mesosphere colleague Alexander Rukletsov on Health Checking: A not-so-trivial task in the distributed containerized world. This talk drew from his experience with the overhaul of Apache Mesos health checks that occurred a year before. At first glance, it seems like doing health checks inside of containers would be easy since they live on a host that has access to them, but it turns out that it’s not. You have PID and networking namespaces to contend with that can turn even the most simple checks into something unexpectedly difficult. In a distributed system you need to consider the costs and benefits to how broadly you scope your checks, noting that a global scoping may be easier to set up, but you then have network overhead, latency and potential for duplication across your cluster. It was a valuable talk from an operations perspective allowing you to understand what goes into a check you’d generally consider “just an HTTP check” or similar.

All Things Open is a tiring conference and I was busy all day, so my day concluded with some nearby chicken and waffles before heading back to my room.

Tuesday opened with more keynotes! The keynotes began with a thoughtful and inspiring talk by Safia Abdalla who, among other things, called upon the audience to build friendships (not just mentorships) and to not just share code in our communities, but to share knowledge. Her talk was quickly followed up by one from Matt Asay, a familiar executive in open source circles and someone whose opinion I respect even if I don’t always agree with his conclusions. As he reviewed key open source successes in the corporate world he spoke to pragmatism as we push for more openness in our industry. He explained that convenience will typically win out and there’s always some amount of lock-in. There is victory even when building into proprietary clouds, they are more open than what we came from and most of what we’re building on top of that is still open source tooling. As someone who so strongly believes in open source as far down the stack as possible, I see his pragmatism argument, and even addressed it in my talk the day before (you can build against proprietary technology if you want, lots of companies do) but I’m glad for the industry that we have choice.

Burr Sutter joined the keynote stage next to give a very fun and accessible introduction to DevOps. He recounted stories of the improvements to code quality when developers shared in the responsibility in production, stressed the importance of on-demand infrastructure, automation and continuous integraton and deployment for fast, high quality development. John Papas was up next to talk about the open web, talking about some of the technology choices that should be considered when making modern websites, including ways to improve load time through priority loading of elements and searching code for module duplication and the presence of too many dependencies that could increase the size of the page being loaded. He concluded by asking us to be kind to one another, we may not see each other as we interact over coding sites and GitHub, but there are real people on the other end. The last keynote was from Jeff Atwood, the creator of StackOverflow and Discourse. His talk was a story through the creation of StackOverflow as a community and company, and how the more community-focused Discourse project came about. It was amusing to learn that having children so deeply impacted his approach to solving problems, or not solving problems. He learned that not every problem is actually in need of a solution, sometimes we just need to be human and empathetic to one another, and that’s enough. With StackOverflow the goal is very strictly about solving problems. With Discourse it’s about building a community and having fun.

The first session following the keynotes I attended was from Mark Voelker on Interoperable Clouds and How To Build (or Buy) Them. I knew of Mark from his work in the OpenStack community, and his talk centered around the work he’s done and lessons learned while participating in the “OpenStack Powered” program, where products can be certified through a testing suite to carry that label. It causes the OpenStack name means something, and ensures that is not haphazardly applied to products that replace pieces of OpenStack with their own tooling, run non-standard versions of the APIs or are otherwise not what the user may expect from something called OpenStack. He dove into some of the strategies the community uses for determining what should be tested and they iterate on the guidelines with every release, drawing from customers and product creators to learn what they should be including, excluding or changing. These lessons extend to what’s popping up in Kubernetes Certified and other programs in this space.

After grabbing lunch with one of my DevOps + open source compatriots, I met up with Spencer Krum, one of my former colleagues on the OpenStack Infrastructure team and current co-conspirator on open source infra work. He was doing live broadcasts for The Root Shell at ATO and I was invited to swing by to chat. It was fun, we talked about my current focus of containers and our work on open infrastructures. When we wrapped up I headed directly for the lightning talks that had already begun, making it in time to see Chris Short talk about DevOps and the joint official launch announcement with Jason Hibbets of the new OpenSource.com DevOps Team, which I happily joined a few weeks ago. I also really loved the Lego hacking lightning talk from Jen Krieger, DYI electronics hacking with Lego looks like a lot of fun and she showed off some neat projects.

Now, in spite of leaving much of my direct operations work behind when I changed jobs last year, Jenkins still looms large in my work today as I am working on a CI/CD demo and frequently give talks about how containerization can be used in this space. As such, I went to a talk by Kristin Whetstone on “7 Habits of Highly Scalable Jenkins Administrators”. The habits included using the latest LTS version of Jenkins, using Jenkins Pipeline and container-based plugins for more self-service job deployments, parallelization of tests and participation in the Jenkins community. What I really enjoyed about this talk was that she not only would describe the problem space of each of these, but gave links to the specific plugins that would help you achieve your goal with them. The day was getting long as I went with the fun choice for the next session, and joined Gareth Greenaway in his “What the FOSS Community Can Learn from 80s Television” talk. It was a funny trip down 80s television memory lane with open source lessons, including: the importance of working as a team, ignoring MacGyver’s failure to document his solutions, improving the bus factor, reducing the impostor feels by new contributors as Sam in Quantum Leap perpetually felt, and the importance of not simply creating new projects (characters) or forks (spin-offs) without a good reason.

In the late afternoon I met up with my friend Laura over at the booth she was running. The plan? A photo of the Ubuntu gurus we knew at the conference. Success! Not many of us are working on Ubuntu any longer, but the friendships we made in that project sure have lasted.


Our connection? Ubuntu!

With that, we were just one talk away from the end of the conference, and my need to get to the airport. The final talk I attended was by my friend and former colleague on Ubuntu work, Michael Hall. His talk had the interesting name of “The Revolution Will Not Be Distributed” which you quickly learn is not about distributed infrastructure, but instead about the way that software itself is delivered. One of my first contributions to an open source project was packaging in Debian, and it was hard. I had a mentor and throughout my work in Debian and Ubuntu, even as I helped others with packaging, I was always consulting the documentation and asking for help from other people about the proper and expected way to do various tasks with Debian packages. At the time I hadn’t thought much about how this wouldn’t scale, and I was right along side everyone else who worked on various efforts to teach more people packaging and to get software creators to package their software for distributions. This included various efforts that the community team at Canonical embarked on to get more software packaged, none of which resulted in a significant, sustained increase in the number of packages or packagers in the community.

Michael ran the numbers for us. There are about three million applications in the Android marketplace. The Ubuntu desktop has about three thousand. You can definitely argue that the Ubuntu includes high quality applications that build a firm and sufficient foundation for a workable desktop, but we all know that there are gaps, and software we install outside of the standard repositories, whether it’s by using a PPA, install script, or continuing to compile it by ourselves. With this baseline gap, along with statistics about the current number of package maintainers per package throughout various distributions, he calculated that the number of maintainers and people-hours involved to get to Android was astronomical. Even to get a respectable fraction was not within reach. Instead of continuing to toil in the world of distributions with their own package formats, he looked to what is already happening with formats like Flatpak and Snappy, making packages that are simpler to make and portable (especially with FlatHub and the Snap Store), as well as the increased adoption of containers. His vision for the future of distributions was less about packaging of software itself into DEBs and RPMs, and more focus on integration of this software with the desktop environment for a more pleasant user experience. It was an interesting talk, and certainly gave me a lot to think about with regard to our approach to the distribution of software.

More photos from the conference here: https://www.flickr.com/photos/pleia2/albums/72157686641873722

It was then off to the airport! I cabbed it over with my friend Stephen, Michael and David, and the latter two ended up being on the same flight as me. Even more amusing, our connections in Charlotte (not a small airport!) were leaving out of a trio of gates right next to each other. It was a lovely way to conclude my ATO adventure.

Huge thanks to Todd and the team who put on All Things Open. It’s a wonderful conference and I was happy to participate again this year.

The Carolinian

On October 22nd I took another US train trip. Like my recent journey with MJ on the Coast Starlight down to Los Angeles, this trip on The Carolinian was a day trip to get me to a conference. This time I went with David from Philadelphia to Raleigh for All Things Open, the train left from 30th Street Station in Philadelphia around 9AM. With the exception of a trip up to NYC from Philly back in June, my big Amtrak trips have primarily been in the double-decker trains that included sleeper cars. This one was not, the top class was Business, which we took, and there was a cafe car, but no formal dining car. I did notice that the Business class car seemed to have new seats though, which was a nice change from all the old equipment I’ve experienced this year on other trains.


Business class cabin, our spot for the eight hours to Raleigh

The early portions of the journey traversed familiar stomping grounds for me, with stops in Wilmington, Baltimore and Washington DC. This was also the relatively quick portion of the trip on the high Amtrak traffic northeast corridor, taken with an electric locomotive.


Coming into Baltimore

Washington DC gave us about a half hour pause for passengers and swapping of the locomotive to a diesel one. Having done this route before, David clued me into the fact that we could head to the front of the train to see this. The power shut off in the trains and we hopped off, the electric engine was detatched and within about 15 minutes the diesel engine came rolling down the tracks to take us the rest of the way.


Electric engine detatched!

Plus, while we waited we got our train selfie!

From there, it was on to Virginia, for hours. It was nice to see the changing foliage in Virginia, somehow Philadelphia mostly consisted of green and brown-leaved trees when I was in town, probably caused by an unusually warm autumn that has confused the trees. The train also passed through fields and farms, through the woods and various small towns that had quick train stops. What I didn’t quite expect on this journey was the small towns, especially in southern Virginia, that had very little to show for in their downtowns. What was clearly once thriving, was now a series of boarded up and closed businesses. Of course I’d heard of the demise of the rural downtown, but being big city dweller that I am, seeing it, even from the window of a train, made an impact. A real estate app on my phone also kept alerting me to nearby home prices, which dipped well below an average of $100k in these areas, aside from the notable exception of an 18th century plantation estate that was going for $2.5 million.


Virginia stretched on for hours, with lots of nice things to see!

Lunch fell on this long stretch in Virginia. The cafe car was equipped with a microwave, which all hot food was unceremoniously prepared in. I’ve not generally been impressed with train food even in a formal dining car, and this was a step below that. I enjoyed my cookie though. There were also tables so we didn’t need to bring our meals back to our seats and were able to enjoy a slight change of scenery. I do admit missing the vista-type viewing cars from some of my earlier Amtrak journeys. Still, the company was good and I was able to get a bunch of reading in during my trip. I also probably spent two hours just staring out the window and taking advantage of the relaxation time that a train ride provides over air travel.

The afternoon brought us to the Roanoke River crossing, in North Carolina! The train enters the state near in the northeast and ends in Charlotte, which means that after going distinctly south for most of the journey, the train began making its way west after the stop in Wilson, NC.


Roanoke River crossing, North Carolina

A stop in Selma was next, and shortly after 5PM and slightly behind schedule the train arrived in Raleigh. The conference venue was just a couple blocks from the station, so a quick walk got us there for conference check-in and then off to the pre-conference social. No post-plane de-stress time needed, I was totally relaxed. As such, it was nice to be able to jump right into visiting with my open source brethren. Bonus: I shared the journey on social media so the most common non-work exchange I had at the conference was about how I had taken the train down, and often sharing details about how they could have done the same.


Off the train in Raleigh!

I’ve called 2017 my year of Amtrak, and though a jaunt up to NYC around the holidays with MJ is not out of the question, this was probably my final long-haul trip of the year. It won’t be my last of my lifetime though, I’ve quite enjoyed seeing the country by train this year. Though trains in the US leave something to be desired, especially when it comes to food, it’s a peaceful and comfortable way to travel.

More photos from the trip are here: https://www.flickr.com/photos/pleia2/albums/72157688445502994

Fires and fleeing

After the Grace Hopper Celebration of Women in Computing in Orlando recently my friends Michael and Michelle were kind enough to put me up for a night in their nearby home. They served me a home-cooked meal and set up space for me to crash in one of the kids’ rooms. I got to visit with their kids, meet their trio of big dogs, but mostly catch up with them and enjoy some down time. It was one of the most relaxing respites I’ve had in months. Thanks again, Michelle and Michael!

From their place it was off to the airport on Saturday afternoon. After a quick layover in Miami, I was on my way back to San Francisco, putting me at home just after midnight. Unfortunately my luggage did not join me. All the luggage from the Orlando flight missed the Miami connection, leaving a third of a plane full of disappointed travelers. Thankfully I had no where else to be but home, so a luggage delay was no big deal, they delivered it to my home the next day.

Following this trip, I should have had nearly two weeks in town to prepare for three weeks of travel abroad, but nearby wildfires had different plans for me. By Sunday night smoke from wildfires in Napa and Sonoma had descended upon the city. It was fun at first. We didn’t realize how close the wildfires were getting to homes and business up in Napa and Sonoma and it was a bit like camping, with the wood smoke smell drifting in through the windows. By Monday morning the reports had started coming in from up north about how bad the fires were, people were losing their homes, some historic buildings were lost, wineries we frequented were taking precautionary measures. At home I was clearly coughing, and come Tuesday my chest was tight and sore from the coughing. This was about when we all learned about airnow.gov and could start tracking the air quality. With the levels remaining at “Unhealthy” on Wednesday I finally heeded the advisories and closed the windows. This made our condo quite warm, getting up to the mid-80s, as we lack air conditioning and usually depend upon the breeze from the windows. This was when MJ first suggested I fly back east to spend time at the townhouse in Philly.

Still, I spent the week as I usually would. MJ and I had some dinners together, Caligula and I chilled after work, one evening I made banana bread when I realized the bananas I had delivered with groceries earlier in the week were a bit more ripe than I had expected. In spite all the tragedy in the reports coming down from wine country and being cooped up, the smoke hanging in the air did make for some fascinating views of the sun and sky. Our sunsets were dripping with pale red, making some doomsday shadows on our walls.

I wasn’t the only one taking photos of the red-tinted sun on Friday evening when MJ and I went to the Synagogue for Shabbat services and Simchat Torah.

Simchat Torah! Jewish congregations read the Torah from scrolls made of parchment and hand written by a scribe in Hebrew. This means there’s a beginning and end of the scroll, and with the Jewish new year behind us, Simchat Torah is when you read the final passage at the end, and then the scroll has to be wound back to the beginning. To celebrate this, our synagogue took the opportunity to fully unravel the scroll and our Rabbi gave us a tour of it as we all held it around the sanctuary. It was a pretty special thing to be a part of, I’m glad we both were able to attend, even if it was earlier on a Friday evening than we tend to go to events, due to proximity to work timing-wise.

That evening after services was when I decided that going back east was not a crazy idea. It actually just meant leaving five days earlier than I had planned on, since I was already planning on going out a day ahead of my trip to Raleigh for the All Things Open conference. MJ helped me find a good itinerary and rebook my ticket for a flight out of San Jose on Sunday night instead of the following Friday.

Saturday night we fled to a nearby hotel to enjoy the climate control which would give my lungs a break until I flew out. Plus, it was a relaxing mini-getaway that allowed us to enjoy their executive lounge, where they had chocolate mousse in chocolate cups!

Plus the views were nice. The sunset was quite a bit less apocalyptic by Saturday evening. In fact, I started reconsidering my trip back east, had I over-reacted?

It turns out I probably hadn’t over-reacted. Though there was some clearing of the air on Sunday and we were able to spend some time outdoors, but fires and winds picked up early in the week to make San Francisco smokey again. I did feel a bit bad for leaving though, I’d miss being home, my husband, my cat, and I wasn’t quite prepared to leave so soon on a trip that was already taking me away for three weeks. Taking care of my lungs was important and we have the townhouse in Philadelphia where I can stay without needing to make prior arrangements or costing us anything more. Childhood asthma seems to have done a number on my lungs, but thankfully MJ isn’t prone respiratory issues like I am, so he seems to have toughed out the smokey conditions with no ill effects, like most other people I know in the city.

Plus, it ended up working out really well for work. Most of my team spent most of the week in Europe, and with our MesosCon Europe planning I had a few 7AM pacific meetings, which is a much more tolerable 10AM eastern time. It also gave us the opportunity to get the sprinkler system in the townhouse inspected on schedule and get some work done on the sprinkler pipe that we had talked about to make room for a future water softener. It’s also my favorite time of year in Philadelphia, I love autumn. The temperature and low humidity was such that I could open the windows and enjoy the fresh air for most of the day. I also lucked out precipitation-wise and ended up with beautiful, clear skies all week.

I was pretty anti-social during my visit though. I took a ride share from the airport to the townhouse upon my arrival on Monday morning and then spent Monday and Tuesday at the townhouse on my own. Wednesday and Thursday evenings I finally made it out to visit with friends, which I’m finally also doing today. Today is my last day in Philadelphia though, tomorrow morning at 9AM I’ll be on am Amtrak train with David down to Raleigh for a Sunday evening arrival, hopefully coming in with enough time to go to the opening social for All Things Open. I have a lot of open source friends and colleagues attending the conference, so I’m really looking forward to it, even if I do have to depart Tuesday evening with enough time to catch a 7:40PM flight that takes me to Charlotte, London and finally Prague for MesosCon Europe. I have a few busy weeks ahead of me, I’m not entirely sad about taking the time to myself to catch up on personal projects.