• Archives

  • Categories

  • Other profiles

  • wallaceadngromit.net

  • Partimus

  • Xubuntu

  • DC/OS

Thanksgiving in Philadelphia

The week I returned from Cuba was a difficult one. I visited an urgent care clinic on Saturday morning to address the ongoing stomach problems I was having. They prescribed the pretty standard round of Ciprofloxacin for travel-related stomach woes, but were careful to mention that it can have nasty side-effects and I should go to the ER if any of them arose. Unfortunately for me, they did. I woke up on Sunday morning with an unusual smattering of hives and went straight to the hospital. That’s where we spent the day, they kept me under observation as we tried a new treatment plan and I was finally able to go home in the late afternoon. Unfortunately I was still sick from the original condition, and that characterized my week. I suffered through work on Monday because I had a lot of work to catch up on, but Tuesday was a straight up sick day. I got better as the week progressed, which was good, since Friday we took a red eye flight to Philadelphia to spend Thanksgiving back east.

That first Saturday in town we caught a nap in the morning, but first I got to see a deer! The contrast between our place in downtown San Francisco and the townhouse is exemplified by moments like that. The rest of the day was taken up with a bunch of shopping before meeting a friend for dinner.

Sunday we went to a wedding! Advertised as an engagement party, my friend Crissi decided to celebrate the engagement to Henry by having a surprise wedding. It was held at a lovely nearby country club where we had enjoyable food and company surrounding the surprise ceremony.

We spent Monday through Wednesday in town working, with a jaunt to downtown Philadelphia on Tuesday to eat at Buddakan, which we’d never been to but I now highly recommend if you’re looking for a nice meal out on the town. The service was great, the food was delicious, with dim sum and small plates able to be ordered as you went. I’d go back again just for the lobster egg rolls.

Wednesday would have been Simcoe’s 11th birthday. We lost her in April but it hit me hard and her birthday remained tough for me. That evening we ended up at Pizzeria Uno near the townhouse where they were having an incredibly appropriate Weyerbacher tap takeover. Why appropriate? Simcoe is a type of hop, one used by many breweries, but Weyerbacher has really made it their own. They had a wonderful Double Simcoe IPA for a few years, and their IPA #2 this year “uses large amounts of Simcoe and Denali hops” in the batch. It was a lovely way to round out the evening. Plus, they gave me a pint glass at the end of the evening!

And then it was Thanksgiving! The daytime was spent watching the MST3K Turkey Day Marathon while MJ slept in, as is my usual tradition. The marathon kicked off with The Day the Earth Froze, followed by Night of the Blood Beast. I also was able to catch up on some volunteer open source work throughout the day, mostly with my European colleagues who were still working on what was a regular Thursday for them. This is also when I discovered how far behind I had fallen with most of the mailing lists I’m on. I have been forced to confront the fact that I burned out a bit last year when my team was laid off and I stepped away from OpenStack. I poured myself in my new role come January, but never did quite pick back up checking email that I had let slide. I haven’t gotten back to where I was productivity-wise since then since most of my work these days is focused on my day job, but I am happy to report that I’m doing a lot better. Catching up on mailing lists and staying on top of them now has made me feel a lot better too.

Thanksgiving dinner was spent at my sister-in-law’s place, as we’ve been doing for the past couple years. We had a really nice time catching up with family, having delicious food, enjoying some good wine and getting to meet their friendly new kitty.

I made plans to meet up with some friends on Friday, and we accidentally ended up at a mall in New Jersey where the restaurant we picked was located. It was Black Friday. Parking was a bit crazy, but it turns out malls are built to absorb holiday crowds and it actually wasn’t a terrible experience. We did some appliance browsing since we have to do some replacements here in the condo in San Francisco. The shopping evening wrapped up with a stop at L.L. Bean where they had a great sale on duck boots, which I’d been meaning to get a pair of, as well as a pile of flannel clothes. I also ordered some stuff to be shipped to San Francisco, my stock of flannel shirts is now complete. The actual visit to New Jersey concluded with a nice dinner with MJ’s great aunt and her daughter.

This trip was also highlighted by getting the O-Lionel scale train set I picked up at a toy fair last year running. I quickly learned upon extracting the train from the crate I had it in that the tracks were rusted. I put the track together on the kitchen island and the tracks were intermittently powered, but I really couldn’t get good conduction. Fortunately I had been advised when I bought it that the tracks would probably have to be replaced, so this wasn’t a huge shock for me.

While we were in New Jersey on Friday we stopped by Toy Train Emporium in Cherry Hill to pick up new track. Their staff was helpful with my need to replace the O-gauge Lionel Tubular Track, and suggested I switch away from the O-27 that came with my original set and go with one that had more forgiving turns. They gave me advice on track cleaning (not letting these get rusty!) and helped me find some smoke liquid to fill the reservoir for generated smoke in my engine, they had several different scents, I went with the root beer. Upon returning home, I quickly learned how tricky the tubular track is to assemble the first time. It requires a bit of patience since it doesn’t simply snap together, you have to re-shape the tubes a bit to get the parts to connect, a fact documented in the train manual. The working of the rails with your hands is a bit tiring, I spent one morning over the weekend taking breaks between fitting track and reading. Thankfully, once assembled, it turns out the island in the kitchen was perfectly sized for my new track, so I got to play around with it there, and even got the smoke going! Though I learned that it has to be going pretty fast for the smoke to start, causing a spectacular derailment while I was testing it out, oops. The train won’t always live in the kitchen, I have plans to set it up in my office area once we move some of our furniture and other items out west.

I spent a nice chunk of time reading during this trip, which was mostly taken up with The Butchering Art: Joseph Lister’s Quest to Transform the Grisly World of Victorian Medicine, by Dr. Lindsey Fitzharris. I pre-ordered it several months ago, and quickly discovered that it was a fascinating page-turner, easily one of the best books I’ve read this year. I also learned that the cover art is from a painting by Thomas Eakins which now resides in the Philadelphia Museum of Art. I’ve been looking for an excuse to go visit, maybe during our trip back east over the holidays.

MJ flew back a couple days early to get back into the office, I stayed through Tuesday continuing to work remotely, so my trip concluded with an introduction to kung fu movies over at a friend’s place. He appealed to my love for MST3K to introduce and explain the ridiculous aspects of so many of the films. I’m still not sure I “get” it, but it was fun anyway.

Holiday cards 2017!

Every year I send out a big batch of winter-themed holiday cards to friends and acquaintances online.


Reading this? That means you!

Even if you’re outside the United States!

Even if we’ve never met!

I met someone at a conference this year who I didn’t know, but to whom I sent a card to one year. Turns out they were having a particularly rough time that year. Made them feel good, made me feel good, everyone was happy. I like it when everyone is happy.

Send me an email at lyz@princessleia.com with your postal mailing address. Please put “Holiday Card” in the subject so I can filter it appropriately. Please do this even if I’ve sent you a card in the past, I won’t be reusing the list from previous years. Edit: I’ll be accepting requests through December 10th.

Disclaimer: My husband is Jewish and we celebrate Hanukkah, but the cards are non-religious, with some variation of “Happy Holidays” or “Season’s Greetings” on them.

CubaConf 2017

I spent the second week of November in Havana, Cuba for CubaConf 2017.

It’s an open source conference, and so much, more as their website explained:

CubaConf, is an international conference about free software and open technologies that will have its second edition in La Habana from November 7th to 9th, 2017. CubaConf is a meeting from the international community of enthusiasts of the free technologie’s world, peer to peer focused and collective development. It seeks the participation of people with different profiles and diverse geographical locations.

From the organization of the conference, it is aimed at the diversity of both topics and projects, as of its participants, so the participation of women, minority groups, in these activities is crucial to build new inclusive spaces of knowledge and dialogue about technology. The main idea is to know people experience and organizations who work with and in free software, with old hardware technologies and little or slow bandwidth, to talk about how free software can help developing countries.

Conference Registration

I learned about the conference from and then tagged along with the OpenNMS crew, Tarus Balog and Alejandro Galue, which I was grateful for. Last week Tarus wrote a great blog post about the conference here: 2017 Cubaconf. It was nice having folks to walk with as we wandered through the streets of Old Havana to find the venue on Tuesday morning. The format of the conference was a day of planned conference talks, one day of unconference and then a workshop day.

I joined them to give my Open Sourcing of Infrastructure talk, but this is one conference where I’m certain I learned more from the attendees and environment they work in than they ever did from me. The day began with a keynote from Ismael Olea who came to the conference from Spain where he participates in HackLab Almería. His talk was in Spanish, and he spoke quickly, but thanks to verbose slides (available here) I was able to make out the gist of his talk. Almería is not a major global city, but they’ve worked to build a thriving community through events and online built around giving people the autonomy to self-organize and do whatever they want, as long as it aligns with their ethos of technological, social and creative experimentation. The broad, generalist approach echoed much of what I heard from Josh Simmons at Linux.conf.au earlier this year: when you live in a less densely populated area, multidisciplinary groups are key to success. The added element of self-organization definitely appealed to the Cuban audience, which was a constant theme throughout the conference.

The first talk after the keynote I attended was by Valessio Brito who presented a career path on how you can make money doing open source software. He outlined opportunities for working in support, consulting, customization and development, and stressed the importance of building an open source portfolio. Even once you’re hired, he expressed the value to your career of staying involved with the open source community instead of being overtaken by the tasks internal to the company where you work. Of course as a developer advocate I’m thrilled when our internal engineers express an interest in being a part of our community, so it was nice to hear from a third party too.

My talk on The Open Sourcing of Infrastructure was next. This is a talk I’ve given at a few different places now, and I think I got the most interesting response yet from this Cuban audience. The talk stresses the importance of controlling your data and resources instead of trusting a third party company to do it, and this really resonated. For the Cubans I met, the ethos of free software (not just open source) was incredibly important. I learned quickly while there in Havana that they have a very resourceful, DYI culture that values the ability to have control of your resources. While reading up on this phenomena upon returning home, I found this article from PBS News Hour in 2015, How communism turned Cuba into an island of hackers and DIY engineers. Had I been more aware going in, I could have shifted my talk away from convincing them to use open source infrastructure tooling, they were already convinced! If I were to do it over, I think I would have focused on how they could do it, instead of the specific technologies (OpenStack, DC/OS, Kubernetes, etc) being a single slide of recommendations.

Unfortunately my talk slot overlapped with my friends from OpenNMS! But I’m sure they did a fabulous job.

At lunchtime we walked over to the nearby Casa de África where they had outdoor event space for us to enjoy sandwiches. It was there that I met an engineer who was deploying DC/OS for a bank in Mexico and several folks from the US and Europe whose attendance at the conference was deeply tied to the spirit and message that free software brought, not just the open source aspect of it. It was interesting to learn their perspectives and what brought them there to Cuba.

After lunch I went to a talk by Christian Weilbach on “Free data and the infrastructure of the commons.” Companies control vast amounts of data today, most obviously by companies like Google and Facebook, but generally by most companies who have a technical presence and customers. He explained the risks here, and his plea was to work to adopt open source methodologies to the free access of data. He didn’t have the answers, but was keen on seeing us get there because of how important data is in today’s world. Indeed, a quote I took from the talk centered around the future money being in data, not in the technology stacks which are being commoditized.

The final talk of the day I attended was by Molly de Blanc on “Freedom Embedded: Certifying fully free hardware.” We’ve met at a few conference, so it was a wonderful surprise to see her at this conference and get the opportunity to catch up. Her talk began with some background of the Free Software Foundation (FSF), where she works. I find their perspective on free software to be on the extreme side, but her talk was about their “Respects Your Freedom” hardware product certification, which is actually really cool. They’re seeking to document hardware that runs with 100% on free software, documenting their criteria for certification here. She also mentioned h-node.org during her talk, which I wasn’t familiar with.

Then we had the conference group photo! Not everyone made it in the picture, but it was a decent chunk of the conference attendees.

CubaConf 2017 Group Photo! Credit: Gabriela Fernández (source)

That evening we made our way to the evening social held at a gallery in Old Havana. They served light appetizers and offered an open bar, where I got to enjoy some rum on the rocks. Rum isn’t usually my drink, but I was in Cuba! That event was where I was able to speak with a developer from cuban.engineer where I learned a bit about how they do software development in Cuba. At his company they rely heavily upon their local LAN instead of having internet access in the office, and collaborate on a local GitLab instance. Collaboration and syncing upstream and online is done, but it’s simply not a part of their constant daily activity. This is in sharp contrast with every software project I’ve ever worked on, being a remote employee for over a decade and involved with online software communities since 2001, internet access is essential to my ability to collaborate. It’s a very different environment than I’m used to, and it was fascinating to learn how they make it work.

Snacks and socializing at the first conference night social event

Wednesday began with a keynote by Dr. Mixael S. Laufer, who I had the pleasure of meeting the day before. The organization he belongs to seeks to equalize access to healthcare, and drawing from similar tenets of open source and Maker movements, encourages a DYI approach to healing. Again, giving the culture in Cuba this was a fantastic message and I’m sure one they already work towards, but it was a fascinating topic for me to learn more about.

The rest of the day was reserved for the unconference, which began by people writing down and pitching their ideas, and attendees voting on sessions which then landed on a schedule put together there on site.

The first discussion I ended up in for the unconference was from Zak Rogoff, who told us a tale of how the Free Software Foundation tried, and failed, to fight back against DRM entering web standards. He recounted the grass-roots campaigns they attempted. I have to admit that I continue to be put off by the campaigns that the FSF launches because I do find them to be on the extreme or childish side. Still, with this campaign they had a very firm foundation in something I believed in, and it’s a shame they failed to get the attention of me and others like me. One of the great things they did do was partner with the EFF and Cory Doctorow which led to An open letter to the W3C Director, CEO, team and membership. I believe they would have had more success if more of these strategic partnerships with organizations I hold in high regard had been made, along ones which could have spoken more directly to the serious challenges DRM presents to security and accessibility. His conclusion was the same. I hope the FSF takes this lesson he presented to heart, for the most part they are fighting the good fight, I’d just like to see a better approach.

Lunch was next! I was able to spend more time chatting with attendees, but unfortunately by the time we were wrapping up in the mid afternoon I wasn’t feeling my best. I had to depart and head back to the AirBnB for the rest of the day, with the exception of a stop at the pharmacy and a futile attempt to eat some chicken soup for dinner. I’m even more disappointed to report that being ill took me completely out of the game on Thursday. That means I missed the women group photo! But it was nice to see a decent representation of women at the conference, I wish I had made time to meet with more of them.

Huge thanks to the organizers of this event who made me feel so welcome. In spite of not being able to attend all of them, I appreciated the existence of social events each night and had a nice time chatting with people throughout the event.

Mi visita a Cuba

I was recently honored to have a speaking engagement accepted for CubaConf 2017. The conference lasted Tuesday through Thursday, so I flew in on Monday and left on Friday, giving me some buffer on each end of the trip to get settled in and explore the city a little bit while I was there. I’ll write about the conference itself soon, but I want to first talk about my experience in Havana.

I learned about the conference from Tarus BALOG in blog post and he was who I reached out to as soon as I knew my talk was accepted. Turns out he had a room in an AirBnB that I could stay in, perfect! The trip quickly turned into a gathering of minds with him and a colleague from OpenNMS giving presentations as well. I was picked up at the Havana airport mid-day on Monday by a cab sent by the AirBnB. But before we get to the AirBnB, let’s talk about cars.

Whenever I saw anything about Cuba, the cars from the 1950’s played a role. They’re on the posters, post cards, advertisements, image searches for Havana, everything. This is caused by the trade embargo which limited imports after that decade. But things have loosened in recent years, how true is the rolling museum of cars in Havana today? There are definitely some modern cars, but wow, there really are classic cars everywhere. From what I’ve read, they’ve also got all kinds of crazy things under the hoods as the resourceful Cubans did whatever they had to parts-wise to keep these old cars running. This is a great article about it all, with tons of pictures: Here’s What Cuba’s Car Scene Looks Like In 2017.

We stayed on the third floor of Gallery III – In the Heart of Old Havana, a building which contained three condo units rented out as “homestays” via AirBnB. Our host Yaima met me upon arrival and kindly shared some cake the staff was having to celebrate her birthday. I wasn’t sure what to expect with this lodging, but was generally pleasantly surprised. The WiFi was not working the first couple of days we were there, and it was only on by-request in the common area downstairs anyway, more on internet access later though. I took the bedroom with the smaller (but still Queen!) sized bed and a bathroom for myself right next to the kitchen which included a stall-less shower and drain in the middle of the room. My roommates for the week took the larger rooms and more modern bathroom, though I did borrow their shower one day. We had a little trouble the first day with hot water, which meant some quick cold showers, but they were able to fix that on subsequent days, even if we did have to strategically time our showers around others in the building.

Everything else was wonderful. The condo itself was beautifully decorated and well-lit. They stocked the refrigerator with bottled water (you can’t drink the tap water), sodas and beers that they billed you for at the end of your stay. In spite of the 80+ degree heat, simply opening the doors on both side of the condo and running a fan made for an incredibly comfortable place to chat, or read, both of which I did a lot of during my stay. The bedrooms had individual units for air-conditioning, which was valuable to a good night sleep, and essential later in the week when I wasn’t feeling well. They also provided breakfast each morning for what amounts to $3/day and there was always at least one staff member there to keep an eye on the cameras, watch for us having trouble getting inside, provide recommendations for food (or pharmacy!) or to turn on WiFi when we needed it. They also had staff who cleaned the condo and made the beds every day, a nice touch I wasn’t expecting.

After settling in a bit I was able to take some time to walk around Old Havana. I walked over to Castillo De Los Tres Reyes Del Morro, another one of the Havana icons. Then made my way past expansive, beautiful hotels, sadly many of which landed on the restricted list for US travelers on November 8th, while I was there.

The rest of my afternoon was spent at the AirBnB condo reading and relaxing following what has been a crazy month of travel for me. My roommates arrived in the evening and we made our way out to the first cocktail stop: Floridita. You see, my friend Tarus is not only is an open source enthusiast, but also runs a Vintage Spirits and Forgotten Cocktails blog! He was the guy to know for cocktails in Havana. Now, the big claim to fame that Floridita has is as the birthplace of the Daiquiri. We ordered a trio of them. Unfortunately they had run out of grapefruit juice so we couldn’t enjoy the papa doble (Hemingway daiquiri). Ernest Hemingway was a daiquiri fan and had them at that very bar! I got my selfie with Hemmingway while I was there and we enjoyed live music. As the evening wore on we decided to also eat at the restaurant there. Specializing in fish dishes, it was a perfect ending to the evening.

Throughout this lively Monday I was tweeting, but that’s only thanks to a very expensive Digicel SIM that MJ bought for me for my trip. $25 for the first 100MB, then $50 for each subsequent 300MB. I’ve grown accustom to my Project Fi SIM working in most of the world, so it was a surprise to my wallet when I had to pay such a premium. Still, I quickly learned why. All the internet I encountered on our trip was mobile internet. Old Havana is dotted with public access points that people buy access to by the hour via codes they buy at local shops. This is what my traveling companions did, and this is also how the AirBnB worked, since they were near (or owned?) one of these public access points and they just covered the costs as part of business, and this is why we needed their help when we wanted to get online. This created a fascinating side-effect: you could always tell when you were near a public access point because everyone was on their phones. Elsewhere? Not so much! It did come as a surprise to me that this is also how many organizations do internet access as well, the university where the conference was held had a LAN for us to upload our slides and share photos on, but no outside access, they advised using a nearby WiFi park with the cards if we needed access. Even people working at tech companies there in Havana told us that they did most of their work locally on their LAN (yay GitLab!) and only synced up to the actual internet at specific intervals. I’m convinced that they must be much better technologists if they don’t have the ability to Google answers to everything, hah!

Tuesday and Wednesday were pure conference days. Tuesday night found us at a conference event, which they put on every evening during the conference at various locations throughout Old Havana. Walking through Old Havana to and from the conference venue and events was a exploration unto itself. There were a few streets shut down to car traffic and filled with tourists and shops. I was able to pick up some post cards and stamps to send to my regular recipients. I skipped the rum since I’m not much of a rum drinker and didn’t want to check my bag on my way home, but I did pick up a couple cigars, though I’ll have to think about who to give them to. No one smokes anymore …but they’re Cuban cigars! The infrastructure there leaves something to be desired, the humid climate wreaks havoc on permanent structures, and there’s evidence of that throughout Old Havana. There are also a lot of pot holes and it’s often difficult to figure out just where you want to walk. Still, walking at night with a group felt pretty safe, and I got to see lots of cats, who appear to come out after dark!

Unfortunately my stomach became the ruler of all my plans starting Wednesday afternoon. I was struck with a brutal bout of travelers diarrhea and forced to remain at the AirBnB for the third day of the conference and the social events, one of which was at a brewery! So sad! I also missed further cocktail adventures with Tarus and Alejandro, no cocktail at the Hotel Nacional for me. In addition to bio-breaks, Thursday was essentially spent napping and listening to audiobooks since I wasn’t even well enough to read or look at a screen. I did get to visit a local pharmacy though, and that was a nice adventure. I was sick for several days though, but thankfully by Friday I was well enough to at least tolerate getting on a plane.

I shall conclude by talking about cars again. A real treat awaited us when we came downstairs to catch our cab to the airport, an orange 1953 Pontiac!

It was my only opportunity on this trip to actually ride in a classic car, and I was really pleased with how well-maintained and improved it was. It lacked seat belts that I instinctively reached for upon getting into it, but it had top notch air-conditioning. The round side mirrors had blinker lights built into them. And everything about the car was shiny and beautiful, like many of the classic cars I’d seen. It’s clear that the owner spent a lot of time maintaining it and it was a beautiful way to conclude my Cuban trip.

Meetups in Germany

Earlier this month I spent several days in Germany. I made some time to do some tourist stuff, but I was actually there for work. I visited and worked from the Mesosphere office for a couple days and host a Meetup. I also made my way over to Berlin for a day to host a Meetup. Throughout my journey I was able to enjoy good German food and sights, and meet lots of friendly people!

The Mesosphere office itself is on the top floor of an office building, at the 7th floor that makes it a reasonably tall building for the city so there are some nice views of the surrounding area.

But Meetups! The first Meetup of the week was over in Berlin. The topic of both the Meetups was “MesosCon Recap” which consisted of me presenting a short slide deck (slides for Berlin) that gave an overview of what MesosCon was, some of the key themes, and what came of events like the new Town Hall on DC/OS. After my introduction, for each location we invited speakers who were local to that region of Germany so we had a different line-up for each evening. Each speaker was given a half hour slot to give a shortened version of a MesosCon EU talk. It was a new format for me, but it was a lot of fun, and I enjoyed bringing a piece of MesosCon to folks who couldn’t make it out to Prague.

The speaker lineup in Berlin consisted of:

  • Till Rohrmann of Data Artisans on “Apache Flink Meets Apache Mesos and DC/OS” (slides)
  • Kevin Klues of Mesosphere on “Running Distributed TensorFlow on DC/OS” (slides)
  • Tim Nolet from magnetic.io on “How to extend Marathon-LB with Canary releasing features using Vamp” (slides)

Huge thanks to all of them for coming out, both Kevin and Tim gave great talks at MesosCon EU, and it was great to have Till join us to present using a slide deck that was not his own, but as a leader in the subject matter he did a fabulous job.

The staff at betahaus, the venue, were also incredibly helpful. They got us all sorted with the projector, pizza delivery and with drinks throughout the evening.

Thursday night I was joined by my colleague Matt Jarvis for the Meetup in Hamburg! At this one Matt and I ran through the introduction slide deck and made some adjustments (slides for Hamburg). This introduction was longer than the one I did in Berlin, partially because the speaker lineup was shorter, and because I had the help of Matt who took time to flesh out some of the impressive statistics about cluster and workload sizes from the keynotes, and he could cover the tracks he was track lead on, so it wasn’t just my perspective from the operations tracks.

Following our introduction, we were joined by our pair of MesosCon EU speakers from Mesosphere for slightly shortened renditions of their talks:

  • Johannes Unterstein on “Marathon and Jobs – Today and Tomorrow” (slides)
  • Adam Bordelon on “Mesos Security Exposed!” (slides)

Logistics were a bit easier since this Meetup was held at the Mesosphere office. They tell me that we don’t normally have a skeleton joining us in the large meeting space, but it was just after Halloween so the decorations were lingering. It was a fun night and the crowd was a bit bigger than the one we had in Berlin, but I admit that I skipped having Meetup pizza for a second night in a row and instead joined Matt for a later dinner at the restaurant in our hotel.

Wall, Egyptians and a friend in Berlin

On November 1st I hopped on a Deutsche Bahn ICE train in Hamburg. A couple hours later stepped off in Berlin. It was drizzling out, but not enough to put me off the plans for the day, which included visiting the remains of the Berlin wall, the Egyptian Collection in the Neues Museum and a late lunch with my friend and Berlin local, Daniel Holbach.

The wall was first. The fall of the Berlin Wall happened in 1989 when I was just eight years old. At the time I was too young to understand what had occurred, but it and the cold war were recent enough that I learned a considerable amount about both at school in subsequent years. Seeing it in person brought up feelings, it was an important stop on my journey through Berlin. Still, I skipped the crowded visitor center and was satisfied with my self-guided walk around the grounds.

From the wall I walked south to the museum. Seeing The Egyptian collection in Berlin, now housed at the Neues Museum, has been on my list for years. It includes the famous Nefertiti Bust, which at over three thousand years old, has been remarkably preserved. I had seen photos of it and it was featured in documentaries, and it recently grabbed headlines through the controversial Nefertiti Hack. I’ll admit right away that the long-standing claims the Egyptian people have to the bust did cause mixed feelings when I went to see it, but my fascination with ancient Egypt won out, I really wanted to see it. It’s hard to tell from flattering photos and angles just how impressive the preservation really is, so seeing it in person was a wonderful experience. No photos were allowed in the exhibit area, but there are plenty of them already.

Stepping back to take in the whole collection is a bit overwhelming. They have a huge room dedicated to their collection of wall segments from ancient tombs and buildings, some are just etched but many of them have period paint still attached to them. I nearly skipped the basement level in the interest of time and energy, but I’m glad I didn’t, that’s where they had some of the more intricate sarcophagi and their collections of animal statues, charms and figures. I’ve seen the collections at the British Museum in London (technically a larger collection) and the Met in NYC, but the presentation at the Neues Museum really stood out for me.

I think part of what made it so impressive for me was the building itself. Built in the mid 19th century, even the rooms in this museum were worth taking pictures of. I’m sure when I took this picture it was all about the room, not the artifacts:

And while I remember that this room was the one housing papyrus, the only reason I remember that was because you can just barely make out the Nefertiti bust at the end of the room, incidentally included in my photo as I was admiring the room.

Once I had concluded my time in ancient Egypt, I did quickly make my way through the other collections in the museum. Of particular note was the Berlin Gold Hat. It’s a tall, conical hat that estimates of similar artifacts has tended to put at around 3000 years old, though unfortunately this artifact itself lacks verified provenance. It’s housed in its own dark room, surrounded by related artifacts. At almost two and a half feet high, it’s definitely something you should see if you visit the museum.

The afternoon was creeping on in as I hopped in a taxi to head south to my hotel. I was able to check in, drop off some of the contents of my increasingly heavy backpack and head out to meet up with my friend Daniel for lunch. We ended up at a Mexican place for a pair of the largest burritos I’ve ever seen. They were pretty good too.

I’ve probably known Daniel for a decade. He was a community manager and developer advocate in the Ubuntu community before there was a name for such roles. His guidance and gentle leadership helped me and so many others be successful members of the Ubuntu community and he’s one of the kindest people I know. He took a break at the end of last year when he left Canonical and I hadn’t really been in touch, so I was thrilled when he replied to my email inviting him to meet up in Berlin.

We had a great afternoon of catching up, wandering through a park, down streets, stopping for coffee. He also know where the event space was for the Meetup I was hosting that evening, so we walked over there as the afternoon wound down and I was able to get settled in before he departed. The Ubuntu community was a very special place for me at a formative time in my life, and I’m so pleased that I have these relationships with people around the world.

I spent that evening hosting the Meetup there in Berlin before walking back to my hotel. I’d say my only mild regret is that I didn’t wake up early enough the next morning to visit the Berlin Zoo before taking the train back to Hamburg. I was starting to get a bit travel weary and I did enjoy the relaxed morning and time it gave me to catch up on some email. I’ll be back some day!

More photos from my visit to Berlin here: https://www.flickr.com/photos/pleia2/albums/72157665780629679

Tiny things in Hamburg

I spent a week in Hamburg, Germany recently to meet with colleagues at our Hamburg HQ and host a couple Meetups. Booking my visit on the heels of MesosCon EU meant that the week I was there coincided with a national holiday. I learned that on October 31, 1517 Martin Luther nailed his Ninety-five Theses on the door of a church, a key point in the Protestant Reformation. Germany was observing the 500th anniversary of this act. I spent this Reformation Day exploring Hamburg.

My exploring had nothing to do with religion, so my first stop on this holiday from work was to the Miniatur Wunderland.

The largest model railway in the world, and one of the most successful permanent exhibitions in Northern Germany.

Now, I have been getting into model railroads, but calling this a model railroad doesn’t quite do it justice. It’s really a whole miniature world, including a model airport that cost 4 million Euros to build. The place takes up three stories of a historic warehouse building, with dozens of people employed to run it all, and along with an impressive control center, there are three racks of hardware at the entrance.

Stats and train nerd stuff aside, it’s a fun place to be. I bought tickets late, so the only ones available online for the day were in the 7-8AM window right as they opened for the day. This timing was not easy with my jet lag situation, but it was worth it. Since it was a holiday, the place was packed with people by 9AM, so getting in early was a good call, I was able to take lots of great pictures before it got too crowded to get close to a lot of the exhibits. Throughout the morning I walked my way through various countries in Europe (various German towns, Switzerland, Austria, Italy) and even a small section representing the United States. The scaled down version of Hamburg itself was pretty cool, you’re able to find the museum itself and I saw many familiar spots even having just been there a couple days. You could spend days exploring the details that were included throughout, I spent about two and a half hours there and feel like I only got an overview of the layout. I’ll have to go back next time I’m in town.

I may have gotten a bit carried away with the picture taking, over 400 more photos from Miniatur Wunderland here: https://www.flickr.com/photos/pleia2/albums/72157665685279559

October 31st is also Halloween. It’s a much bigger holiday in the United States than most other places, and over the past few years I’ve been traveling on Halloween more often than not so I often don’t get to properly celebrate. I wasn’t going to let this year get away from me though, so I booked morning tickets at The Hamburg Dungeon, conveniently located right next to Miniatur Wunderland. These dungeons are a chain, there’s even one in San Francisco, and the idea is that they hire actors to act out dramatized versions of some of the more gruesome history in their respective cities. It’s silly, but a lovely haunted house-like way to get a taste of Halloween as a solo tourist.

I then took a walk to find some food and ended up at Kartoffelkeller, potato cellar! So many potato dishes. It was good and I finally had a beer. Well-fed, I spent the rest of my late afternoon back at my hotel, 25hours Hotel Hamburg HafenCity, where they have ship cabin themed rooms and a whole floor dedicated to games and socializing, but for me made a great spot to get some writing done as I enjoyed the herbal tea they had available to guests. Yum. It was back to work and a detour off to Berlin, which I’ll write about soon, before I returned to Hamburg on Thursday evening for a Meetup.

Now, the other thing I learned about Hamburg during my visit was how water-centric the city is. I knew Hamburg was in the north of Germany, but I had never zoomed in on a map and noticed that it was connected to the North Sea by the huge Elbe River. This river not only winds its way through Hamburg (so many bridges!) but also creates a huge harbor that industry in the city appears to revolve around. Friday at lunchtime I went out with my colleague Jörg who took me up to the Elbphilharmonie, a concert hall just a few blocks from the office. Every day you can get free timed tickets to take some very long escalators up into the concert hall to walk the perimeter and see some amazing views of Hamburg and the harbor.

Lots of water in Hamburg!

As work wound down on Friday I was able to make time to visit the International Maritime Museum, and even better, since they close at 6PM I learned that after 4:30PM the tickets are only €6! An hour and a half isn’t a lot of time to explore that museum, it’s several stories and takes you through about three thousand years of maritime history, but my visit was long enough for me to enjoy it. I will say that one of the most interesting things about this museum was how many ship models there were. The top floor was entirely made up of models, but every floor had models of varying scales and types, from boats and ships, to engines. I was starting to get the impression that Germans, at least those local to Hamburg, really like models.

The highlight of the museum for me was their display of Enigma machines. I’d seen an Engima machine before, but this was the first time I’d seen a pair in a museum in Germany. The visit to the museum was worth it just to see these.

It was a bit of a whirlwind visit, and I really didn’t get to see much of the city or explore the local cuisine much. Hopefully my work will bring me here again in the near future.

More photos from wandering around Hamburg here: https://www.flickr.com/photos/pleia2/albums/72157665634578459

MesosCon EU 2017

I attended my first MesosCon was back in September in Los Angeles. In October I head the pleasure of participating in my second, this time the European version in Prague. As I’ve mentioned previously, this was a busy week for me. My participation in All Things Open kicked off the week, and I landed in Prague on Wednesday evening with just enough time to check into my room, say hello to some folks I knew who were in town for the Open Source Summit and then head off to a planned dinner with work folks at Restaurant Mlýnec.

The next morning arrived quickly as I was up and available for a 7:30AM breakfast with some of the MesosCon Europe keynote speakers. Part of the breakfast was spent chatting with the panelists who would join me on stage that morning to participate in a panel on “SMACK in the Enterprise” that I was moderating. At 8:30 it was time to go down to the keynote room and see that everything was on schedule for our 9AM start. The folks at the Linux Foundation do a great job coordinating these events, it was a pleasure working with them as a speaker and track lead throughout the event.

At 9AM Ben Hindman opened the conference with a talk on the current state of Apache Mesos, reflecting on past MesosCons and the past year of developments. Improvements in the platform have included things like nesting of containers, and the creation of the Container Storage Interface (CSI). Fault domains and the promise of multi-cloud also made an appearance in his keynote.

Directly following Ben’s talk was one that, as a pure open source enthusiast, made me really happy to attend. Rich Bowen of the Apache Software Foundation who came to talk to us about The Apache Way. Given my background and current role, I’m familiar with the history of the foundation and loosely keep tabs on their current work. Still, seeing a history presented with a message is always an enjoyable way to consume it, and Rich did a masterful job weaving the history in with where we are today, and explaining what “Apache” means in the Apache Mesos project. From bottom-up leadership to collaborative decision-making and they way they approach conflict-resolution, there’s a lot to admire. He also drove home the importance of transparency in a project, with no decisions being made privately or in ways that are difficult to document for future participants of the project. He also touched upon where the project was today, with over 100 projects within the Apache Software Foundation and the Apache License remaining strong in the industry.

The SMACK (Spark, Mesos, Akka, Cassandra, Kafka) in the Enterprise panel was next, and as moderator I was thrilled to be joined by representatives from Audi, Deutsche Telecom and ASML. From connected cars to how they’re making innovations on mobile networks, learning how companies are using Mesos and the entire stack to do fast data processing really brings all the work we do into focus. Innovation is happening across various industries as we all become more familiar with what we’ll need to succeed in a world with so much data coming at us, and I’m proud to be a part of it.

Thanks to the Apache Community for taking this photo (source)

The keynotes concluded with one from Netflix Engineering Director, Katharina Probst. Internally at Netflix they’ve built a massive infrastructure that is, at a high level, managed day to day by a relatively small team of Site Reliability Engineers. This has been accomplished through the tooling they built called Mantis, that allows not only for the essential autoscaling that a company like Netflix requires, with viewing numbers dropping considerably when each region has their work day, and picking up in the evening hours, but also real time testing and metrics from their platform. The key for them has been not only monitoring to detect that there is a problem somewhere, but finding out exactly where it is and reporting to the engineers what the problem may be, so a fix can be developed without a long session of troubleshooting first. As someone who used to work in operations, this was something I could really appreciate, I’ve spent many long evenings chasing down problems, evenings that could have been made much shorter with the addition of some automation to rule out the usual suspects quickly. The scale of their operations also never ceases to amaze me, and she was able to share some statistics around that as well.

This first day of the conference landed me as the track lead for the DevOps and Ops track. The day began with a talk from David vonThenen around external storage. In this talk he gave a cost/benefit evaluation of local versus external storage, and then, not keeping to just files, he ran the same evaluation when looking at storage for databases, both traditional like Postgres and MySQL and the newer distributed databases. Tying this in to Mesos, he touched upon CSI work and mentioned that REX-Ray is currently being used by DC/OS to handle the attachment of external volumes. These were interesting considerations in what has been a quickly growing part of our ecosystem as demands upon reliable storage solutions for containers quickly increase. His talk was followed by one from Ádám Sándor titled “Knee Deep in Microservices” and which had a Doom theme to demonstrate the new “demons” we have let loose as we’ve also taken all the benefits of migrating to microservices. He cited the key elements of DevOps, resilience, elasticity, resource abstraction, and tooling that helps us monitor containers as key improvements in the microservices platforms that are helping us tame the demons we have unleashed. It was also nice for me to hear these things, these massive distributed systems are complicated, we need to be continually improving our toolkit (weapons!) to effectively manage them.

After lunch Adam Bordelon and Alexander Rojas joined the track to give a popular talk titled “Mesos Security Exposed!” I won’t enumerate them all here, but by digging into how Mesos works, they were able to pull back the curtain and explain what needed securing in your cluster, from API endpoints to use of TLS and the handling of private data (secrets). Julien Stroheker then spoke on “Doing Real DevOps with DC/OS” where he gave a live demonstration of a continuous integration pipeline with Jenkins, Docker, Gradle and Vamp. The sessions for the day concluded with Zain Malik giving us a tour of mesos2iam, “IAM Credentials for Containers Running Inside a Mesos Cluster”.

But the day was not over! Appetizers and drinks at the attendee reception were directly followed by Town Halls, where attendees could casually gather in groups to talk about Mesos, Marathon, DC/OS or Kubernetes. My colleague Matt led the DC/OS session, with me taking notes and pitching in here and there. Our Town Hall began with introductions, with 15-20 people in attendence as the 90 minute evening session progressed we had a nice mix of people from a variety of unrelated industries. With the ice broken, it was easy to get people talking about pain points they had with DC/OS and we had Adam Bordelon in the room to give history and insight into specific features and concerns that people were bringing up. At the end of the session we had a nice list of shared struggles, but the attendees also were able to swap knowledge with each other. Empathy goes a long way, and in my own work I know how valuable it is to know that concerns you have are being shared and solved by others too.

Attendees file in for the DC/OS town hall Thursday evening

The Town Halls wrapped up at 8:30 and I headed straight to bed. Between the jet lag and the 13+ hour day, I was exhausted.

On Friday the keynotes also began at 9AM, but the keynote slot was shorter in order to squeeze more track sessions in during the rest of the day.

After an welcome and opening remarks from Jörg Schad, the first keynote came from Yrieix Garnier who gave a more Enterprise-focused take on “The Future of Apache Mesos and DC/OS” than Ben had explored in his keynote the day before. He was able to pull up statistics about the data-processing power of the platforms, sharing that 50% of DC/OS clusters were running some form of the SMACK or ELK stacks. The big news from his talk was unveiling of TensorFlow support in DC/OS. We then had Pierre Cheynier join us to share a talk that he had originally proposed during our CFP, but we upgraded to a keynote, “Operating 600+ Mesos Servers on 7 Datacenters @Criteo”. The ability to scale is a key feature of Mesos, so it was fascinating to learn about the scale of their work and just how much data they were storing (171PB on Hadoop!). He also shared a series of tips for other organizations looking to operate at this scale, including effectively automating everything (configuration management, scaling, CI system), defensive configuration (things will go wrong, be prepared), visibility to operations as to what is going on (metrics, alerts, tracing), and the importance of doing networking right, and addressing problems like QoS and “noisy neighbors” during design. He also covered some incidents which gave further insight into the sorts of things they were able to effectively prevent, or not.

In a slight shift from Thursday, I spent Friday as the track lead for the Users and Ops track. The first talk came from the oldest company in the lineup, the publisher Houghton Mifflin Harcourt. I was looking forward to this talk, and as Robert Allen got into the details of “How HMH Went from Months to Minutes for Infrastructure Delivery” I was not disappointed. In a common theme for the day, he brought up the slow, inconsistent, old technology team was not keeping up with either the industry or their own product lines, as the publishing industry is rapidly changing to serve a more tech savvy customer base. He walked us through the creation of their “Bedrock tech services” team, and their DevOps focused goals, including comprehensive developer involvement from idea to production, a continuous delivery approach that encouraged small, frequent updates, and a change in culture that made them shift from feeling like they must prevent failure, and instead acting as if failure will occur and planning accordingly. He then dove into the technologies used, beyond Apache Mesos, they’ve also been using Apache Aurora, Terraform for orchestration, Vault for secrets, and a Jenkins plus Artifactory CI/CD pipeline. He also stressed the importance of metrics and logging, all things close to my own interests as well!

Tim Nolet then joined us to give a talk on “Advanced Deployment Strategies and Workflows for Containerized Apps on DC/OS” where he too walked us down the painful memory lane of the massive, error-prone deployments of the 2000s, the rise of DevOps and today the productization of a lot of the DevOps tooling, including Vamp, which he works on the development of. As a product, Vamp seeks to simplify and tame the ecosystem of microservices you have running by simplifying the process of deployments. He then showed a demo of load being distributed across different versions of a deployed application, as well as different versions being served up to customers using different clients.

In our next user story, we heard from Jay Chin on “Optimising Mesos Utilization at Opentable”. He began his talk with a quick production history of the infrastructure at OpenTable, sharing that they made the move to microservices in 2013. During this time they were orchestrating their microservices via standard configuration management tooling, a process that turned out to not only difficult to maintain at scale, but was actively disliked by the engineers who had to work on it. In 2014 they switched to Mesos, and through resource abstraction and running a consolidated cluster, they were able to simplify application-level operations. Additionally, it allowed them to easily create centralized metrics and logging. It’s a story I’d heard before from other companies, and one I was thrilled to hear again, but where OpenTable really led the way here was by open sourcing some interesting tools they were using, including the following mentioned during his talk:

The day continued with Ilya Dmitrichenko on “Time Traveling in the Universe of Microservices and Orchestration” where he used his own career as a baseline for the changes he was seeing in the industry with regard to the rise of microservices. His dabbling with old Sun boxes and awareness of things like openvz is consistent with some of my own hobbiest and day job work early in my career. Indeed, containers have always been with us. He went on to track the rise of Docker simplifying the space and tooling coming together to make management of a microservices infrastructure feasible for smaller organizations. The culmination of this story was his current work at Weaveworks which, just like Vamp improves deployments, improves and simplifies your network policy.

Jorge Salamero of Sysdig joined us next with the amusingly titled talk, “WTF, My Container Just Spawned a Shell”. I was immediately fond of this talk because after a sea of Macs, Jorge uses Xubuntu. He began his presentation by talking about how we consume container images, and the rise in static scanning of these images for libraries with vulnerabilities and more, but that this only goes so far. There are things that these scans miss because they need constant updating, and which behavioral analysis while the container is running will catch. He introduced how Sysdig’s metrics tooling used with Falco can give you a comprehensive view inside the containers you’re running. Suddenly you have access to security tracking that can show you every command taking place inside of your containers, and from there you can train it to be aware of problem behavior. He also talked about SysDig Inspect, the open source project that they have built their Sysdig Secure product with.

Julien Stroheker then joined us again to talk about his DC/OS autoscaler. This is a talk I’d already seen twice, so I won’t go into it, but it is a cool project that he’s always looking for help on! The final talk of the day came to us from Fredrik Lindner of Tunstall Nordic AB, who shared “There and Back Again: How Tunstall Healthcare Built an IoT Platform for Health Monitoring Using Mesos Cluster on Azure.” Just like so many other industries, elder care is undergoing a transformation with the help of IoT technology. He shared details of their “Evity” platform they were developing on top of DC/OS to help manage the data coming in from these devices so they can effectively and quickly meet the needs of the families they work with. It was a good story to end with, as with much of the technology we saw showcased throughout this MesosCon, as consumers we just assume it will all work. My FitBit will reliably track my steps and share them with my friends, my car will show me traffic, and when something goes wrong it’s unexpected and unsettling. This is even more pronounced with you get into the space of technologies that help with things like healthcare, where reliability, accuracy and speed have even more urgency. We’re building the platform that people are using to make sure these all run well, and that’s pretty exciting.

When I finally stepped out of that track, all the booths had been taken down and most of the conference attendees had gone their own ways. I was able to finally leave the hotel with Matt and Jörg to explore some of the old city in what turned into a lovely night that thankfully didn’t keep me out too late. The conference was great, but predictably exhausting, especially coming on the heels of All Things Open on another continent.

Huge thanks to everyone involved in organizing, and to all the speakers in my tracks who made the event really interesting by sharing their stories, tools and expertise. This event was a little smaller than the one in Los Angeles, but it didn’t feel like it, and the quality of the event was top notch.

More of my photos from the event can be found here: https://www.flickr.com/photos/pleia2/albums/72157689885991786 and more photos, slides and videos are hosted by the Linuxd Foundation at http://events.linuxfoundation.org/events/mesoscon-europe

Tourist in Prague

I’m in the midst of a bunch of travel for work, with a trip to Prague directly following the fun I had in Raleigh at All Things Open. I took a flight out of Raleigh on Tuesday evening, making connections in Charlotte and London before finally landing in Prague on schedule on Wednesday evening. I was there for MesosCon EU, but I’ll write about that later, first I want to talk about what I got to see Friday evening and on the Saturday I spent there to do some tourist things.

The Open Source Summit was happening when I got into town, so I knew a whole lot of people who were there for that event. I was thrilled to meet up with Gina Feichtinger, who I’d long known online through LinuxChix, but never had the opportunity to meet. Unfortunately our schedules didn’t work out to spend more than a few minutes chatting before I had to run off to dinner on Wednesday night, but that was enough time to get a picture!

The conference concluded on Friday night and I was able to get my first real taste of the city as I went out with some colleagues for drinks at Lokal and then a nice dinner together over at U Sádlů. Prague is beautiful at night, and I enjoyed wandering around Old Town all evening.

Looking over at Church of Our Lady before Týn (not a castle!)

Saturday was my big tourist day. I met up with my colleagues Matt and Jörg to head up to Prague Castle. Prague Castle does not look like a castle, I’d call it a palace. There were tours but with just a couple hours of morning on our hands we decided to just explore the outside. We did make our way inside St. Vitus Cathedral, which is inside the Prague Castle complex. It’s a beautiful cathedral.

After visiting inside, we walked around it and then through the village area that surrounds the castle and took in some beautiful views. We then slowly made our way down the steep hill and back to the river. We parted ways as we got back to old town and they had flights to catch. This is when I made my way to a nearby hotel to meet up with my second Matt of the day, this time a friend and former colleague from OpenStack work.

Our first stop was for some lunch at U Fleků. It’s was founded in 1499 and claims to be the oldest brewery in Prague. They’ve really capitalized on this, it’s a prime tourist spot. They wander around with trays of beer, mead and becherovka. The food was nothing to write home about and it was a bit touristy for me, especially once the accordion began to play. Still, I’m glad we stopped by and I certainly enjoyed the beers, mead and the sip I stole of Matt’s becherovka. From there we took the long walk out to Vyšehrad. This is essentially the other castle site in the city, but it’s now a public park instead of a residence, though the cathedral and famous cemetery are in use for their intended purposes. We stopped at a cafe while wandering around to address some beer/mead sleepiness from lunch.

The views from here were beautiful as well, allowing for selfie opportunity!

While in Prague I also enjoyed tram watching. Had I bothered to figure out how to ride them I would have saved myself an incredible amount of walking. Maybe next time. Our journey back to our respective hotels did bring us back to the old town, this time in full weekend swing. It was packed with people. I was pretty eager to be on my way, though stopping for a filled Trdelnik was tempting, the whole neighborhood smelled like them.

My evening was spent with yet another friend who was in town for the Open Source Summit, Nigel! Nigel and I met through our involvement in the Ubuntu community. We’ve both gone our own ways at this point, but we’ve kept in touch and as we’ve both followed open infrastructure paths there’s frequent cause for us to chat. We met at my hotel and went to a nearby restaurant where I enjoyed both the dark and the light lager with lots of fun conversation.

That last night in Prague was an early one, as I had a 7AM flight Sunday morning to head off to Hamburg for my next adventure.

Lots more photos from Prague here: https://www.flickr.com/photos/pleia2/sets/72157687755704821/

All Things Open 2017

The goal of the trip down to Raleigh last week was to attend All Things Open. I attended back in 2014 and was eager to attend again, but was hampered by conflicting events. This year I committed to going, in spite of an unfortunate overlap with the Open Source Summit in Prague, which is important to me because MesosCon Europe happens on the Thursday and Friday of that week. This means I had to get on a plane immediately at the conclusion of All Things Open on Tuesday in order to make it to Prague by Wednesday night to begin my MesosCon EU activities. It was a long week.

But the week began in the most relaxing way possible as I stepped off the train in Raleigh Sunday night. Conference badge was picked up and I was off to a pre-conference social to say my first hellos to fellow conference-goers. I was quickly able to find some friends, but the social venue was quite busy and I was eager to find some dinner.

On the way out of the social I collected some folks and five of us ended up at an empanada and tequila restaurant not too far from the conference venue. Highlight of the night besides good company? They had pumpkin pie emapandas as a dessert. Heavenly.

The next morning the conference began with a series of keynotes. Todd Lewis, the grand architect of All Things Open and super nice guy, kicked everything off with welcomes and piles of gratitude. His personal kindness is one of the main reasons I love this conference so much. It’s also the largest open source conference on the east coast, this year drawing 3000 attendees, which is a new record for the event. That’s how I convince my employers to support the event year after year.

The first keynote came from Tim Yeaton of Red Hat who began his talk with a fun walk through a typical day that brought him to Starbucks, Chili’s, Target and then on to a United flight, all companies who are major users of open source and helping to drive the incredible growth throughout our industry. Jake Flomenberg of Accel joined us next to talk about the rise of open innovation, with companies like Intel and Goldman Sachs choosing open source software first and proprietary only as a last resort, a distinct change from what we were seeing a decade ago. He shared that there’s also a rise in venture capital backing of open source-driven companies who are driving innovation, it’s no longer just Red Hat making money in the world of open source business, others are following the path of Project, Product and Profit.

Danese Cooper, who has worn many hats in her lengthy open source career but is currently the Head of Open Source Software at Paypal, then graced the keynote stage. She began her talk by walking us thorough a history of why many people got involved with open source in the early days, recounting a story from Mark Shuttleworth. He explained his motivation behind investing his wealth in an open source project and company was because he cared and wanted to build something that mattered. The rest of us in open source software weren’t much different, even if we didn’t have millions to invest, my own passion drove my involvement through years of unpaid work. She shared that while many people today are getting involved for the very valid reason of being paid for their work, we can’t forget the lessons we learned along the way. To this end, we need to document our history and motivations so that future generations will know how we got to where we are, as well as mentor new people, serve as board members in open source organizations, and encourage good practices within our communities and company. She concluded by also stressing that we all must continue to follow our moral compass as we contribute, and that money to pay our mortgages will follow. That’s certainly true in my case.

Perhaps the most exciting keynote for me of the morning was from Sara Chipps, the CEO of Jewelbots. Jewelbots are tech-enabled friendship bracelets that have default modes to let you know when friends are nearby, but you can also extend them by writing C++ programs. She gave some history of the company, sharing that she wanted to build something for girls and noticed that they already had traditional friendship bracelets. By creating a tech-enabled one she discovered that the drive to extend the devices functionality-wise would “trick” the girls into having to write code. It worked, of the 10,000 units shipped, 44% of the users have extended them by writing C++ code, an incredible conversion rate. Finally, all the code for the devices has been made open source and is available on GitHub. Super inspiring keynote, it makes me so happy to see things like GoldieBlox and Jewelbots in the market for girls these days.

The final keynote of the day came from Kelsey Hightower of Google and Kubernetes fame. His talk was entirely demo-driven as he stressed that containers were not hype, they do simplify infrastructure by putting applications in nice little boxes. But that they aren’t a panacea, and while they help, we haven’t gotten to our future of flawless automation with them or any infrastructure technology. Much to our delight, he then was able to use Google-driven voice commands to show off not just a Kubernetes deployment on Google’s cloud, but an on the fly upgrade which he sees as the next big evolution in maintainability of large clusters.

With the keynotes behind us, the first talk of the day I attended was by Alicia Carr on Home Automation. The talk itself was a tour of the home automation devices she used in her day to day life, exploring how they interacted with each other (or not!) and her favorites on the market. She addressed the topic of security, admitting that we’re still in the early days of home automation and it’s not where it needs to be to be entirely safe and fool-proof for full, mainstream adoption, but that the companies in the space were aware of these limitations and working on it. Unfortunately there wasn’t much open source to this talk, some of the devices did allow you to code against them, and trigger tooling like IFTTT can allow you do do interesting things. I’d really love to see a talk about the home automation ecosystem of free software hackers that’s emerged, and the open source considerations therein. What devices are the most friendly to hacking? Which can be independently secured most easily?

The next talk I went to was John Mertic on “Accelerating Big Data Implementations for the Connected World” where he focused on the ODPi which seeks to strengthen the engagement model between upstream projects and end users. Five years ago the industry started adopting tooling like Hadoop to do big data processing at a large scale, but even then we weren’t at a place where we are today with the emergence of the Internet of Things and vastly more data coming in than we ever imagined. He shared that that the major challenge now is standardization. The options today for companies seeking to wrangle their data tend to be relying upon raw open source projects, which are frequently not productized enough for simple use, or being led by a vendor who may not have a standardization strategy in mind, instead steering you towards their version of productized piece of open source software. Companies need a resource to help fill in the gap there, and this is were the ODPi comes in.

After lunch I went to Corey Quinn’s talk on AWS cost control. He’s a fun and engaging speaker so I always enjoy going to his talks, but it was also incredibly informative. Though I’m not personally seeking to reduce AWS costs, it was fascinating to learn from his experience about what finance departments, C-level executives and company boards actually care about. Many consulting companies around AWS cost reduction focus on a series of technical strategies that can reduce cost, but his talk wasn’t about those, instead he focused on understanding what you’re running from a business perspective. It’s great to be efficient, but at the end of the day a large bill is actually fine if you can account for what you’re spending it on. Tracking workloads and having an itemized, defendable listings of costs goes much further than a painful cost reduction that knocks a few percents off of what is still an expensive black box as far as finance is concerned. Even better, when you do a project to start tracking your workloads, you’ll inevitably find services that aren’t being used and other extra fat that can be trimmed from your bill, and you do end up with the cost reduction too. Given the audience, he also shared the Open Guide to AWS and his own Last Week in AWS newsletter, both valuable resources to folks administrating AWS.

At 2:30 I gave my talk of the conference! I spoke again on The Open Sourcing of Infrastructure (slides here), taking the audience through the work we’ve done to choose open source as our infrastructure platform and how we can use those lessons moving forward in a world of proprietary cloud services. I gave a version of this talk back at FOSSCON in Philadelphia, but it’s one of those talks that’s always evolving based on audience feedback. The talk went well and afterwards I had a chat that will lead to some additional slides about security. Most specifically, there have been high profile security vulnerabilities in open source software and the major open source tooling that’s using it tends to have patches out within hours of disclosure. We don’t have that with proprietary tooling, in fact, the latest WPA vulnerability has left millions of proprietary devices without patches, and no indication to the customers about when patches will be available, assuming they will be and the devices and software are able to be upgraded at all.

The next talk I went to was from Elsie Phillips of CoreOS in the business track about building an selling an open core product. She walked through business models around training, support and consulting, sharing that these are still valuable avenues of making money, but that they generally weren’t enough and had very slim profit margins. There’s also a balancing act that must be made when you have open core software, making sure that the open source version is valuable and that the “Enterprise” version simply adds features, not that the open source version is restricted in some way. She stressed that the proprietary version should add polish like automation and other tooling that is not impossible to do on your own with the open source version, but saves the company engineering investment if it’s done for them out of the box. I happen to work for a company that uses this model, so while my open source heart bristles at some of the decisions made about what goes into the open source product, we do all have bills to pay and I often find myself in agreement as to where the line is drawn.

The final talk I attended of this first dway was by my Mesosphere colleague Alexander Rukletsov on Health Checking: A not-so-trivial task in the distributed containerized world. This talk drew from his experience with the overhaul of Apache Mesos health checks that occurred a year before. At first glance, it seems like doing health checks inside of containers would be easy since they live on a host that has access to them, but it turns out that it’s not. You have PID and networking namespaces to contend with that can turn even the most simple checks into something unexpectedly difficult. In a distributed system you need to consider the costs and benefits to how broadly you scope your checks, noting that a global scoping may be easier to set up, but you then have network overhead, latency and potential for duplication across your cluster. It was a valuable talk from an operations perspective allowing you to understand what goes into a check you’d generally consider “just an HTTP check” or similar.

All Things Open is a tiring conference and I was busy all day, so my day concluded with some nearby chicken and waffles before heading back to my room.

Tuesday opened with more keynotes! The keynotes began with a thoughtful and inspiring talk by Safia Abdalla who, among other things, called upon the audience to build friendships (not just mentorships) and to not just share code in our communities, but to share knowledge. Her talk was quickly followed up by one from Matt Asay, a familiar executive in open source circles and someone whose opinion I respect even if I don’t always agree with his conclusions. As he reviewed key open source successes in the corporate world he spoke to pragmatism as we push for more openness in our industry. He explained that convenience will typically win out and there’s always some amount of lock-in. There is victory even when building into proprietary clouds, they are more open than what we came from and most of what we’re building on top of that is still open source tooling. As someone who so strongly believes in open source as far down the stack as possible, I see his pragmatism argument, and even addressed it in my talk the day before (you can build against proprietary technology if you want, lots of companies do) but I’m glad for the industry that we have choice.

Burr Sutter joined the keynote stage next to give a very fun and accessible introduction to DevOps. He recounted stories of the improvements to code quality when developers shared in the responsibility in production, stressed the importance of on-demand infrastructure, automation and continuous integraton and deployment for fast, high quality development. John Papas was up next to talk about the open web, talking about some of the technology choices that should be considered when making modern websites, including ways to improve load time through priority loading of elements and searching code for module duplication and the presence of too many dependencies that could increase the size of the page being loaded. He concluded by asking us to be kind to one another, we may not see each other as we interact over coding sites and GitHub, but there are real people on the other end. The last keynote was from Jeff Atwood, the creator of StackOverflow and Discourse. His talk was a story through the creation of StackOverflow as a community and company, and how the more community-focused Discourse project came about. It was amusing to learn that having children so deeply impacted his approach to solving problems, or not solving problems. He learned that not every problem is actually in need of a solution, sometimes we just need to be human and empathetic to one another, and that’s enough. With StackOverflow the goal is very strictly about solving problems. With Discourse it’s about building a community and having fun.

The first session following the keynotes I attended was from Mark Voelker on Interoperable Clouds and How To Build (or Buy) Them. I knew of Mark from his work in the OpenStack community, and his talk centered around the work he’s done and lessons learned while participating in the “OpenStack Powered” program, where products can be certified through a testing suite to carry that label. It causes the OpenStack name means something, and ensures that is not haphazardly applied to products that replace pieces of OpenStack with their own tooling, run non-standard versions of the APIs or are otherwise not what the user may expect from something called OpenStack. He dove into some of the strategies the community uses for determining what should be tested and they iterate on the guidelines with every release, drawing from customers and product creators to learn what they should be including, excluding or changing. These lessons extend to what’s popping up in Kubernetes Certified and other programs in this space.

After grabbing lunch with one of my DevOps + open source compatriots, I met up with Spencer Krum, one of my former colleagues on the OpenStack Infrastructure team and current co-conspirator on open source infra work. He was doing live broadcasts for The Root Shell at ATO and I was invited to swing by to chat. It was fun, we talked about my current focus of containers and our work on open infrastructures. When we wrapped up I headed directly for the lightning talks that had already begun, making it in time to see Chris Short talk about DevOps and the joint official launch announcement with Jason Hibbets of the new OpenSource.com DevOps Team, which I happily joined a few weeks ago. I also really loved the Lego hacking lightning talk from Jen Krieger, DYI electronics hacking with Lego looks like a lot of fun and she showed off some neat projects.

Now, in spite of leaving much of my direct operations work behind when I changed jobs last year, Jenkins still looms large in my work today as I am working on a CI/CD demo and frequently give talks about how containerization can be used in this space. As such, I went to a talk by Kristin Whetstone on “7 Habits of Highly Scalable Jenkins Administrators”. The habits included using the latest LTS version of Jenkins, using Jenkins Pipeline and container-based plugins for more self-service job deployments, parallelization of tests and participation in the Jenkins community. What I really enjoyed about this talk was that she not only would describe the problem space of each of these, but gave links to the specific plugins that would help you achieve your goal with them. The day was getting long as I went with the fun choice for the next session, and joined Gareth Greenaway in his “What the FOSS Community Can Learn from 80s Television” talk. It was a funny trip down 80s television memory lane with open source lessons, including: the importance of working as a team, ignoring MacGyver’s failure to document his solutions, improving the bus factor, reducing the impostor feels by new contributors as Sam in Quantum Leap perpetually felt, and the importance of not simply creating new projects (characters) or forks (spin-offs) without a good reason.

In the late afternoon I met up with my friend Laura over at the booth she was running. The plan? A photo of the Ubuntu gurus we knew at the conference. Success! Not many of us are working on Ubuntu any longer, but the friendships we made in that project sure have lasted.

Our connection? Ubuntu!

With that, we were just one talk away from the end of the conference, and my need to get to the airport. The final talk I attended was by my friend and former colleague on Ubuntu work, Michael Hall. His talk had the interesting name of “The Revolution Will Not Be Distributed” which you quickly learn is not about distributed infrastructure, but instead about the way that software itself is delivered. One of my first contributions to an open source project was packaging in Debian, and it was hard. I had a mentor and throughout my work in Debian and Ubuntu, even as I helped others with packaging, I was always consulting the documentation and asking for help from other people about the proper and expected way to do various tasks with Debian packages. At the time I hadn’t thought much about how this wouldn’t scale, and I was right along side everyone else who worked on various efforts to teach more people packaging and to get software creators to package their software for distributions. This included various efforts that the community team at Canonical embarked on to get more software packaged, none of which resulted in a significant, sustained increase in the number of packages or packagers in the community.

Michael ran the numbers for us. There are about three million applications in the Android marketplace. The Ubuntu desktop has about three thousand. You can definitely argue that the Ubuntu includes high quality applications that build a firm and sufficient foundation for a workable desktop, but we all know that there are gaps, and software we install outside of the standard repositories, whether it’s by using a PPA, install script, or continuing to compile it by ourselves. With this baseline gap, along with statistics about the current number of package maintainers per package throughout various distributions, he calculated that the number of maintainers and people-hours involved to get to Android was astronomical. Even to get a respectable fraction was not within reach. Instead of continuing to toil in the world of distributions with their own package formats, he looked to what is already happening with formats like Flatpak and Snappy, making packages that are simpler to make and portable (especially with FlatHub and the Snap Store), as well as the increased adoption of containers. His vision for the future of distributions was less about packaging of software itself into DEBs and RPMs, and more focus on integration of this software with the desktop environment for a more pleasant user experience. It was an interesting talk, and certainly gave me a lot to think about with regard to our approach to the distribution of software.

More photos from the conference here: https://www.flickr.com/photos/pleia2/albums/72157686641873722

It was then off to the airport! I cabbed it over with my friend Stephen, Michael and David, and the latter two ended up being on the same flight as me. Even more amusing, our connections in Charlotte (not a small airport!) were leaving out of a trio of gates right next to each other. It was a lovely way to conclude my ATO adventure.

Huge thanks to Todd and the team who put on All Things Open. It’s a wonderful conference and I was happy to participate again this year.