Unfortunately most of my plans didn’t pan out, but I was at least able to turn Sunday around! I swapped some evening plans with them for daytime hangout plans with a couple of my friends in New Jersey and their little ones. A daytime visit meant I could more easily visit with the newest additions to their family (a child and a kitty). They have really been there for me since early in the pregnancy, I didn’t reach out often, but it was a relief to know I had someone to talk/rant/ask advice from as I stumble through all of this. It was great to finally catch up in person, and as I’ve casually been browsing telescopes lately, get shown their Celestron telescope (the brand I’m looking at) and get some ideas about how I may set up my own when I finally make the leap.

Being in town did mean I could get some tasks done at the townhouse. I was available Monday morning as a bed was delivered for the guest room. Until now we ha just gotten by with an old metal bed frame for that room, but we recently decided we wanted a more finished room. A platform bed was delivered and assembled, and I’m happy with how it turned out. Tuesday morning I took the MDX to the shop to get an annual check-up. Since we only use it when we’re in Philadelphia, it doesn’t have enough miles on it to warrant mileage-based check-ups, but we do like having an oil change and a look over it at least once a year. Wednesday morning I met with a handyman who sorted out the ventilation problem in our attic. If you recall, ventilation issues caused frost in the attic over the winter and extended my holiday visit to work with the remediation company that came in with heaters and dehumidifiers. The saga of the attic is not over, but the exhaust fans in the master bathroom are safe to use again, and he was able to look at a couple other things while he was over.

New guest bed!

My social plans not working out made for a bit of a lonely week. I wish I could say that I got a lot of project work done each evening instead, but it wasn’t really the case. I felt pretty down and watched a bit more TV than I would have liked. Thankfully MJ joined me on Thursday and I perked up as we met up with friends that evening to see Krull get the RiffTrax treatment at the nearby theater at Neshaminy. Krull is a ridiculous fantasy movie from the 80s that my family owned when I was a kid. It was the early 80s, so you may be asking how we had it, VHS? Betamax? Laserdisc? None of these! We had one of the rare CED players, which played videodiscs. I think one of the most amusing things to come out of the RiffTrax treatment of Krull was my tweeting this particular fact, being retweeted by RiffTrax, and then having fellow internet nerds start geeking out about CEDs. One of my friends even chimed in with a whole 30 minute video made a couple years ago that covered some of the history and technology of CEDs. For a few hours that day, I remembered that there can sometimes be a considerable amount of joy from connecting with random people on social media.

And RiffTrax: Krull itself? I’m very glad I didn’t remember the movie, it was terrible, and likely pretty unwatchable today without the delightful commentary of Bill, Kevin, and Mike. Afterwards we headed over to Unos for a late dinner.

Saturday was FOSSCON! I come into town for this conference every year, and it’s often the one time each year I can connect with the open source tech crowd in Philadelphia, many of the members of which I’ve known for well over a decade. I always meet new people as well, and this year was no exception.

The conference kicked off with a round table on effectively promoting FOSS to businesses, which focused on “selling” organizations solutions rather than banging the drum about freedom and other things that get us excited about open source. From there I attended a talk by Angel Rivera of CircleCI on “Build for Production using CI/CD Pipelines & Docker.” It’s a small conference, but I was glad that he gave a talk that was so complementary to mine. He covered the CI/CD fundamentals in depth before launching into his demo that used CircleCI to build and execute an entire pipeline.

After lunch (cheesesteaks!) I settled in to prepare a cluster for my own upcoming CI/CD talk, and have some lovely “hallway track” time catching up with folks. When I was ready, I got set up in the auditorium to prepare for my talk. The talk focused on the benefits of using containers (with a focus on Apache Mesos and DC/OS) for your CI/CD pipeline. I used the demo I’d completed successfully dozens of times, including on stage twice. Unfortunately I finally had to pay my dues to the “demo gods,” my live demo failed! I didn’t have a backup strategy for running the demo because I was so confident in the success. Instead I talked through the steps and was able to show off what the results should have looked like. In retrospect I suspect there was just too much latency on the cloud platform I was using, since the demo had problems from the start. If I had the time, I would have re-launched the cluster in a region that was geographically closer, and I may see about using a local deployment on my laptop as a total network failure backup. Regardless of the demo, I think I got my points across and the audience seemed generally sympathetic when I was later answering questions at the end of the presentation. Slides from the talk are available as pdf here.

Thanks to Angel Rivera for this photo during my talk! (source)

The conference continued with a talk about running Apache Ignite on Kubernetes, and then a series of lightning talks. As things wrapped up, I found the annual Oreo cake (my favorite!) that Jim Fisher brought along and shared with anyone who was walking by as the event concluded. We then collected a few people for an unofficial after event dinner at the nearby City Tap Room.

Being familiar with the organizers, staff and many of the attendees means this conference is probably the one where I feel most comfortable. It’s still a long, exhausting day, especially as I’m recovering from a cold that made a minor comeback in the form of a sore throat and continued cough, but I had a great time. Huge thanks to Jonathan Simpson and the rest of the FOSSCON crew who spend so much time and effort into putting it together each year.

A few more photos from FOSSCON here: https://www.flickr.com/photos/pleia2/albums/72157699031220361

Sunday we flew back to California! We used miles to upgrade to some international-style first class seats on a 767 that was doing an unusual trip direct from Philadelphia to San Francisco. I usually struggle to be productive on flights, but I’ve been more focused lately on flights, and the five hour flight gave me time to catch up on some reading for work that I’d been meaning to do for weeks, along with drafting up most of this blog post.

I’m now back in California for less than 48 hours before I leave for the Open Source Summit in Vancouver tomorrow evening. It’s enough time for me to get some work squared away (including adding some finishing touches to my talk!), do a bit of laundry, and make sure Caligula hasn’t forgotten who I am. After that, I’m in town for a few weeks, which will give me time to finish some big projects at work before I start traveling again in October and November for the five conferences I have lined up.

Thanks to Nathan Handler for getting a picture during the meeting (source)

Later that week I met up colleagues early Friday morning in San Francisco for a team off-site in Sonoma and day of wine tours. We stopped at Benziger for a quick tram tour, then went off to Imagery to enjoy a boxed lunch at a table I’d used my membership to help arrange. From there we went to Petroni, which was a new one for me, and enjoyed a wine tasting and stories of the history of the family and winery in their caves. It was a nice to get to know some of my colleagues who I don’t work with every day, and in spite of being on the warm side, Sonoma is always beautiful and a pleasure to visit.

At the end of June we flew out to Philadelphia for a week. The trip was aimed at spending time with family and going through lots of stuff to organize and decide what we’re keeping there at the townhouse and what we’re going to ship out west to use and store here at the much larger house in California. It was an exhausting week, so I’m glad we took a nap on Saturday afternoon after stepping off our flight and spent the first evening with MJ’s sister Irina and her husband Sam over at Longwood Gardens. I went with a friend last year to see their fountain light show, and it featured a delightful mix of classic jazz. This year they changed it! Instead of the standard spot for viewing the show, we stood on the bridge in the middle of the fountains and they had switched to a soundtrack of pop music reaching back to the 70s. I was a big fan of the Pinball Wizard bit. The company was good as well, it was nice to grab some dinner at their outdoor beer garden, wander through some of the gardens, and generally catch up for a couple hours.

Mid-week my cousin Melissa was driving down the east coast and stopped by for a visit and dinner. She’s one of the cousins I see the most because we both travel and our paths cross often, especially when work or events bring her out to San Francisco every couple of years. She had her dog with her so I scoured restaurant reviews to find one with an outdoor area, and we lucked out with a 5PM thunderstorm that broke the heat wave a bit just before she arrived. We had a lovely dinner outside in the freshly minted 75 degree post-storm weather, which had earlier been in the 90s and humid.

The 4th of July was spent at MJ’s father’s house with BBQ and some extended family joining us as well. Plus, there was cake! We also got to meet up with family later in the week for a birthday at a nearby hibachi restaurant.

All the family visits aside, those boxes of stuff were front and center in this trip. We were able to get through everything and now have a solid inventory of it all with decisions about what we’ll be sending out west. We’re a bit concerned about whether everything will fit into the container we ordered to ship everything out, but we’ll find out next week. We’re taking another Friday night flight to get us into Philly around 8AM on Saturday, then spend the weekend finalizing what we need to do before the movers arrive. I will be so thrilled to have most of the boxes, totes and extra furniture finally out of the den we’ve designated as our shared office, in no small part because of all the comments and questions I get when I’m on work video calls about the sea of boxes I live with.

Our trip concluded by taking SEPTA down to 30th Street Station in Philadelphia where MJ had lunch before parting ways. He was headed out west on a plane for a quick work trip and I was taking the Amtrak Acela north to Boston for work of my own.

The last time I was at an OpenStack Summit was in Barcelona 18 months prior, during my last week with HPE. As a high end to much of my OpenStack work, I spent that week prepping for and then delivering one of the keynotes where I demonstrated live addition of OpenStack-powered clouds to to the production Nodepool in the OpenStack project CI system. At the time, the key messaging was OpenStack as an “integration engine” that empowers organizations to embrace a wide breadth of proven technologies, from storage to compute-focused virtualization to software-defined networking. I think in these 18 months we’ve started taking those features for granted as the virtualized server side of the market becomes more commoditized and what really resonates with companies today is a desire to avoid vendor lock-in. With this change of pace, the OpenStack Summit this time around had strong messaging around “Open Infrastructure” so you’re not bound to a single cloud provider. Key to the product strategy in the container space, it’s messaging that I’m familiar with and generally resonates with my own goals in the free software movement.

As far as the OpenStack Summit itself goes, for the first time I wasn’t there to collaborate with my peers on OpenStack, so it was a very different experience than past years. As an industry observer this time, I could enjoy the keynote explorations into edge computing and the improvements in hardware involvement in virtualization, from CPUs to networks. There was the general collaboration between would-be competitors throughout the keynotes was what I’d come to expect from this amazing community (with the notable exception of Canonical…). In addition to their general “open infrastructure” messaging, there also has started to be a shift in the role of the foundation, with support of additional projects that, while complementary, aren’t strictly tied to OpenStack itself. This was highlighted with announcements around the independent projects Kata Containers and Zuul CI. Amusingly, containers and CI systems are both things that are now solidly in my wheelhouse, so it appears my own career trajectory is currently mirroring what we’re seeing there. The OpenDev conference I was there for solidly showed their commitment to CI/CD systems, but massive container tracks at the event make me now see the value in Mesosphere being a stronger part of the conversation at future summits.

OpenDev was great. I wrote about my keynote and the event in broad swaths over on the company blog in Open CI/CD Systems Gaining Traction. My keynote was about doing CI/CD for microservices in containers, where the bulk of the 14 minute talk was a demo where I showed deploying Jenkins, Github and a brand new Git repository for a website, with tests, on DC/OS in 12 minutes. It was a nerve-wracking adventure up there on the stage, but it succeeded! If you’re curious, the demo is fully open source so you can even try it yourself: video, slides and demo.

I also enjoyed finally meeting Benjamin Mako Hill (who I overlapped with a bit in Ubuntu work very early on) and seeing him speak on a topic he has written about, Free Software Needs Free Tools (video). He walked through the lessons Linux learned from their trouble with the proprietary BitKeeper software and stressed the importance of using free tools for the development of free software. It was an incredibly popular talk and message for this crowd, as much of the work we’re all doing is focused on an open source ecosystem for software development. A talk from Fatih Degirmenci and Daniel Farrell on Continuous Delivery Across Communities was also fascinating (video). Collaborating at OpenCI.io there are multiple open source projects that now do some degree of hooking their CI systems together to test specific changes against each other, it was fun to sit down with them on day two of the conference to see what the next steps are for making software development better across open source by using forward-thinking strategies that cross the boundaries between communities.

The conference had several talks spanning the open source CI/CD ecosystem today, from Spinnaker to Zuul to a Kubernetes-driven implementation of CI/CD tooling. What was most valuable to me though were the collaboration sessions, which get several practitioners in a room together, many of whom had never met, to talk through common problems and start coming up with solutions and action items. I can catch up with the talks online post-event, but it was energizing to be in a room with others who share my interest in these topics and to tackle popular operations and culture topics outside of the general DevOps umbrella, where I’m more accustom to seeing these discussions happen.

I was really happy with how this event turned out, and it was a pleasure to be invited on the program committee for it. The OpenStack Foundation succeeded in pulling of a flawless event that pulled in a lot of the right people far beyond the crowd who’d normally come to anything OpenStack related. It was really nice to see that several people had come in just for this event itself, and that made it so important voices some of us hadn’t heard before were able to add an amount of diversity to the conversations. Plus, there were cupcakes.

More photos from the OpenStack Summit and OpenDev here: https://www.flickr.com/photos/pleia2/albums/72157691445454010

On Thursday OpenDev was behind me, and I spent the afternoon in a few final sessions at the summit itself and catching up with people before taking the train to the airport. But the morning was reserved for something a lot more fun than a conference, I had booked a seaplane tour! When I visited Vancouver last time I watched longingly as the seaplanes took off and landed all week, but I decided to go to amazing Vancouver Aquarium instead on my free day. I wouldn’t miss the seaplanes this time! So I Thursday morning I dragged (ok, it didn’t take much convincing) my friend Steve along with me down to the port, which was conveniently right next to the conference venue.

We did the “Extended Panorama” (Adventure Tour) from Harbour Air, where our charming and amusing pilot took us up for about 45 minutes through the nearby mountains for much of it. The tour kicked off with a reminder from the pilot that seaplanes don’t have brakes (the water does that) and limited steering ability. Hah! Once airborne, we flew around the harbor and then into the snow-capped mountains. The tour offered gorgeous glimpses of mountain lakes and other large waterways and concluded by flying over the city of Vancouver before splashing back down into the harbor. At $200/person it’s not something I could do regularly, but I probably wouldn’t say turn down the opportunity to do it again when I’m in Vancouver again at the end of August for the Open Source Summit.

Lots more photos from the seaplane ride and some of the yummy food I enjoyed on my trip here: https://www.flickr.com/photos/pleia2/albums/72157697222323075

My last public talk was in November, and I skipped KubeCon + CloudNativeCon in Austin to spend some time at home after an incredibly busy month on the road. Instead, I spent the early part of this month visiting a couple companies to give talks to their internal events. This isn’t a regular thing for me since my role is designed to be very public, but it does give me the opportunity to connect with users and community members who are less likely to be attending the bigger conferences. The insight gleaned from these internal conversations can help my team build up engagement goals and strategies for the upcoming year. In both these cases, I was invited by professional-colleagues-turned-friends who I met through OpenStack. It’s always a pleasure to spend time in proximity to these women and their extraordinary work.

These talks also unexpectedly brought me to beautiful offices with breathtaking views. The first was in San Francisco in an office at One California Street.

It’s not the tallest skyscraper in the city, but when a building is near the bay it doesn’t take much to claim some some nice views of downtown and glimpses of the bay.

The talk itself was focused on maintenance of containerized environments (logging, metrics, upgrades) but the most valuable part of my visit was actually the Q&A when this group of experienced Mesos administrators grilled me on features and upcoming plans. The company works in the space of artificial intelligence, so the support for GPUs that came out last year of particular interest.

The next skyscraper talk was a week later in Philadelphia. Comcast invited me to speak at their internal open source event, which lined up very closely with my existing plans to spend the end of the year back east, I just had to fly in a couple days earlier than planned. Weather on the east coast in December is much different than San Francisco. Grey skies replaced blue, and a winter coat and mittens became required as I made my way downtown via regional rail on Friday morning.

Comcast Center is the biggest building in Philadelphia, and I’d been in it once before when attending an after-party for a conference with MJ a few years ago. This was the first time I got to see out the 45th story windows during the day.

And then it snowed all afternoon!

Warm inside, it was actually quite a pleasure to see the snow come down and watch city almost disappear below us.

The talk itself was a variation on my open sourcing of infrastructure deck. The open source infrastructure message of this talk is a solid one, but perhaps my favorite part about giving this talk is that afterwards it helps me discover all the geeks of my generation who recognize the infrastructure path of proprietary to open source to cloud that I describe and reflect upon. Even better, giving this talk in Philadelphia again means that all my own Philadelphia connections early in my career mean that much more to an audience that has a lot of locals in it.

Thanks to Shilla Saebi‏ for taking a picture during my talk! (source)

It was a pleasure coming out to the event and spending time with my fellow speakers and Comcast employees throughout. And this was my last talk of any type this year!

It’s an open source conference, and so much, more as their website explained:

CubaConf, is an international conference about free software and open technologies that will have its second edition in La Habana from November 7th to 9th, 2017. CubaConf is a meeting from the international community of enthusiasts of the free technologie’s world, peer to peer focused and collective development. It seeks the participation of people with different profiles and diverse geographical locations.

From the organization of the conference, it is aimed at the diversity of both topics and projects, as of its participants, so the participation of women, minority groups, in these activities is crucial to build new inclusive spaces of knowledge and dialogue about technology. The main idea is to know people experience and organizations who work with and in free software, with old hardware technologies and little or slow bandwidth, to talk about how free software can help developing countries.

Conference Registration

I learned about the conference from and then tagged along with the OpenNMS crew, Tarus Balog and Alejandro Galue, which I was grateful for. Last week Tarus wrote a great blog post about the conference here: 2017 Cubaconf. It was nice having folks to walk with as we wandered through the streets of Old Havana to find the venue on Tuesday morning. The format of the conference was a day of planned conference talks, one day of unconference and then a workshop day.

I joined them to give my Open Sourcing of Infrastructure talk, but this is one conference where I’m certain I learned more from the attendees and environment they work in than they ever did from me. The day began with a keynote from Ismael Olea who came to the conference from Spain where he participates in HackLab Almería. His talk was in Spanish, and he spoke quickly, but thanks to verbose slides (available here) I was able to make out the gist of his talk. Almería is not a major global city, but they’ve worked to build a thriving community through events and online built around giving people the autonomy to self-organize and do whatever they want, as long as it aligns with their ethos of technological, social and creative experimentation. The broad, generalist approach echoed much of what I heard from Josh Simmons at Linux.conf.au earlier this year: when you live in a less densely populated area, multidisciplinary groups are key to success. The added element of self-organization definitely appealed to the Cuban audience, which was a constant theme throughout the conference.

The first talk after the keynote I attended was by Valessio Brito who presented a career path on how you can make money doing open source software. He outlined opportunities for working in support, consulting, customization and development, and stressed the importance of building an open source portfolio. Even once you’re hired, he expressed the value to your career of staying involved with the open source community instead of being overtaken by the tasks internal to the company where you work. Of course as a developer advocate I’m thrilled when our internal engineers express an interest in being a part of our community, so it was nice to hear from a third party too.

My talk on The Open Sourcing of Infrastructure was next. This is a talk I’ve given at a few different places now, and I think I got the most interesting response yet from this Cuban audience. The talk stresses the importance of controlling your data and resources instead of trusting a third party company to do it, and this really resonated. For the Cubans I met, the ethos of free software (not just open source) was incredibly important. I learned quickly while there in Havana that they have a very resourceful, DYI culture that values the ability to have control of your resources. While reading up on this phenomena upon returning home, I found this article from PBS News Hour in 2015, How communism turned Cuba into an island of hackers and DIY engineers. Had I been more aware going in, I could have shifted my talk away from convincing them to use open source infrastructure tooling, they were already convinced! If I were to do it over, I think I would have focused on how they could do it, instead of the specific technologies (OpenStack, DC/OS, Kubernetes, etc) being a single slide of recommendations.

Unfortunately my talk slot overlapped with my friends from OpenNMS! But I’m sure they did a fabulous job.

At lunchtime we walked over to the nearby Casa de África where they had outdoor event space for us to enjoy sandwiches. It was there that I met an engineer who was deploying DC/OS for a bank in Mexico and several folks from the US and Europe whose attendance at the conference was deeply tied to the spirit and message that free software brought, not just the open source aspect of it. It was interesting to learn their perspectives and what brought them there to Cuba.

After lunch I went to a talk by Christian Weilbach on “Free data and the infrastructure of the commons.” Companies control vast amounts of data today, most obviously by companies like Google and Facebook, but generally by most companies who have a technical presence and customers. He explained the risks here, and his plea was to work to adopt open source methodologies to the free access of data. He didn’t have the answers, but was keen on seeing us get there because of how important data is in today’s world. Indeed, a quote I took from the talk centered around the future money being in data, not in the technology stacks which are being commoditized.

The final talk of the day I attended was by Molly de Blanc on “Freedom Embedded: Certifying fully free hardware.” We’ve met at a few conference, so it was a wonderful surprise to see her at this conference and get the opportunity to catch up. Her talk began with some background of the Free Software Foundation (FSF), where she works. I find their perspective on free software to be on the extreme side, but her talk was about their “Respects Your Freedom” hardware product certification, which is actually really cool. They’re seeking to document hardware that runs with 100% on free software, documenting their criteria for certification here. She also mentioned h-node.org during her talk, which I wasn’t familiar with.

Then we had the conference group photo! Not everyone made it in the picture, but it was a decent chunk of the conference attendees.

CubaConf 2017 Group Photo! Credit: Gabriela Fernández (source)

That evening we made our way to the evening social held at a gallery in Old Havana. They served light appetizers and offered an open bar, where I got to enjoy some rum on the rocks. Rum isn’t usually my drink, but I was in Cuba! That event was where I was able to speak with a developer from cuban.engineer where I learned a bit about how they do software development in Cuba. At his company they rely heavily upon their local LAN instead of having internet access in the office, and collaborate on a local GitLab instance. Collaboration and syncing upstream and online is done, but it’s simply not a part of their constant daily activity. This is in sharp contrast with every software project I’ve ever worked on, being a remote employee for over a decade and involved with online software communities since 2001, internet access is essential to my ability to collaborate. It’s a very different environment than I’m used to, and it was fascinating to learn how they make it work.

Snacks and socializing at the first conference night social event

Wednesday began with a keynote by Dr. Mixael S. Laufer, who I had the pleasure of meeting the day before. The organization he belongs to seeks to equalize access to healthcare, and drawing from similar tenets of open source and Maker movements, encourages a DYI approach to healing. Again, giving the culture in Cuba this was a fantastic message and I’m sure one they already work towards, but it was a fascinating topic for me to learn more about.

The rest of the day was reserved for the unconference, which began by people writing down and pitching their ideas, and attendees voting on sessions which then landed on a schedule put together there on site.

The first discussion I ended up in for the unconference was from Zak Rogoff, who told us a tale of how the Free Software Foundation tried, and failed, to fight back against DRM entering web standards. He recounted the grass-roots campaigns they attempted. I have to admit that I continue to be put off by the campaigns that the FSF launches because I do find them to be on the extreme or childish side. Still, with this campaign they had a very firm foundation in something I believed in, and it’s a shame they failed to get the attention of me and others like me. One of the great things they did do was partner with the EFF and Cory Doctorow which led to An open letter to the W3C Director, CEO, team and membership. I believe they would have had more success if more of these strategic partnerships with organizations I hold in high regard had been made, along ones which could have spoken more directly to the serious challenges DRM presents to security and accessibility. His conclusion was the same. I hope the FSF takes this lesson he presented to heart, for the most part they are fighting the good fight, I’d just like to see a better approach.

Lunch was next! I was able to spend more time chatting with attendees, but unfortunately by the time we were wrapping up in the mid afternoon I wasn’t feeling my best. I had to depart and head back to the AirBnB for the rest of the day, with the exception of a stop at the pharmacy and a futile attempt to eat some chicken soup for dinner. I’m even more disappointed to report that being ill took me completely out of the game on Thursday. That means I missed the women group photo! But it was nice to see a decent representation of women at the conference, I wish I had made time to meet with more of them.

Huge thanks to the organizers of this event who made me feel so welcome. In spite of not being able to attend all of them, I appreciated the existence of social events each night and had a nice time chatting with people throughout the event.

The Mesosphere office itself is on the top floor of an office building, at the 7th floor that makes it a reasonably tall building for the city so there are some nice views of the surrounding area.

But Meetups! The first Meetup of the week was over in Berlin. The topic of both the Meetups was “MesosCon Recap” which consisted of me presenting a short slide deck (slides for Berlin) that gave an overview of what MesosCon was, some of the key themes, and what came of events like the new Town Hall on DC/OS. After my introduction, for each location we invited speakers who were local to that region of Germany so we had a different line-up for each evening. Each speaker was given a half hour slot to give a shortened version of a MesosCon EU talk. It was a new format for me, but it was a lot of fun, and I enjoyed bringing a piece of MesosCon to folks who couldn’t make it out to Prague.

The speaker lineup in Berlin consisted of:

• Till Rohrmann of Data Artisans on “Apache Flink Meets Apache Mesos and DC/OS” (slides)
• Kevin Klues of Mesosphere on “Running Distributed TensorFlow on DC/OS” (slides)
• Tim Nolet from magnetic.io on “How to extend Marathon-LB with Canary releasing features using Vamp” (slides)

Huge thanks to all of them for coming out, both Kevin and Tim gave great talks at MesosCon EU, and it was great to have Till join us to present using a slide deck that was not his own, but as a leader in the subject matter he did a fabulous job.

The staff at betahaus, the venue, were also incredibly helpful. They got us all sorted with the projector, pizza delivery and with drinks throughout the evening.

Thursday night I was joined by my colleague Matt Jarvis for the Meetup in Hamburg! At this one Matt and I ran through the introduction slide deck and made some adjustments (slides for Hamburg). This introduction was longer than the one I did in Berlin, partially because the speaker lineup was shorter, and because I had the help of Matt who took time to flesh out some of the impressive statistics about cluster and workload sizes from the keynotes, and he could cover the tracks he was track lead on, so it wasn’t just my perspective from the operations tracks.

Following our introduction, we were joined by our pair of MesosCon EU speakers from Mesosphere for slightly shortened renditions of their talks:

• Johannes Unterstein on “Marathon and Jobs – Today and Tomorrow” (slides)
• Adam Bordelon on “Mesos Security Exposed!” (slides)

Logistics were a bit easier since this Meetup was held at the Mesosphere office. They tell me that we don’t normally have a skeleton joining us in the large meeting space, but it was just after Halloween so the decorations were lingering. It was a fun night and the crowd was a bit bigger than the one we had in Berlin, but I admit that I skipped having Meetup pizza for a second night in a row and instead joined Matt for a later dinner at the restaurant in our hotel.

The next morning arrived quickly as I was up and available for a 7:30AM breakfast with some of the MesosCon Europe keynote speakers. Part of the breakfast was spent chatting with the panelists who would join me on stage that morning to participate in a panel on “SMACK in the Enterprise” that I was moderating. At 8:30 it was time to go down to the keynote room and see that everything was on schedule for our 9AM start. The folks at the Linux Foundation do a great job coordinating these events, it was a pleasure working with them as a speaker and track lead throughout the event.

At 9AM Ben Hindman opened the conference with a talk on the current state of Apache Mesos, reflecting on past MesosCons and the past year of developments. Improvements in the platform have included things like nesting of containers, and the creation of the Container Storage Interface (CSI). Fault domains and the promise of multi-cloud also made an appearance in his keynote.

Directly following Ben’s talk was one that, as a pure open source enthusiast, made me really happy to attend. Rich Bowen of the Apache Software Foundation who came to talk to us about The Apache Way. Given my background and current role, I’m familiar with the history of the foundation and loosely keep tabs on their current work. Still, seeing a history presented with a message is always an enjoyable way to consume it, and Rich did a masterful job weaving the history in with where we are today, and explaining what “Apache” means in the Apache Mesos project. From bottom-up leadership to collaborative decision-making and they way they approach conflict-resolution, there’s a lot to admire. He also drove home the importance of transparency in a project, with no decisions being made privately or in ways that are difficult to document for future participants of the project. He also touched upon where the project was today, with over 100 projects within the Apache Software Foundation and the Apache License remaining strong in the industry.

The SMACK (Spark, Mesos, Akka, Cassandra, Kafka) in the Enterprise panel was next, and as moderator I was thrilled to be joined by representatives from Audi, Deutsche Telecom and ASML. From connected cars to how they’re making innovations on mobile networks, learning how companies are using Mesos and the entire stack to do fast data processing really brings all the work we do into focus. Innovation is happening across various industries as we all become more familiar with what we’ll need to succeed in a world with so much data coming at us, and I’m proud to be a part of it.

Thanks to the Apache Community for taking this photo (source)

The keynotes concluded with one from Netflix Engineering Director, Katharina Probst. Internally at Netflix they’ve built a massive infrastructure that is, at a high level, managed day to day by a relatively small team of Site Reliability Engineers. This has been accomplished through the tooling they built called Mantis, that allows not only for the essential autoscaling that a company like Netflix requires, with viewing numbers dropping considerably when each region has their work day, and picking up in the evening hours, but also real time testing and metrics from their platform. The key for them has been not only monitoring to detect that there is a problem somewhere, but finding out exactly where it is and reporting to the engineers what the problem may be, so a fix can be developed without a long session of troubleshooting first. As someone who used to work in operations, this was something I could really appreciate, I’ve spent many long evenings chasing down problems, evenings that could have been made much shorter with the addition of some automation to rule out the usual suspects quickly. The scale of their operations also never ceases to amaze me, and she was able to share some statistics around that as well.

This first day of the conference landed me as the track lead for the DevOps and Ops track. The day began with a talk from David vonThenen around external storage. In this talk he gave a cost/benefit evaluation of local versus external storage, and then, not keeping to just files, he ran the same evaluation when looking at storage for databases, both traditional like Postgres and MySQL and the newer distributed databases. Tying this in to Mesos, he touched upon CSI work and mentioned that REX-Ray is currently being used by DC/OS to handle the attachment of external volumes. These were interesting considerations in what has been a quickly growing part of our ecosystem as demands upon reliable storage solutions for containers quickly increase. His talk was followed by one from Ádám Sándor titled “Knee Deep in Microservices” and which had a Doom theme to demonstrate the new “demons” we have let loose as we’ve also taken all the benefits of migrating to microservices. He cited the key elements of DevOps, resilience, elasticity, resource abstraction, and tooling that helps us monitor containers as key improvements in the microservices platforms that are helping us tame the demons we have unleashed. It was also nice for me to hear these things, these massive distributed systems are complicated, we need to be continually improving our toolkit (weapons!) to effectively manage them.

After lunch Adam Bordelon and Alexander Rojas joined the track to give a popular talk titled “Mesos Security Exposed!” I won’t enumerate them all here, but by digging into how Mesos works, they were able to pull back the curtain and explain what needed securing in your cluster, from API endpoints to use of TLS and the handling of private data (secrets). Julien Stroheker then spoke on “Doing Real DevOps with DC/OS” where he gave a live demonstration of a continuous integration pipeline with Jenkins, Docker, Gradle and Vamp. The sessions for the day concluded with Zain Malik giving us a tour of mesos2iam, “IAM Credentials for Containers Running Inside a Mesos Cluster”.

But the day was not over! Appetizers and drinks at the attendee reception were directly followed by Town Halls, where attendees could casually gather in groups to talk about Mesos, Marathon, DC/OS or Kubernetes. My colleague Matt led the DC/OS session, with me taking notes and pitching in here and there. Our Town Hall began with introductions, with 15-20 people in attendence as the 90 minute evening session progressed we had a nice mix of people from a variety of unrelated industries. With the ice broken, it was easy to get people talking about pain points they had with DC/OS and we had Adam Bordelon in the room to give history and insight into specific features and concerns that people were bringing up. At the end of the session we had a nice list of shared struggles, but the attendees also were able to swap knowledge with each other. Empathy goes a long way, and in my own work I know how valuable it is to know that concerns you have are being shared and solved by others too.

Attendees file in for the DC/OS town hall Thursday evening

The Town Halls wrapped up at 8:30 and I headed straight to bed. Between the jet lag and the 13+ hour day, I was exhausted.

On Friday the keynotes also began at 9AM, but the keynote slot was shorter in order to squeeze more track sessions in during the rest of the day.

After an welcome and opening remarks from Jörg Schad, the first keynote came from Yrieix Garnier who gave a more Enterprise-focused take on “The Future of Apache Mesos and DC/OS” than Ben had explored in his keynote the day before. He was able to pull up statistics about the data-processing power of the platforms, sharing that 50% of DC/OS clusters were running some form of the SMACK or ELK stacks. The big news from his talk was unveiling of TensorFlow support in DC/OS. We then had Pierre Cheynier join us to share a talk that he had originally proposed during our CFP, but we upgraded to a keynote, “Operating 600+ Mesos Servers on 7 Datacenters @Criteo”. The ability to scale is a key feature of Mesos, so it was fascinating to learn about the scale of their work and just how much data they were storing (171PB on Hadoop!). He also shared a series of tips for other organizations looking to operate at this scale, including effectively automating everything (configuration management, scaling, CI system), defensive configuration (things will go wrong, be prepared), visibility to operations as to what is going on (metrics, alerts, tracing), and the importance of doing networking right, and addressing problems like QoS and “noisy neighbors” during design. He also covered some incidents which gave further insight into the sorts of things they were able to effectively prevent, or not.

In a slight shift from Thursday, I spent Friday as the track lead for the Users and Ops track. The first talk came from the oldest company in the lineup, the publisher Houghton Mifflin Harcourt. I was looking forward to this talk, and as Robert Allen got into the details of “How HMH Went from Months to Minutes for Infrastructure Delivery” I was not disappointed. In a common theme for the day, he brought up the slow, inconsistent, old technology team was not keeping up with either the industry or their own product lines, as the publishing industry is rapidly changing to serve a more tech savvy customer base. He walked us through the creation of their “Bedrock tech services” team, and their DevOps focused goals, including comprehensive developer involvement from idea to production, a continuous delivery approach that encouraged small, frequent updates, and a change in culture that made them shift from feeling like they must prevent failure, and instead acting as if failure will occur and planning accordingly. He then dove into the technologies used, beyond Apache Mesos, they’ve also been using Apache Aurora, Terraform for orchestration, Vault for secrets, and a Jenkins plus Artifactory CI/CD pipeline. He also stressed the importance of metrics and logging, all things close to my own interests as well!

Tim Nolet then joined us to give a talk on “Advanced Deployment Strategies and Workflows for Containerized Apps on DC/OS” where he too walked us down the painful memory lane of the massive, error-prone deployments of the 2000s, the rise of DevOps and today the productization of a lot of the DevOps tooling, including Vamp, which he works on the development of. As a product, Vamp seeks to simplify and tame the ecosystem of microservices you have running by simplifying the process of deployments. He then showed a demo of load being distributed across different versions of a deployed application, as well as different versions being served up to customers using different clients.

In our next user story, we heard from Jay Chin on “Optimising Mesos Utilization at Opentable”. He began his talk with a quick production history of the infrastructure at OpenTable, sharing that they made the move to microservices in 2013. During this time they were orchestrating their microservices via standard configuration management tooling, a process that turned out to not only difficult to maintain at scale, but was actively disliked by the engineers who had to work on it. In 2014 they switched to Mesos, and through resource abstraction and running a consolidated cluster, they were able to simplify application-level operations. Additionally, it allowed them to easily create centralized metrics and logging. It’s a story I’d heard before from other companies, and one I was thrilled to hear again, but where OpenTable really led the way here was by open sourcing some interesting tools they were using, including the following mentioned during his talk:

• Mesos Stats
• sous
• Request Timeline

The day continued with Ilya Dmitrichenko on “Time Traveling in the Universe of Microservices and Orchestration” where he used his own career as a baseline for the changes he was seeing in the industry with regard to the rise of microservices. His dabbling with old Sun boxes and awareness of things like openvz is consistent with some of my own hobbiest and day job work early in my career. Indeed, containers have always been with us. He went on to track the rise of Docker simplifying the space and tooling coming together to make management of a microservices infrastructure feasible for smaller organizations. The culmination of this story was his current work at Weaveworks which, just like Vamp improves deployments, improves and simplifies your network policy.

Jorge Salamero of Sysdig joined us next with the amusingly titled talk, “WTF, My Container Just Spawned a Shell”. I was immediately fond of this talk because after a sea of Macs, Jorge uses Xubuntu. He began his presentation by talking about how we consume container images, and the rise in static scanning of these images for libraries with vulnerabilities and more, but that this only goes so far. There are things that these scans miss because they need constant updating, and which behavioral analysis while the container is running will catch. He introduced how Sysdig’s metrics tooling used with Falco can give you a comprehensive view inside the containers you’re running. Suddenly you have access to security tracking that can show you every command taking place inside of your containers, and from there you can train it to be aware of problem behavior. He also talked about SysDig Inspect, the open source project that they have built their Sysdig Secure product with.

Julien Stroheker then joined us again to talk about his DC/OS autoscaler. This is a talk I’d already seen twice, so I won’t go into it, but it is a cool project that he’s always looking for help on! The final talk of the day came to us from Fredrik Lindner of Tunstall Nordic AB, who shared “There and Back Again: How Tunstall Healthcare Built an IoT Platform for Health Monitoring Using Mesos Cluster on Azure.” Just like so many other industries, elder care is undergoing a transformation with the help of IoT technology. He shared details of their “Evity” platform they were developing on top of DC/OS to help manage the data coming in from these devices so they can effectively and quickly meet the needs of the families they work with. It was a good story to end with, as with much of the technology we saw showcased throughout this MesosCon, as consumers we just assume it will all work. My FitBit will reliably track my steps and share them with my friends, my car will show me traffic, and when something goes wrong it’s unexpected and unsettling. This is even more pronounced with you get into the space of technologies that help with things like healthcare, where reliability, accuracy and speed have even more urgency. We’re building the platform that people are using to make sure these all run well, and that’s pretty exciting.

When I finally stepped out of that track, all the booths had been taken down and most of the conference attendees had gone their own ways. I was able to finally leave the hotel with Matt and Jörg to explore some of the old city in what turned into a lovely night that thankfully didn’t keep me out too late. The conference was great, but predictably exhausting, especially coming on the heels of All Things Open on another continent.

Huge thanks to everyone involved in organizing, and to all the speakers in my tracks who made the event really interesting by sharing their stories, tools and expertise. This event was a little smaller than the one in Los Angeles, but it didn’t feel like it, and the quality of the event was top notch.

More of my photos from the event can be found here: https://www.flickr.com/photos/pleia2/albums/72157689885991786 and more photos, slides and videos are hosted by the Linuxd Foundation at http://events.linuxfoundation.org/events/mesoscon-europe

The event began on Wednesday with a Mesos community Hackathon and a customer-focused DC/OS Day run by Mesosphere. Given my role and background, I joined the Hackathon where my colleague Jörg Schad kicked off the event by encouraging attendees to collect ideas for projects they wanted to tackle throughout the day. Topics ranged from documentation improvements, creation of a Kubernetes-related demo for SMACK, work on frameworks and further work on an autoscaler for several cloud platforms where DC/OS runs. Plus, there were mini cupcakes!

The main event for MesosCon began Thursday morning with an introductory keynote from Ben Hindman, Co-Creator of Apache Mesos (and my boss), where he covered some of the latest features in Mesos over the past year that he’s most proud of. These included the introduction of nested container support, adherence to standards through the Open Container Initiative project and beyond, and general expansion in usage and community. Mesosphere Co-Founder Tobi Knaup also got on stage, joined by Tim Hockin of Google for a demonstration of the new beta availability of Kubernetes on DC/OS. One of the really distinctive things about this implementation is that it’s running pure upstream Kubernetes, among other things, this allows for migration of a pure Kubernetes environment (on Mesos or not) as well as known functionality with the tooling written today for pure upstream Kubernetes.

Ben Hindman sharing new and noteworthy improvements to Mesos

These talks were broken up by a pair of panels, the first talking about Mesos in the Enterprise with Michael Aguiling at JPMC, Larry Rau (Verizon), Cathy Daw (Mesosphere), Stefan Bauer and Hubert Fisher (Audi), and Josh Bernstein from {code} as a moderator. At the enterprise level, a lot of interest was around distribution of resources across cloud vendors and on-prem, the introduction of open source software in companies unaccustomed to it, general scaling needs required by large companies and a standards-driven future so they can continue to integrate container technologies into their infrastructure.

The final keynote panel of the morning had Ben Hindman sitting down with Neha Narkhede (co-founder and CTO of Confluent) and Jonathan Ellis (co-founder and CTO of Datastax) for a fascinating discussion about the role of Apache Cassandra and Apache Kafka in the fast-data driven world powered increasingly by the SMACK (Spark, Mesos, Akka, Cassandra and Kafka) stack. The main message in the panel was how fast data is essential in the interaction with customers today, and their Internet of Things gadgets that create an expectation of immediate response. This panel was also where I heard a theme that would be repeated as the conference progressed: people move to microservices incrementally. There are a few ways of moving an environment which is microservices driven, but whether they convert old systems slowly or adopt a policy that all new projects must be built with them, it’s incredibly common across the industry. As a developer advocate these days, it was also nice to hear from Neha that the open source nature of the components are empowering developers, which makes my role in getting out there to talk to developers and operations folks who are on the ground with the technology particularly important.

The keynotes concluded and we dispersed into three tracks, on SMACK, Ops, and DevOps, and a new addition to the conference called MesosCon University where attendees could attend an 80 minute hands-on session on making applications production-ready, securing Mesos clusters, or building stateful services.

I was the track lead for the DevOps track along with Julien Stroheker from Microsoft. Leading up to the conference Julien and I reached out to the speakers to get slide decks and answer any questions, then the two of us met up to chat about the track in San Francisco the week before. The day of the track everything went smoothly, with Mark Galpin of JFrog getting us started by talking about the development and deployment of three key projects: Artifactory, Bintray and Xray. They learned quickly that the application and configuration layers had to be configured separately and that it was best to plan for enterprise and high-availability early on, rather than trying to build it in later. He also discussed importance of good startup scripts and package management in a microservices world, since you don’t simply use an RPM anymore to install things… but you do still need RPMs because not every customer will want to adopt a microservices environment just yet! Standardization was also important, customers wanted the ability to hook into their existing storage and network backends, and often had unusual requirements (for example, “my nodes have no outbound internet access”). He concluded by stressing that everyone in the DevOps organization should be involved from the beginning when working to develop the products and how they’re being deployed so that they consider every aspect of how the tooling will be used and deployed.

The next talk was by Aaron Baer from Athena Health who brought us deep into the world of healthcare information services where monoliths rule and shared his experiences working to bring new technologies build on microservices into the fold. Like I heard earlier in they keynote panel, he’s been working on a path to incrementally move from monolithic and relatively inflexible services over to microservices. In addition to building new infrastructure where they can begin using containers, they’ve also been using their legacy database as a canonical resource and storage, and doing data processing and delivery through faster, more modern data storage and processing tools. He’s also a proponent of code as infrastructure (me too!) so it was great to hear him talk about how essential API-driven infrastructure engineering and automation is to getting our infrastructures to the next level.

Aaron Baer on the progression of monolith to microservice

After lunch the track continued with Chris Mays and Micah Noland from HERE Technologies sharing details about the Deployment API they built for DC/OS. They wanted a simplified version of the core DC/OS API which was deployment-centric and needed an open source option that would improve upon Marathon-LB. The API they’ve developed has just recently been open sourced and can be found here: https://github.com/heremaps/deployment-api. During the talk Micah provided an in depth tour of how straight-forward their YAML configurations were for a sample deployment of a Jenkins pipeline. The API theme continued in the next talk by Marco Palladino of Mashape, the makers of Kong. Kong is an open source gateway for APIs which allows you more control over access, including security, authentication and rate limiting. He stressed how important this whole ecosystem of support is for microservices, and his talk also touched upon the steady migration needed by most organizations from their old stack to the new, which also meant keeping a lot of pieces like their legacy authentication system intact but that adding too many layers can leave to unforgivable latency.

Lively booth area break before the final two sessions of the day

The day concluded with talks from Will Gorman of Cerner on Spinnaker and Imran Shaikh of YP on hybrid clouds. I enjoyed the entire DevOps track, but these two talks really went to the heart of what I’ve focused on infrastructure-wise in the past five years. Give my CI/CD background, I’ve known about Spinnaker for some time, but never made diving into it a priority. This talk inspired me. As you may have guessed, Spinnaker is a continuous delivery platform and Will began his talk by covering the basics about it before getting into integration with DC/OS, including the ability to hook into Metronome for jobs. He also touched upon controls Spinnaker has for the deployment phase to complement Marathon’s own attempts to deploy and restart processes. I’ll definitely be digging into it on my own very soon. Imran’s talk on the importance of being responsible for your own infrastructure was very satisfying for me. He led his talk by talking about how moving to the cloud doesn’t absolve you of the responsibility of making sure you have a good disaster recovery plan, reminding the audience that your business relies upon a good failover plan, and your cloud provider has different goals. Throughout his talk his talking points revolved around vendor independence, distributing risk and the importance of your processes being portable to other solutions.

Will Gorman showing how to execute a Metronome job as a step in a Spinnaker pipeline

Thursday evening had another MesosCon first, Town Hall sessions. A big hit with attendees, these were casual evening gatherings on various topics (Mesos, DC/OS and Marathon/Chronos). They allowed community members to gather and talk about whatever they wanted, swap war stories, share tips, anything! Sadly I had to miss this, participation in the Open Source Summit earlier in the week and with a panel to prep for, I was running low on energy. I’m looking forward to attending the ones we host at MesosCon EU coming up at the end of October.

Friday morning came quickly, and I’ll let you in on a little secret: I had breakfast with my Future of Cluster Management panelists. The panel was made up of Sharma Podila (Netflix), Sam Eaton (Yelp), Ian Downes (Twitter) and Zhitao Li (Uber). As the panel moderator I had worked with them on questions and spoke with them via video conference, I hadn’t met any of them in person. A casual, leisurely hotel breakfast was a great way to break the ice and answer any lingering questions folks had, including myself (“none of you are going to surprise me with the answer you give about the importance of open source, right?”). I highly recommend such a gathering for other moderators if it’s possible, I’ve been a panelist a few times and on stage ice-breaking is not always the best way to go.

Jörg was the MC for day two of keynotes, and after a community-focused introduction to this second day (the first day was more enterprise-focused), he brought us on stage. Talking about the future is always fun, and this panel was no exception. After introductions, we covered the importance of open source in our ecosystem, something that is near and dear to my heart. Theme-wise we talked about the move from perfecting deployments on containerized systems to a focus on maintenance, and the tools they’ve had to built outside of what Mesos provides to manage their clusters. The question of how they handle mixed workloads (stateless vs. stateful, batch vs. long-lived) and what improvements could be made here. The conclusion centered around the future of artificial intelligence of clusters, which led us to learn that some of them are already using machine learning to teach their clusters about load and behavior, which was later explored in a track talk from folks at Twitter who are doing automated performance tuning using Bayesian optimization. The last question I had for the panelists was what they wanted to see in 5-10 years, and their answer was identical: they don’t want to care about the underlying tech, they just want their workloads to run.

Keynote panel: Future of Cluster Management, thanks to Julien Stroheker for the photo (source)

The morning keynotes continued with a talk from folks at the National Geospatial-Intelligence Agency that spoke to how they’re using Mesos at scale inside of their organization. Keynotes concluded with one from Ross Gardler from Microsoft who explored the question of where we were in the hype/actual production cycle for microservices and containers. He expressed that there’s still a lot of diversity in the field, and that with so many early deployments there’s no clear “winner” in the space of containerization. Today, he shares, the focus should be sharing best practices and methodologies across platforms so that all of the next iterations of our various containerization platforms can be that much better.

Just like Thursday, after keynotes we shuffled off into various tracks, with Friday featuring tracks on Mesos Internals, Ops, SMACK and the one I took over as track lead for, Mesos Frameworks. My focus in my role at Mesosphere had largely been on higher level operations, so it was a very interesting track for me to sit in on for the day. There’s a lot of really interesting work being done in the Mesos ecosytem that I’d love for my team to be a part of drawing more attention to. This day was great for that.

The first talk was on the juggling that happens around optimizations, service guarantees and how those trade-offs manifest by Sharma Podila of Netflix, who I’d just met earlier in the keynote panel. He began by asking some questions about your cluster and explaining the heterogeneous nature of the hardware they have at Netflix, you never know which generation hardware your workload is going to land on. He then dove into the challenge of scaling down, showing off the open source Fenzo, “a scheduler Java library for Apache Mesos frameworks that supports plugins for scheduling optimizations and facilitates cluster autoscaling.” His talk was followed by one from Wil Yegelwel for TwoSigma who joined us to talk about simulation testing. In spite of my deep interest in CI/CD, simulation testing is something that’s largely escaped my radar. Thankfully he gave an introduction to the concept and then shared how they used the scheduler they built at Two Sigma, Cook, to intelligently handle scheduling of workloads. The intersection of an intelligent scheduler and a policy of simulation testing meant they could run tests that changed configurations and see how they could optimize things as much as possible, and he explained that this has sometimes resulting in some surprising, non-intuitive discoveries that boosted performance. He also credited simulation testing with helping engineers become more familiar and comfortable with the infrastructures they’re running, which are growing increasingly complicated and difficult to fully understand.

After lunch we dove right back in with Joshua Cohen and Ramki Ramakrishna from Twitter who came to speak on the aforementioned Bayesian optimization talk. It’s well known that Twitter’s infrastructure is built on a lot of microservices. At that scale, they explained, manual performance tuning of hundreds of JVM options doesn’t scale, is error-prone, time-consuming and honestly leads to goal-driven engineers copying existing configs to get going, without fully understanding why the optimizations exist in specific places. They went on to stress that even if you do manage to get everything running well, it’s effectively undone as soon as you change or upgrade any component. Instead they shipped the problem off to an internal machine-learning driven technique that uses Bayesian inference to optimize performance. Bayesian became popular in my own operations world for spam analysis, but the methodologies as I understand them make a lot of sense here as well. Unfortunately they haven’t open sourced the Bayesian Optimization Auto-Tuning (BOAT) tooling, but knowing that it’s been done and is successful for them is a good start for others looking to do similar machine-learning inspired performance tuning.

Next up was Tomek Janiszewski from Allegro who gave us 8 tips for Marathon Performance, summing them up:

1. Monitor — enable metrics
2. Tune JVM
3. Optimize Zookeeper
4. Update 1.3.13
5. Do not use the event bus
6. Use a custom executor
7. Prefer batching
8. Shard your Marathon

Marathon is a popular project in the Mesos ecosystem, so the room was quite full for this talk and the questions about it went well into the coffee break we had between sessions!

After the coffee break we were in the home stretch, only two more talks to go before MesosCon concluded! First up in this final pair was Dragos Dascalita Haut and Tyson Norris from Adobe Systems who where sharing details about their use of OpenWhisk as a Mesos framework. They quickly covered OpenWhisk, an event-driven serverless platform for deploying code. Their lively presentation was full of demos and they shared their Akka-based mesos-actor project before pulling in the OpenWhisk bits that brought together all the pieces of the work they were doing.

The final talk brought a pair of engineers from Verizon Labs, Tim Hansen and Aaron Wood, to share their work on fault-tolerant frameworks without Docker. They share some of my own opinions about the over-use of Docker where something slimmer would suffice, so I knew going in that I’d enjoy this talk. The talk explained why using Docker and the entire Docker daemon was often overkill for their work, specifically Tim saying that there was “no real need for an extra daemon and client when Mesos can containerize tasks.” So they instead directed the audience toward use of the Mesos containerizer and the CNCF’s Container Network Interface (CNI). To this end they developed Hydrogen, “High performance Mesos framework based on the v1 streaming API” and their own Mesos SDK, a general purpose Golang library for writing mesos frameworks.

As always, as tiring as the week was it was still sad to see it conclude. It was great to meet up with so many people, huge thanks to everyone who take time to let me pick their brain, or simply sat down for a quick chat. I learned a lot while there and had a wonderful time.

Clockwise from top left: With Yang Lei (IBM), Dave Lester (Apple), Mesosphere community team (minus Jörg, plus Ravi!), and Aaron Baer

More photos from MesosCon NA 2017 here: https://www.flickr.com/photos/pleia2/albums/72157689480391705

First on my noteworthy agenda was speaking on The SMACK stack for the Dublin Apache Kafka Meetup by Confluent. It was held at the ZenDesk office just south of St Stephen’s Green. This meant I could enjoy a lovely mile long walk south from where I was staying near Trinity College to get to my destination. When I arrived, my gracious host for the evening was Andrei Balcanasu, an Infrastructure Engineer there at Zendesk. As attendees trickled in he made me feel very welcome and we had some great chats about the direction of infrastructure tooling over the past decade.

The presentation was an overview of the SMACK stack (Apache Spark, Apache Mesos, Akka, Apache Cassandra, and Apache Kafka) itself, but I did try to stress the value of Kafka specifically in this model. My experience thus far has been that Mesos and Kafka are the two components that are least likely to be swapped out as people tailor SMACK for their own infrastructure environment and industry. I concluded with Achim Nierbeck’s Iot Fast Data analytics demo which plots buses from the Los Angeles Metro on a map and is pure SMACK stack. Slides from the talk can be found here (PDF).

The evening concluded with beer and pizza, during which I indulged in my first Bulmers Cider of the trip as I spoke with attendees about their infrastructures and where they were planning on going tooling-wise. While I’d never call myself a social butterfly, getting able to geek out about open source infrastructure tooling is one of the areas where I can let my guard down and get really into it, so enjoyed the evening. During these conversations something that stood out most for me was a comment about the simplicity of the data source for the demo, as the stream coming in from the LA Metro is not what any of us today would call “fast” (just a few requests per second) so the use of Kafka is a bit overkill. It was a very fair point and I’ll mention it up the next time I use it in a presentation, but it is just a demonstration of a pipeline, and at that it succeeds at being reliable and effective.

Unbeknownst to me when I booked my travel, I came into town the same week as the Usenix SREcon17 held right there in Dublin. This meant that a whole bunch of people seeing my Dublin tweets assumed I was in town for that, and reached out to me to meet during the conference. Had I known about the overlap, I would have probably participated, but the CFP has closed long before I made plans for this trip and it was sold out attendee-wise before I knew about it. Still, this was how I ended up with the opportunity to meet up with DC/OS community members Levente Lajko, Luis Davim and Matthew Allen from PTC!

PTC has been an active participant in Working Groups we run for DC/OS so I was delighted to meet with them in person as they took time away from the conference venue to meet for lunch. Discussions centered around improvements we could make to the community involvement side of things, including being more public about our road maps so they can plan accordingly with what they build and lack of engagement around some tickets and pull requests. Nothing here was new for me and my team is actively working to make real progress on these points, but hearing it directly from community members strengthened my confidence that we’re working on the right things. Even better, they were a fun group of folks to spend a meal with and they were eager to make clear that the technology is worth the community-related road bumps.

Now to be clear, I don’t always intend on making vacation-centric trips include work stuff, I do need time away too, but I’m glad I did this time. With only 8 months under my belt at Mesosphere I was eager to spend as much time as I could interacting with community members, and in-person meetings like this are an important part of that. Having visited Ireland before also helped, I didn’t feel like I was missing out on anything, so breaking up tourist time with some work ended up being a pretty healthy use of my time.

We had dinner at the nearby Toscana 52 Italian restaurant before settling in for the night. I saw them off the next morning before starting my Friday morning of work. That evening my friend Danita came over to spend the night, a weekend trip to New York City together, which I wrote about here, beckoned!

Monday arrived much too quickly. The day was spent once again working from the townhouse, which is working out really well. My office may be where we are storing piles of boxes, totes and furniture (most of which will end up in California), but in the midst of all that I have a really comfortable setup with everything I need to be productive. I just plug in my work laptop and go. Unfortunately during this trip I learned that the personal laptop that I keep there as my desktop has finally failed. The blink codes indicate that it’s the CPU or motherboard, which essentially means the laptop has reached the end. I’ll have to bring out a replacement next time I come into town.

However, Monday was eventful for another reason, it was August 21st, total solar eclipse day! I had piles of friends who braved the crowds to experience the totality zone, but in Philadelphia we just had a partial eclipse. During much of it we had cloud cover, but after the high point the clouds parted and I was able to get some nice glimpses through my pinhole camera box as I sat on the back porch. While sitting out there I flash backed to being 12 years old after school and doing the same thing. I have always enjoyed a good eclipse.

Monday evening I drove out to PLUG West to give an Introduction to DC/OS presentation (slides). Surprisingly, up until this point most of my DC/OS presentations had been more goal-oriented, so I really enjoyed doing a more holistic presentation about the “datacenter operating system” space, getting able to dive into more of the infrastructure geek bits of how DC/OS works as a platform on top of Linux for my fellow Linux-loving attendees present that evening. I also still know a number of folks who have attended PLUG meetings over the years, so it was nice to catch up and meet some new folks as a number of us adjourned to a nearby restaurant for a late dinner.

My next two evenings gave me time to catch up with friends over dinners, Tuesday night I had dinner with my friend Crissi. It had been a little while since just the two of us caught up in person, so it was really nice to get that time. Thursday I met up with my friend David to see RiffTrax Live: Doctor Who – The Five Doctors. I’d really been looking forward to this, and almost bought tickets for the actual live event in Nashville when they went on sale in the spring. This was actually a replay from the previous week, but as a Fathom Event in a theater, it’s hard to tell the difference. It was totally worth seeing. I don’t remember if I’d actually seen The Five Doctors before, I have seen a lot of old episodes but this movie is a bit ridiculous, even by old Doctor Who standards. A replacement of the first doctor (who had died a decade earlier), a cameo of the fourth doctor taken from footage shot for another production, and plot holes and continuity problems rounded out the basics of what made it so amusing. The Rifftrax of it was hilarious, even if they frequently took the opportunity to make fun of us old school Whovians.

On Friday morning MJ got into town! He flew in to visit for a few days, attend FOSSCON and so we could depart together on our trip to Ireland Monday night after work.

It’s hard to believe I was in Philadelphia for almost two weeks during this trip. With every visit the townhouse feels more like a home, I can relax and really feel comfortable there. Not sure yet when we’ll be back, but the end of year holidays are always a good opportunity since we have some off from work.