• Archives

  • Categories:

  • Other profiles

The Coast Starlight

I’ve wanted to take a journey on Amtrak’s Coast Starlight for years. Alas, timing was always a challenge. It’s so hard to justify spending 11+ hours on a train when a short flight will suffice, but 2017 has been my year of trains. Over Memorial Day MJ and I took the California Zephyr and Capitol Limited to get across the country and this month I’ll be hopping on the Carolinian with a close friend, and fellow train adventurer, from Philadelphia to Raleigh.

We finally decided to take the Coast Starlight down to Los Angeles a couple weeks ago to get me to the Open Source Summit and MesosCon.

Now I’m going to pause for a moment to talk about my feelings about Amtrak. It’s the only long-haul passenger service in the United States, and I love trains. It makes it my de facto home train so I have a special fondness for it. But Amtrak is expensive, often more so than flying, and the cars are extremely dated. The food even in the dining cars isn’t great, and there is no variation in the menu between routes that I’ve seen thus far. They serve a fair amount of pork, which excludes much of the menu for me and making the menu even smaller. I can’t imagine riding on it as a vegetarian unless you packed your own food. The air conditioning leaves much to be desired and even if you get on a train that advertises having WiFi, your best bet is to assume it’s not working. The WiFi situation often isn’t their fault, coverage is simply spotty through some of their service area, which leads me to another thing out of their control: the United States is big and taking a train that typically must travel below 79 MPH is going to be slow.

Still, it’s my train and I enjoy taking trains. The views, the disconnection, the dull roar of the rails. I can relax, get some reading done or chat with my fellow passengers in the lounge car.

MJ and I departed early on a Saturday morning to begin our trip at an Amtrak bus station near the Embarcadero here in San Francisco. Unfortunately our train was delayed, as I hear often happens with this route, so after the bus dropped us off at Jack London Square station we had a couple of hours to kill while we waited for the train. It did finally arrive though!

In spite of not doing an overnight this time, we sprung for the Roomette so we could have have private space to retreat to during our 11 hours on the train. This class of room got us access to the famous Pacific Parlour Car. The status of the six Pacific Parlour style cars is tracked on pacificparlourcar.com, explaining that five of them are in operation by Amtrak on the Coast Starlight on train numbers 11 southbound and 14 northbound. These are pretty special car in the Amtrak fleet, they’re Hi-Level lounge cars by The Budd Company in 1956 for the Santa Fe Railway, not Amtrak. They’re also shorter than the other cars, which you notice a little when walking between the cars but it’s definitely noticeable from the outside of the train. As for the features of the car itself, this page has a photo tour that takes you through the bar area, seating area, dining area and downstairs to the movie theater.


The Parlour Car

In the afternoon the parlour car also hosted a wine tasting, but while it was fun to move around a bit and have a few sips of wine, the wine snob in me was not terribly impressed with the selection. I did manage to have a Stone IPA during dinner though, which somewhat to make up for the lackluster showing from the wines.

The downstairs movie theater was pretty cool, unfortunately they weren’t showing any movies. When I asked about the movie theater one of the staff members said the royalties were too expensive so they currently couldn’t show any. I’ve also read on the forums that they’ve struggled with the technology aspect of it, switching to DVDs when satellite feeds wouldn’t suffice. I couldn’t stop thinking about this for the rest of the trip. What an opportunity for someone interested in developing a Creative Commons program! Old movies! Modern, CC-licensed cartoons! So many great options! Maybe some folks would be disappointed that it wasn’t known blockbusters, but it’s quality content and certainly better than nothing. Develop a reliable SSD-based box to stick in there to play the content on the TV and then you get to avoid DVD and satellite issues. I actually now want to be the crazy tech lady who runs the movies on the Coast Starlight.

All that said, I probably wouldn’t have watched a movie during the trip. Hah! As my first journey on the Coast Starlight I was eager to see the scenery down the whole route.

Down through the bay area we had the predicable salt marshes, a group of hikers here and there. As we went further south we passed through a lot of farm land. Here and there the backdrop of mountains made for some views worth seeing.

I also managed to get a panoramic photo as the train navigated the horseshoe curve at Cuesta Pass. Zoom in on the photo, you’ll see both the front and end of the train! Our car was near the middle of the train, and it takes a few seconds to take the panorama, making the following photo possible.

And then just south of San Luis Obispo we got to the amazing part of the trip. The train heads west and runs along the coast line. We enjoyed dinner as we looked out over the cliffs and beaches.

No joke, RVs, surfers and palm trees ruled down here.

The sun set as the clock ticked past 8PM and the rest of the trip was taken in darkness as we passed through Oxnard and Burbank. It was interesting to learn that the Burbank airport is attached to the Amtrak station, which is a rare thing here in the US, but obviously makes a lot of sense. We got into Union Station in Los Angeles just after 9:30PM on Saturday evening. Onward to our Sunday of adventures in Los Angeles!

Sub-optimal air-conditioning aside, I really enjoyed the trip down and I’m glad we finally took the time to do it. More photos from the trip down here: https://www.flickr.com/photos/pleia2/albums/72157685483861352

Galway and the Cliffs of Moher

Our last day in beautiful Ireland this year was spent on the west coast. MJ had been out there, but it was the first time for me. He’d also driven on the left side of the road before, so when it came to renting a car, he was the driver. That morning we left the hotel early to pick up or lovely BMW rental car and make our way out to Galway!

The drive took about two and a half hours, we found parking and quickly discovered we were in the midst of a city that was extremely excited about the hurling championship game that Galway was playing in. I knew nothing about the game when we walked into The Quays for lunch. I still know very little about it, but we had a lovely lunch with one of MJ’s colleagues and his wife who also happened to be on that side of the country for the weekend. Plus, I got my beloved fresh north Atlantic oysters.

After lunch the sun came out and we walked around town a bit, walking right past Lynch’s Castle (a medieval “town house”) without realizing it as it’s a bank these days. We then walked past a couple of old churches before making our way down to the Spanish Arch and then back to the car. The morning had been rainy in Dublin, but the stunningly blue sky that met us after we saw the arch made me very happy that MJ had checked the weather prior to planning the weekend. It was the perfect day to be on the west side of Ireland. The harbor there in Galway was beautiful.

Thankfully we were able to flee the city before the parties over the hurling game win broke out, and we were on our way to the Cliffs of Moher.

There are two things I’ll say about the roads out there in western Ireland. First, it’s beautiful out there. The countryside is dotted with castles and various ruined towers, livestock grazing in fields that have stone walls (I started calling them “cow castles”), several rivers and lakes, and so much green.

The second thing is that the roads are terrifyingly narrow! I’m not sure it’s a trip I’d repeat soon given how queasy I became as we drove through the countryside. The feeling when a giant tour bus came careening down the road partially in our lane is not one I wish to repeat in the near future. Still, we did stop a few times to soak in the views, especially as we drove up through the hills.

It was late in the afternoon by the time we made it to the Cliffs of Moher. It was worth the narrow road drive, the cliffs are gigantic and amazing! We got there before the clouds rolled in, and while I was walking around taking pictures the fog started to take over for a bit. The day transformed from warm and sunny with clear visibility to cool and damp with fogged over cliffs very quickly. The change completely altered the feel for the place. One moment feeling alive and fresh, the next gloomy and somewhat ominous. I’m grateful we were there to experience the cliffs in both moods.

After seeing the one side, we walked up the other side to look out over in the direction of O’Brien’s Tower. We did venture a bit past the confines of the visitor center grounds, but I had no compulsion to jump any other fences. We also generally kept our general wandering to a minimum, with MJ still walking on his bad foot and my ankle acting up from the sprain I got at Newgrange on Friday. We were quite the pair, but at least we matched.

There were several selfies taken, and MJ got this shot of me as the clouds poured between the cliffs, but my favorite picture was the one MJ took with his phone and didn’t show me until we got home. It’s probably my favorite recent picture of us!

We left for the long drive back to Dublin just as the sun was starting to set. If Ireland hadn’t already made me fall in love with her back in 2010, this drive back would have done it. With every mile through the countryside my heartstrings were pulled as I gazed at the rolling hills we were driving through at dusk. Photos don’t do it justice. Ireland clearly didn’t want us to leave.

More photos from Galway and the cliffs here: https://www.flickr.com/photos/pleia2/albums/72157689480206285/

We got back to Dublin just before midnight and had one final meal at Zaytoon before parking the rental for the night, which we’d drop off at the airport on our way home in the morning.

The spoiled American in me would miss some of the food and creature comforts if I ever moved away from the United States, but Ireland is on my short list of places I could see myself living. Every time I visit it feels like I’m going home and my heart sinks every time I leave. Alas, it was time ! Caligula was waiting for me, and I’d been away from San Francisco for nearly a month.

A Castle and the National Transport Museum in Howth

I spent Friday during my stay in Dublin visiting Newgrange and the Hill of Tara on my own, with twenty of my newest tour group friends. Come Saturday MJ and I had the weekend together! We planned for one day to include a trip to Galway and the Cliffs of Moher, and the other to spend local to the Dublin area. After consulting weather reports, Saturday became the day we stayed near Dublin. However, we’d both already done all the major sights in Dublin itself. Instead we decided to hop on a DART train and head out to Howth. What’s in Howth aside from some nice views? A privately owned castle you can wander around, and the National Transport Museum!

We had a leisurely morning that allowed us to get to Howth in time for lunch. We ended up with a great lunch outside at the restaurant attached to the Howth train station, The Bloody Stream. Gotta love Viking-inspired naming! We then walked over to the transport museum.

The transport museum is tucked behind Howth Castle. It’s home to dozens of vintage vehicles housed in a pair of warehouses, with a strong preference for buses used throughout the 20th century. There were also work vehicles, a firetruck going back more than a hundred years, and a few things that ran on rails. I’m a rail fan, so some quick research on their website ahead of time clued me in to the handful of vehicles that would be of particular interest to me.

The first was their tram No. 253. With the exception of tourist buses and Amtrak, double-decker vehicles are pretty rare in the United States. I’ve never seen a double-decker tram (streetcar) running before, as none of the lines I’ve visited were double-decker capable. However, they’re incredibly common in Europe. From vintage trams to modern buses, the double-decker style in the British Isles seems to be the norm rather than the exception. This means that tram 253 and the other two I visited were all double-deckers. What a treat!


Tram 253

The next tram I was visiting was right nearby, decked out in varnished grained teak was tram No. 9. I was more excited about 253, but this one was just stunning. It had been beautifully restored and shined from it’s corner of the museum.


Tram 9

After visiting these trams, I caught sight of No. 224, one of the most popular vehicles in their collection. At first glance it’s a tram, but closer inspection reveals the truth: it has tires, and “false tramway truck frames were fabricated, giving the tram a highly authentic appearance.” Not only that, it’s a former trailer that’s been so extensively rebuilt that it is now serving as a replica of the original Dublin No. 224, a former trailer, which records show met it’s demise around 1923. The full story is on their website, a fascinating read for people interested in the sorted history which also includes the death of a passenger in 1898 that led to a change in how trailers were used in Dublin at the end of the 19th century.


Replica Tram 224

I do happen to like fire trucks as well, so photos of them and various other vehicles in the transport museum made their way into my album for the museum: https://www.flickr.com/photos/pleia2/albums/72157687000431984

Once we finished walking around the Transport museum we made our way back to Howth Castle. Since it’s a private residence, tours are only done on a booked schedule, and only certain days of the week. Still, the owners allow people to walk around parts of the grounds, so it was nice to walk around a bit and take some photos. See, as the stereotypical American, I love castles.

More photos of the castle and other adventures around Dublin throughout the week are in my Dublin and Howth album: https://www.flickr.com/photos/pleia2/albums/72157689479792585

Newgrange and the Hill of Tara

At the beginning of September MJ and I spent some time in Ireland. With MJ wrapping up his work week in the Dublin office, I spent the Friday on a tour bus visiting some sights. He had already been to the Neolithic passage tomb at Newgrange, so that was number one on my list. The day long tour I booked also included a stop at the Hill of Tara (Cnoc na Teamhrach).

The weather was beautiful. The tour began with a quick drive through the south of Howth, just outside of Dublin, where we saw some beautiful views across Dublin harbor. There was then a half hour stop in the main harbor area for some fresh air.

From there we were off to Newgrange. The tour bus got us there just before noon and we all went to the visitor center. At the visitor center we browsed for a few minutes as we all waited for the tram which would take us up to the site itself.

Archaeologists estimate is that it was built around 3200 BC, predating both Stonehenge and the great pyramids in Egypt.

When it was rediscovered by archaeologists in modern times, it was overgrown and a lot of the stones had been taken away from the site to build nearby structures. Several excavations happened throughout the 20th century, and the controversial reconstruction we see today, which includes the retaining wall, was worked on from the 1960s through the 1970s by Professor Michael J. O’Kelly. It is a little sad to know that the exterior is a restoration, however researched and informed, but it was still an impressive place to visit.

The tour took us past the “most photographed stone in Ireland” – the entrance stone which is carved with swirls of meaning that has been lost to time. I took a picture of it too. As advertised, past the stone is the entrance to the mound, where we were lead down a low and narrow passageway which reaches about one third of the way into of the mound. At the end of the passage there’s a trio of alter-like spaces which are believed to be ceremonial, as well as being where period human remains were found. Lending further credence to the ceremonial speculations, on the winter solstice the entrance is aligned with the rising sun, an experience that people enter a lottery annually to get a chance at participating in it. They do a mock sunrise with artificial lights during the tour, to give visitors a hint of what the experience is like.

After visiting the interior, we had another half hour or so before the tour bus returned to pick us up. I took the time to walk around the mound to take more photos, and sprain my ankle. Only one of these things was intentional, but they were related, fiddling with my camera while walking on uneven paths is clearly too dangerous for me. A sprain is not the souvenir I planned on returning home with! Unfortunately I then continued to walk on it, not just that day, but over the whole weekend, at a conference two weeks later, and I’m just now trying to properly attend to letting it heal.

Once back to the visitor center we had time to get lunch in the cafe, stop in the shop and take a quick browse around the on site museum. The pace of guided tours is often unfortunate for me, I could have spent more time there, but we had another stop on our itinerary, the Hill of Tara!

I’ll be honest, in spite of the historical significance of the Hill of Tara, there’s not much to see there. The story goes that it was the seat of the High Kings of Ireland for centuries. As the tour guide brought us through the complex he explained some of the known roads and buildings that evidence was found for. Another passage tomb is also located there, known as the Mound of the Hostages. But the site is mostly a series of rolling hills, not optimal on my ankle, but I was still confident in my ability to walk it off.


Mound of the Hostages

The most impressive spot is the the Stone of Destiny (Lia Fáil), said to be where the High Kings were crowned. The guide explained the legend of the stone, sharing that the true king, having met a series of challenges, would touch the stone during the ceremony and the stone would scream, being heard throughout Ireland. I touched the stone, it didn’t scream.


Touching The Stone of Destiny (Lia Fáil) atop the Hill of Tara

At the conclusion of the tour the guide was kind enough to drop me off at MJ’s office, as he was heading in that direction to drop off a couple anyway. I was able to get a quick tour before we headed back downtown and had a wonderful dinner at The Bank Bar and Restaurant right near our hotel. There we had some drinks and dined with the bust of James Connolly.

With Friday winding down, the weekend awaited us!

More photos from my day here: https://www.flickr.com/photos/pleia2/albums/72157688735097416

Open Source Summit NA 2017

I recently attended my first Open Source Summit, formerly LinuxCon. It’s strange that having worked in the space of Linux for so long that I kept missing these Linux Foundation events, I think the only event of theirs I’ve been to was the Linux Collaboration Summit in 2010. I always had conflicts, then differing priorities with regard to events, especially since I’ve rarely been paid to actually do work directly on Linux, it’s always been a hobby or base for other projects and infrastructures I’ve worked on. The co-location with MesosCon and shift to general open source conference changed the game for me, so I could finally attend and speak at this North America edition in Los Angeles!

I was thrilled to have a talk accepted at the Open Community Conference run by Jono Bacon. My talk was on Building Open Source Project Infrastructures where I explored the current state of proprietary infrastructures used to assist open source project and asked the audience to consider the benefits of self-hosted and open source solutions for their projects. The OpenStack project and others listed on opensourceinfra.org have open sourced various parts of their project infrastructure to allow for outside contributions and even assistance from the community in hosting. Both the Xubuntu and DC/OS projects have websites that are hosted in revision control. Several projects are using open source tooling for their continuous integration work, and publishing your build system helps contributors replicate releases when adding their own patches and improvements.

I then walked the audience through steps to opening up their infrastructures more, including finding talent (systems administrators do exist in open source projects!), determining what hosting is required, and seeking funding to support anything your project may end up needing to pay for. The talk concluded with a closer look at OpenStack and Xubuntu to walk through the different ways these projects are maintained and the various pieces of the infrastructure we put in the hands of community members. Slides here (PDF).


Thanks to Shilla Saebi for taking a photo during my talk! (source)

On Tuesday I spoke in the ContainerCon track on Advanced Continuous Delivery Strategies for Containerized Applications Using DC/OS. This was actually a talk submitted by a colleague who wasn’t able to participate, so he handed it off to me. I was able to take some slides from other talks, but I always have to do a fair amount of rewriting to make a presentation my own. In this talk I walked through modern continuous deployment pipelines and stressed two things in a containerized infrastructure: running everything in containers and organizing workloads efficiently. DC/OS does this by running both services (Jenkins, GitLab, etc.) inside of containers and software actually being tested. Jenkins has a really nice plugin for Mesos which allows it to farm workloads out to the Mesos cluster automatically, negating the need for something like Nodepool that we used in OpenStack land. The use of containers lends to the second bit, being able to run multiple, isolated workloads across the same hardware instead of using discrete, dedicated servers for various services. Additionally, by using something like DC/OS you can give developers the flexibility they want to use their own tooling without the additional infrastructure administrative needs that exist on for each team running their own hardware/VMs themselves. Everyone gets a chunk of the Mesos cluster and they install what they want, only having to worry about maintaining versions of the applications themselves.

It was after this talk that I learned something fascinating about my references to container. My colleague Jörg mentioned to me that a lot of folks he interacted with automatically assumed “container” to mean “container image” rather than the concept of a running any generic container in the operational sense. I was puzzled by this until I went to a talk later that day by another one of my colleagues, Jie Yu who spoke on Containerization in Mesos, Embracing the Standards. It dawned on me that the communities Jörg tended to work in were developer-focused and mine were operations-focused. It makes sense that we’d see those views in our work. It did cause me to be more cautious about my definitions though, the assumptions I made about what people mean when they say “container” and would indeed be a bit confusing if the assumption is that I’m talking about an image.

Terminology revelations aside, Jie’s talk was an interesting one. He gave some introductory details about how Mesos works, but then dove into the history of containerization in Mesos. This was a history I had mostly pieced together over the past 8 months as I learned about the Mesos containerizer and how Docker fits into the Mesos world, but it was great to see it presented formally, in chronological order so I could more deeply understand the progression and design decisions throughout. He then got to the heart of his talk, where he linked many of the changes being made to improvements users were asking for, as well as the importance of standards in the container space, specifically saying we needed:

  • Stable interfaces
  • Backward compatibility
  • Multiple implementations
  • Vender neutrality
  • Interoperability

He specifically called out efforts by the Open Container Initiative, the Container Network Interface (CNI) and the Container Storage Interface (CSI).

Moving on from my own focus at the conference, I enjoyed the keynotes I saw on Monday and Tuesday (most of the videos can be found here). Though I wasn’t nearly as star-struck as some in the audience, it was particularly interesting to hear from Joseph Gordon-Levitt about hitRECord where they’ve built a community very similar to online open source communities, but for content creation. I believe in the principles of open source, but it’s still inspiring to see that the key concepts of online collaboration are transferable to things beyond software. He also dove into monetary compensation for artists (and developers), which is something we’ve spent many years shying away from as a community. Many of us are paid for our work in open source today, but as someone who came up as an unpaid community member for most of my career, honest, open, discussions about it are not as frequent as I’d like. Video here.

I also really appreciated the keynote from Dan Lyons, author of Disrupted: My Misadventure in the Start-Up Bubble. The talk began by poking fun at the culture of startups, but a third of the way in took a turn for the dark about the dangers of startups and what the fun is hiding. Workers are increasingly being exploited in and around our industry, we need to see this turned around. Video here

During the summit I had the pleasure of meeting Ed Warnicke who gave a lightning talk on Tuesday afternoon about Bringing Network and DevOps People Together. He’s a Principal Engineer at Cisco and was raising awareness around the disconnect between networking and systems folks. This talk hit home for me, literally. When we got married, my husband was working as a network engineer and I was working as a systems engineer. Whenever we talked about networking-related topics, we’d talk past each other. This became even worse when I was working on OpenStack, and then my OpenStack book. I had a networking chapter that I wrote with a fellow systems engineer in the OpenStack community and as soon as I showed it to network engineers, they were confused. I probably spent more time on this chapter than any other because I had systems and networking reviewers pushing back and forth at terminology and ways I explained things and more fundamentally how OpenStack was doing certain things.

Coming back to the talk, Ed stressed that with the future we’re heading into with microservices, we can’t afford to talk past each other and the DevOps world shouldn’t be ignoring the expertise of the networking community that’s been solving problems for decades. His own call to action was for all of us to talk to someone from the other discipline. Done and done! Video here.

There were bunch of other talks I enjoyed as I moved between the Open Community and Containers tracks. It was a little disappointing on Wednesday when my attention was pulled over to the MesosCon Hackathon, but I did make it upstairs to see the puppies.

The event was also great for meeting up with people I hadn’t seen in a while and meeting new people. I enjoyed a women of open source lunch on Monday where we heard an intro blurb from every woman in the room (there were dozens!) and I just happened to sit down next to someone I know online via following each other on Twitter.


First two photos: source, source, Mesosphere Developer Advocates, LinuxChix LA!

…and this collage only captures a fraction of the people I managed to chat with. Great conference.

More photos from the Open Source Summit here: https://www.flickr.com/photos/pleia2/albums/72157687000663014

MesosCon NA 2017

I recently attended my first MesosCon, the North American edition hosted in Los Angeles. There are three such events per year, the other two held in Asia and Europe. These events bring together various companies and other organizations working with and on Apache Mesos so I was really eager to finally meet some of the folks I’ve only interacted with online.

The event began on Wednesday with a Mesos community Hackathon and a customer-focused DC/OS Day run by Mesosphere. Given my role and background, I joined the Hackathon where my colleague Jörg Schad kicked off the event by encouraging attendees to collect ideas for projects they wanted to tackle throughout the day. Topics ranged from documentation improvements, creation of a Kubernetes-related demo for SMACK, work on frameworks and further work on an autoscaler for several cloud platforms where DC/OS runs. Plus, there were mini cupcakes!

The main event for MesosCon began Thursday morning with an introductory keynote from Ben Hindman, Co-Creator of Apache Mesos (and my boss), where he covered some of the latest features in Mesos over the past year that he’s most proud of. These included the introduction of nested container support, adherence to standards through the Open Container Initiative project and beyond, and general expansion in usage and community. Mesosphere Co-Founder Tobi Knaup also got on stage, joined by Tim Hockin of Google for a demonstration of the new beta availability of Kubernetes on DC/OS. One of the really distinctive things about this implementation is that it’s running pure upstream Kubernetes, among other things, this allows for migration of a pure Kubernetes environment (on Mesos or not) as well as known functionality with the tooling written today for pure upstream Kubernetes.


Ben Hindman sharing new and noteworthy improvements to Mesos

These talks were broken up by a pair of panels, the first talking about Mesos in the Enterprise with Michael Aguiling at JPMC, Larry Rau (Verizon), Cathy Daw (Mesosphere), Stefan Bauer and Hubert Fisher (Audi), and Josh Bernstein from {code} as a moderator. At the enterprise level, a lot of interest was around distribution of resources across cloud vendors and on-prem, the introduction of open source software in companies unaccustomed to it, general scaling needs required by large companies and a standards-driven future so they can continue to integrate container technologies into their infrastructure.

The final keynote panel of the morning had Ben Hindman sitting down with Neha Narkhede (co-founder and CTO of Confluent) and Jonathan Ellis (co-founder and CTO of Datastax) for a fascinating discussion about the role of Apache Cassandra and Apache Kafka in the fast-data driven world powered increasingly by the SMACK (Spark, Mesos, Akka, Cassandra and Kafka) stack. The main message in the panel was how fast data is essential in the interaction with customers today, and their Internet of Things gadgets that create an expectation of immediate response. This panel was also where I heard a theme that would be repeated as the conference progressed: people move to microservices incrementally. There are a few ways of moving an environment which is microservices driven, but whether they convert old systems slowly or adopt a policy that all new projects must be built with them, it’s incredibly common across the industry. As a developer advocate these days, it was also nice to hear from Neha that the open source nature of the components are empowering developers, which makes my role in getting out there to talk to developers and operations folks who are on the ground with the technology particularly important.

The keynotes concluded and we dispersed into three tracks, on SMACK, Ops, and DevOps, and a new addition to the conference called MesosCon University where attendees could attend an 80 minute hands-on session on making applications production-ready, securing Mesos clusters, or building stateful services.

I was the track lead for the DevOps track along with Julien Stroheker from Microsoft. Leading up to the conference Julien and I reached out to the speakers to get slide decks and answer any questions, then the two of us met up to chat about the track in San Francisco the week before. The day of the track everything went smoothly, with Mark Galpin of JFrog getting us started by talking about the development and deployment of three key projects: Artifactory, Bintray and Xray. They learned quickly that the application and configuration layers had to be configured separately and that it was best to plan for enterprise and high-availability early on, rather than trying to build it in later. He also discussed importance of good startup scripts and package management in a microservices world, since you don’t simply use an RPM anymore to install things… but you do still need RPMs because not every customer will want to adopt a microservices environment just yet! Standardization was also important, customers wanted the ability to hook into their existing storage and network backends, and often had unusual requirements (for example, “my nodes have no outbound internet access”). He concluded by stressing that everyone in the DevOps organization should be involved from the beginning when working to develop the products and how they’re being deployed so that they consider every aspect of how the tooling will be used and deployed.

The next talk was by Aaron Baer from Athena Health who brought us deep into the world of healthcare information services where monoliths rule and shared his experiences working to bring new technologies build on microservices into the fold. Like I heard earlier in they keynote panel, he’s been working on a path to incrementally move from monolithic and relatively inflexible services over to microservices. In addition to building new infrastructure where they can begin using containers, they’ve also been using their legacy database as a canonical resource and storage, and doing data processing and delivery through faster, more modern data storage and processing tools. He’s also a proponent of code as infrastructure (me too!) so it was great to hear him talk about how essential API-driven infrastructure engineering and automation is to getting our infrastructures to the next level.


Aaron Baer on the progression of monolith to microservice

After lunch the track continued with Chris Mays and Micah Noland from HERE Technologies sharing details about the Deployment API they built for DC/OS. They wanted a simplified version of the core DC/OS API which was deployment-centric and needed an open source option that would improve upon Marathon-LB. The API they’ve developed has just recently been open sourced and can be found here: https://github.com/heremaps/deployment-api. During the talk Micah provided an in depth tour of how straight-forward their YAML configurations were for a sample deployment of a Jenkins pipeline. The API theme continued in the next talk by Marco Palladino of Mashape, the makers of Kong. Kong is an open source gateway for APIs which allows you more control over access, including security, authentication and rate limiting. He stressed how important this whole ecosystem of support is for microservices, and his talk also touched upon the steady migration needed by most organizations from their old stack to the new, which also meant keeping a lot of pieces like their legacy authentication system intact but that adding too many layers can leave to unforgivable latency.


Lively booth area break before the final two sessions of the day

The day concluded with talks from Will Gorman of Cerner on Spinnaker and Imran Shaikh of YP on hybrid clouds. I enjoyed the entire DevOps track, but these two talks really went to the heart of what I’ve focused on infrastructure-wise in the past five years. Give my CI/CD background, I’ve known about Spinnaker for some time, but never made diving into it a priority. This talk inspired me. As you may have guessed, Spinnaker is a continuous delivery platform and Will began his talk by covering the basics about it before getting into integration with DC/OS, including the ability to hook into Metronome for jobs. He also touched upon controls Spinnaker has for the deployment phase to complement Marathon’s own attempts to deploy and restart processes. I’ll definitely be digging into it on my own very soon. Imran’s talk on the importance of being responsible for your own infrastructure was very satisfying for me. He led his talk by talking about how moving to the cloud doesn’t absolve you of the responsibility of making sure you have a good disaster recovery plan, reminding the audience that your business relies upon a good failover plan, and your cloud provider has different goals. Throughout his talk his talking points revolved around vendor independence, distributing risk and the importance of your processes being portable to other solutions.


Will Gorman showing how to execute a Metronome job as a step in a Spinnaker pipeline

Thursday evening had another MesosCon first, Town Hall sessions. A big hit with attendees, these were casual evening gatherings on various topics (Mesos, DC/OS and Marathon/Chronos). They allowed community members to gather and talk about whatever they wanted, swap war stories, share tips, anything! Sadly I had to miss this, participation in the Open Source Summit earlier in the week and with a panel to prep for, I was running low on energy. I’m looking forward to attending the ones we host at MesosCon EU coming up at the end of October.

Friday morning came quickly, and I’ll let you in on a little secret: I had breakfast with my Future of Cluster Management panelists. The panel was made up of Sharma Podila (Netflix), Sam Eaton (Yelp), Ian Downes (Twitter) and Zhitao Li (Uber). As the panel moderator I had worked with them on questions and spoke with them via video conference, I hadn’t met any of them in person. A casual, leisurely hotel breakfast was a great way to break the ice and answer any lingering questions folks had, including myself (“none of you are going to surprise me with the answer you give about the importance of open source, right?”). I highly recommend such a gathering for other moderators if it’s possible, I’ve been a panelist a few times and on stage ice-breaking is not always the best way to go.

Jörg was the MC for day two of keynotes, and after a community-focused introduction to this second day (the first day was more enterprise-focused), he brought us on stage. Talking about the future is always fun, and this panel was no exception. After introductions, we covered the importance of open source in our ecosystem, something that is near and dear to my heart. Theme-wise we talked about the move from perfecting deployments on containerized systems to a focus on maintenance, and the tools they’ve had to built outside of what Mesos provides to manage their clusters. The question of how they handle mixed workloads (stateless vs. stateful, batch vs. long-lived) and what improvements could be made here. The conclusion centered around the future of artificial intelligence of clusters, which led us to learn that some of them are already using machine learning to teach their clusters about load and behavior, which was later explored in a track talk from folks at Twitter who are doing automated performance tuning using Bayesian optimization. The last question I had for the panelists was what they wanted to see in 5-10 years, and their answer was identical: they don’t want to care about the underlying tech, they just want their workloads to run.


Keynote panel: Future of Cluster Management, thanks to Julien Stroheker for the photo (source)

The morning keynotes continued with a talk from folks at the National Geospatial-Intelligence Agency that spoke to how they’re using Mesos at scale inside of their organization. Keynotes concluded with one from Ross Gardler from Microsoft who explored the question of where we were in the hype/actual production cycle for microservices and containers. He expressed that there’s still a lot of diversity in the field, and that with so many early deployments there’s no clear “winner” in the space of containerization. Today, he shares, the focus should be sharing best practices and methodologies across platforms so that all of the next iterations of our various containerization platforms can be that much better.

Just like Thursday, after keynotes we shuffled off into various tracks, with Friday featuring tracks on Mesos Internals, Ops, SMACK and the one I took over as track lead for, Mesos Frameworks. My focus in my role at Mesosphere had largely been on higher level operations, so it was a very interesting track for me to sit in on for the day. There’s a lot of really interesting work being done in the Mesos ecosytem that I’d love for my team to be a part of drawing more attention to. This day was great for that.

The first talk was on the juggling that happens around optimizations, service guarantees and how those trade-offs manifest by Sharma Podila of Netflix, who I’d just met earlier in the keynote panel. He began by asking some questions about your cluster and explaining the heterogeneous nature of the hardware they have at Netflix, you never know which generation hardware your workload is going to land on. He then dove into the challenge of scaling down, showing off the open source Fenzo, “a scheduler Java library for Apache Mesos frameworks that supports plugins for scheduling optimizations and facilitates cluster autoscaling.” His talk was followed by one from Wil Yegelwel for TwoSigma who joined us to talk about simulation testing. In spite of my deep interest in CI/CD, simulation testing is something that’s largely escaped my radar. Thankfully he gave an introduction to the concept and then shared how they used the scheduler they built at Two Sigma, Cook, to intelligently handle scheduling of workloads. The intersection of an intelligent scheduler and a policy of simulation testing meant they could run tests that changed configurations and see how they could optimize things as much as possible, and he explained that this has sometimes resulting in some surprising, non-intuitive discoveries that boosted performance. He also credited simulation testing with helping engineers become more familiar and comfortable with the infrastructures they’re running, which are growing increasingly complicated and difficult to fully understand.

After lunch we dove right back in with Joshua Cohen and Ramki Ramakrishna from Twitter who came to speak on the aforementioned Bayesian optimization talk. It’s well known that Twitter’s infrastructure is built on a lot of microservices. At that scale, they explained, manual performance tuning of hundreds of JVM options doesn’t scale, is error-prone, time-consuming and honestly leads to goal-driven engineers copying existing configs to get going, without fully understanding why the optimizations exist in specific places. They went on to stress that even if you do manage to get everything running well, it’s effectively undone as soon as you change or upgrade any component. Instead they shipped the problem off to an internal machine-learning driven technique that uses Bayesian inference to optimize performance. Bayesian became popular in my own operations world for spam analysis, but the methodologies as I understand them make a lot of sense here as well. Unfortunately they haven’t open sourced the Bayesian Optimization Auto-Tuning (BOAT) tooling, but knowing that it’s been done and is successful for them is a good start for others looking to do similar machine-learning inspired performance tuning.

Next up was Tomek Janiszewski from Allegro who gave us 8 tips for Marathon Performance, summing them up:

  1. Monitor — enable metrics
  2. Tune JVM
  3. Optimize Zookeeper
  4. Update 1.3.13
  5. Do not use the event bus
  6. Use a custom executor
  7. Prefer batching
  8. Shard your Marathon

Marathon is a popular project in the Mesos ecosystem, so the room was quite full for this talk and the questions about it went well into the coffee break we had between sessions!

After the coffee break we were in the home stretch, only two more talks to go before MesosCon concluded! First up in this final pair was Dragos Dascalita Haut and Tyson Norris from Adobe Systems who where sharing details about their use of OpenWhisk as a Mesos framework. They quickly covered OpenWhisk, an event-driven serverless platform for deploying code. Their lively presentation was full of demos and they shared their Akka-based mesos-actor project before pulling in the OpenWhisk bits that brought together all the pieces of the work they were doing.

The final talk brought a pair of engineers from Verizon Labs, Tim Hansen and Aaron Wood, to share their work on fault-tolerant frameworks without Docker. They share some of my own opinions about the over-use of Docker where something slimmer would suffice, so I knew going in that I’d enjoy this talk. The talk explained why using Docker and the entire Docker daemon was often overkill for their work, specifically Tim saying that there was “no real need for an extra daemon and client when Mesos can containerize tasks.” So they instead directed the audience toward use of the Mesos containerizer and the CNCF’s Container Network Interface (CNI). To this end they developed Hydrogen, “High performance Mesos framework based on the v1 streaming API” and their own Mesos SDK, a general purpose Golang library for writing mesos frameworks.

As always, as tiring as the week was it was still sad to see it conclude. It was great to meet up with so many people, huge thanks to everyone who take time to let me pick their brain, or simply sat down for a quick chat. I learned a lot while there and had a wonderful time.


Clockwise from top left: With Yang Lei (IBM), Dave Lester (Apple), Mesosphere community team (minus Jörg, plus Ravi!), and Aaron Baer

More photos from MesosCon NA 2017 here: https://www.flickr.com/photos/pleia2/albums/72157689480391705

Chester Beatty Library and some whiskey

As I mentioned in my last post, at the end of August I traveled to Ireland with MJ for the second time. My first visit there in 2010 was pure tourism, and I hit all the major tourist attractions and historical sites and was able to visit with a local friend to learn about the Vikings, go to the Dublin zoo, see some Leprechauns, tour the Wicklow Mountains, take part in Guinness and Jameson tours, check out the Book of Kells, Dublin Castle and a couple cathedrals and visit Malahide Castle after feeding some seals in Howth. I even went to an Ubuntu release party. It was a whirlwind week, but I had a wonderful time and fell in love with Ireland on that trip.

It took me seven years to return and one of the first things I noticed was how much construction was going on this time. When I was there in 2010 the area was immersed in an economic downturn, but things had picked up considerably since then. The basic historic points of the city remained untouched, but between them were cranes and building of all kinds, particularly out where my husband’s office is down by Dublin Bay. Still, it was the Dublin I knew and loved, and my first order of business upon arriving on Tuesday morning off of an overnight flight was to visit some sights I hadn’t seen before. My jet lagged day began by making my way to the Chester Beatty Library, recommended to me by my friend Walt.

The Chester Beatty Library is now one of my favorite places in Dublin. As explained on their website:

Manuscripts, miniature paintings, prints, drawings, rare books and decorative arts complete this amazing collection – all the result of the collecting activities of one man – Sir Alfred Chester Beatty (1875-1968). Egyptian papyrus texts, beautifully illuminated copies of the Qur’an, the Bible, European medieval and renaissance manuscripts are among the highlights on display. In its diversity, the collection captures much of the richness of human creative expression from about 2700 BC to the present day.

Admission is free, photos are not allowed inside (though their online image gallery is nice), and it’s not a large museum, but it’s meticulously laid out to take you through a tour of what is largely a collection of religious manuscripts of various faiths. The illuminated manuscripts across various faiths were breathtaking, their Witness to the Word: Chester Beatty Library New Testament Papyri exhibit was fascinating, and I enjoyed seeing the mid-18th Century Astrolabe. However, the two things that stood out for me were the Qibla compass and the Mughal-era Indian collection of composite images. Walt told me about the Qibla compass so I made sure I specifically sought it out, I’m glad I did, I would have missed it if I hadn’t been on the lookout for it. As wikipedia explains, “a Qibla compass is a modified compass used by Muslims to indicate the direction to face to perform ritual prayers.” It’s a simple, yet clever device, which works, again consulting wikipedia, as follows:

To determine the proper direction, one has to know with some precision both the longitude and latitude of one’s own location and those of Mecca, the city toward which one must face. Once that is determined, the values are applied to a spherical triangle, and the angle from the local meridian to the required direction of Mecca can be determined.

I wish I had jotted down some notes about it while I was there, but just-off-a-plane brain wasn’t interested in taking notes. I have sent a quick note to the library through their contact form to learn more, fingers crossed that someone will get back to me with even the most basic details that accompanied the display.


Qibla compass at the Chester Beatty Libraryimage source (I wish I could be more specific as to source)

EDIT: They got back to me! Details:

Universal Qibla Finder
Made by Barun al-Mukhtari
Turkish text
AD 1738 (dated AH 1151)
Istanbul, Turkey
CBL T 443

Depicted on one half of this device is the Ka‘ba in Mecca and various near-by sites associated with the pilgrimage. On the other half is a map of Europe, Asia and Africa (regions north of the equator only), with numbers indicating the location of almost 400 cities, the names of which are provided in a colour-coded, numbered list beneath the map. A long pointer is attached at one end to the city of Mecca and at the top of the map is a small compass.

To determine the direction of Mecca and hence the direction of prayer from one’s current location, find the city in the list beneath the map, rotate the pointer until it points to the number (and colour) of that city on the map, and move the device until the compass needle points due north-south. When correctly aligned, the pointer will indicate the qibla, the direction of prayer.

As mentioned, the composite animal images (creatures whose bodies consist in whole or in part of human and animal figures) were the other thing that grabbed my attention. Specifically there was one of “A Peri Riding a Composite Lion” that captured the imagination. I knew nothing about the Indian tradition of composite animal art before visiting this museum, but seeing a collection together was really something. I stopped at the gift shop on my way out specifically to see if I could get a post card or a book with the composite lion image, and they didn’t disappoint, €7.95 later I came home with a coloring and postcard book of the images in their collection, including the lion!


“A Peri Riding a Composite Lion” circa 1700, Kashmir

I now consider the whole museum a must see when visiting Dublin.

Cultural enrichment behind me for the day, my next stop was a new whiskey distillery that I heard about, Teeling Whiskey. I learned there that the current iteration of the business launched in the 2015. However, the background of the brand, discussed on the wikipedia page and recounted during the tour, is interesting. In short, the Teeling family founded a whiskey distillery in 1782, but it sold and eventually shut down as a series of unfortunate events caused the entire Irish whiskey market to crash in the early 20th century and whiskey production move out of Dublin. After our history lesson, we were able to get up close to their on site distilling setup, including their big pot stills that the whiskey goes through for the traditional three-phase distilling process for Irish whiskey.

As we were tasting some of “their” three+ year old whiskey at the conclusion of the tour I joked about being pretty good at math when I asked “how can I be drinking a six year old whiskey when you were founded less than three years ago?” The history of their family explains, the current batches being sold under the Teeling brand actually came from the father of the founders, who founded the Cooley Distillery north of Dublin in 1987 and gave his sons a large collection of minimally aged whiskey. The results of these older barrels under the Teeling branding have already made their way stateside, but I’ll be keeping an eye out over these next few years for the ones with vintages older than 2015.


Teeling tasting

Teeling I did on my own, but a few days later was the final full day MJ and I spent in the Dublin area itself found ourselves concluding the evening with a stop at the Bow St location of Jameson. They have long done their distilling elsewhere, even in 2010 when I did the tour for the first time. However, this location is a nice tourist attraction that has a large bar and a tour. During my last trip it was more tourist-y and a bit kitschy, but I actually enjoyed it. We learned that week that they had recently redone their whole building and the tour, a perfect excuse to go back!


Fancy redone bar at Jameson

The tour is simplified now, much more refined with a greater focus on history and how the whiskey is made. They’ve also added several other tours for people who are more into Irish whiskey. I admit missing the old style tour a bit, but I’m a sucker for tourist stuff. The concluding tasting was enjoyable though, everyone gets to sample an American whiskey, Scotch, and some Jameson, to compare the styles and become our own experts at telling the difference. It was fun, and afterwards we all had a drink ticket for one final drink on the way out. Contrary to my typical “on the rocks” preference, my day called for a nice bit of Jameson, neat.

Our Ireland trip wasn’t all museums and whiskey though! Our trip to a transit museum and the long journey to the Cliffs of Moher are due in my next posts.

Chats about DC/OS and Apache Mesos in Dublin

At the end of August MJ made plans to spend a week in Dublin for work with a team he has out there. This seemed like a great opportunity for me to go back to Ireland as well! The last (and first) time I went to Ireland was also to tag along, back in 2010 with his previous job when he was doing a rotation in the office out there. That trip landed in October of 2010 and I took the entire week off to explore the city and surrounding areas, this time I only took a couple days off, choosing to also spend some time getting to know some of the DC/OS and Apache Mesos folks there in Dublin.

First on my noteworthy agenda was speaking on The SMACK stack for the Dublin Apache Kafka Meetup by Confluent. It was held at the ZenDesk office just south of St Stephen’s Green. This meant I could enjoy a lovely mile long walk south from where I was staying near Trinity College to get to my destination. When I arrived, my gracious host for the evening was Andrei Balcanasu, an Infrastructure Engineer there at Zendesk. As attendees trickled in he made me feel very welcome and we had some great chats about the direction of infrastructure tooling over the past decade.

The presentation was an overview of the SMACK stack (Apache Spark, Apache Mesos, Akka, Apache Cassandra, and Apache Kafka) itself, but I did try to stress the value of Kafka specifically in this model. My experience thus far has been that Mesos and Kafka are the two components that are least likely to be swapped out as people tailor SMACK for their own infrastructure environment and industry. I concluded with Achim Nierbeck’s Iot Fast Data analytics demo which plots buses from the Los Angeles Metro on a map and is pure SMACK stack. Slides from the talk can be found here (PDF).

The evening concluded with beer and pizza, during which I indulged in my first Bulmers Cider of the trip as I spoke with attendees about their infrastructures and where they were planning on going tooling-wise. While I’d never call myself a social butterfly, getting able to geek out about open source infrastructure tooling is one of the areas where I can let my guard down and get really into it, so enjoyed the evening. During these conversations something that stood out most for me was a comment about the simplicity of the data source for the demo, as the stream coming in from the LA Metro is not what any of us today would call “fast” (just a few requests per second) so the use of Kafka is a bit overkill. It was a very fair point and I’ll mention it up the next time I use it in a presentation, but it is just a demonstration of a pipeline, and at that it succeeds at being reliable and effective.

Unbeknownst to me when I booked my travel, I came into town the same week as the Usenix SREcon17 held right there in Dublin. This meant that a whole bunch of people seeing my Dublin tweets assumed I was in town for that, and reached out to me to meet during the conference. Had I known about the overlap, I would have probably participated, but the CFP has closed long before I made plans for this trip and it was sold out attendee-wise before I knew about it. Still, this was how I ended up with the opportunity to meet up with DC/OS community members Levente Lajko, Luis Davim and Matthew Allen from PTC!

PTC has been an active participant in Working Groups we run for DC/OS so I was delighted to meet with them in person as they took time away from the conference venue to meet for lunch. Discussions centered around improvements we could make to the community involvement side of things, including being more public about our road maps so they can plan accordingly with what they build and lack of engagement around some tickets and pull requests. Nothing here was new for me and my team is actively working to make real progress on these points, but hearing it directly from community members strengthened my confidence that we’re working on the right things. Even better, they were a fun group of folks to spend a meal with and they were eager to make clear that the technology is worth the community-related road bumps.

Now to be clear, I don’t always intend on making vacation-centric trips include work stuff, I do need time away too, but I’m glad I did this time. With only 8 months under my belt at Mesosphere I was eager to spend as much time as I could interacting with community members, and in-person meetings like this are an important part of that. Having visited Ireland before also helped, I didn’t feel like I was missing out on anything, so breaking up tourist time with some work ended up being a pretty healthy use of my time.

August visit to Philadelphia

As I’ve already written, in mid-August I road tripped with my aunt and mother up the east coast from central Florida to Philadelphia. They left on Wednesday, leaving me on my own until that evening when I had to drop the MDX off at the shop to get the windshield replaced due to a very unfortunate incursion with a rock while driving down the highway when MJ was in town back in July. I had dinner with a friend who picked me up there. Thankfully the windshield replacement went without a hitch and was completed by Thursday afternoon, so I was able to craft my work schedule that day to start early, get the car mid-day and be home to work until I had my next house guests, my Aunt Meg and cousin Melissa! The timing worked out well, they were driving up to New England from the Carolinas so I invited them to spend the night on their way up as a nice midway drive resting spot.

We had dinner at the nearby Toscana 52 Italian restaurant before settling in for the night. I saw them off the next morning before starting my Friday morning of work. That evening my friend Danita came over to spend the night, a weekend trip to New York City together, which I wrote about here, beckoned!

Monday arrived much too quickly. The day was spent once again working from the townhouse, which is working out really well. My office may be where we are storing piles of boxes, totes and furniture (most of which will end up in California), but in the midst of all that I have a really comfortable setup with everything I need to be productive. I just plug in my work laptop and go. Unfortunately during this trip I learned that the personal laptop that I keep there as my desktop has finally failed. The blink codes indicate that it’s the CPU or motherboard, which essentially means the laptop has reached the end. I’ll have to bring out a replacement next time I come into town.

However, Monday was eventful for another reason, it was August 21st, total solar eclipse day! I had piles of friends who braved the crowds to experience the totality zone, but in Philadelphia we just had a partial eclipse. During much of it we had cloud cover, but after the high point the clouds parted and I was able to get some nice glimpses through my pinhole camera box as I sat on the back porch. While sitting out there I flash backed to being 12 years old after school and doing the same thing. I have always enjoyed a good eclipse.

Monday evening I drove out to PLUG West to give an Introduction to DC/OS presentation (slides). Surprisingly, up until this point most of my DC/OS presentations had been more goal-oriented, so I really enjoyed doing a more holistic presentation about the “datacenter operating system” space, getting able to dive into more of the infrastructure geek bits of how DC/OS works as a platform on top of Linux for my fellow Linux-loving attendees present that evening. I also still know a number of folks who have attended PLUG meetings over the years, so it was nice to catch up and meet some new folks as a number of us adjourned to a nearby restaurant for a late dinner.

My next two evenings gave me time to catch up with friends over dinners, Tuesday night I had dinner with my friend Crissi. It had been a little while since just the two of us caught up in person, so it was really nice to get that time. Thursday I met up with my friend David to see RiffTrax Live: Doctor Who – The Five Doctors. I’d really been looking forward to this, and almost bought tickets for the actual live event in Nashville when they went on sale in the spring. This was actually a replay from the previous week, but as a Fathom Event in a theater, it’s hard to tell the difference. It was totally worth seeing. I don’t remember if I’d actually seen The Five Doctors before, I have seen a lot of old episodes but this movie is a bit ridiculous, even by old Doctor Who standards. A replacement of the first doctor (who had died a decade earlier), a cameo of the fourth doctor taken from footage shot for another production, and plot holes and continuity problems rounded out the basics of what made it so amusing. The Rifftrax of it was hilarious, even if they frequently took the opportunity to make fun of us old school Whovians.

On Friday morning MJ got into town! He flew in to visit for a few days, attend FOSSCON and so we could depart together on our trip to Ireland Monday night after work.

It’s hard to believe I was in Philadelphia for almost two weeks during this trip. With every visit the townhouse feels more like a home, I can relax and really feel comfortable there. Not sure yet when we’ll be back, but the end of year holidays are always a good opportunity since we have some off from work.

Summer weekend in NYC

Several months ago I learned that they had made a Broadway musical based on the full-length animated feature from 1997, Anastasia. I quickly scoured the dates that Anastasia the Musical was being shown at the Broadhurst Theatre in New York City and learned that it would indeed overlap with one of my Philadelphia trips! I got a quick confirmation from my friend Danita that she was interested in seeing it and was available over a weekend I was in town, then booked us tickets for August 19th. We drove up Saturday morning after breakfast to park and drop stuff off at the hotel before we made our way over to the American Museum of Natural History.

This museum occupies a special place in my heart. I’ve always loved dinosaurs, and when I was a kid I visited my grandparents in north New Jersey. They brought my middle sister and I up to NYC for the first time and this museum is where I saw my first dinosaurs. My first impression? I thought they’d be bigger. Now, this was the 80s and since then we’ve learned much about dinosaurs, including discovering even larger ones that my kiddo self may have been more impressed with. During our visit this time I got to see the Titanosaur, a cast of a 122-foot-long dinosaur finally given the scientific name, Patagotitan mayorum just days before our visit. It was in a very large room, but it didn’t quite fit. In addition to wandering through the history of mammal fossils and more, we saw a show in the planetarium.

After the museum we stopped in to Heartland Brewery and Chophouse for beer and a huge pretzel snack and dessert before the show. It was a hot summer day in New York, so I was appreciative of the time we took to pause.

And then it was finally time for the evening show. Much waiting in line before finally getting into the theater. Inside the Broadhurst is lovely!

I hadn’t ready anything about the show before going in, so I wondered what they’d do about zombie Rasputin and his little bat sidekick. Turns out, they replaced them both with the more historically accurate and less fanciful Bolsheviks. It worked out well, and I totally have a crush on the Bolshevik general Gleb Vaganov. “The Neva Flows”? My new favorite song from the original cast recording!

After the show we grabbed another quick bite before a very late walk back to the InterContinential New York Barclay. On Sunday morning we enjoyed brunch at the nearby Lexington Brass before driving back down to Philadelphia. In all, an awesome weekend and allowed for a calm closing to my Sunday before getting back to work.

More photos from our weekend here: https://www.flickr.com/photos/pleia2/sets/72157685760317003