API Goodness with Mailgun, Trello, and gspread

We’re about to open registration for OpenStack Upstream Training, and we needed to glue together Google Spreadsheets, Trello, and email to streamline the registration process.

The gspread and Trello APIs were easy to use, as expected.

I had my doubts about Mailgun, because there was no suggested Mailgun library. Instead, the docs recommended developers just install Requests. I expected to write some boilerplate as a result, but they proved me wrong.

Check out the Mailgun docs for sending an email. Examples in 6 languages and curl, and the examples update with your API endpoints and auth data if you’re logged in. That’s some Twilio level documentation, and I’m now a huge fan.

If you want to see how we tie the systems together, check out the openstack-upstream-registrar code. I’d like to morph that short script into a small web application that will guide developers through the ATC process. Reach out if you’d like to help!

Back on the topic of the training itself: we’re always looking for more in-class mentors and assistants! If you’ll be in Tokyo on October 25th and 26th, please consider registering as an assistant.


I’m thinking about getting in to Ceph development, and so I’m thinking about how to quickly provision and destroy and reprovision tiny Ceph clusters at home.

For love of blinking lights, I’ll probably build a cluster with Banana Pi boards in the future, but for now I’m just using libvirt and KVM.

Ceph-libvirt-clusterer lets me clone a template virtual machine and attach as many OSD disks as I’d like in the process. I’m really happy with the tool considering that I only have a day’s worth of work in it, and I got to learn some details of the libvirt API and python bindings in the process.

On the road map:

  • Add and remove disks from running machines
  • Build a command line interface

Build Virtual Machines for Multiple Hypervisors with Jenkins and Packer

Building a virtual machine image for one target hypervisor probably doesn’t cut it anymore. If your organization is like most today, you run VMware in production, and you’re investigating AWS or OpenStack for burstability or a full blown migration, and your developers all run VirtualBox on their laptops. Except for Marvin. Marvin runs KVM because he’s a contributor to the project.

We can define cross-hypervisor machine definitions with Packer.

We can run Packer on a variety of machines to provide full hypervisor coverage with Jenkins.

Packer and Jenkins make a great team. They’re like peanut butter and chocolate. Or coffee and chocolate. Or pastry and chocolate.

/me grabs a chocolate bar from his drawer and gets back on track.

Many organizations use a single Jenkins server to run continuous integration builds, but Jenkins supports any number of slave nodes to distribute load. Further, we can assign labels to those nodes that define capabilities or other characteristics (host OS, installed software, special hardware architecture, etc).

Jobs can be pinned to nodes with specific labels, and the NodeLabel Parameter Plugin lets us use node labels as a parameter when building a job.

I give each of my build nodes a label of packer-${builder-name}, then I can build any machine in my Packer Open Source Appliances repository on all of my available local hypervisors and cloud accounts. My personal list includes qemu, docker, virtualbox-iso, and openstack (with Rackspace).

I’m pretty excited about this, and I’ll share more details and possible next steps in a future post. Until then, you can snag the job definition from my repository.

Better Storytelling and Positive Deviancy

I mark passages in books while I read, and then I return to the books later and copy the marked passages into my personal notes.

Among other interesting snippets, I marked and starred four whole pages in Atul Gawande’s Better. I think that people who highlight entire pages just don’t quite understand the concept of highlighting, so my left eyebrow instinctively rose while I attempted to figure out what the hell I was thinking when I marked such a large passage.

It talks about the difference between 99.5% and 99.95% health rates, and I’m already very familiar with that difference since they come up in service level agreements all the time, but this passage doesn’t recite facts. It tells the story of 99.5% versus 99.95% with a fantastic narrative that left me with a little lump in my stomach, and those four pages are enough for me to recommend the whole book, but you can read that excerpt right now thanks to NPR.

Atul could replace those four pages with a brief exposition of facts: the difference between being 99.5% and 99.95% successful with a treatment regimen sums up to a 16% chance or 83% of making it through a year without complications. Instead he tells a story in four pages that made me stop, think, and mutter “damn.”

And the impact of a good story isn’t just a momentary pause. Stories help us retain information. That’s why I thought of Shlemiel the Painter while full grown adults bickered about string encoding in C today. Joel Spolsky wrote Back to Basics in 2001, and I still remember the story of Shlemiel.

Street Lines

Learning to clean toilets even makes a great story when Joel tells the tale.

Returning to Better before I close out this post, I want to share two paragraphs from the Afterword (Becoming a Positive Deviant)

Wherever doctors gather - in meeting rooms, in conference halls, in hospital cafeterias - the natural pull of conversational gravity is toward the litany of woes all around us.

But resist it. It’s boring, it doesn’t solve anything, and it will get you down. You don’t have to be sunny about everything. Just be prepared with something else to discuss: an idea you read about, an interesting problem you came across — even the weather if that’s all you’ve got. See if you can keep the conversation going.

A Review of My Life in Advertising

Pepsodent Ad

My Life in Advertising is the autobiography of Claude Hopkins. It’s out of copyright, so you can also get a legitimate copy from The Internet Archive. Everyone reading this post is in sales and marketing, and so everyone reading this post can gain something from this freely available book.

Three themes repeat throughout the book:

  1. Know your customer, show them value, get the sale.
  2. Trace your advertising efforts through the sale.
  3. Work is play, if framed correctly.

I hosted a book discussion on The Power of Habit. If you’ve read that book, Claude’s work on Pepsodent will sound familiar. Charles Duhigg wrote about Claude’s work on Pepsodent that helped usher in an age of tooth brushing in America.

Internet marketers pat themselves on the back for A/B testing. Claude did that in print, and with real, physical product on the line. Everything old is new again.

What Every Web Developer Should Know about HTTP

What Every Web Developer Should Know About HTTP has a long title, but it’s a pretty short read. I didn’t time my reading, but my Kindle tells me that 30 minutes remain in the book if I open to the first page.

I bought this book because I’ve considered writing something similar concerning other topics like DNS or SSL, and I can report that it’s a decent tour through HTTP.

If you’ve never made a HTTP request by running telnet against a web server, or never reviewed requests and responses in Wireshark or a developer console, this book is for you. If you have a junior developer on the team that’s just learning about web programming, send them a copy of this to get them started. Examples are in C#, so stay away if you hate C# and you’re too shallow of a programmer to look past that choice (and consider yourself judged for that shallowness).

Kallithea CI Service Proposal

Kallithea is a new source code management system based on the GPL origins of RhodeCode. The project needs a continuous integration service running in the open to sufficiently test incoming patches across a matrix of configuration variables. I’ve started looking into adding SSH authentication capabilities, so solid automated testing platform is personally interesting.

I am building a Jenkins server for this purpose. The server build process is automated with Packer and Puppet so that we can easily host the system in a large variety of environments.

Builds will run in Docker containers through the Jenkins Docker Plugin. The plugin makes Docker hosts look like a cloud provider to Jenkins. Since Docker now supports nested containers, each Jenkins slave container can launch any number of other containers to orchestrate multi-node tests.

This approach offers at least three benefits:

  • The system can start on a single host
  • We can build rich single use test environments that launch quickly
  • The system can grow as our testing needs grow

At the least I imagine test jobs to run the following:

  • Unit tests
  • Integration tests with various infrastructure components (for instance: sqlite, MySQL, PostgreSQL)
  • Application upgrade tests

I’m happy to pay for the hosting of the CI service, but the open and automated deployment definitions will allow anyone to build and run their own system as well.

The first scraps of configuration are in the following two repositories:

Please let me know if you have any thoughts, questions, or concerns. In addition to the mailing list, you can find me in the #kallithea IRC channel as timfreund.

Review - Becoming a Change Artist

Becoming a Change Artist is not bad, but it’s also not in the top three books that I’d recommend from Gerald Weinberg.

Read it if you have three to four hours to spare, and you’re interested in the dynamic of change in an organization. Don’t read it if you’ve heard great things about Mr. Weinberg and want to see what all the fuss is about.

Start instead with Becoming a Technical Leader or The Secrets of Consulting. You don’t need to be an official consultant to benefit from The Secrets of Consulting. Most of consulting involves facilitating work among groups of humans, and that’s probably a big part of your job even if you don’t want to admit it.

Notes for Enterprises Investigating OpenStack from eNovance

Nick Barcet of eNovance gave two presentations at the OpenStack Summit in Atlanta that resonated with me as we investigate running OpenStack at work.

Are Enterprises Ready for the OpenStack Transformation? slides

Key points:

  • Vanilla OpenStack is a framework, not a product. It will never be ready out of the box.
  • Licensing alone isn’t reason enough to use open source: people, hardware, and power still make up the lion’s share of data center budgets.
  • OpenStack provides agility with infrastructure to enable rapid application deployment and iteration.
  • This agility is worthless if the business doesn’t capitalize on it by shifting mindsets and accelerating cycle times.
  • Start with new apps in a small scale at first to get practice with the shift. No need for everything to move all at once.

Delivering OpenStack Clouds as a Factory

Key points:

  • Remember the agility in the last presentation? Part of that includes the platform itself: OpenStack releases every 6 months.
  • No cloud platform is simple: lots of moving parts.
  • OpenStack runs a sophisticated continuous integration system with Jenkins
  • eNovance runs a duplicate CI infrastructure chained from the OpenStack CI system.
  • eNovance also runs a duplicate CI infrastructure chained from their corporate CI systems in customer cites.
  • They’ve built tooling to allow progressive upgrades and rollbacks of installed clouds: rack by rack, for instance.
  • They often run as little as two weeks behind the stable branch.

As a counterpoint to this, I met with someone who was struggling with an upgrade for over 6 months. They installed OpenStack’s Diablo release years ago, and they’ve given up on the idea of smoothly migrating to Havana (released in November of 2013). Instead, they’re dropping in a new cloud on new hardware. If you don’t stay up to date, you’ll get into a treadmill of invasive installations and migrations.

OpenStack Upstream Training

I grabbed one of the last tickets to the OpenStack Upstream Training that ran the weekend before the Atlanta OpenStack Summit.

I’ve used open source for longer than I care to remember, and I’ve even submitted a few patches along the way, but I’ve never become a member of a project. I felt a little under- and over-qualified simultaneously. How could I feel overqualified if I had never become a repeat contributor? I know how to program, but contribution is a lot more than programming. See if any of the phrases sound familiar:

  • I just need to sit down and figure it out. (repeated weekly)
  • I’m going to work on bug $X tomorrow. (repeated weekly)
  • I hope I do this right, I don’t want to look stupid.

Here’s a hint: “I just need to sit down and figure it out” repeated more than once is code for “I don’t really want to do this” or “I have no idea where to start, and I could use a little guidance.” In my case, it was the latter.

The size and sophistication of OpenStack intimidated me. I’m no stranger to code review, continuous integration, and automated testing, but OpenStack is big. There are more hosts running or ready to run automated tests for OpenStack than we use to service our biggest application at $DAYJOB. (Check out the “Job Stats” graph on the Zuul Status Page to see a live count of hosts.)

My biggest problem was a people problem: I was convinced that while navigating the OpenStack build and review environment for the first time I’d make a mistake, and people would think I’m stupid. In retrospect, that was stupid of me. Of course I’d make mistakes, but the people and the processes exist to help us build a better product, accounting for the possibility of mistakes along the way. OpenStack doesn’t get built in large chunks of caffeine fueled code tossed over a wall. We build it one small, well reviewed, well tested patch at a time.

The OpenStack Upstream Training broke down into three major components:

  • The Training Class
  • The Design Summit
  • The Mentoring

The Training Class

Scheduled in the two days prior to the Summit, we soaked in the technical and social processes of OpenStack, and we didn’t have time forget it all before the Summit began.

The bulk of our classroom training focused on the people and the social processes in open source, and specifically OpenStack. Behind every patch and every review comment there is a human being with hopes, goals, stress, problems, and dreams. We all want to build a great system, and we’re all constrained to 168 hours each week. The PTL that doesn’t respond to an email isn’t ignoring you on purpose, she’s drowning in email and constantly trying to dig out. Gerald Weinberg reminds us in his writing that “no matter how it looks, everyone is trying to be helpful.” We can never go wrong with empathy.

Even if you don’t attend Upstream Training, you should definitely read Communication Gaps and How to Close Them.

The Design Summit

Summits are more than just a traditional conference. Design discussions to shape the next release fill more than half of the week. Upstream students’ most important assignment for Summit week was to find and meet the team members that would review our chosen work.

Introduce yourself, tell them what you’re interested in working on, and how much time you will commit each week. They are busy, they will love you.”

I met a large chunk of the Designate team, and I learned how little I know about running DNS at scale. That’s OK, though, I’ll learn more as I go, and the team will help along the way.

The Mentoring

The class followed by the summit made for an exhausting week. It would have been easy to fly home, sleep for a day, and get lost in $DAYJOB work for a week trying to catch up. “I’ll do some OpenStack work next week once I’ve caught up on home and work.”

Wrong. The habit of OpenStack contribution isn’t yet built, and taking a week off will destroy any momentum. The program includes weekly mentoring sessions to keep students moving forward. Loic set aside valuable time on Wednesdays and Saturdays to check in on students and keep them moving toward their goal, and it was vital for my success. I stayed up until 4:00 AM the night before my first mentoring session because I didn’t want to let him down, and I wasn’t happy with my progress thus far.

I now have one patch merged, one under review, and one in work. I’m also working on improving the automated testing for Designate, and I suspect I’ll work on documentation in coming months as well.

Should you attend?

You, too, should attend OpenStack Upstream Training if you feel capable but stuck when considering open source contribution. The training will give you the social and process knowledge to succeed, and the encouragement to keep moving forward until you build up your own momentum. It’s hard work, but it’s fun work.

The Upstream University training changed the trajectory of my career. I’ll forgive you for dismissing that as hyperbole since I’m still so early in the process, but I already feel addicted to the scale of OpenStack and the hum of hundreds of sharp developers and operators improving it one patch at a time.