Shipping five times a day

Shipping code, I'd say, is one of the most satisfying aspects of being a developer. Shipping is fun! Your code is running in production. Customers use it. They might even be happy about it and tell you. Or, things might not work the way you expected and you'll hear about that as well (and learn from it and get better). Either way, it can make you feel great and give you this warm fuzzy feeling of meaningfulness. Think Maslow's hierarchy of needs for developers:

Maslow's_Hierarchy_of_Needs.svg

So, one of the things I really like about my current developer job at Stack Exchange is the fact that, thanks to continuous integration and deployment, I can ship code to production basically as often as I want. Occasionally, that's as often as five times a day. That would be the case when it's my turn to fix bugs reported by users on meta.stackoverflow.com or by our paying customers. 

Of course, getting brand-new features out the door often takes longer. Getting company pages right took a few months and several iterations, for instance. That doesn't mean code just sits there hidden somewhere in branches and subbranches or even on developer machines. It's still getting checked in all the time, the new stuff is just hidden behind feature flags. In-progress features light up on our internal dev environment. Heck, I'm still gonna call that shipping for now in Maslow's sense. I've shipped my code somewhere, even if just to a small internal audience for now. Still fun. I'm happy. The real shipping - removing the feature flags, announcing to customers etc. - happens later and is even more fun, of course.

Now, throughout my developer career, I've worked in several larger organizations where this kind of instant-gratification, developer and customer happiness producing way of getting code out to production seemed impossible. Shipping typically meant all or a subset of:

  • Painful end-of-milestone branch integrations with merge conflicts all over the place.
  • Testing and re-testing. Does everything still work after all those merges?
  • Long meetings with many stakeholders to discuss "the deployment", schedules, dependencies, downtimes...
  • Follow-up emails to those meetings. Follow-up emails to those follow-up emails.
  • Multiple hand-offs (give binaries to QA team, they will then give them to the operations team...)
  • Dealing with bureaucracy: deployment documents with step-by-step instructions. Sign-off meetings for those documents. Deployment checklists.
  • More discussions on "what we'll do if something goes wrong". Typical outcome: we need rollback instructions! In a separate document!
  • Long and uncertain working hours at "deployment time".
  • And the worst of all: your ready-to-ship feature is getting "postponed" (that means "cut") and doesn't ship at all.

You've already guessed it - shipping code this way can be anywhere from tolerable to simply awful. If you're a manager of developers on such a project, better up their morale with a big end-of-milestone "shipping party" - that'll at least address some of their needs in that orange area of the pyramid above.

My point? If you're a developer and about to join a new team, talk to them about shipping before you sign.

(Next post coming up: tools & culture aspect of continuous deployment, and how I broke something and didn't get executed)



Written on September 28, 2013.