Quick Decisions, in a Software Release and on the Staten Island Expressway

Caution: If you have a weak stomach and don’t like reading about baby puke and quick project decisions, you might not want to continue reading.

So, I guess my 18 month old daughter doesn’t like my driving. Since we don’t own a car and just rent a few times a year, her body probably isn’t prepared for the shock of it all. Regardless of how and why it happened, she had an unfortunate up-chuck while we were just slowing down in a huge mass of weekend traffic on our way back through Staten Island this past Sunday night. It was quite the event too… anyone who’s seen Monty Python’s “Life of Brian” can picture a similar scene. So, we’re stuck in the car with a wailing puke-stained baby and a 3 year old who keeps yelling, “stop crying, you make me sad!”

What to do? Do we try to pull over on the lack of a shoulder lane and try to change her into clean clothes? Do we pull off at an exit and see if we can find somewhere to stop and clean her up? Do we just try to keep driving and see how quickly we can get back to Brooklyn (granted, this is New York traffic on a Sunday night in June, seems like everyone left the city for the weekend and they’re all coming back now).

As I was weighing my options, I was reminded of similar situations that I’m in when I am managing software releases for my company. (Ok, so maybe I really didn’t drift off to such thoughts while in such a stressful predicament in the car, but it sounds funny so I’ll tell the story this way).

Our last large new feature launch was in the beginning of May, and we do regular bug releases and small feature launches. Inevitably things will go wrong, regardless of the amount of testing we do while on our QA and staging sites, there’s always something we find when we put the new work live. This can cause varying levels of panic, especially when it’s late at night and our project team wants to go home, or its early morning and we’re trying to finish up before our clients come into the office and want to start using their websites. When these things happen, we have a few different options –

1. Put a programmer on it ASAP and see if this new issue can be resolved and tested quickly and hope that no new related problems are created by the changes.

2. Roll-back the entire release and try again the next day, after spending time working on the new issue and resolving on the test sites.

3. Leave the issue on the live site, make the client aware of the issue and work on getting the fix thoroughly tested and live within the next couple of days.

It all depends on the severity of the issue, but what I can always count on is a project team looking to me to make the decision. I don’t always like making quick decisions, but I know it’s my job in these situations and I know my team is looking to me to make the call. It’s almost funny how much they count on me, on different occasions the programmers or the QA team and even the more senior members of the staff have shied away from making crucial decisions when it comes to software releases and potential impact on our clients. It’s as if nobody wants to be responsible if the decision made sends us into the deep dark sea, they’d prefer I be responsible and as the manager go down with my sinking ship. So far so good…we haven’t sunk yet. I wouldn’t call myself caption of the year, but we’ve been floating along somehow. This part of the job can keep things exciting, and terrifying at the same time.

So, what did we do with the sad sick baby in the car? We kept on driving, opened the windows up all the way to keep the smell from making me sick myself, and gave her a bottle of water. The bottle actually made her very happy, and within maybe 20 minutes she was calm and maybe even forgot that she was covered with slop. We figured since it was already late and she was tired, better to try to just get her home than risk getting stuck in an even worse crowd of traffic. We eventually got out of Staten Island and were able to get home to Brooklyn to get everyone hosed down. I was glad my husband and I (yes, he helped in the decision making, too) had decided not to stop because once I had started to clean the crime scene I found that it was a lot worse than her clothes, even in clean clothes she would have been sitting in a stink and gotten dirty all over again. Today she’s clean and happy and hopefully has forgotten the whole thing.

And can I even try to remember the show-stopping bug that plagued my project team the last time we did a release? Barely…