Reflections on "The Sprint"

I am really enjoying reading the 'Slack' book by Tom Demarco (Slack: Getting Past Burnout, Busywork, and the Myth of Total Efficiency). It's setting in enough that I am pressuring myself to not do real work right now and write something for the blog instead. He makes many great points, and ones that I've already been trying to push on my colleagues (for example, by ordering one of the support staff to not stay a minute past quitting time and "Go home!").

One section I found interesting was his description of the "Sprint" at the end of a project. He describes one way of handling the sprint, where team members pull marathon weekend work shifts, taking naps on the office couch or overdose on caffeine to keep going. The team will stick it out together until Monday morning, and (hopefully) will finish victorious. Another way is to have people work regular overtime in the last weeks/days before the launch to finish everything up, rather than crashing all of the resources at the very last few days. He explains that each one of these methods has a different effect on the team morale. To do the sprint over a couple days rather than the last few weeks, although exhausting, can create a real bond and almost a 'high' for the team as they pull together their last bit of energy to bring themselves to the finish. The extended overtime will more likely lower the morale, cause stress and problems with personal life, etc.

The description of the weekend marathon sprint brought back memories of college, of the last days before my senior engineering project was due. Three other Mechanical Engineering pals and I had spent the year together designing, building and testing a small low-cost and feul efficient turbojet engine. In the last days before our project was due the engine was built to the best of our ability and we were gathering our reports, writing up our research and putting it into as presentable a thesis as possible (engineers trying to write a paper...funny). We were in and out of the engineering building at all hours of the night, taking turns sleeping and writing, and thankful for the one or two 24 hr takeout places nearby. It was an amazing bonding experience, and we did finish in the end and get our report written, nicely bound and in our advisors office by that Monday at 5pm.

I was 22 back then, and had alot more energy and the drive to do those marathon work sprints. The best I can compare it to now is the first few months after each of my daughters were born, staying up at all hours of the night to handle feeding & changing and whatever other needs they had, trying to get daddy to help with whatever he was biologically able to do, although even that was a project sometimes.

But, enough reminiscing about those wonderful sleepless nights, I started to think about the sprints I've been involved with for my software projects. Not too many I'm happy to say, and I even wonder if the ones I was involved in were very effective, especially when it comes to quality of product.

Writing, fixing, and migrating code is not exactly something you want to do when you're tired, there's way too high of a risk of error and more problems as a result. We like to joke "Hey, don't drop the database by accident" when our programmers write scripts on the database, because someone did once, by accident of course and thankfully not on the live database. Making critical decisions about a project is not something you want to do when you're tired, it's too easy to get caught up in the moment and not be able to think clearly. System test and bug checking is not something you should do when you're tired, you don't have the focus you should have and will be too quick to rush through testing because you feel so close to the finish.

The times when my project team has had to do these types of sprints to get a project done have not ended in disaster, for the most part things were ok (except for that one time years ago when we realized at 2am Monday morning that we had to roll back an entire project and months of programming work, by the time people got into the office at 9). And even though it might have been a nice bonding experience, I can't say I love doing it this way. I would like to think that in an ideal world we won't be in the situation where we have to pull marathon work shifts to get the project out in time, and I'd like to say that in the interest of quality we should avoid this all-together.

Is it possible to always avoid? Probably not. I am fortunate to have a client who allows me flexibility most of the time with my launch dates, and I guess I should just be grateful for that!

"To Catch a Bug" - New Quality Assurance related post on Flightpath blog

This latest is nothing about adventures with contracters, but stuff about bugs that's been floating around my head for a while and wanted to get it all written down...

To Catch a Bug: Quality Assurance at all Stages of the Project

My Failed Project, and Lessons Learned

I'll probably come up with a couple more posts about my contracter before this is all over, he's finally back in touch and doing some work for us. It's always interesting having him around, especially when I can reflect on how work we do with him can relate to good or really bad project management.

This time I might as well take the blame.

We asked Fernando to install two wall unit air conditioners in each of our daughters small and stuffy rooms. For reasons I wont go into, window unit A/C's were not an option and fans weren't cooling down the rooms enough (nothing like getting woken up at 2am by a very sweaty baby, I felt terrible about it and we had to fix it as soon as possible).

So, what went wrong? Well, the air conditioners were cemented into the walls and put through to the back, but without a sleeve to hold them. So, two big problems here - the mechanics in the back of the A/C are exposed and not protected by the elements, and there is no easy way to remove the A/C if and when it breaks down. How did this big (expensive) mistake happen? A serious of bad decisions...

Let's start with the purchase of the appliances. My husband (who is a wonderful father and very intelligent and talented in his field of English Literature, but not exactly a handy fixit kindof guy) went to buy the wall units. Ideally we would have went online beforehand and picked out exactly what we wanted and done the research to help us figure out what accessories we needed, but we didn't.

Mistake #1: Lack of research & planning.

We rushed to get the A/Cs so that Fernando's guy could install them ASAP, and before the next brutal heatwave. Apparently wall units are sold without the sleeves, because they assume you might already have one in the wall. So, the sales guy was happy to sell us the A/C, but when my husband asked if he needed anything else, the guy said absolutely not.

Mistake #2: Bad communication and transfer of knowledge (assuming that the sales guy knew exactly what our situation was and would know enough to make the right recommendations).

So, we purchased two nice new wall unit A/Cs and brought them home (without the sleeve, of course). Fernando came by to look it over and raised the red flag. He said we bought the wrong thing, these were without a sleeve and we could not put them into the wall without one. I think it was morning when we came to this realization, and I was juggling getting the girls dressed and fed and getting my own stuff together so that I could get everyone where they needed to be and myself to work on time, and had not planned for this little hitch in my schedule. So, I wasn't focusing enough on the problem and trusted that Fernando would be able make everything ok. We looked through the box and took out all the pieces. I asked Fernando if there was any way to make it work, or what we should do. He said he'd figure something out. Sounded good to me... I dashed downstairs, fed the girls and got us all out the door.

Mistake #3: Rushing the schedule and not taking the time to properly reflect on the issue at hand and manage the new risks of the project.

So, Fernando's guy (who is an electrician by trade and not an A/C installation expert) started working on putting the holes in the wall. Within a couple of days the units were in the wall. The units were cemented back in, so no chance of anything sliding out. When I went to look at the work that was done, it was only then that I realized how important that sleeve was, because I saw the 3-4 inches of mechanics exposed in the back of the A/C. I asked Fernando if this was going to be a problem and he said only in the winter. So, he promised to get it covered by winter.

The next day my neighbors (one of whom is an expert in heating & air conditioning I have recently learned) raised about 10 red flags and told me that we needed to cover that A/C as soon as possible, before the first rain. So, now I have my contractor (who I've trusted for many years) telling me one thing and our neighbors (who we also trust) telling me another. I'm just using common sense here and figuring my neighbor is probably right. Another issue that my neighbor mentioned and I'm kicking myself for not realizing sooner is that if we ever needed to remove the unit, we'd have to break through all that concrete. Without a sleeve, there's no easy way to get the units out of the wall.

So, what was the end result? We cut our losses and decided that rather than remove both units from the wall now to put in a sleeve (and risk breaking the new units while separating from the concrete), we'd cover the exposed parts of the units and 5-10 years from now (hopefully not sooner) when they break down for good, we'll break the concrete and put the sleeves in.

I regret not spending more time to properly think the whole process through and expecting that each player would know exactly what to do. At this point I'm just keeping my fingers crossed that the units aren't damaged by the rain we've had in the last couple of weeks and we can atleast salvage our investment.

The bright side is that I learned some great lessons. I try to be super careful with planning, risk management and communication with the software projects I manage from 9 to 5 (I only wish it was from 9 to 5!) but maybe it's time to be more careful with the stuff that happens after 5.

If only I could work with my outside contracts the same way the NY DOT does

Anyone who's run a project with an outside vendor or freelance, contract worker, or third party application knows that it's best to plan alot of buffer into the schedule because anything that could go wrong, probably will. I've learned this the hard way a couple times, from communication breakdown, to technical difficulties, to technical difficulties caused by communication breakdown. It would be so much easier if everyone working on your project was working under the same roof, on the same schedules, with the same work style, etc.

But, if this were the case than project management wouldn't be so much fun!

We worked with an outside vendor on the west coast last year and the only way I was able to get them to respond to my email was to mark them high priorty with the red exclamations, and make sure all relavant parties were copied. Kindof annoying, I know...but it worked.

If only I could work with my outside contracts the same way that the street crew works with theirs...

So, its finally time for my street to get repaved. They've been putting up signs for the last week, letting us know when we can and can not park on the street. Yesterday they did the block west of me and this morning was time for my block. Now, the residents of my neighborhood who own cars and park them on my street are like outside 'contracters' who can have a direct impact the schedule & progress of the street paving project. At 7am this morning there were still a few cars parked on my block, so immediately the street paving project was delayed. Within a few minutes, we hear out the window, very loud announcements from a truck with a megaphone attached "Good Morning. Please remove your car from Chester Avenue. If you do not remove your car we will have your car towed and issue you a ticket."

What I love most is not the threat, but the way it was delivered. I happened to have been awake already because I have young kids who got me up at 5am, then again at 6:20am, but for those who have a bit more time to sleep in the morning, to be woken up by loud annoucements from the street is just beautiful. This reminds me of my camping days, when the evil counselors would yell at the kids or find some way to be really obnoxious to get them out of bed (of course I never did that as a counselor, I was very nice).

I looked out the window while the truck was going by, and saw a nice big smile on the driver's face. You know he was just loving this job, I bet he was once one of those evil camp counselors. If only I could use these same tactics to keep my outside vendors on schedule and communicating regularly. But, they probably wont ever want to work with me again...

(ps - this is not to say all outside contracts are bad, I happen to be working with one now who is totally on schedule and even more ahead of the game than the rest of us are...maybe he'd like to use the megaphone on us!)

Everything I learned about (what not to do in) Project Management, I Learned From My Contractor

Well, let me preface all this by saying that my contractor Fernando is the most wonderful person in the world, and we have worked with him for years now and (almost) never regretted it. He's bent over backwards for us when we really needed him, rushing to finish up our bathroom renovations in our old apartment so that when I was coming home from the hospitol with my first child I could have a finished bathroom to use. Then he came back late at night to fix the bathroom floor when I (in my drunken, sleepless new mother state) stepped on the still drying tiles and messed them up.

That said, if he was to be my mentor when I started out as a PM, I probably wouldn't be a PM anymore. Let's start from the beginning...

The Estimate
I called Fernando because I wanted to have him walk around the house with me and show him 4 things I wanted an estimate for. He stopped by and I showed him the 4 things needing work:
- installing a storm door
- fixing a leak & ceiling damage in a side room
- installing a cabinet & counter around a dishwasher (we couldnt afford the cabinets last year, just the dishwasher and wood box around it)
- making holes in the wall in my daughter's rooms and installing wall unit A/C's

He didn't write anything down, but I did (it was the PM in me nudging). I called him up 2 weeks later and he gave me my price. He gave me one number, no ranges, no conditions, no specifics about the deliverables, just the flat price. When I asked about specifics of the 4 deliverables, he broke down the work estimates for me. He didn't specify whether the price covered labor and materials (which burned me in the past because I assumed that the price covered materials and I was wrong). He didn't mention how the cost might change if the wall was harder to bust through then he expected (its a 100+ year old rowhouse, needing repairs), just gave me the flat fee.


I remember from my PMP class that this is one type of contract, which can be very beneficial to the buyer but can be dangerous to the seller.

The Schedule
Ok, so anyone who has a contracter knows that they are completely unreliable and if they show up 2 of the 5 days of the week then we should be grateful. Fernando and his team are the same way at times. So, I've stopped asking how long something will take, because I think it will just get done when it's done and I have no idea when that will be. He does good work, so in the end that's most important to me. Fernando doesn't exactly know about the triple constraint, but as long as he does good work I don't bug him about it (so much).



Change Control

..or lack thereof really. Fernando is so nice, he will let us take advantage of him sometimes. When he's in the house doing work, I'll say "Fernando, can you take a look at ___ or ___ ". And he will, and he'll help us out with other stuff, but what he really should be saying is "I can help with this, but it will have this impact on the schedule of the current work I am doing for you and will affect the cost of the labor by this." Not that I'm complaining really, but I know not to adopt this process when I manage my projects!



Revising Estimates in Project Execution

Last year we needed Fernando to redo all of our electrical wiring because it was more than 50 years old and in dire need of replacement. We got our price quote (7K) and his electrician started his work. This guy was just as unreliable as Fernando, and came maybe 2-3 days a week. When he was there, he worked pretty slow (not that I'm complaining so much, electrical stuff should be done carefully!). The entire work effort took about 5 months, and throughout the project the electrician asked for installments of the 7K because he was running low on supplies. It seemed to me that all of the money was going to supplies, what was going to labor? They never asked me for more than 7K, but it started to seem like they severaly underestimated the work and were totally losing money on the job. Maybe this is why they only came half-time, they were putting their time into more profitable jobs. In hindsight, I kindof wish that Fernando would have come back to me and said its going to take 12K and this will ensure the project will complete more quickly and efficiently, and my employees will get the compensation they deserve. Not that I love throwing money into projects, but that's what home equity loans are for!



Fernando's business is thriving, he does great work and he's always got projects lined up. So, obviously he must be doing something right. We probably shouldn't expect our contracters

to follow PMBok, but maybe some happy place in between might be ok.



Maybe it's just my contractor, who knows!

Quick Decisions, in a Software Release and on the Staten Island Expressway

Caution: If you have a weak stomach and don’t like reading about baby puke and quick project decisions, you might not want to continue reading.

So, I guess my 18 month old daughter doesn’t like my driving. Since we don’t own a car and just rent a few times a year, her body probably isn’t prepared for the shock of it all. Regardless of how and why it happened, she had an unfortunate up-chuck while we were just slowing down in a huge mass of weekend traffic on our way back through Staten Island this past Sunday night. It was quite the event too… anyone who’s seen Monty Python’s “Life of Brian” can picture a similar scene. So, we’re stuck in the car with a wailing puke-stained baby and a 3 year old who keeps yelling, “stop crying, you make me sad!”

What to do? Do we try to pull over on the lack of a shoulder lane and try to change her into clean clothes? Do we pull off at an exit and see if we can find somewhere to stop and clean her up? Do we just try to keep driving and see how quickly we can get back to Brooklyn (granted, this is New York traffic on a Sunday night in June, seems like everyone left the city for the weekend and they’re all coming back now).

As I was weighing my options, I was reminded of similar situations that I’m in when I am managing software releases for my company. (Ok, so maybe I really didn’t drift off to such thoughts while in such a stressful predicament in the car, but it sounds funny so I’ll tell the story this way).

Our last large new feature launch was in the beginning of May, and we do regular bug releases and small feature launches. Inevitably things will go wrong, regardless of the amount of testing we do while on our QA and staging sites, there’s always something we find when we put the new work live. This can cause varying levels of panic, especially when it’s late at night and our project team wants to go home, or its early morning and we’re trying to finish up before our clients come into the office and want to start using their websites. When these things happen, we have a few different options –

1. Put a programmer on it ASAP and see if this new issue can be resolved and tested quickly and hope that no new related problems are created by the changes.

2. Roll-back the entire release and try again the next day, after spending time working on the new issue and resolving on the test sites.

3. Leave the issue on the live site, make the client aware of the issue and work on getting the fix thoroughly tested and live within the next couple of days.

It all depends on the severity of the issue, but what I can always count on is a project team looking to me to make the decision. I don’t always like making quick decisions, but I know it’s my job in these situations and I know my team is looking to me to make the call. It’s almost funny how much they count on me, on different occasions the programmers or the QA team and even the more senior members of the staff have shied away from making crucial decisions when it comes to software releases and potential impact on our clients. It’s as if nobody wants to be responsible if the decision made sends us into the deep dark sea, they’d prefer I be responsible and as the manager go down with my sinking ship. So far so good…we haven’t sunk yet. I wouldn’t call myself caption of the year, but we’ve been floating along somehow. This part of the job can keep things exciting, and terrifying at the same time.

So, what did we do with the sad sick baby in the car? We kept on driving, opened the windows up all the way to keep the smell from making me sick myself, and gave her a bottle of water. The bottle actually made her very happy, and within maybe 20 minutes she was calm and maybe even forgot that she was covered with slop. We figured since it was already late and she was tired, better to try to just get her home than risk getting stuck in an even worse crowd of traffic. We eventually got out of Staten Island and were able to get home to Brooklyn to get everyone hosed down. I was glad my husband and I (yes, he helped in the decision making, too) had decided not to stop because once I had started to clean the crime scene I found that it was a lot worse than her clothes, even in clean clothes she would have been sitting in a stink and gotten dirty all over again. Today she’s clean and happy and hopefully has forgotten the whole thing.

And can I even try to remember the show-stopping bug that plagued my project team the last time we did a release? Barely…

The limits of quality, in a crane

I walked past a crane yesterday. I have to say, being New Yorker and hearing a little too regularly about the construction accidents that have been happening, makes me not want to walk so close to those cranes.

There were a few construction accidents earlier this spring, one of them involved a crane that was improperly secured to a building, falling onto a neighboring building. Very recently, a crane on 1st Ave and 91st Street collapsed onto a 23 story building and killed the crane operator.

Many critics will say that the problem is that there is too much construction going on and too much rushing to get the projects done. Seems to make sense to me, especially after reading about the repeated complaints and stop-work orders on that crane that collapsed most recently. If there was not such an urgency to get these new high rises up, then maybe the project could have stopped and the crane properly evaluated and replaced.

I've been especially sensitive to these types of issues lately because I've been reading the very interesting book titled "The Challenger Launch Decision: Risky Technology, Culture, and Deviance at NASA", by Diane Vaughan. It's a slow but very interesting read, and from what I've got so far it's about the 'production' pressures faced in the organization and the 'amoral' judgement and behavior used by the managers there. In both of these cases, the team just wasn't cautious enough, or chose to ignore the warning signs.

What stunned me was the statement New York Mayer Bloomberg made after the accident, something like (not exact quote) "there are only so many inspections that can be made and so many inspectors, sometimes problems aren't caught." When it comes to ensuring the quality of your system when the failure of that system can mean loss of human life, then the warnings have to be taken extremely seriously. I'm not exactly an insider in the construction world, but it looks like the warning signs were not taken seriously enough.

In my software projects we take quality assurance and control seriously and are trying to find ways to improve what we're doing.

Do we miss things?
Of course!

Do we put new bugs into the system when we launch a new project, I think I've got a perfect record on that one, there's always something we have to fix post launch. I'm happy to be in the field where a bug in my system will not cause the loss of human life. I guess I expect a little more from the more dangerous fields of work out there.