Ecommerce and Entrepreneurship Blog | About | Contact | Store

Recovering from a Serious Website Error: Four Simple Steps

Posted on April 29, 2009 by Trevor

Oh no. It's happened. I was making a fairly ordinary update to one of our eCommerce websites. I had tested it pretty thoroughly and gone through the whole procedure of preparing it for launch. I held my breath and launched it-- and got an error screen. I refreshed the site homepage, and my heart sunk. The site, the entire site, was broken. We lose literally thousands of dollars an hour when the site is down, so this was a huge deal.

As a developer, what do you do when something goes very, very wrong? You're not expecting it: you may have prepared for it but you didn't believe it would happen. And now it's staring you in the face. And depending on how you handle it, you can either recover quickly with minimal losses or you can flounder and make it even worse. Here are a couple of points I've learned that can help avoid the latter.

  1. Calm Down - I can't tell you to stay calm, because you probably lost that when things started going wrong. But you can calm yourself, and you're going to need to if you are to respond properly. Rash mistakes can easily compound the problem, making it many times harder or even impossible to recover from. So take five seconds, sit back, breathe, and tell yourself, "OK, this is bad, but panicking would only make it worse. So I'm going to be careful and do things right." Then just follow through, solving the problem as you would any other. The urgency of the situation is useless in helping you solve it, so forget about it. You'll be more likely to see solutions you otherwise wouldn't, and less likely to miss further dangers.

  2. Be Honest - Your first impulse may be to try to cover up any mistakes you made or anything that could lead blame to fall on you. However, your first priority is to fix whatever's gone wrong, and that often runs directly counter to that impulse. The first step to solving a problem is figuring out exactly what the problem is, and that can't happen if everyone involved is afraid to release any information in case it might inculpate them. Remember that you need to be honest with yourself, as well. Don't try to convince yourself that it wasn't your fault or it's not that bad if it was or is. This is also a management issue: managers should make it clear in their policies and behavior that, while subpar performance isn't acceptable, no one will be punished or scapegoated for owning up to their mistakes (my own managers were very understanding!).

  3. Look Forward - Computers are (usually) programmed to obey logical rules. They perform the task given to them, exactly as it is given to them. They don't worry about things like pressure or situation. Humans, however, aren't made that way. We have emotions and operate, more often than not, on general heuristics and hunches rather than logic. This is good in many cases; we can do more with less than computers. But in certain situations it's a problem, and one of these comes up often in crisis situations. One of our rules of thumb that we tend to follow is that if we've already invested ourselves in a certain course of action, we try to protect that investment and, if necessary, follow it with more. Unfortunately, if it's a bad investment, we end up "throwing good money after bad", losing more and more while becoming more and more attached to one course of action. This shows up in crisis situations when we get fixated on one course of action, trying more and more variations on the same ineffective theme. We can even end up stupidly repeating the same action over again, hoping that this time it'll work. Instead of doing this, try to forget about what you've already done and think instead about where you want to go. When making a change to a website, for example, I might be tempted to think, "Well, since we've already put the changes on the site, we should try to fix those changes." However, my immediate priority is to get the site working; if it's faster to do that by undoing all my changes and going back to the original version of the website, that ought to be my course of action instead.

  4. Be Lazy - This one may surprise you. Being lazy isn't usually thought of as a good quality, and of course I don't mean that you should be slow and unwilling to do the work needed to fix things. However, there is a place for laziness in being smart about your time and effort. Let's take an example to clarify. Suppose I've got a problem that has two possible solutions. There's a 90% chance that I need to undo all my work, figure out what the problem is, fix it, and put it back in place. However, there's a 10% chance that all I have to do is re-apply my change because it got corrupted the first time. My initial impulse might be to go the undo-and-fix route. However, if I'm smart and lazy, I could realize that I could at least try the other option. Even though it's less likely to fix the problem, it takes an insignificant amount of time and energy. That way I'm more likely, in total, to get the problem fixed sooner.

That last point actually turned out to be the case when the site went down on me. While I was starting a large roll-back to a backup version of the site which would require a lot of cleanup, someone else suggested simply restarting the site on the server. That turned out to be the problem, and it was back up with a minimum of downtime and cleanup work. While I learned that point the hard way, I plan to keep all of these in mind in the future.

Kohler is arguably one of the most innovative brands in the home improvement industry. The new Karbon faucet has completely transformed the kitchen and more specifically revolutionized the kitchen faucet. Meanwhile Kohler seems to effortlessly create bathroom fixtures that are not only sleek but save water, like the Escale toilet.

blog comments powered by Disqus