Why is software so hard? Why do projects routinely run over budget, over time, and require more people and hardware than forecast? Why is software development so far behind the other engineering disciplines when it comes to projects and budgeting? These are common questions for anybody working in or with the software industry.
Programmers will usually talk about complexity, changing requirements, mis-management, or pace of technology. While these are all real issues about why writing software is so hard and time consuming, it doesn't necessarily address why we are so often mistaken about the amount of effort a given project requires. Even being fully aware of the afore mentioned problems, most programmers will still give you a rosy eyed estimate that will be woefully off target (and almost always in the under estimating of the time and resources required for a given task).
I have been working at this for over 10 years now, and I am sad to say I don't feel any better at this than when I started. All I have learned is a healthy respect for the amount of time and effort those little boxes "encapsulate", and a near phobia of commitment to estimates of how long any given "piece" will take.
Central to the issue is our biological need to feel that we understand a situation in order that we might act. It seems a deep seeded reality that we are wired to generalize, categorize, and abstract messy reality so that we can make some forward progress in this crazy world. We are masters of this, and whisk away ugly detail with ease, boxing up thorny problems with the stroke of a dry erase marker creating orderly corrals where our shiny new abstractions live in words like "storage", "service", and "analysis".
When you are dealing with tigers jumping out of the woods at you or a fist speeding towards your face, this ability to take a messy situation and make it simple is a very useful trick. While pontificating about the underlying motives of the fist wielder or determining the exact species of tiger hurtling toward you may be all excellent strategies for handling the situation if time were no object, in our reality they just get you dead. In our reality, over estimating the amount of knowledge you posses in order to more quickly come to a decision can be a huge advantage when time critical situation are on the menu. In fact, this was the usual case until fairly recently in human history. It's no wonder that we are geared to handle these types of situations given the frequency and risk/reward of each type.
Unfortunately, this can leave us arrogant yet delusional, which can be a pretty nasty combo. Our tendency is to quickly feel like we understand a situation and are ready to act on it, while in reality we have very little understanding of the actual problem. This leads us to over estimation of our understanding of the factors involved which leads to an under estimation of the time involved (since we have to not only implement what didn't understand but also figure it out). This combined with the inconvenient tendency of reality to be more fractile than linear in nature means that things tend to get more complicated as you investigate them. The devil's in the details as they say.
For a much better account of our epistemic arrogance as it relates to many interesting (non-software) issues, check out Nassim Taleb's excellent book The Black Swan.
