Uncertainty, Arrogance, and Software

Why is software so hard? Why do projects routinely run over budget, over time, and require more people and hardware than forecast? Why is software development so far behind the other engineering disciplines when it comes to projects and budgeting? These are common questions for anybody working in or with the software industry.

Programmers will usually talk about complexity, changing requirements, mis-management, or pace of technology. While these are all real issues about why writing software is so hard and time consuming, it doesn't necessarily address why we are so often mistaken about the amount of effort a given project requires. Even being fully aware of the afore mentioned problems, most programmers will still give you a rosy eyed estimate that will be woefully off target (and almost always in the under estimating of the time and resources required for a given task).

I have been working at this for over 10 years now, and I am sad to say I don't feel any better at this than when I started. All I have learned is a healthy respect for the amount of time and effort those little boxes "encapsulate", and a near phobia of commitment to estimates of how long any given "piece" will take.

Central to the issue is our biological need to feel that we understand a situation in order that we might act. It seems a deep seeded reality that we are wired to generalize, categorize, and abstract messy reality so that we can make some forward progress in this crazy world. We are masters of this, and whisk away ugly detail with ease, boxing up thorny problems with the stroke of a dry erase marker creating orderly corrals where our shiny new abstractions live in words like "storage", "service", and "analysis".

When you are dealing with tigers jumping out of the woods at you or a fist speeding towards your face, this ability to take a messy situation and make it simple is a very useful trick. While pontificating about the underlying motives of the fist wielder or determining the exact species of tiger hurtling toward you may be all excellent strategies for handling the situation if time were no object, in our reality they just get you dead. In our reality, over estimating the amount of knowledge you posses in order to more quickly come to a decision can be a huge advantage when time critical situation are on the menu. In fact, this was the usual case until fairly recently in human history. It's no wonder that we are geared to handle these types of situations given the frequency and risk/reward of each type.

Unfortunately, this can leave us arrogant yet delusional, which can be a pretty nasty combo. Our tendency is to quickly feel like we understand a situation and are ready to act on it, while in reality we have very little understanding of the actual problem. This leads us to over estimation of our understanding of the factors involved which leads to an under estimation of the time involved (since we have to not only implement what didn't understand but also figure it out). This combined with the inconvenient tendency of reality to be more fractile than linear in nature means that things tend to get more complicated as you investigate them. The devil's in the details as they say.

For a much better account of our epistemic arrogance as it relates to many interesting (non-software) issues, check out Nassim Taleb's excellent book The Black Swan.

Google App Engine

If you haven't checked out google app engine yet you yeally need to. I wrote my first facebook app backed by app engine. The barrier to entry has now actually dropped to zero. It is amazing: marketing through facebook fiends is free, the host and database are now free, and all the tools required to write each piece are free. I look forward to seeing what comes out of this emerging platform.

Nuclear Power and the Cost of War

While doing a little BSing with a friend on IM about the costs and merit of war for oil, started wondering what it would look like if we instead spent this money on nuclear power plants. Now perhaps this isn't exactly your favored solution, but relax, I don't make policy, and this is just a thought experiment.

According to the ever popular Wikipedia we in the US consumed 29000 TWh of juice in 2005, equivalent to an (average) consumption rate of 3.3 TW. Likewise we are informed that the capital costs of a new plant are in the 5,000-6,000$ range per kilowatt, or lets say 5.5$/w capacity. Now, in our little war on Iraq we are likely spending around 4 trillion dollars (see past posts for references). A little math reveals that would buy us $4,000,000,000,000/$5.5 * 1w ~ 730GW of capacity. Given we consume around 3,300GW of energy, that would be about 730/3,300*100 = 22% of our needs. Now obviously we would have to pay to run the plants, guard them, and dispose of the wastes, I will lead it up to the reader to figure out those costs as I need some sleep now.

In summary, we could crank out a lot of nuclear power plants for the cost of this war for oil. Food for thought.

Objectivism Revisited

Rereading Atlas Shrugged for the second time, and was struck by this paragraph where Francisco is talking to Dagny:

"Look around you," he said, "A city is the frozen shape of human courage--the courage of those men who thought for the first time of every bolt, rivet and power generator that went to make it. The courage to say, not 'It seems to me,' but 'It is'--and to stake one's life on one's judgment. You're not alone. Those men exist. They have always existed. There was a time when human beings crouched in caves, at the mercy of any pestilence and any storm. Could men such as those on your Board of Directors have brought them out of the cave and up to this?"

- Page 510 in the first Plume printing 1999

I have been a pretty hard nosed skeptic for some time, and have often debated with people about what one can and cannot know (in the epistemological sense). Having agreed with Descartes in his Discourse on Method only as far as "I think therefore I am", suffice to say I don't think we can truly know much of anything.

Yet, we live in a seemingly real-time universe where we exist and function to some degree or another. Each thought, each decision leads to some action that we take in the praxeological sense. There is no way for the skeptic to hide from action indefinitely, and in fact, what little time there is really gone in the blink of an eye. For I can decide that I either have sufficient knowledge to start or continue an undertaking or I do not, but there is no escaping the decision to act. Either I build a bridge, create a piece of software, invent a new model of cognition or I do not. Either way, I have made my choice, and that choice results in inescapable action without the luxury of assured knowledge or truth.

I have never really dug in to see if I buy Objectivism or not, and while I certainly do agree with many of the resulting conclusions, I have been quite reluctant to accept the main pretexts on which the philosophy seems to be grounded. But if in this paragraph, and in the resulting thoughts it brings to mind, the underpinnings are to be found, then perhaps I need to reconsider what it means to be an Objectivist, and if I consider myself one.