As a consultant I’ve had the opportunity to see first-hand how software development works at some of the most cutting-edge .Net shops in the Denver area. One of the things I’ve seen over and over is a phenomenon I call “Smart Guy Disease”.
The basic concept is this. The biggest, most destructive mistakes are always made by the “smart” programmers, not the “dumb” programmers. The smarter the programmer, the greater the capacity for catastrophe.
It seems counter-intuitive, but it’s true. Think about it for a minute. “Dumb” programmers…. actually I have to stop a moment because I don’t really like the term dumb programmers. One of the reasons I like software development is that I consistently get to work with smart people. Even programmers who aren’t up on the latest technologies and are just showing up to do their job every day tend to be very intelligent people. However, I think we all have worked with people who just weren’t up to the task. They consistently ship code rife with bugs, you have to code review every line they commit with a fine tooth comb, they just aren’t very good at programming and they consistently produce bad code. Bad programmers might be a better term, but I’ll keep using “dumb” because it contrasts well with smart.
Anyway, “dumb” programmers do produce crappy code, but they almost never make the kind of massive mistakes that cripple an entire organization’s ability to ship software. They may implement features that perform poorly, but they don’t deploy code that destroys performance on a global scale. Nope, when you want to cause that kind of damage, you need a smart guy.
I’ve seen it. It happens. The smartest guys in an organization are the ones that make the truly big mistakes.
Shipping architecture instead of software
The first mistake I’ve seen is shipping architecture instead of software. Companies take the smartest guys in the organization and tell them that they’re architects now and they are no longer responsible for shipping an actual application. Instead they will create a company-wide enterprise architecture, or Core Framework, that other app development teams, who do ship production applications, will be required to build on top of.
I’ll go into more detail on why this is a bad idea in a later post, but for now just know that in this situation the incentives of the architects and the app developers are not aligned. The architects’ priority is to create architecture, something that implements cutting edge technologies, looks great in diagrams, is impressive when presented to C level executives, and if it’s overly complex… well that just demonstrates how smart they are. The app development team’s priority is to produce code that performs well, is easy to maintain, and ships on time. The architects really have no idea if their designs are easy to implement or perform well until they are implemented by the app team. Not a good way to ship code.
Overly complex solutions
Another common problem, that I alluded to above, is overly complex solutions. You might be tempted to dismiss this one as no big deal. Don’t. This is probably the most common and most damaging mistake that I’ve seen in the real world. Smart programmers usually like to show how smart they are. You can’t really do that if you produce code that is easy to understand and maintain. Why implement a simple solution when you can do something clever instead?
So what’s the cost of extra complexity? If you look only at the first 9 months of an application’s lifecycle then the answer is “not much”. However, if your application is successful and has a lifecycle that extends beyond the first year then that extra complexity becomes a tax that you pay over and over, every time someone touches that code base. That’s the real problem. The price for additional complexity isn’t just paid by that initial developer, it’s paid again and again by every developer who touches that code for the entire lifetime of the application. It’s far too common for smart developers to ship a new product with needlessly complex architectures, then declare the project a success and move on, leaving a maintenance nightmare for the “average” developers who come after them.
Complexity also leads to brittle code. We all understand intuitively that the more complex something is, the more likely something will go wrong and it will break. That’s bad enough by itself. Now think a year down the road when that complex system is being modified by a developer who doesn’t necessarily understand the intent of the original “smart” programmer. Now you’ve got a high probability that something will get broken.
The crazy thing is that teams sometimes look at the complexity of their code as a badge of honor. If you ever catch yourself saying something like “We are so cutting edge that we can only hire the best developers, and even then it takes them a good 4 months before they’re productive with our architecture”…. maybe you should ask yourself if that’s really a good thing.
Smart on paper, disaster in reality
One last mistake, solutions that look great on paper but are a complete disaster when implemented in the real world. The best example I’ve seen of this was when a company decided that they wanted to rewrite their ASP.Net WebForms applications so that they could be managed with a 3rd party CMS. The idea was that the WebForms pages were mostly made up of .ascx controls anyway, so it should be possible to make every .ascx responsible for loading and saving it’s own data, then publish that data to and coordinate with the other controls on the page at runtime. These individual controls could then be composited together in any combination by designers or business people using the CMS. There was some kind of limitation with the CMS that required all of the heavy lifting to be done in the browser with JavaScript and Ajax. Did I mention this was WebForms code with update panels and all that nonsense. Remember how fun it was to do real JavaScript and manage DOM Ids browser-side with update panels?
To be fair, it’s been a while and I don’t really remember the details. I just remember that the big brains decided this made sense, and I’m sure it did make sense in a high-level planning meeting. But, every hands-on-keyboard developer who worked with those apps immediately recognized that this was a terrible idea. Every developer who actually shipped code could see that even if a miracle happened and this idea worked, it would make the code much, much harder to maintain and it would slow development to a crawl. It did by the way.
How to avoid smart guy disease
So what’s the point? Is this just a rant because I hate smart people? No of course not. I like smart people and I like working with smart people. The point is that it’s possible, even common, for the smartest developers, who should be our greatest asset, to instead create massive problems. But it doesn’t have to be that way.
I have 2 simple rules that can help prevent smart guy disease from crippling your organization.
Rule 1. Make sure your smartest developers are always in a role where they are shipping production code.
Most of the problems mentioned above happened when the smartest developers became disconnected from the reality of building and maintaining production software. That disconnect messes up the incentives and puts your smartest programmers at odds with the programmers who actually ship code.
Instead, keep your best and brightest developers embedded in teams that are shipping production applications. Let them be a team leader or a team member. Give each application team their own architect. Or, if they can’t be a permanent team member, at least let them be a temporary resource embedded inside the team. They can work hands-on-keyboard with the other developers to actually implement their architectural designs. That kind of hands on experience will help everyone involved and will keep those incentives aligned.
Rule 2. Value simplicity.
Make simplicity a cultural value, a goal that every developer strives to attain. Make it clear that simple solutions are valued and needlessly complex solutions won’t be tolerated. Keep a close eye on any developers who want to use multiple layers of base classes, Unity Interceptors, or other techniques that are used to make things happen automagically.
Never, ever boast that your architecture is so complex that only the very best developers can work on it. Instead your boast should be that your architecture is so well designed that any junior developer can come in off the street, understand the basics in a day or two, and be productive within the week.
