Assigning Blame 101

It soon became clear that the performance problem was caused by inept and poorly executed design. The database was multitiered but with so many convoluted tiers that it brought tears to the consultant’s eyes. Fred submitted a comprehensive review of the system and included a number of suggestions to get the DBMS (database management system) back on track.

It seems that Fred’s advice was already doomed—the company had decided to switch from SQL Server to Oracle. Apparently, some decision-maker had been told Oracle was far faster than SQL Server and would easily solve all of the company’s performance problems. Before Fred could take another breath, he was informed that the Oracle consultant who made these claims didn’t know how to write T SQL (SQL Server’s unique SQL language), and the company wanted Fred to teach it to him.

Roll the clock forward several years. What would you expect to be the current state of this megacompany’s DBMS? Personally, I would expect it might actually be better. Of course, this assumes that the Oracle consultant (who didn’t know T SQL) was eventually replaced with one that had more skill and that this new consultant took a careful look at the design, figured out what was wrong, and spent the next six months fixing it using SQL Server or the next two years redesigning the DBMS to match Oracle’s strengths. On the other hand, the DBMS performance and stability would more likely be worse, or at least no better, given the way management had been able to fathom Fred’s advice.

Porting a poorly designed DBMS from one vendor to another rarely (if ever) improves the design and even more rarely improves performance.

Who Should Take The Blame?

But who is to blame here for the poor performance? I think there is plenty of blame to go around. First, the original design (and thus the original DBMS architect) might (only might) be to blame. Multitiered designs can scale better, but they don’t necessarily perform better.

However, if the concept and design were brilliant but the implementation of this design was flawed or clumsy, then a portion (only a portion) of the blame can be shared with the development team and the development team’s management. Management can’t just lay down timelines, hire a staff, and let staff members do their thing without staying involved with the day-to-day details in how the design is being implemented. Consider that the person writing the code usually is doing the best she can. Developers might be in over their head and fail to recognize how other, more efficient approaches can be implemented. This isn’t stupidity; it’s ignorance—a common malady when the architectures, tools, and infrastructure change every few years.

Blame ADO Or The Database?

Getting back to the original newsgroup thread, another consultant whom I respect said the most common factor in performance problems was poorly written data access (ADO) code. That is, he saw too many developers making very common mistakes: fetching entire tables, accessing tables instead of stored procedures, overloading deployed client applications with schema-dependent code, and using other sloppy coding techniques that many of us have been railing against for decades.

Another consultant in the thread, whom I also respect, disagreed. She said that the most common performance bottleneck was poor index design, poor query design, and other database-centric issues.

Personally, I think they’re both 100% right. In a good design review, ADO-skilled developers see data access issues and DBMS-skill developers (and DBAs [database administrators]) see DBMS issues. Unfortunately, there are precious few consultants that are highly skilled in both disciplines, and we can hardly afford to hire those who are. Your performance problems might be the cause of several implementation teams who fail to work together as a team.

How do you protect yourself when, years down the road, you discover that the DBMS you chose is not any better than the current choice (perhaps the one chosen by the IT manager’s son-in-law)? IMHO, it makes sense for decision makers and managers to listen openly to more than one point of view, hire a team you can trust and really knows your business issues, and bring in trainers (a good consultant is really a trainer) that can help your team implement the design as specified.

Of course, this assumes your architect is disciplined and skilled enough to write a comprehensive specification before a line of code is written. It also assumes that management and the decision makers up and down the line are not open for output only; they listen to the advice they’re paying for.