The road to a CIO position typically goes through application development, and my journey is no exception. Before my time at the State of Nebraska, I began working in Operations. Then, I moved to Infrastructure, where I spent the majority of my foundational years. Regardless of where my journey led, whether in private or public organizations, the customers share a common presumption about their ailing or newly implemented systems, throwing more computing power at an issue, will resolve it.
This band-aide treatment, of course, only masks the problem. In time, the band-aide always falls off, and the situation becomes more costly and challenging to resolve the next time it rears its ugly head.
I have read other articles that say it makes sense to buy the hardware. The logic is economically founded that “developers are expensive and hardware is cheap.” The cost of hardware is only a fraction of the overall cost when factoring supporting infrastructure, storage, backups, and support costs for hardware. Looking at the specs of modern hardware, it is tempting to go all-in. However, if you do, you run the risk of robbing the development team to pay the infrastructure and operations teams.
New hardware offers immediate scalability, but while scaling your infrastructure is quick to do, scaling your support staff is not so easy. The right support team will be proficiently skilled in specific areas and have the required training on your system, but the additional support costs will quickly make this move more costly. Throwing hardware at a performance problem is not a substitute for developing the right team to design your product in the first place.
Next, apply ‘Technical Debt.’ A poorly designed application acquires technical debt, like credit interest. Here is how it happens: The development team releases a poorly designed application because the programmers took shortcuts to hit deadlines and/ or budget constraints. The developers did not have sufficient time to address the minor performance issues. Over time, the team fails to address them, even if they intend to provide remediation in a future release. Ultimately, minor flaws become expensive problems, and no hardware can dig you out of the hole. The compounded problem is your technical debt.
Technical debt has other causes, but I see it commonly in situations where developers are under pressure to build additional functionality into an application within a concise amount of time. In these situations, the developers find that requirements lack definition, or the customers left something out during the requirement-gathering phase.
I believe any development team can help customers minimize or eliminate their technical debt by following three simple rules:
1. Resist the temptation to call a project complete. When the team discovers minor coding issues, do not ignore them or mask them with additional processing power. Take the time to address the code before it becomes impossible to handle. The sooner, the better.
2. Identify precisely where the performance problems are and scrub only the identified areas to make an immediate impact. Then rinse and repeat. In my experience, the problem is often inefficient interaction with a database or a poorly optimized database; nested loops initiating queries that initiate other questions; excessive table scans; etc.
3. The changes that you are making must ultimately result in lower rates. The customer must see this as a long-term win. It makes no sense to spend expensive development time with very little payback. Ensure there are significant gains in performance before you begin. Never optimize beyond your stated performance goals. Know when it is time to stop and move on.
Along the journey, a technology leader should educate decision-makers to invest their resources appropriately and make the right judgment calls for their organizations’ best interests. As technology leaders, we can be accountable to our product owners by educating them on the actual cost of hardware. Where I stand in my journey, I intend to help the State of Nebraska to make decisions that reduce hardware processing costs, infrastructure costs, and, therefore the State taxpayers’ total cost of ownership.