Custom tech can be a pitfall

"I was actually alarmed by the amount of custom technology we procure," Estes said.

Although not identified as a central issue in audits of problematic programs such as NCTracks, the effort and expertise needed to design and maintain custom-built systems is something the state needs to consider going forward, according to Estes and other experts. 

Most computer users can find an "off-the-shelf" program to do what they need. There's not need, for example, to build a custom word processor when Microsoft Word or another ready-made product can fill the need. 

But governments often need computers to handle big tasks that aren't common in the private sectors, such as processing Medicaid claims or evaluating whether someone is eligible for food stamps. 

However, governments aren't alone in their need to process large amounts of data securely, and more and more big private sector players are trying to find off-the-shelf solutions.

"They understand, it's not only the initial cost to implement it, but it's the ongoing cost to upgrade it," Estes said. "Once you customize a piece of software, you are forever customizing it."

This customization problem is something that North Carolina's thornier IT projects share with problematic systems in other states and at the federal level. Healthcare.gov, the problem-plagued interface for the Affordable Care Act, for example, was custom built for the federal government.

Estes said he has been grappling for the past year with why governments rely so heavily on custom-built features.

"Part of it is, sometimes the law gets so specific about the requirements is there's no technology that can do what the law says. So we can't buy an off-the-shelf solution like a private company might buy, because we have to comply with the law," Estes said.

In the future, he said, lawmakers may want to consider the IT implications of the laws they are working on before those requirements are handed down to state agencies.

Estes isn't alone in pointing to the complexities of designing and maintaining big, custom-built IT systems as a contributing factor to cost overruns and implementation delays. 

State Auditor Beth Wood has questioned whether the state can keep the necessary talent on staff to maintain complex computer systems.

"I'm not sure the state of North Carolina, with the salaries that we pay, has the necessary people on board that can see everything that needs to be seen for these more complex systems," Wood said.

Still, sticking with pre-crafted software doesn't guarantee a problem-free experience. Case-in-point, Home Base, the constellation of systems rolled out by the N.C. Department of Public Instruction to manage grade reporting, teacher evaluation and other student information, was basically an off-the-shelf product produced by the Pearson School Systems using technology developed by big-name manufacturers like Oracle.

At point this year, Oracle updated its Java software in a way that didn't jibe with the software put together by Pearson. Since most school systems automatically receive updates from Oracle, the change caused a widespread system outage.

"It shut down everything," said Phillip Price, the chief financial officer in the Department of Public Instruction and the man responsible for managing Home Base's roll out.

Price, like others managing big IT projects before him, has learned that even the best IT roll out is going to come with problems and glitches. For DPI, those normal growing pains were aggravated by serving 115 local school districts and 127 charter schools, which use Home Base in slightly different ways and need it to produce different reports. 

Gathering, tracking and solving these issues is a logistical challenge like few others in state government – and one for which the state could have better prepared, Price said. 

"If I had this to do over, I would have hired a firm, a corporate entity, that does support centers," said Price said. "I would have hired them for three to six months of implementation, to make sure that we had techniques and information at the proper level."