@NCCapitol

@NCCapitol

For state-funded technology, failure a likely option

Posted December 18, 2013
Updated December 19, 2013

— Over the last seven years, thousands of children have passed through the exam rooms of Dr. Albertina Smith's office off Capital Boulevard.

The evidence is all over the walls.

From floor to ceiling, Sunshine Pediatrics is plastered with hundreds of drawings in crayon and marker. They represent just a fraction of Smith's 3,800 patients, 90 percent of whom are on Medicaid.

Educated in North Carolina, Smith attended school through a scholarship program that encouraged the care of needy patients. She's taken a special interest in caring for families on Medicaid, the joint federal-state program that provides health care for the poor and disabled.

 Troubled Tech

Wednesday's story is the first of a two-part series on the systemic problems with state technology projects. Read part two for a look at how state leaders could fix North Carolina's ailing process for acquiring software.

"For the past seven years, it's been good for me," Smith said. "In fact, because I had so little capital when I started the practice, it was easy to start because Medicaid is typically very reliable when it comes to payment."

When the state flipped the switch on its new Medicaid system in July, it created a big problem for Smith – about 90 percent of her payments just stopped coming.

"In my mind, the beginning of July was like 'Apocalypse Now,'" Smith said. "As far as I was concerned, my revenue came to a halt."

She wasn't alone.

DHHS Secretary Aldona Wos and Chief Information Officer Joe Cooper

Over the last few months, providers and lawmakers have demanded answers from the state Department of Health and Human Services about the rocky roll-out of NCTracks, a massive technology revamp meant to streamline reimbursement and make it easier for policy-makers to track often erratic Medicaid costs. As state officials have noted in legislative testimony and scathing audits, the implementation has had its share of problems.

But technology experts say the deeper issues with the state's new Medicaid system go back several governors and DHHS secretaries to the start of the contract process. Bad estimates, poorly written contracts and lax oversight that cost taxpayers millions of dollars are among the root causes. Experts say the issues with NCTracks are reflective of broader problems plaguing the purchase and development of big technology projects for every state government in the country, including North Carolina.

"These projects are huge. They're very complex," said State Auditor Beth Wood, whose office examined the issue of state technology in an April report. "I'm not sure that, on the front end, we've taken the time or that we have the talent to plan a huge project the magnitude of NCTracks properly."

IT failure a widespread problem

Formula for failure

A Standish Group analysis of 1,741 large state technology projects from 2003 to 2012 found that only about 12 percent come in on time and on budget.

ResolutionTotalPercentage
Successful21112.1%
Challenged1,02658.9%
Failed50428.9%

According to data from the Standish Group, a Boston-based IT consulting firm that tracks the status of more than 50,000 technology projects across the country, the vast majority of state government IT initiatives in the U.S. either fail completely or come in significantly over budget and behind schedule.

From 2003 to 2012, data analyzed by Standish for WRAL News show that only about 12 percent of large state government projects are successful – in other words, on time and on budget.

An eight-year project in Florida to implement a child welfare information system scheduled for launch in 1998, for example, was delayed until 2005. When it was done, according to a Standish report, the final cost was $230 million, nearly seven times the original estimated budget.

In North Carolina, challenges like these mean big business.

Across all state agencies, 116 technology initiatives are now in various stages of development, according to state Chief Information Officer Chris Estes. That portfolio represents a total of about $1.6 billion, a mix of state and federal taxpayer dollars.

An April audit by Wood's office of the 84 trackable state technology projects found they took an average of 65 percent longer to complete and cost more than twice original agency estimates. That's an additional $356.3 million, the audit showed.

Problem projects

From state audits to media reports, a lot of problems have emerged from state technology initiatives over the past few years.

NC FAST $48.2 million
NC FAST, the state's new system for processing and delivering food stamps and other social services, saw a budget increase of about $23 million. Yet when the software, provided by IBM and integrated for the state by Accenture, began rolling out in earnest in 2013, county social services directors saw big problems that evolved with each update.

In July, records show DHHS downplayed major glitches that delayed benefits to almost 70,000 families.

Home Base $32.3 million (estimated)
The N.C. Department of Public Instruction's Home Base software, built by Pearson Education, is actually a suite of tools that allows students, teachers and parents to manage and view grades, assessments and other reports.

With federal Race to the Top funding hanging in the balance, the state found ways to advance the time table for migrating data from the old system to Home Base. In many counties however, the implementation led to delays of progress reports and report cards.

TIMS $97.3 million
The Department of Revenue's Tax Information Management System saw a massive increase from $525,000 to $97.3 million, according to an April 2013 audit.

Information Technology Services pointed out in the audit that the increase coincided with a massive expansion in the scope of the project. But auditors left TIMS in their analysis to "demonstrate the large monetary impact that inaccurate estimates for a few large IT projects can have on state spending."

Source: ITS, state auditor data

The larger price tags often haven't come with better results.

About a decade ago, under a Democratic governor, the state awarded a $171 million contract to Affiliated Computer Services to replace North Carolina's 15-year-old Medicaid information system. The project was originally scheduled to last two years.

But when that deadline came, Wood said, it was clear to state officials at the time that the company couldn't deliver. With $5.6 million already spent, the state terminated the contract. Settling the resulting lawsuit by ACS cost the state another $10.5 million.

"Because of the poor contract that we had, we paid them $16 million to go away," Wood said.

The current contract was awarded to CSC in January 2009. By then, the cost of the project had jumped to $265 million.

Like its predecessor, CSC had more than two years to do the job, and just like its predecessor, the firm sought to extend the contract past the August 2011 launch date. In July 2011, the state gave CSC two more years to finish and launch. The final cost as of July 2011 was $484 million, close to triple the price tag of the original contract. The federal government paid for about 90 percent of the amount.

NCTracks has been the most high profile of those problematic systems, but is far from an outlier.

Problems with another DHHS system, NC FAST, delayed processing of food assistance claims this summer. Students across the state, including some in Wake County, saw report cards and other important reports delayed by growing pains associated with Home Base, a computer system rolled out by the Department of Public Instruction this year.

For complex tech, contract is key

In almost every problematic case Wood's office has examined, including the long saga of NCTracks, the issues began with bad contracts.

She said state workers normally tasked with hiring companies and purchasing goods aren't knowledgeable enough about the technology they're looking to build or buy.

Read the documents

Explore the documents used to report this story on your own at the links below.

"I'm not sure the state of North Carolina, with the salaries that we pay, has the necessary people on board that can see everything that needs to be seen for these more complex systems," Wood said.

As a result, vendors – not state officials – write the contracts, and Wood said these companies don't always have the state's best interest at heart.

"Once you sign that contract with the vendor, you can't break it. You're theirs," Wood said. "They can take us to the bank, and we'll never see it coming."

Cost overruns have played out repeatedly in North Carolina projects, Wood's audits show. The new food stamp system implemented by DHHS jumped from an estimated $25.2 million to an actual $48.2 million. The Turnpike Authority's toll collection system went from $19.8 million to $41.1 million.

As the audits make clear, some of the largest spikes in cost come from an expansion of the size of projects – the change from a small pilot program to a whole-state solution, for example. But many more are simply the result of bad estimates that go unchallenged by the state's Information Technology Services division, Wood said.

"We've even had people at ITS tell us that you really don't know how much a project's going to cost until you sign the contract and get started," Wood said. "That is just wrong, because the vendor has all the cards. They have all the power, and they know it."

From the vendor's perspective, according to Red Hat National Sales Director John Punzak, it can be difficult to balance both a competitive bid and a risky project complicated by politics. 

"You just can't possibly know everything you're getting into as a vendor. There's just no way to uncover all the risk and all the issues," Punzak said. "At the end of the bid cycle, the winner can't price enough risk into it, and they can't understand the resistance in implementing the change."

State officials make these risks more complicated, Wood said, when they can't adequately explain the requirements of the project at the beginning.

"When I'm talking about upfront planning, I'm talking about making sure that you define what the system needs to do," Wood said. "Then, let the vendor tell you how that needs to be built."

Over the course of long-term projects, rapidly evolving technology prompts government officials to change these requirements in the middle of the process.

Chris Estes, state CIO

Estes, who as CIO heads up the state's ITS division, compares the process to buying a car.

"If we buy a car in 2008 to drive it in 2013, we want it to have all the features that a 2013 car has. Now we're trying to convert a 2008 car into a 2013 car while we're driving it. We've got to stop doing that," he said. "We've got to lock down and say we're buying a 2008 car that can only do 2008 things, and if we want to upgrade to a 2013, that should be a separate project with a separate timeline and a separate budget."

Until now, Wood's audits show, ITS has played a nominal role in the oversight of state agency technology projects. 

"You don't wait until you get to the end of a project like NCTracks and find out that it blew up," Wood said. "There's testing that's done along the way, there's building that gets done by phase and nobody's really stepping back and challenging the agency responsible for it."

She said she hopes that will change with Estes, who took the job under Gov. Pat McCrory in January.

"We're in the process of trying to fix that," Estes said. "But specifically, it's not working as well as it should."

For users, technology failures have consequences

Whether the problems with systems such as NCTracks originate in the planning or the execution, it's the state's current leaders who hear from angry users like Smith, the pediatrician who is still struggling to get paid five months later.

It's a message state Sen. Jeff Tarte, R-Mecklenburg, a member of the General Assembly's IT oversight committee, says he gets loud and clear.

7 years of patients

"Everyone wants to blame prior administrations, but at the end of the day, we're the ones who accepted it on our watch. We said, 'Yeah, it's good to go, turn it on.' So, we're the ones that have to be held accountable," said Tarte, a former partner at Ernst & Young who now runs his own technology consulting firm. "At some point, it's not about assigning blame to anybody. Let's just get it working. Let's fix it."

Smith said DHHS still owes her about $24,000 in Medicaid reimbursement fees, an amount that represents at least a dozen pay periods to her small practice.

She's been in frequent contact with DHHS officials, who granted her an emergency loan in August to help her stay afloat. Since then, she said, she's gotten assurances each month that payment will come soon.

DHHS spokesman Ricky Diaz said in an email that the agency is working to address Smith's issue, which is specific to only 80 providers out of 77,000. He added that even one provider impacted negatively is "one too many."

Although DHHS officials have been helpful, Smith said, she gets the impression that, much like herself, they "didn't know what they were getting into."

"At the very least, there should have been a higher level of due diligence in making the decision in hiring any company," she said. "Then, you put in incentives to make sure they get the work done. You give them incentives to get it right. You actually give them responsibility to get it right."

Those suggestions match many of the recommendations in Wood's audit, nearly all of which Estes said his staff will work to implement over the coming years.

In the meantime, Smith said she'll keep waiting.


Wednesday's story is the first of a two-part WRAL News series on the systemic problems with massive state technology projects. Check part two for a look at the ways state leaders could fix North Carolina's ailing process for acquiring and building new software.

38 Comments

This story is closed for comments. Comments on WRAL.com news stories are accepted and moderated between the hours of 8 a.m. and 8 p.m. Monday through Friday.

Oldest First
View all
  • Capt Mercury Dec 19, 2:53 p.m.

    "Government should have zero involvement."

    You've got to be kidding.

  • free2bme Dec 19, 2:40 p.m.

    I just love what Senator Tarte stated. Sick of the blame game. It is time for someone to take some responsibility and fix or try to fix these problems. Just too much money is being wasted with still no results or resolutions. Wos has provided no time frame or date as to when the NC Tracks problems will be resolved. Diaz has no clue what he is talking about either, I think it is time to bring in people that have a clue or that can put an effective plan in place. Too many tax dollars are going down the drain continually on the same issue.

  • A cold hard dose of Franz Dec 19, 2:31 p.m.

    Get the government OUT of healthcare and stupid stuff like this wouldn't happen.

  • Dr Sanchez Dec 19, 2:10 p.m.

    "when are they going to fire the state CIO….that's what any corporation would do..." -ProzacDispenser

    It's not his fault the State cannot hire people to do the jobs necessary. He can only pay what the Legislature and the OS-HR tells him, and the career banding is a joke. The only people who work in IT now are fresh out of college, and when they gain some work experience, leave to get paid at a private company. The only people who get paid in State Government are the 1000+ appointees McCrory has now.

  • whatelseisnew Dec 19, 1:19 p.m.

    "With nobody minding the store for the government, guess what happened to the treasury. Poof!"

    UH HUH as opposed to what happened to the money that was supposed to go into all the government trust funds? Oh yeah, they spent it and created debt. Privatizing is the way to go, but you have to ACTUALLY privatize it. Government should have zero involvement.

  • whatelseisnew Dec 19, 1:17 p.m.

    The really simple solution to this is to get rid of Medicaid. That would reduce the level of taxpayer enslavement and completely eliminate the cost of this system. People like the Doctor in this story can then go and establish a real practice instead of being a leech on the backs of taxpayers.

  • whatelseisnew Dec 19, 1:14 p.m.

    "From floor to ceiling, Sunshine Pediatrics is plastered with hundreds of drawings in crayon and marker. They represent just a fraction of Smith's 3,800 patients, 90 percent of whom are on Medicaid."

    Ah yes a Medicaid mill. Shut it down.

    "Sorry mrapex, we tried contracting government out during the Reagan years and it failed miserably."

    Contracting out only fails when Government does it because they still leave Government involved. What we need to do is get Government down to its basic functions and get them out of about 98 percent of the stuff they have gotten into with their greedy fingers. It won't happen, even bankruptcy does not get rid of the criminal politicians.

  • Capt Mercury Dec 19, 12:57 p.m.

    "The only solution is to eliminate all government employees and contract all work out." - mrapex

    Sorry mrapex, we tried contracting government out during the Reagan years and it failed miserably. Same result in Iraq with contractors. It's not that the contract workers were not higher quality (and better paid too), but their managers were always looking for another way to suck more money from the government. With nobody minding the store for the government, guess what happened to the treasury. Poof!

  • mike275132 Dec 19, 10:44 a.m.

    "They represent just a fraction of Smith's 3,800 patients, 90 percent of whom are on Medicaid."

    When you base your business on a Government Welfare program,
    you deserve to go out of business.

  • nic Dec 19, 10:43 a.m.

    It's not just in IT, it happens with a lot of state contracts. One example is the building of Central Regional Hospital. They hired a contractor to build a psychiatric hospital who never had experience building any hospital after all was said and done the project cost 420 million more do to the numerous changes that needed to be made after the fact. Like all new doors, handles and shower heads due to risk of hanging. Shouldn't this have been something the builder or the project managers should have known to start so it wouldn't have to be changed after the fact? Like another poster said know one managing this project listened to the front line workers who addressed the concerns about small doors ways that couldn't fit wheelchairs, no storage space, suicide risk concerns. If they would listen on the front end they wouldn't have to go back and waste money making changes.

More...