Records reveal data gaps in one of NC's key COVID-19 metrics

On a weekend in mid-April, with COVID-19 infections rapidly approaching 5,000 cases across North Carolina, something among daily updates from state officials about the spread of the disease appeared to be a bright spot.

The steady rise in the number of hospitalized patients was declining after topping more than 400 on April 10, Good Friday. The number ticked down to 362 the next day, then 331 the next. By Monday, the state was reporting 313 hospitalizations as a result of COVID-19 ⁠— the lowest total in a week.

The good news wouldn’t last past Easter Weekend.

While the patient count was dropping, so was the number of hospitals responding to a daily survey the N.C. Department of Health and Human Services uses to tally hospitalizations, bed usage and ventilator stocks.

When about 25 additional North Carolina hospitals added in their missing numbers that Tuesday, the number of hospitalized patients leapt by more than 100, the largest increase since state officials began publishing the data.

An analysis of limited data provided by the state shows significant gaps exist in responses to the survey, sometimes from the state’s largest hospitals. But a month after a collaborative group of newsrooms requested that survey data, DHHS has yet to produce it.

And those aren't the only records reporters are awaiting that could shed more light on the spread of the virus and its impact on the state.

More than 100 hospitals have responded in some capacity to the DHHS survey over the past few months.

The agency started reporting hospitalizations and the related hospital survey response rate on March 27. Health officials have provided few details on how the agency arrives at that response rate, although they’ve pointed out that it’s based solely on the raw number of hospitals that respond ⁠— or don’t.

"This data, as incomplete as it is, is the best we have on a systematic basis of what's really going on in our hospitals across the state," said Aaron McKethan, senior policy fellow at the Duke-Margolis Center for Health Policy.

The former chief data and analytics officer for DHHS, McKethan in recent weeks has been part of a team of researchers and public health experts using the state’s hospitalization data in part to model future impacts of COVID-19 on the healthcare system.

"We've tried to address some of the shortcomings of a manual, voluntary survey instrument like this in the way you would in any other data survey collection process," McKethan said in an interview Wednesday.

Given the variation in hospital size across the state, a single large hospital’s failure to report could have a big impact on the numbers.

Until now, there hasn’t been a clear picture of just how large those reporting gaps are.

"We’ve heard that those hospitals that do not consistently report are smaller hospitals whose responsible staff are also tasked with other COVID-19 related needs that are more directly related to planning for and providing patient care," said Cynthia Charles, with the N.C. Healthcare Association, in an email this week.

But smaller hospitals aren’t alone in occasionally failing to complete the survey, according to an analysis of limited data provided by DHHS in response to a public records request submitted by the six-newsroom collaborative. That data is messy and filled with inconsistencies — respondents misspell hospital names and home counties as they manually enter information every day.

Yet it provides at least some detail on the gaps in reporting.

Since the state started publishing its hospitalization numbers, only a handful of North Carolina's hospitals have provided uninterrupted daily responses to the survey through May.

Responses appear to be especially problematic on Sunday, which the data shows is skipped more often than any other day of the week. That drives down weekend numbers and, early in the week, the rolling average the state uses to focus on broader trends.

Some facilities skipped just a single day of reporting since late March.

That was the case with Atrium Health's main campus in Mecklenburg County, which submitted nothing to DHHS about its more than 1,000 licensed beds on March 29. Same with Rex Hospital, which skipped a report on April 4.

“It is always our intent to file all requested reports and submit all necessary data,” Rex spokesperson Alan Wolf said in an email. “Occasionally a report is missed.”

Others, like Duke University Hospital in Durham, started responding more consistently in late March. The hospital has been providing daily reports like clockwork since March 28, after a nearly two-week gap. The same goes for WakeMed in Raleigh.

But some of the state's largest hospitals have gone much longer without reporting.

Duke Regional Hospital, just five miles from its university counterpart, has missed its reporting more than a dozen times over the past six weeks.

In a statement, Duke Regional Hospital President Katie Galbraith said that, although the facility was “committed to meeting the data submission requirements for COVID-19,” those requirements were numerous and need manual entry at each hospital.

“As this was a new process, we had a few gaps in reporting initially, and Duke Regional is aware of a gap in April as a result of a change in its internal process,” Galbraith said in an emailed statement Wednesday. “We are committed to ensuring there are no future gaps in reporting.”

Novant Health's Forsyth Medical Center has 865 licensed beds, making it the state's third largest hospital. The facility failed to respond to the survey on seven different days last month, including over Easter. That weekend saw a precipitous decline in the number of reported hospitalizations, which hit the lowest point in days.

A spokesperson with Novant Health said Wednesday afternoon the hospital “routinely responds to surveys requested by the federal and state governments, including the voluntary COVID-19 DHHS survey.”

“In addition to surveys, we work closely with county and state officials to provide regular updates concerning COVID-19 admissions, supply capacity and bed availability across the Novant Health system,” an emailed statement read.

New Hanover Regional Medical Center, the state's 10th largest hospital by licensed beds, also failed to respond to the DHHS survey the weekend of April 10. The hospital skipped another two days on April 18 and 19, another weekend when DHHS reported a downtick in the number of hospitalizations.

New Hanover spokesperson Julian March said the hospital was previously providing data from the weekend as part of its Monday report to DHHS.

“We switched to daily reporting once it became apparent that was the state’s preference,” March said in an email Wednesday.

Cape Fear Valley Medical Center in Cumberland County, a hospital with more than 500 licensed beds, missed at least four days of reporting.

“Cape Fear Valley Health reports its COVID-19 patient census numbers on a daily basis unless there are extenuating circumstances. We report the numbers both to the state and federal COVID-19 database portals,” Donnie Byers, a spokesperson with the hospital, said in an email Wednesday.

Despite the gaps in information, Charles, with the hospital lobby, says her organization still thinks the survey data is trustworthy.

“Hospitals are working hard to provide timely and accurate information to help state public health authorities make sound public health decisions,” she said in an email.

That data is missing from the hospitalization numbers isn’t necessarily a dealbreaker, McKethan said. The figures can still be helpful if they’re viewed with caution ⁠— and interpreted “appropriate to what’s missing.”

"For those who spend their careers in data and data science, this isn't a big tragedy, or not an anomaly," McKethan said. "But it's particularly challenging in a situation where there's a pandemic that is causing an urgent need to get the numbers right and to get them available every day."

For her part, DHHS spokesperson Chris Mackey maintains that the agency has “worked hard to find the balance between transparency and accuracy in publishing data gathered by the state.”

“We’ve worked to be clear in sharing that all of the data has limitations, which is why no one data point can be looked at in isolation,” Mackey said in an email Wednesday. “The composite picture of data helps inform policy decisions."

That picture may become more clear in the coming weeks.

Mission Health Hospital, in Asheville, missed four total days of manual reporting in the last month and a half.

But at the same time, the hospital has been submitting data separately to DHHS as the first participant in a pilot program to automate the survey process. Although that automated data is not yet flowing into the state health agency’s public-facing portal, DHHS Chief Operating Officer for Technology & Operations Charles Carter verified Wednesday that his office had received reports from Mission Health uninterrupted since the pilot became operational in mid-April.

“We saw a problem, saw a gap, and fortunately Mission was willing to be a guinea pig,” he said.

With other hospitals like UNC Health, Cape Fear Valley and Duke coming on board in the coming weeks, Carter said the plan is to soon use the automated system to feed public updates that DHHS publishes daily. The goal, he said, is to sign every hospital up for the system to both improve the quality of the data and reduce the burden on the healthcare workers who are now forced to enter the information by hand.

“They’re tired of having to hunt down all this data,” Carter said. “It stretches their staff pretty thin.”

Whether DHHS will publicly release that automated survey data in response to records requests is another question entirely.

Mackey, the DHHS spokesperson, said she expects to fulfill reporters’ month-old request for the manual survey data by the end of the week, noting the agency is working to respond to “multiple requests as our team addresses the COVID-19 pandemic.”

The survey responses aren’t the only records detailing the impacts of the COVID-19 pandemic reporters have been waiting for.

“As you are aware, the department has not released the names of facilities with a COVID-19 outbreak and nor have we released data specific to the number of positive cases or deaths in those settings,” said the April 26 email from Mark T. Benton, assistant secretary for public health. “This has been our practice thus far in this epidemic, and it has been our advice to each of you to follow.”

The next day, the collaboration filed a records request for a copy of the previous advice DHHS gave to health departments to withhold the names of congregate facilities with outbreaks. DHHS legal communications specialist Charles Epstein said last week the department was still working on the request.

On April 7, the collaboration asked emergency management officials for a list of requests submitted by hospitals to the NC Healthcare Web EOC ⁠— a procurement system for healthcare facilities during emergency situations.

During an emergency, hospitals can request additional resources such as specialized personnel and equipment. In the case of COVID-19, hospitals have utilized this system to request things like personal protective equipment.

In response, Susan Smith, the public records lead for the N.C. Department of Public Safety, said the request was “extensive” and the department would respond, but did not give a timeframe.

“While we fully appreciate and support cooperation and transparency, please be aware that the fulfillment of your request may be delayed due to the unprecedented workload associated with ensuring the health, safety and security of staff members and the public during this time of pandemic,” Smith said in her April 9 email.

One month later, reporters inquired about the status of the request, but received no response.