Durham city, county leaders respond to cyberattack
The managers and IT leaders of the city and county of Durham update their progress in response to a cyberattack last week.
status following a cyber malware attack. I need to emphasize. First said, we believe that these attacks were two separate attacks for two separate organizations. Okay, but the city and the county city Manager, Tom Bonfield, will come to you first, along with the city's chief information officer. Who is Carrie? Good. And they're going to talk to you about how the city is responding and how this occurred afterward. County officials will speak to the county status, and then following that, we will open it up for questions. Okay, Tom. Thank you, Beverly. Good afternoon, everyone. On Friday evening, the city of Durham Data Networks experienced a serious cyber malware malware attack which affected our entire network. Thanks to our advanced notification systems are technology solutions Department acted quickly to take networks and phones off line, which greatly minimize the damage to the city's operating systems. The malware has been contained, and we are in recovery mode with city staff and other cyber security professionals working around the clock to get everything back up and running. There is no mistaking. This was a serious incident in the city's advanced threat detection and cyber security software allowed us to respond quickly a forensic investigation is underway. But due to the nature of the event and the kind of malware that was used, the cyber security professionals has have confirmed and were highly confident that no personally identifiable information was compromised as a result of this breach, including city employees and resident data at this time, most city networks and phones remain intentionally offline during the initial stages of the recovery process. With assistance from the Fire and police departments, Derm Emergency Communications Center continues to receive 911 calls and has been dispatching police fire and E. M s responses as needed since the incident occurred while phones were down. City residents can still access service is and make payments via our Durham N C dot gov website, which was not affected by the attack. This includes putting in a request that during one call via our website or phone app and paying water bills through the pay mentis application again, these systems were not affected. I'm very grateful to our technology solutions department who made sure that we were prepared, responded quickly and continue to work tirelessly as we work through the recovery process. Also want to give my thanks to these agencies who are helping with the recovery effort, including the National Guard Cyber Security Team, Duke University, I t specialists in Carolina. I t At this time I'd like to introduce Carry Good the city's see io, who will provide additional details on the timeline in recovery process. I'm Carrie, good at CEO and the Director of Technology Solutions. So let me begin with by saying we have a cyber security program and we have planned for this day to occur. We have a contingency plan that we had built. And so immediately upon notification from the monetary system, we activated our response plan and that called bringing together a team of professionals, including M S I sack, including Carolina Sovereign Response Team and including the state D i t. We came to bear on this problem. So what we did initially is what it's on both of the city, man. Just say it was can't contain it. We turn off our course switch so that the spread will will be contained. And at that point, we started doing forensics. I understand what we were dealing with. And based on the M s I sack and huntress. I was monitoring Team. We clearly identified the viruses are y U K Barry's and its eyes one of the premier Mayor where ransomware type of virus so, so immediately way met Saturday morning. We had a meeting with my response sing and then we had a meeting with M s. I sock, we gather boy information and then we created a A plan of action to move forward with the containment and Ebert eradication. We created a A plan because I Black, gray and white plan was that the network Presley is black. We need to turn our network into white. So we did is take every device of ending that work. And we went through a process of clearing that device using tools and analysis. And once the device is, we think it's clear we're moving to the gray ray. We will test it on a separate network to ensure that it doesn't have ransomware, and then we will plug it back into our see the wind network, which is going to be the white network. So we have a three tier program and also at the same time we asked the question, Is there any better tools out there to the fan to say that we quickly identified a tool that can catch that this type of ransomware and eradicated. And we're putting that on the work stations as added precaution. So every work states go, the network will have this agent that will identify in the further contamination. And it was shut the workstation. They are reported to us as well. So we have put together some technology teams led by technology solutions. This is the resource is came from Duke I t. National Guard Carolina Tea. We have approximately 20 additional PC specialists. That will happen. Let's go around. And two clean. Approximately 1000 workstations that was contaminated. We had 80 work stages, 80 servers in a data center contaminated. So we have a process currently restoring our core business servers being we use minutes and when I last left were 86% restored because that's not the biggest server is probably approximately 20 to 30 terabytes inside. So it takes a little bit of time to download from my cloud backup system. We expected to be back online soon. The next two of the hours it will be back online and This is our system that does. I'm a core business. HR payroll utilities. Then we bring it up work order, system of one call system, which is not as big, and they should be coming online tomorrow. We think our data center will be fully recovered in two days. Then the remain of the work will be along the lines of restoring each workstation and given that connective it back to the employees, that's all the game plan. And of course, of every plan we run into contingencies, and if we do it, make slow us down. But to see that could be assured that I would back ups or very good because they're in beautiful, which means that ransom where it cannot consume our backups, we use a product or Reuben is one of the leading back up systems you can purchase. And one reason we purchased it was caused. It was a backup system that could not be, um, consumed by ransomware. And at this time, that's the update that I have for you as faras. How are we going to move forward with this? Are clearing and we're going to see the back into production on our computer systems. Thank you. Good afternoon. I'm Deborah Craig Ray, Durham county general manager for strategic planning and innovation. Now it's time to hear from our county manager, Wendell Davis, who will talk about the county's perspective and impact during this attack. Mr Davis. Thank you, Deborah. And good afternoon. So Durham County was notified on late Friday evening about the malware attack on our county. Resource is soon thereafter. Our I S and T team called in additional cybersecurity. Resource is to investigate, perform forensics and determine the extent of the impact. I am comfortable that we have and will acquire. The resource is to address this breach. Our goal is to continue to provide Citizen's service is as appropriately as possible during the time that our systems are down. That could mean some manual processes and other workarounds that will be brought to bear during this period. But it is important that we take our time to fully investigate and restore our business systems to ensure we provide enhanced security so that this situation does not re occur. The recovery period is critical, and we must do it right. At the end of the day, we ask our citizens and our employees for their patients. As we work through this business interruption at this time, I'd like to recognize our I S and T director Greg Marrow, who will report to you on our current situation and how we move forward from this point. I also want to take the occasion to walk Bank Cisco Systems, our National Guard Cyber Security unit as well as Microsoft for their assistance in this effort. In addition to Grant coming forth, we also have some additional department heads President of room to ask any subsequent questions that may come up around public health Emergency service is an emergency management on DME or so Greg, Thank you. Ah, County Manager. My name again is Greg Merrill, CEO with the county of Dharam. And as the county manager mentioned similar to the city, we, uh, experienced a ransomware attack on Friday evening as well, and we executed our incident plan on Friday evening. As county manager mentioned, we are taking ah slow process in terms of investigating the cause of the attack and looking deeply into our systems to ensure that when we do our backups, we don't affect, uh, we don't have this problem a week from now, and so we're following a very similar process to the city, not going to repeat that process. But we're going through an investigative stage. I think it's fair to say we are wrapping up our investigation stage. We do know it's a ransomware attack. We now know where imminent, you know from and how it how it entered the county network in terms of restoration. We're beginning that process now. So we're moving out of the investigation stage into the restoration process, and our focus at the county is beginning to look at social service is public health to ensure that citizens can quickly and expeditiously began to, uh, Thio utilize service's that are provided through whether it's online, whether through computers, whether it's through our call center. And so that's our focus over at the county. As Kerry mentioned, our numbers look about the same. We have about 1000 computers or so that we need to re image on the county side. We've decided that we're going to take the precaution and re image all death stops in all laptops and in terms of our data center, we have about 100 service out that we have decided Thio rebuild from scratch just for precautionary precautionary measures. And lastly, I'll just mention, as the Count County manager said, we have a full team of, ah folks working with us from the National Guard to the state of North Carolina to other agencies within, uh, uh, North Carolina, who have gone through this process who are now here working with us. And we also have several of our business partners Microsoft, Cisco and a few other partners who are actively working with us to help us get back to normal. And so I think that's pretty much it from the from the county side. So thank you. We're going to open it up for questions. And after our questions, we're gonna have a statement from Mayor Shool and Commissioner Jacobs. So we do have a mic and Okay. All right. Do you have any idea how this even got into the work stations? That somebody open up email or down a little something? I'm gonna ask our experts to come up and talk to you about that. I don't know if you heard the question, could you repeat the question? Do you know how this Ransomware even got onto the server or network. I'll go first from Carrie, good CEO for this CDO of Durham. Based on our forensic analysis up to this point, we had several and was, Look at our patient zero information. We have identified five work states that could possibly be patient zero. And based on the announce, it looks like I e mail was the way that is infiltrated into our network. So when clicking on a attachment within an email, Greg Marrow again from the county side similar to the city, Um, we've identified 22 laptops, and we believe that the virus entered our malware into the county through someone clicking on a female as well. And this question is for Mr Marrow, with respect to the county. Has any data information on, say, voter registration records been compromised if they've been stolen? Lost? I've heard a number of comments from the public about voter registration information. What can you say to that end? So, based upon our investigation to date, we have no indication that any data has been stolen or tampered with. And it's also important for me to say that all of our data sitting at rest or in transit is encrypted, but a CZ part of the forensic investigation. We have no indication that any data has been tampered with in your forensic investigation. Have you had any indication of why Durham city and County were targeted in this, huh? No, we haven't saying we don't know. We don't know of any specific reason other than, uh, the hackers, just cyber threat actors. They don't need a reason to attack you other than they want to do it. Money. And it also we haven't received any ransom notes yet. We've been looking forward, but we haven't see received any screens, any kind of ransom requests. Well, that leads me to my question. What was the nature of the email attachment? Do you know we can get that information for you? But we don't have that information right now. The analysts didn't didn't share that with us because we didn't ask. Uh, okay, I am curious. And which specific office inside city and county government did this email, I guess. Infiltrate. Yeah. On the on the county side, all we've have identified is a particular lap leapt up. We have been gone too you know? OK, what what offices just connected with or what person is disconnected with. Right now, all we're concerned with is understanding the nature of the situation is communiqu to the echo. Whether break Mirrors said I focus was understanding the how not to who at this time will there be any type of education among your employees, since it seems to be a user error that this style at the city of during we've been doing since cyber security awareness and we even test them to see they are cognizant of what we did when we train them. And we had a very high score of employees compliant with not clicking on fishing. The male attempts. Of course, we're not not all employees past, but it was a high degree of employees that passed the test. Can employees continue to work at their offices over the next couple of days? Are all right? Do they have to work from home? What does this mean for the employees who can access records and so forth? Certainly the work is adjusted. All of our employees received notice that they are to report to work, and they are. They are at their work. Their work may be different. In some cases, it is as rudimentary is paper and pencil. In other cases, there are some some systems that you know that are allowed Thio accumulate data. But the most important thing is we want to be available as soon as the phone systems are up. And certainly City Hall the doors were open and we have facilities all over the city, and all operations were available. Thio to meet the public today Any suggestion that this hack might have had anything to do with the elections? That is an election year. No, I mean the the technical folks can can answer that question. Well, while it may be perceived that Durham was targeted or Durham City and County were targeted From what I understand, Carrie can confirm. I mean, these attempts are going on all over the country all over the world. This particular virus is prolific, has has retained a significant amount of ransom being paid by a lot of people. And I think this was really just, you know, something that was it wasn't identified, identified just as the city or the county of Durham. We just happened. Thio have had the misfortune of you know that this happening simultaneously care. You wanna talk about it, right? Oh, Tom. Buffy is correct. Opera ends the information. Haven't seen any specific reason why we were target. But we do know that this particular ransomware I have air in three big and dollars up to this point based on the analysis. So it is a high earning type attack by the cyber threat actors on. I think they just wanted to see if they can penetrate and get us until they get our backups consume. We can always recover. And I, like not, like other cities are backups were not consumed by the by the mayor. Where? So just time element for us to recover. Thank you. Did you go? So there was a question about whether or not our employees are working. Obviously, all of our employees are at work today. We do business continuity, planning. And so we essentially just activated. It activated those plans. For the most part, it means a number of different things for various departments. Some folks what? They're apprentice to work and work off line along with their computers and things of that nature. And so all of our employees are working. And if you know, we get ultimately to a point that we have to exercise telecommuting practices and things of that nature, we will. But we are not at that point. I think that we're in a good recovery place. Um, can you tell us why the public wasn't notified about this until two days after the attack happened? I'm not sure. That is exactly correct. Um, we've issued we issue statements when we have information to share while we get a lot of questions, the staff are staffed. City staff was working around the clock to try to understand what was going on and what was what was the extent of the problem. And we wanted to wait until we had our hands around handle around that before we issue the official public statement. I think that was was that yesterday. But this was not something that we were, you know, there was anything to be kept secret about. We're just very concerned about the misinformation coming from from guessing what happened until we were sure what happened. No, I don't. I don't have anything. I can add to that. My question. Will this incur any additional costs? Well, I think it depends on how you how you described additional costs. Obviously, we've got additional staff time. Additional resource is the city does have cyber security insurance. We have had cyber security insurance for quite some time. Our cyber security insurer has been put on notice and at some point we will make an assessment about what additional costs that we incurred that would be eligible. Thio come under that coverage again. I would just echo what Tom said on the front end of his statement. We do have that same insurance as well on the county side as we go through these experiences. Obviously we will learn some things about our system and as we learn those things, we may have to make additional investments. But we're not at that point yet to understand precisely what that means in the broader context of this investigation. But if we discover that additional investments need to be made, then we will have that conversation internally and we'll have some conversations with IRA elected leaders. When was the last time employees were trained on cyber security awareness at the city of Derm? It was last fall around November, the summer time frame, and at the county we do cybersecurity training on an ongoing basis. So is part of a new employee orientation. Cybersecurity training is included in that we're always doing something on a monthly basis in terms of training and warning and preparing employees for these kinds of incidents. And let me add Our train is continuous need new employees or trained on cybersecurity throughout into the orientation. Also on that screen savers we have training that they see every time that screen goes into screen saver mode. We warned them about phishing emails and about spirit, spearfishing and and how cyber security is very important to how they use their computers. And anyone working the city sees that screen saver every day. And just just to kind of something's up again. You've addressed this here and there, but to summarize, When do you anticipate everything will be quote back to normal when things will be functioning normally at the city and county level. And to say that I have a great, great deal of confidence that within two weeks we'll be back fully operational, okay, And on the county side, I would say that, you know, we're still and we are a little bit more cautious. And so we're starting the restoration process today. Um, I would say on the on the you know, if everything goes great, it could be within a week. And then if there's some extenuating circumstances that that, you know, come up it maybe two weeks now, we're gonna hear from Commissioner Chair Jacobs and Mayor. Sure. I just want to say what while they're coming up that trout said, we're going to keep you up to date on our social media forums and just get information out whenever we can and need to the public to keep you apprised of what's going on. Good afternoon. I'm Wendy Jacobs, chair of the Durham County Board of Commissioners. And I just want to share with our residents that, um we are in very good hands. We have outstanding. I t staff are our I T director Greg Marrow. And all of our I T staff are fantastic. People have been working as hard as they can around the clock. This, unfortunately, is something that they have prepared for and trained for. As you heard from from comments are all of our employees and even me personally as a county commissioner, have been trained to watch out for fishing expeditions. But we know that in today's world, unfortunately, these types of attacks are very, very skillfully crafted. All of us know from the e mails that we get personally from the phone calls, even that we get that these attacks can look like a bank statement or an order purchase order. It is. It is very, very difficult. And it is very easy for somebody to just click on an email or an attachment. Um, and this is what happens. I think the good news is that we have prepared for this. We have The county has invested very heavily in, um, preparing for this moment, and I have full confidence in our staff. Uh, and we just ask people to be patient. We ask our residents to just be patient and that everybody is doing doing their best to make sure that all of our residents are having the service's and programs that they need and that that is our number one priority. So everyone is working as hard as they can. Social service is public health MSR non one center, the office of the sheriff, making sure that we're taking care of residents in our community. So again, I just want to thank our county manager. All of our department heads who were here, all the county staff, um, you know, every once learning to talk to each other, face to face and talking on the phone things that we maybe don't do as often as we should. And we will. We will all get through this together. Thank you, everyone. Thank you, Commissioner Jacobs ends. Thank you all for being here today. I want to echo what Wendy has just said. We in the city are very fortunate that we have super capable people in our technology solutions department who are doing a great job. And we're prepared for this. Um, let me just say a little bit of our cyber security. One of the things that I don't believe Kerry did say is we back up all of our data in the city every two hours. So when something like this happens, we knew lose no more than two hours worth of data. So that is one of our cyber security protections. Another one of them is something that are both Greg and carry of also mentioned, as did Commissioner Jacobs, which is we off. We are all the time educated about these phishing attempts, and here's some of the ways we're educated. One is we have on our screens. Every day something comes up on my screen that educates me about not answering something that looks like a phishing email. But there's another thing that happens in the city, which I think is even more powerful, which is our technology solutions department sends us fake e mails to see if we will open them. I've gotten these. I don't know how often because they're I try not to open them on. What I frequently do is I will send us to carry and say, This looks like a phishing. Email is this and he will send back Yes, this is one of our tests. So we are being educated in that very practical way as well. And then one of the questions important question is, you know about Durham being targeted. My colleague Mark Anthony Milton, who's here today, just said to me, I thought which is a really good metaphor for this we're not being targeted. What's happening is these viruses are just rattling doorknobs there rattling doorknobs and they're seeing what's open, and this happens were attacked. But the city and the county thousands of times a year by people attempting to break into our system. This time they succeeded because of someone or someones opening an email that was a fishing attempt. But this is not rare. This is common on, and I think the way that I think we need to all of you. This is This was not a question of if this was gonna happen. This was a question of when this was going to happen. And the question is, are we prepared? And the answer is we are prepared, and I'm very confident that we will be back up soon that our major systems will be back up by either late this afternoon or tomorrow and then subsequent to that, we mainly have to do work on workstations, getting each making sure that each of the 1000 workstations itself is not contaminated. And that will be that will take a little bit of time. S O. I appreciate everyone being here and have again just really appreciation for our county colleagues. A cz well as our own staff. So thank you. Thank you. Five years. Yeah. Yeah. Just so you know, I don't have to worry about getting all that injects before five, because I'm gonna use just him from