The bushy-tailed rodents put on a daily show outside Clara Price's day care.
"We've got seven regular ones here that the children enjoy watching, but they can cause a lot of damage too at times," she said.
Many people complain the squirrels are tearing up gardens and worse. For example, one squirrel once climbed into a transformer on Price's street.
"I don't know if they were chewing on the wires or what, but anyway, it was the squirrels' fault that we didn't have power for several hours," Price said.
Town Manager Ray McDonald said squirrels have also chewed wires running into people's homes. He said that makes them potentially dangerous.
"If it were to start a fire and someone would lose their life, we'd feel bad about that," he said.
People who have had squirrel trouble say they don't want to see them all disappear, but they believe reducing the squirrel population in town will reduce the problems. Some angry neighbors have threatened extreme action.
"One thing we don't want people doing is shooting them," McDonald said.
The town plans to put out traps, baited with pecans. Officials said they do not want to hurt the squirrels, just relocate them.
"Try to move them to a wooded area outside our town," McDonald said.
However, some people like having the squirrels around. They feed on Cathy Noonan's pecan tree.
"We see several a day here in the yard," she said. "It's one less pecan I have to pick up or rake, so it's no problem for me."
Officials said this year's squirrel population is one of the largest Mount Olive has ever seen. They believe this year's large pecan crop is probably attracting the squirrels.