Fire did heavy damage to a three-story building in downtown Durham early Wednesday, authorities said.