Butner, N.C. — Disgraced financier Bernard Madoff will serve his 150-year fraud sentence at the federal prison in Butner, CBS News reported Monday.

Madoff, 71, was sentenced two weeks ago to a maximum 150 years in prison for bilking thousands of investors out of billions of dollars.

CBS News said federal law enforcement sources confirmed he had been moved from a correctional center in New York and was headed to Butner to serve his sentence.

Federal prison officials said they don't comment on prisoner transfers because of security concerns. Nevertheless, the news of Madoff's transfer made its way quickly around the community Monday evening.

"He has got to serve his time somewhere, so I reckon why not Butner?” Butner resident Scottie Norwood said.

“I would think that he would be no threat to anybody, other than somebody wanting to get their money back. Maybe it is just security reasons for him,” Butner resident Stan Riley said.

A court-appointed trustee is trying to identify assets that could be sold to compensate victims of the $65 billion scheme. Investors have filed more than 15,400 claims against him.

Madoff won't be the first high-profile inmate to serve time in Butner. Israeli spy Jonathon Pollard, World Trade Center bombing mastermind Omar Abdel-Rahman, former televangelist Jim Bakker, Hustler publisher Larry Flynt and John Hinckley, who tried to assassinate former President Ronald Reagan, all spent time behind bars at the federal prison.

The Butner complex holds 3,400 inmates and includes two medium-security facilities, a low-security facility and a hospital, according to the Bureau of Prisons Web site.