Neighbors of several men federal authorities have accused of being part of a terrorist group said FBI agents had tracked the men for some time.