Banks deny accusations that they're hoarding money from the federal bailout, but builders maintain few people are able to buy homes in the current restrictive lending environment.