"That's what people are saying, they've been abandoned, they didn't get their job finished," Shane Perry admitted.
We met Perry, the founder of a faith-based nonprofit group called Rebuild South Texas, last November in Patton Village.
He and his group of volunteers were going to surprise a man named Pedro who lost his home to Harvey.
Keith Rusnak gave Perry his FEMA money to help Pedro rebuild his home.
"It didn't work out like we had wanted it to, unfortunately," Rusnak said.
Shane Perry didn't finish Rusnak's house either like Rusnak says Perry promised he would.
"They're coming at me. Why? Because I'm here everyday volunteering," Perry said.
Several families in Rosharon are accusing Perry of taking off with $76,000. It was their FEMA money to rebuild.
Some of them are now living underneath their houses.
"They did receive FEMA for rental property to rent homes. They didn't give that to me. They just spent it on something else. I do feel sorry for them, but they should have been more responsible with their FEMA money," Perry said.
Perry denies taking the $76,000 in FEMA funds.
One man says Perry took off with his 16-year-old step daughter.
"There's 100 witnesses who say this 16-year-old girl was kicked out of her house for accepting Jesus Christ," Perry said.
We pointed out to Perry by his own admission he didn't finish Pedro's house, Rusnak's house and others.
"You guys aren't going to the houses we did finish and Pedro is a successful chiropractor in Mexico, did you know that?" Perry said.
Rusnak, who knows Pedro, finds that hard to believe.
"He can barely get around. I don't see how he's cracking backs or anything like that. He can't get around," Rusnak said.