We just figured they were doing what we were doing, getting a piece of land and getting off the grid," said Anderson, who moved to New Mexico from Seattle with his wife seven years ago.
As the months passed, however, they stopped seeing the smaller children playing in the area. They also stopped hearing guns fired off at a shooting range on the property, he said.
Jason Badger, who owned the property where the compound was built, said he and his wife had pressed authorities to remove the group after becoming concerned about the children. The group had built the compound on their acreage instead of a neighboring tract owned by Lucas Morton, one of the men arrested during the raid.
"I started to try and kick them off about three months ago and everything I tried to do kept getting knocked down," said Badger said.
A judge dismissed an eviction notice filed by Badger against Morton in June, court records said. The records did not provide further details on the judge's decision.
After the raid, Anderson went over and looked at the property for the first time in months.
"I was flabbergasted from what it had turned into from the last time I saw it," he said.
Authorities said the compound shielded by old tires, wooden pallets and an earthen wall studded with broken glass had been littered with "odorous trash."
The 11 children found living at the encampment described as a small trailer embedded in the ground had been without clean water and appeared to have not eaten in days, according to Taos County Sheriff Jerry Hogrefe.
At a news conference in Taos, Hogrefe described FBI surveillance efforts in recent months that included photographs of the compound and interviews. He said the images were shared with the mother of Abdul-ghani but she did not spot her son, and that the photographs never indicated the boy's father was at the compound.
"I had no probable cause to get a search warrant to go onto this property," the sheriff said.
He said FBI officials were invited to the news conference but declined to attend. An FBI spokesman did not immediately return a call seeking comment.
Hogrefe said the "breaking point" in seeking a search warrant came when Georgia authorities received a message that may have originated within the compound that children were starving inside.
It was not clear who sent the message or how it was communicated. Georgia detectives forwarded it to the Taos County Sheriff's Office.
Authorities returned to search the compound after interviews on Friday and Saturday led them to believe the boy might still be on the property.
"We discovered the remains yesterday on Abdul's fourth birthday," Hogrefe said, appearing to fight back tears.
Aleks Kostich, managing attorney in the Taos County public defender's office, said the office was gathering information and assigning attorneys to the defendants. He declined to comment on their behalf, saying the case was in its early stages.
However, he questioned the "legal sufficiency" of the criminal complaints filed against the men and women, saying they were vague.
"I'm not sure how much investigating has been done," he said. "I'm not sure how much law enforcement knows and how long they've known it for."