We always asked if our guest wanted to eat with us.
Then one of three things happened.
1: The Guest ate with us.
2: The Guest went home because it was time for him to eat at home.
3: The Guest had already eaten, or would eat later, so he either continued to do whatever he did in another room or sat at the table having a conversation while we ate.