I was aware of the background most of the time, but I didn't have a plain background in their houses.
Then move them further away from the background and use a wider aperture. Or - shock horror - get them to go outside the house Then the background automatically buggers off out of the way.