I think it's justified for the Japanese to think whatever, but I think they should mostly speak to them in Japanese at first, assuming they may know at least a little be, because that is the country's language. Just like how foreigners come to the US, many people assume everyone knows some English. It's more of the responsibility of the person that entered the country not knowing the country's language to let others know they don't know the language, and residents that do know the language should assume who they're talking to knows at least some of the language.

Keiichi