I'm finding more and more Japanese people here in Osaka who don't give a second thought as to whether or not I speak Japanese. People will just waltz up to me and blurt something out expecting me to understand, and I wish I did (btw I'm a 6 foot, brown hair, green eyed white guy).

The other day I was waiting for the tram when a man pulled up next to me on his bike and asked for directions to somewhere, in Japanese. I didn't really know what he said so I just replied, "sumimasen, wakaranai". Then he skipped to the next person who gave him the directions he needed.

I reckon the people who are most surprised that you speak Japanese and that you can use chopsticks are the ones who have had previous (although not a lot of) exposure to Westerners and English. I.e. Japanese who have gone on a holiday to Hawaii, can speak a few English phrases, maybe know someone from OS.