Here's where you're going wrong: you don't ask if you can stick to your original plans, you say what your plans are and if friend B doesn't like it he can do something else. It's your trip.
If that's not diplomatic enough for you, "Grandma says there's not enough room for three people in her (presumably small) Japanese house, and I'm not comfortable imposing on her anyway. We'll catch up with my cousin at a bar somewhere."