I would definitely recommend Himeji Castle. During my round the world trip a couple of years back I ended up in Japan twice (long story) and both times I visited that wonderful place.
Curiously enough at both times I regretted not having time visiting the museum. All because I spent the while wandering about, soaking in the magic.
I like castles; it's a legitimate sickness.