I think that those interviews are designed to root out smart people. You did the right thing.
However if I were the manager, I would actually have hired you because you walking out signals that you have initiative and will not listen to crap but do what you have to.