About "disrespect" and literature

When I told someone recently that I "hated" The Da Vinci Code, I had to clarify that I don't disrespect people who like it. I don't understand why people take their literary taste so seriously.

A couple of years ago, I went to a presentation on poetic language, and the speaker went out for coffee with a few of us later. I recognized Owen Barfield as an influence in his viewpoint, and we talked a little about that, then turned to Charles Williams, whom he didn't like. I said, "Well, as far as that goes, I don't like Walt Whitman either."

The people around the table said, "Ooooo," as if we were on a school ground and I'd just kicked dirt into his eyes.

I must be culturally tone deaf, but I was taken aback by the response. All I meant was something along the lines of, "It takes all kinds," and everybody else at the table took it as, "You don't like my writer, well, I don't like yours. So there!"

The truth is, I don't think Charles Williams is "great literature," though I have my reasons for enjoying him. And I do think Whitman is "great literature," though I have my reasons for not enjoying him. Why should that cause a problem for Whitman fans? When I say, "I don't like sushi," sushi fans say something like, "Not fond of raw fish, eh?"

I would phrase the thing differently -- or keep my mouth shut -- if I were talking to Whitman himself, but even there -- isn't it just a matter of reality that some people won't like our work? What is it to the author of hard space opera if Frannie Romannie says she doesn't like it?

This is an honest question, if anyone cares to speculate about the answer with me. I think it has something to do with how we view ourselves, and how we think our literary tastes mark us. But is that a valid perception? Am I really going to think less of my friend because she doesn't like Notes from Underground and I do? Or is our discussion of what we like and dislike about the book more likely to open aspects of the work we hadn't seen? As in the case of Frannie Romannie, the parts of the hard space opera she didn't like may be the parts that her friend Jeremy Rocketer likes the best. And the parts she manages to appreciate about it could help him to see aspects of the story he hadn't seen before. Or whatever.

Or is there some hierarchy of "right" vs. "wrong" literature, which one violates at one's peril? (And how come I never get those memos?)