It's like those who rate too highly the bowl season in football: Discounting an entire season based on a single game. Everyone loves the upsets in the NCAA tourney and the fact that it can happen to any team, but then it does happen, and it means the losing team wasn't as good as they appeared over the course of the previous four months?
And the overall quality of a conference matters a great deal, because of the tougher challenges the top teams in that conference face night after night, week after week.