Have you ever tried to ref a game, in any sport? It is SO hard. As fans, we are just sitting or standing there and watching the ball. Refs have to be in constant motion, don't get to use replay on 99% of their calls, and have to watch the entire court.
My Dad always tells a great story. When he was younger, the umps didn't show up a baseball game he was playing in. The teams decided to let the players call the game, one from each team behind the plate and on the field. First pitch thrown with my Dad behind the plate, fastball, right down the middle, waist high. He calls it a ball. Said it was the easiest strike call of his life, and the word "ball" is what came out of his mouth. I know he wasn't trained, and wasn't doing it for a living, but officiating high level sports is not easy at all.