Yup. The best example is the series finale of MASH - 106 million viewers. That was 1983. No non-sporting event since has come close, and never will. As such, you can't make comparisons to ratings from the previous century.
What might hurt college basketball in the ratings department is the fact the on-the-court product is so much worse than decades past. I was going to college back in the early-mid 1980s. The college stars then (besides Wayman of course) were Michael Jordon, Patrick Ewing, Chris Mullin, Karl Malone, Hakeem Olajuwon, James Worthy, Clyde Drexler, and on and on and on. There's obviously nothing close to that level of talent now in the college game because they don't stay there long enough to develop their game at the college level.