Exactly, it isn't the bird. It's just "The Cardinal." (I wonder if they have an arch-rival that is "The Ordinal"?) Then again, it is very similar to Harvard ("The Harvard Crimson"), and I guess it could be referring to "those who are cardinal in color" the same way that "the annointed" would refer to an implied group who were annointed.

Looking over a list of college sports nicknames at http://www.smargon.net/nicknames/ I see that many colleges in fact are simply named after a color. A sampling:

Blue --- Wellesley College (Wellesley, Massachusetts)
Big Blue --- Millikin University (Decatur, Illinois)
Big Blues --- Bluefield State College (Bluefield, West Virginia)
Big Green --- Dartmouth College (Hanover, New Hampshire)
Big Red --- Cornell University (Ithaca, New York) and Denison University (Granville, Ohio)
Blues --- Mississippi University for Women (Columbus, Mississippi)
Crimson --- Harvard University (Cambridge, Massachusetts)
Crimson Storm --- Southern Nazarene University (Bethany, Oklahoma) (changed in August 1998 from Redskins)
Crimson Tide --- University of Alabama (Tuscaloosa, Alabama)
Crimson Wave --- Calumet College of St. Joseph (Whiting, Indiana)

However, far more entertaining names are to be found away from colors:

Hustlin' Quakers --- Earlham College (Richmond, Indiana)
Ichabods --- Washburn University (Topeka, Kansas) (men's team)
Medics --- Thomas Jefferson University (Philadelphia, Pennsylvania)
Swordsmen --- Life Bible College-East (Christiansburg, Virginia)
Women of Troy --- University of Southern California (Los Angeles, California) (women's team)
Wonder Boys --- Arkansas Tech University (Russellville, Arkansas) (men's team)
Hatters --- Stetson University (Deland, Florida)
Black Flies --- College of the Atlantic (Bar Harbor, Maine)