By Julia Sommer, Public Affairs
Posted February 3, 1999

At a Jan. 29 forum on "Ranking Research Universities: A Discussion of Purpose and Influence," representatives from the National Research Council (NRC) and U.S. News and World Report explained and defended their methods and results, at the same time acknowledging that both are imperfect and that data collection and accuracy can be improved.

Hugh Graham, co-author of the book "The Rise of American Research Universities," began the discussion by noting that the release of university rankings create an "explosion of discussion, hurt feelings and chortling."

The two main forces distorting rankings, he said, are biases in favor of size and prestige, or "the halo effect." Making accurate rankings even more difficult, he said, was a "sea change" in graduate education, including new PhD programs and an increase in interdisciplinary work and specialization.

Several speakers echoed the difficulty of ranking departments in an age when multidisciplinarity is the order of the day, new fields are emerging, and increasingly creative names are being chosen for departments.

While praising NRC graduate school rankings, Graham's co-author, Nancy Diamond, noted that, "by and large they get it right at the top and bottom, but there's distortion in the middle."

Graham and Diamond have found that there are "occasional but significant mismatches" between objective criteria, such as faculty publications and citations, and rankings, especially when it comes to younger, smaller schools.

Charlotte Kuh, executive director of the NRC Office of Scientific and Engineering Personnel, called the once-a-decade NRC survey of PhD programs in the U.S., "the Rolls Royce of academic rankings." The next NRC survey will be released in 2003. It is funded by private foundations.

Kuh noted that the differences among the top schools are "infinitesimal."

In the future she would like the NRC survey to include agriculture and nutritional science programs and address student outcomes, reputation of PhD programs outside the U.S., interdisciplinary and emerging fields, information technology, the environment for doctoral education and the differing missions of PhD programs.

Amy Graham, director of data research at U.S. News and World Report, said that universities are wasting money on consultants who advise on how to improve rankings. Instead, she recommended that universities call her office directly or check the U.S. News web site, where underlying data for U.S. News rankings are published.

A PhD in economics from the University of Virginia, Graham is responsible for data collection and analysis for America's Best Colleges and America's Best Graduate Schools, published annually by U.S. News.

Graham acknowledged that U.S. News measures "our opinion of what matters," which includes objective measurements, expert opinion, and opinions from higher education reporters.

She noted that U.S. News gets very little criticism from students who use its surveys, and that there is little evidence of colleges and universities not telling the truth.

But "the data collection burden on universities is a real issue," she acknowledged.

Graham said U.S. News rankings have "encouraged development of standards for data reporting," such as college graduation rates and MBA placement rates, and that they "have created tremendous pressure on some schools to improve rankings. That's not necessarily a bad thing," she said, "but it's not an easy thing either ... Rankings are relative. Our goal should be improving education, not improving rankings."

She said U.S. News will continue to improve its data collection methods, re-examine its selection of measures and weights, and add interactive features to its web site.

"We do not change methods to shake up the rankings," Graham stressed.

During a question-and-answer period following panel presentations, Hugh Graham noted that the U.S. News rankings obviously fill a consumer need, and wondered how rankings can be made helpful to students in sub-fields.

Kuh said that the NRC has difficulty finding objective measures for ranking PhD programs in the arts and humanities, and that the number of library books has become less important. "Differential access to library resources has disappeared," she said.

Joseph Cerny, vice chancellor for research and dean of the Graduate Division, said it was a "national disgrace" that the federal government did not require completion rates for PhD programs similar to graduate rates for undergraduates. "Time-to-degree and attrition rates should be publicized," he said.

The forum was presented by the Center for Studies in Higher Education and moderated by its interim director, Professor Arnold Leiman.

The Office of the Vice Chancellor for Research helped fund the event, which was followed by a meeting with Charlotte Kuh and Jim Voytuk of the NRC to discuss possible changes in methodology for the next NRC study.

The Center for Studies in Higher Education is building a website on Ranking Research Universities, including material presented at the forum, at www.ishi.lib.berkeley.edu/cshe/events/ranking.html

February 3 - 9, 1999 (Volume 27, Number 21)
Copyright 1999, The Regents of the University of California.
Produced and maintained by the Office of Public Affairs at UC Berkeley.
Comments? E-mail berkeleyan@pa.urel.berkeley.edu.