Web feature

[an error occurred while processing this directive]

Smarts, for sure — but what other qualities make a good lawyer?

New research proposes assessing the judgment and people skills of law-school applicants — something the LSAT alone can't do

By Carol Ness, Public Affairs | 4 August 2009

BERKELEY — The LSAT is, in tandem with an applicant's GPA, the gold standard for U.S. law-school admissions. While the test may do a great job identifying potentially stellar law students, picking the ones who will ultimately make the best lawyers takes a broader approach, according to groundbreaking research by two Berkeley experts.

Marjorie Shultz, a Berkeley professor of law, and Sheldon Zedeck, an industrial-psychology expert (and Berkeley's vice provost for academic affairs and faculty welfare), have designed a way to do just that — according to tests of their research on 1,100 graduates of Berkeley Law and Hastings College of the Law in San Francisco.

Their Law School Admissions Project identified 26 factors that make for effective lawyering, going beyond the sheer brain power assessed by the Law School Admissions Test to cover personal and interpersonal attributes such as integrity, empathy, situational judgment, and communication and people skills. Then they devised tests to measure those attributes, and validated the tests by trying them out on the alumni, whose abilities as lawyers also were rated by their peers and supervisors.

"Having correlated the subjects' test scores with their ratings on those factors, we are able to say we can predict the effectiveness of an attorney much better than the LSAT does," says Zedeck.

The results of the work, if re-confirmed by broader testing, could mean profound changes for law schools and the legal profession, asserts Berkeley Law Dean Christopher Edley, who has supported Zedeck and Shultz's work.

"I can't overstate the potential of this work to transform the way law schools conduct selected admissions and, perhaps, the way law firms think about training and evaluating young attorneys," says Edley.

However, coming up with the funding to take the research to the next level — a national study — has proven a challenge, since its principal sponsor, the Law School Admission Council (LSAC), has balked at going further, according to Edley and the two researchers. LSAC, a nonprofit organization, owns and administers the LSAT, in addition to marketing extensive LSAT prep materials and admissions software for law schools.

The LSAC hasn't formally declined Zedeck and Shultz's request for more funding, and when contacted by e-mail through a spokesperson, council President Dan Bernstein didn't comment directly, instead praising their work and talking about "the challenge" of building on it. Asked to clarify, the spokesperson said the council "is moving ahead on several fronts to see how noncognitive factors might be of assistance to the law school admission process," but she could not be more specific.

Whatever LSAC does, Zedeck and Shultz say their own talks with the council over further testing have stalled for so long that they believe the issue is moot. What's the holdup? For one thing, says Zedeck, the council has expressed concern that their tests might be coachable. His response: "You could ask the same question about those who do well on the LSAT."

Edley comments, "Given the extraordinary quality of this research, the only excuse I can imagine for LSAC refusing to invest more is that they don't want to undermine the market power of the LSAT. That's the problem with private-sector funding for truth-telling research."

Dismaying realizations

The project got its start 11 years ago, as the number of minority students entering Berkeley Law (then Boalt Hall) plunged after Proposition 209, passed by voters in 1996, put an end to affirmative action in the public arena in California. That meant an end to the use of any racial preference in admissions at public schools, including the University of California.

Shultz and other law faculty were concerned. A committee appointed in 1998 by Herma Hill Kay, then Berkeley's law dean, and headed by law professor Malcolm Feeley analyzed the post-209 admissions process and came to realize, with dismay, how much it depended on standardized tests, according to Zedeck and Shultz's report on their work.

That trend has only intensified with the advent of such highly publicized law-school rankings as U.S. News and World Report's, which lean heavily on LSAT scores, the professors wrote.

The committee was well aware, Shultz has said, of research suggesting that the factors measured by standardized tests like the LSAT are ones that whites and some Asians tend to do well on, while African Americans and Latinos don't — but that those aren't the only factors that make for a good lawyer.

"Everyone said it would be great if we could measure what would make a great lawyer, but that it can't be done," Shultz recalls. But Feeley knew of Zedeck and his expertise in employment-selection and validation models, with a particular emphasis on reducing adverse impact against minorities. (He helped design the federal fair- employment standard that says a test for employment must be related to the job.)

 Zedeck went to work with Shultz, a Berkeley law professor for 32 years before retiring in 2008. Now, some $600,000 and 15 boxes of paper records later, they've nailed down 26 factors — including intellectual skills — that could be used to evaluate a would-be attorney.

How to tell if a question's telling

A test of situational judgment might ask something like this, Zedeck says: Say you're in charge of a Democratic candidate's election committee, and it's scheduled to speak to a Republican group about the candidate's stands on health, crime, and taxes. Just before the event, the tax expert on the committee is in an accident and can't make it. Do you cancel the meeting, go but don't talk about taxes, or go and talk about taxes?

The questions were tested on Berkeley and Hastings graduates to determine which answers correlated with good lawyering. If highly effective lawyers (as evaluated by peers and superiors) picked the first answer to a given question, and bad lawyers picked the third, the question was judged to be telling.

The 26 factors include but go beyond the skills tested by the LSAT — analysis, reading and logic.  "Your LSAT score won't tell you whether you have integrity or diligence," Shultz says.

In addition, while research indicates that standardized tests that focus on academic skills put minority candidates at a substantial disadvantage, Shultz says "there is not a race or ethnicity difference in performance on our tests."

"That's why I say fairness demands that you have a system that's a bit broader than the one currently used to pick applicants," she adds.

Since their report came out last fall, at least 18 law school deans from around the country and some 40 law faculty have expressed interest in the findings, Shultz says.

They've also heard from many lawyers, including Berkeley Law alumni. Some were concerned that the work was tantamount to undermining the school's effectiveness by lowering its standards, Shultz says. She invited conversations that she says ended up moving all the naysayers at least to neutral on the question.

"I wanted them to know that our tests are race-blind, that we are not putting our thumbs on the scale," she says.

Even UCLA law professor Richard Sanders, one of the country's best-known affirmative action skeptics (his work contends that affirmative action makes it harder for African Americans to succeed in law school), told the California Bar Journal that he had no problems with Shultz and Zedeck's approach, though it needed broader testing, the journal wrote last month.

Mostly, the response has been "wow, this would be really great if it could actually work," Shultz says. "We've gotten many emails from lawyers who say, 'I am your poster child, I had terrible LSAT scores but I had this very important and satisfying legal career.' "

The Bar Association of San Francisco, which recently launched its own Destination Law School to help diverse students across the college-to-law-school gap, is extremely interested in the Berkeley research and invited Shultz to address several conferences.

"This is really, really important," says Yolanda Jackson, BASF deputy executive director and diversity director. "A lot of college students who have dreamed about being lawyers get stopped at the LSAT point." Broader testing would reveal the intelligences and life skills they do have, she says.

The Society of American Law Teachers and the Clinical Legal Education Association have also shown an interest, according to Shultz.

And some law firms have asked to use Shultz' and Zedeck's materials in their own hiring decisions.

But for now, Shultz says, the researchers want to keep the focus on law school admissions, and a national test of the findings, "because that's where we are going to expand the pool of potential lawyers."  Law school is an avenue to significant amounts of money and influence, and it's important that access to such a scarce and valuable resource be fair, she adds.

A new phase of the project would try out Zedeck and Shultz's tests on graduates from a wide range of law schools. If it finds funding and turns out as well as the work so far, Edley says Berkeley Law "definitely will participate in an experiment with other leading law schools to test it in practice."

The full Law School Admissions Project report is available online (PDF format).