Empirical Methods for Measuring Effectiveness of Innovation
I. “Customer Satisfaction” surveys
A. Current students: The Law School Survey of Student Engagement (http://lssse.iub.edu/ )
1. Question 8b: “To what extent does your law school emphasize ... encouraging the ethical practice of law?”
2. Question 10n: “To what extent has your experience at your law school contributed to your knowledge, skills and personal development in ... developing a personal code of values and ethics?”
3. Complete survey instrument (2006) (pdf)
B. Recent graduates
1. James Moliterno, Professional Preparedness: A Comparative Study of Law Graduates· Perceived Readiness for Professional Ethics Issues, 58 Law & Contemporary Problems 259 - 286 (1995)
a. “How well did your law school experience prepare you for ______?”
b. Suggested emphases for law schools
2. The Canadian Lawyer (http://www.canadianlawyermag.com/ ): Annual Law School Survey (asks recent graduates to rate their own school)
a. 2007 Survey Results (pdf)
II. Psychological tests: Kennon M. Sheldon & Lawrence S. Krieger, Does Legal Education have Undermining Effects on Law Students? Evaluating Changes in Motivation, Values, and Well-Being, 22 Behavioral Sciences and the Law 261-286 (2004)
A. Subjective Well-Being
1. The Positive Affect/Negative Affect Schedule
2. The Satisfaction with Life Scale
3. The Beck Depression Inventory
B. Motivation and Values
1. The Aspirations Index
2. Personal Goal Assessment
a. Five law school goals (“things you will be trying to accomplish in the coming year”)
b. Rate among four reasons for pursuing each goal
(1) two autonomous reasons (intrinsic motivation)
(2) two non-automous reasons (extrinsic motivation)
3. Career preferences (from list of 15 careers)
a. Five service-oriented
b. Four money-oriented
c. Seven “other”
C. Other data
1. Demographic
2. LSAT
3. GPA
III. Performance based assessments: Effective Lawyer Client Communication (ELCC): An International Project to Move from Research to Reform Web Site Address http://law.gsu.edu/Communication/
A. Standardized Patients in Medical Education
1. Ker J.S., Ramsay J., Hogg G., Dewar G., & Ambrose L., Medical Use of Standardised Patients (pdf file) The Role of Simulated Patients in the Education of Doctors (Powerpoint Presentation) (Clinical Skills Centre: Ninewells Hospital and Medical School, University of Dundee, Scotland) (presented January 7, 2005 at the 7th Annual Conference of the Learning in Law Initiative, University of Warwick, England)
2. David Stern, Outside the Classroom: Teaching and Evaluating Future Physicians, 20 Georgia State University Law Review 877 (2004)
3. Lawrence M. Grossberg, Standardized Clients: A Possible Improvement for the Bar Exam, 20 Georgia State University Law Review 841 (2004)
4. Lynne Robins, The Establishing Focus Protocol: Training for Collaborative Agenda Setting and Time Management in the Medical Interview, Families, Systems & Health (2001)
5. Robert Smith et al, The Effectiveness of Intensive Training for Residents in Interviewing, 128 Annals of Internal Medicine 118-26 (1998)
6. The Use of Standardized Patients in the Teaching of and Evaluation of Clinical Skills, 6 Teaching & Learning in Medicine No. 1 (1994) (Table of Contents to Special Issue)
7. National Board of Medical Examiners (U.S.A.)
a. --Home Page
b. --Article on Standardized Patient Exam
c. --Video Explaining Use of Standardized Patients
8. Association of Standardized Patient Educators (ASPE)
a. -- History of Standardized Patients in Medical Education
b. -- Standardized Patient Methodology for Teaching and Assessment: How to Get Started (pdf)
B. Standardized Clients in Legal Education
1. Karen Barton, Clark D. Cunningham, Gregory Todd Jones & Paul Maharg, Valuing What Clients Think: Standardized Clients and the Assessment of Communicative Competence, 13 Clinical Law Review 1 (2006) (pdf) (reporting empirical evidence that standardized clients can provide reliable assessment of client interviewing skillls) (presented at the 6th International Clinic Conference as Do We Value What Clients Think About Their Lawyers? If So, Why Don't We Measure It?)
2. -- Excerpts from Valuing What Clients Think (pp 1-6, 35-41, 50-54, appendices 4-6) (pdf) (to print using Adobe Acrobat, select "shrink large pages" for "Page Scaling" and un-check the box marked "Auto-Rotate and Center")
3. Applied to “board certification” for lawyers: The Signet Accreditation Program for Lawyers in Scotland
4. The New Hampshire Bar Exam Pilot Project
a. Linda S. Dalianis and Sophie Sparrow, New Hampshire’s Performance-Based Variant of the Bar Examination: The Daniel Webster Scholar Program, 74 Bar Examiner, 23-26 (November 2005)
IV. Measurements of Moral Development
A. James R. Rest & Darcia Narvaez (eds), Moral Development in the Professions: Psychology and Applied Ethics (1994)
1. James R. Rest, Background: Theory and Research (Ch 1)
a. Lawrence Kohlberg: developmental stages of moral judgment
(1) Discusses Carol Gilligan critique of Kohlberg
b. The Moral Judgment Interview (Anne Colby)
c. The Defining Issues Test (DIT)
d. The Four Component Model (FCM)
(1) Moral sensitivity (interpreting the situation)
(2) Moral judgment (judging which action is morally right or wrong)
(3) Moral motivation (prioritizing moral values relative to other values)
(4) Moral character (having the courage, persistence, and skills to implement the choice produced by 1-3)
2. Muriel J. Bebeau, Influencing the Moral Dimensions of Dental Practice (Ch. 7) (Dental Ethics Curriculum at the University of Minnesota: “subjected to regular review ... among the most documented and intensively studied programs ... most dental schools have incorporated some parts of it”) For more information: Assessment of Education Programs in the Sciences, Center for the Study of Ethical Development, University of Minnesota: www.centerforthestudyofethicaldevelopment.net/
a. The Dental Ethical Sensitivity Test (DEST)
(1) students watch an interaction on video
(2) at a point in the video, students take on the role of the professional and respond on audiotape as if they were the professional
(3) using scoring manual developed in collaboration with practicing dentists
(4) the transcription is rated for the extent the student recognized the significant patient characteristics and professional responsibilities presented in the situation
b. The Defining Issues Test (DIT)
c. The Professional Role Orientation Inventory (PROI)
d. Role Concept Essay: “What Does It Mean to you to Become a Professional?”
(1) Scored for the presence of six obligations that distinguish occupations from professions
(2) Common reasons for giving low scores
(a) failure to see the limits of obligation
(b) failure to attribute the obligation to oneself
(c) narrow interpretation of the obligation
B. Muriel J. Bebeau, The Defining Issues Test and the Four Component Model: contributions to professional education, 31 Journal of Moral Education 271-95 (2002) (comprehensive literature survey, including studies of law)
C. Larry J. Shuman et al, Can Our Students Recognize and Resolve Ethical Dilemmas?, Proceedings of the 2004 American Society for Engineering Education Annual Conference (pdf download)
D. Law
1. Steven Hartwell, Promoting Moral Development Through Experiential Teaching, 1 Clinical Law Review 505 - 539 (1994-1995) (see especially pp 508 - 513 and 522 - 28, Defining Issues Test shows consistent improvement in moral judgment after completion of experiential courses in professional responsibility)
2. Thomas E. Willging & Thomas G. Dunn, The Moral Development of the Law Student: Theory and Data on Legal Education, 31 Journal of Legal Education 306 (1981) (neither the first year of law school nor a traditional third year legal ethics class produced any improvement in DIT scores)
3. Maury Landsman & Steven P. McNeel, Moral Judgment of Law Students Across Three Years: Influences of Gender, Political Ideology and Interest in Altruistic Law Practice, 45 South Texas Law Review 891-919 (2004)