|
Developing and Implementing Assessment of Student Learning Outcomes Andrea Serban and Jack Friedlander
1. A Perspective on Good Practice in Community College Assessment, Trudy W. Banta, Karen E. Black, Susan Kahn, Julia E. Jackson
Observations and Reflections 1. University level faculty focus more on research and other priorities within the academy than on high-quality education for students (Massy,2003). 2. To gain our constituent’s support, we must gain their trust by emphasizing service, innovation, and flexibility in the education we provide. 3. Early attempts to assess student learning in community colleges (1990s) often relied on standardized instruments (ACT). Faculty at both the 2 and 4 year institutions found such instruments to be of limited value. 4. Faculty have begun to align assessment approaches with locally developed, campus-specific goals for student learning and accomplishment. 5. Butler Community College (Kansas) developed an ongoing, systematic assessment program for general education. a. Faculty develop their own course-based assignments designed to assess specific, desired outcomes such as critical thinking skills. b. Students complete the assignments and receive grades on them. c. During the following semester, samples of the student work are analyzed by interdisciplinary groups of faculty using a common scoring rubric. d. Findings are submitted to an assessment team that aggregates the data and reports institution-wide results. e. These data are discussed and further analyzed in departmental faculty meetings. f. During the third semester of the assessment cycle, the director of assessment and a faculty representative from the assessment team meet with faculty and administrators to review findings and discuss needed improvements in curriculum, planning, budgeting, or the assessment process. g. To begin the next cycle, the director of assessment identifies for each department the courses from which samples of student work will be drawn. h. Faculty then generate the assignments that will be assessed, and the process continues. 6. Characteristics of good practice in the three phases of assessment. a. Planning 1. Involves stakeholders from the outset to incorporate their needs and interests and solicit later support. 2. Begins where the need is recognized. 3. Has a written plan with clear purposes that are related to goals that people value. 4. Bases assessment approaches on clear, explicitly stated program objectives.
b. Implementing 1. Has knowledgeable, effective leadership. 2. Recognizes that assessment is essential to learning. 3. Includes faculty and staff development to prepare individuals to implement assessment and use the findings. 4. Uses multiple measures. 5. Assesses process as well as outcomes. 6. Occurs in an environment that is receptive, supportive, and enabling. 7. Incorporates continuous communication with constituents concerning the activities and findings. c. Improving and Sustaining 1. Produces credible evidence of learning and organizational effectiveness. 2. Ensures that assessment data are used to improve programs and services. 3. Provides a vehicle for demonstrating accountability to stakeholders. 4. Encompasses the expectation that outcomes assessment will be ongoing, not episodic. 5. Incorporates ongoing evaluation and improvement of the assessment process itself.
2. Assessment of Student Learning Outcomes at the Institutional Level, Andreea M. Serban
“Assessment is the systematic collection, review, and use of information about educational programs undertaken for the purpose of improving student learning and development” (Banta,1999).
“Student outcomes assessment is the act of assembling and analyzing both qualitative and quantitative teaching and learning outcomes evidence in order to examine their congruence with the institution’s stated purposes and educational objectives” (Volkwein,2003).
Observations and Reflections 1. Effective assessment begins with clear, overall institutional goals and values. 2. The following areas of educational activity have distinct importance for linking purposes, resources, and educational outcomes. a. Basic skills education b. General education c. Major d. Student’s personal and social development 3. Basic skills assessment needs to include exit proficiencies and tracking of student’s transition and performance in college-level courses. a. Include comparisons between performance in college-level courses of students who transitioned from basic skills verses students who did not need remedial education. b. Include non-cognitive factors of self-reported measures of attitude, satisfaction, and goal completion. c. Incorporate institutional interventions. 1. San Diego Community College District instituted mandatory enforcement of all course prerequisites. 2. Create collaborative partnerships between community colleges and feeder high schools. 3. Implement computer-aided instruction systems. 4. General education – “An essential aim [of general education] is to activate the verbal, numerical, and visual skills necessary to analyze and synthesize information, construct arguments, and identify and solve problems” (Banta,1999). The broad competencies that are the subject of assessment related to general education are: a. Cognitive abilities – critical thinking, problem solving. b. Content literacy – knowledge of social institutions, science, and technology. c. Literacy skills – communication, information skills. d. Value awareness – multicultural understanding, moral and ethical judgment. e. There are three different types of approaches to assessment of general education. 1. Individual course 2. Multi-course 3. Non-course f. Methods and instruments used for assessment of general education include: 1. Standardized tests 2. Self-reported measures of progress obtained through locally developed or published surveys, essays, class discussions, class exit assignments, course-embedded assessment activities, portfolios, and capstone courses. 3. Reviews of course syllabi. 4. Inventories of class assignments. 5. Common final exams. 5. Students’ personal and social development. a. Student services influence the personal and social development of students (Volkwein,2003). b. The assessment techniques most frequently used to assess this growth include: 1. Surveys 2. Interviews 3. Focus groups 6. Reporting and using assessment results a. In order to ensure the primary purpose of assessment (to improve teaching and learning), improvement can occur if adequate, timely, and systematic feedback is provided to all who are involved in assessment including faculty, administrators, and the community. b. Design the reports to be audience specific. c. Design reports for major areas (basic skills, general education, etc.) d. Link the reports to internal processes such as program review, planning, and budgeting. 7. Assessment expertise and skills a. One of the major challenges in building, sustaining, and effectively utilizing student learning outcomes assessment is having the expertise and skills on campus. 1. Vision – There must be a broad and comprehensive grasp of institutional goals and purposes with a clear view of how assessment processes and outcomes can be used to advance these goals and purposes. 2. Understanding of the college – There must be a clear conceptual understanding of how the college functions and the strengths and limitations of faculty and administrators as they perform their roles. 3. Functional knowledge – A thorough knowledge of measurement theory, statistical methods, and research design is needed. 4. Technical know-how – Should possess the practical knowledge of techniques of data collection and analysis. 5. Understanding of relevant concepts. 6. Good communication skills. 7. Academic qualifications. 8. Teamwork
3. Engaging Faculty and Students in Classroom Assessment of Learning, Fay Rouseff-Baker, Andrew Holm
Observations and Reflections 1. Engaging the faculty and students in authentic assessment of student learning at the classroom level through the use of CATS.
4. Assessment at the Program Level, Trudy H. Bers
Observations and Reflections 1. Programmatic assessment “helps determine whether students can integrate learning from individual courses into a content whole. It is interested in the cumulative effects of the educational process” (Palomba and Banta, 1999). 2. It focuses on the extent to which each student in the program acquires the knowledge, skills, beliefs, and feelings specified as program outcomes. 3. It focuses on gauging the learning of a group of students in the program rather than each student in it. 4. Assessing a group of students in a program is of more value to the department or institution as it can be used to improve courses, programs, and services. 5. Types of programs: a. A sequence of prescribed courses – nursing b. The general education component of an Associate Degree c. Courses in a specific discipline – psychology d. Precollegiate or remedial courses e. Special programs for selected students 6. Assessment approaches a. Capstone courses – for this, course learning outcomes can be interpreted as program level outcomes. b. Vendor or industry certification examination where there is external validation that the student has acquired the knowledge and skills identified by the vendor or organization as essential for a particular job or credential. c. Institutional or departmental testing that requires faculty to agree on one or more standardized or industrially developed tests that cover all or most essential elements of a program. These tests are administered to all students at the completion of the program. d. Satisfaction surveys that reflect student and alumni self-reported estimates of learning as indirect evidence of student learning outcomes. e. Portfolio assessment that contains documentation of learning and development across the spectrum of program objectives. f. Performance manuals that lists and briefly describes behaviors a student should be able to execute with competency at the conclusion of a program. Teachers evaluate students’ abilities to perform these behaviors, which may be observed in a class, at clinical settings, through service learning, or elsewhere. g. Narratives – Brenner (1999) suggests a novel approach to assessment: having students recount their experiences through stories while faculty assess learning by listening to and questioning students to determine whether they demonstrate understanding of the context and content of the situations they describe. h. Culminating project that must be broadly defined and reflect learning and ability to integrate information from across the curriculum. i. Transfer to and success in another institution as an indirect indicator of student learning by acceptance at and transfer to a four year college or university. 7. Challenges of implementation a. Defining programs. b. Identifying students who are at the end of a program. c. Faculty concurrence on key learning outcomes. d. Resources to implement some assessments. e. Sporadic or missing feedback on external certification or licensure examinations. f. Sustaining the assessment effort.
5. Implications of State Performance Indicators for Community College Assessment, Joseph C. Burke, Henrik P. Minassians
Observations and Reflections 1. The 1990s moved accountability for community colleges from outcomes assessment to performance reporting. 2. In theory, performance reporting asserted state priorities, adopted indicators assessing their achievement, and allowed comparisons among peer institutions. It pledged a shift from process and inputs to outputs and outcomes. 3. Leaders in both community colleges and four-year campuses have failed to develop robust indicators of student learning outcomes capable of competing with input and process measures of quality and of winning acceptance in the academic community and in state capitols. 4. Community colleges should have reporting indicators that reflect their distinct goals. 5. Since performance reporting programs become increasingly invisible on campuses below the level of vice president, a system of internal reporting from academic departments on a limited list of core indicators is needed to develop an unbroken chain of performance reporting.
6. The Role of Student Learning Outcomes in Accreditation Quality of Review, Barbara Beno
“In recent years, accreditation standards developed and used by most of the regional accreditors have changed to incorporate the assessment of student learning as a central process in evaluating institutional effectiveness.”
Observations and Reflections 1. Accountability refers to the public’s fundamental interest in what students are supposed to learn and how ell they are learning. 2. Accreditation and student learning a. In assessing institutional quality, accreditors are evaluating the student learning produced by the institution in the context of the institution’s own mission, its stated learning objectives, and its identified means of assessing student learning. b. The college must identify the expected student learning outcomes in the context of our mission and our curriculum and develop means for assessing student learning. c. Institutional effectiveness requires institutional assessment and improvement strategies that ultimately support learning or result in improved student learning. d. Accreditors will require community colleges to collectively attribute meaning to the results of learning assessment and to plan institutional improvements that will result in learning. e. Accreditors will want to examine evidence that the institution is meaningfully engaged in assessing student learning, and they will want to see some evidence of student learning. f. Community colleges must demonstrate setting appropriate student learning goals for course, program, certificate, or degree offered. 1. These must conform with the college’s standards for quality. 2. They must be consistent with higher education standards of quality. 3. They must address the content and level of learning students are expected to achieve. 4. They must be in writing and communicated to faculty and students. 5. Evidence might include course syllabi, official course outlines, or other documentation. 6. There must be evidence of course and program goal alignment through recorded discussions, review, and revision. g. Colleges must collect evidence on how well students are learning. 1. Institutional strategies. 2. Faculty strategies (formative and summative). 3. Methodologies must be valid and reliable. 4. Keep records of experimental strategies. h. Document student achievement of expected learning goals of the institution. This includes results compiled by individual faculty, program, degree, or certificate. It could include learning portfolios. i. The institution will be evaluated based on its degree of engagement with the assessment of learning all the way to the level of the individual. 1. Demonstrate an institution-wide understanding of educational quality. 2. Demonstrate institution-wide commitment to improved learning. 3. Demonstrate ways assessment and interpretation have informed college planning. j. Mesa Community college’s example of seven areas of learning for its general education program: 1. Communication 2. Problem solving and critical thinking 3. Numeracy 4. Arts and humanities 5. Scientific inquiry 6. Information literacy 7. Cultural diversity In each of these areas, the college designed general education outcomes that are each measured with a different assessment tool. Students are assessed at the beginning of study and at the point of completion of general education. A faculty committee interprets results and develops themes of interdisciplinary faculty work and development over subsequent years to enhance curriculum and pedagogy in order to improve learning. k. Prompts for faculty evaluation of specific learning outcomes covered in their courses. 1. Is the outcome explicitly stated as an outcome for the course? 2. Are the students asked to demonstrate their competence through course work that is assessed by the instructor? 3. Do students receive formal feedback on their mastery of this learning outcome from the instructor? l. The time frame for full implementation is between 10 and 15 years.
7. Assessment of Online Education: Policies, Practices, and Recommendations, John Milam, Richard A Voorhees, Alice Bedard-Voorhees
Observations and Reflections 1. There is relatively little empirical research to guide deep policy discussions about assessing online education. 2. If learning is different in an online environment, assessment and methodology should be different, too. 3. In fully online environments, multiple measures, formative and summative assessment over the course timeline, and electronic interaction with the learner are sound assessment practices. 4. Western Governors University has been a leader in the measurement of online learning. 5. Assessment tools: a. “Assessing Learners Online.” b. Western Cooperative for Educational Telecommunications c. Student satisfaction surveys and retention rates
8. Learning Outcomes for the Twenty-First Century: Cultivating Student Success for College and the Knowledge Economy, Cindy L. Miles, Cynthea Wilson
Observations and Reflections 1. Can students do what their degrees imply? 2. The adoption of “21st Century Skills.” a. Communication skills (reading, writing, speaking, listening) b. Computation skills (understanding and applying mathematical concepts and reasoning, analyzing and using numerical data) c. Community skills (citizenship; appreciation of diversity and pluralism; local, community, global, and environmental awareness) d. Critical thinking and problem-solving skills (analysis, synthesis, evaluation, decision making, creative thinking) e. Information management skills (collecting, analyzing, and organizing information from a variety of sources) f. Interpersonal skills (teamwork, relationship management, conflict resolution, workplace skills) g. Personal skills (ability to understand and manage self, management of change, learning to learn, personal responsibility, aesthetic responsiveness, wellness) h. Technology skills (computer literacy, Internet skills, retrieving and managing information via technology) 3. Implementation of all learning outcomes across the curriculum for all students is the goal. a. Implementation in discrete courses. b. Implementation in some programs or disciplines. c. Implementation across the curriculum. |