Assessment Guide

This guide provides an overview of the program learning outcomes (PLOs) assessment process. In particular it provides strategies for how to collect, analyze, and use evidence to refine PLOs. For more help with developing PLOs, please click here. If you have any questions, please feel free to contact us at crte@ucmerced.edu.

A. Timeline and Goals

Generally, the time-frame for assessment plans is five years, with one Program Learning Outcome reviewed on an annual basis.
This timeline is recommended to fit with accreditation cycles, though teams are encouraged to design plans that best fit their assessment goals.  Not only does this process relate to accreditation, it also aligns with program review processes.  More broadly speaking, annual assessment reports serve as a program's historical record of growth and development.

Sample program goals

B. Outline of Program Learning Outcomes

Learning outcomes are statements of the knowledge, skills, (and possibly attitudes or values) that graduates of the program will possess. To formulate these statements, you might ask the question: “What will students know or be able to do upon degree completion?”
Sample program learning outcomes
Can Program Learning Outcomes be revised?

C. Evidence

Emphasis has been placed on learning outcomes because actual student work is the best evidence of how and what students learn.
What is the difference between direct and indirect evidence?
Why not grades?  What is the difference between grading and assessing?
Embedded Questions and Assignments

D. Process

Assessment involves the collection of student data, its analysis, and thoughtful evaluation and use of the results also known as ‘closing the loop.’
Using rubrics to articulate performance criteria and to communicate and evaluate standards
Why is it important to triangulate data?
What kind of financial support is currently available to fund these efforts?

E. Participants

Can students (undergraduate and graduate) be included in the assessment process?

F. Minor Program

Tips for how to focus assessment with minor programs

 

A. Timeline and Goals

Sample Program Goals

“Develop an issue-based approach to anthropological knowledge and practice that emphasizes common topics shared by multiple sub-fields.” Rationale: Students majoring or minoring in Anthropology at UC Merced develop a holistic view of the complexities of human societies past and present and around the world. Students conduct anthropological research, critically analyze anthropological research, and demonstrate the ability to communicate anthropological knowledge in different mediums to a range of audiences. Finally, as students are majoring or minoring in Anthropology understand the social worlds of others, they demonstrate that they better understand their own world, including their place in an increasingly globalized world. -- Anthropology Program

“Build a community of life-long learners that use the analytical and computational tools of mathematics to solve real-world problems.” Rationale: The Applied Mathematical Sciences program at UC Merced provides students with knowledge of the foundations of mathematics and the skills needed to apply mathematics to real-world phenomena in the social sciences, natural sciences and engineering. The program uses a “core + emphasis” model which teaches students the fundamentals while building expertise in an application area through the emphasis tracks. There is a core set of courses all mathematical sciences students take. Beyond these classes, students complete an emphasis track consisting of courses in other fields. With the depth and breadth of training of the major, students are well suited for immediate placement within specialized positions in industry or for further educational advancement. -- Applied Mathematical Sciences Program.

B. Outline of Program Learning Outcomes

Sample program learning outcomes

Graduates from the Physics B.S. program will have demonstrated the following learning outcomes:

  • Physical Principles. Students will be able to apply basic physical principles---including classical mechanics, electricity and magnetism, quantum mechanics, and statistical mechanics---to explain, analyze, and predict a variety of natural phenomena.
  • Mathematical Expertise. Students will be able to apply advanced mathematical techniques (e.g., calculus, linear algebra, probability, and statistics) in their explanations, analyses, and predictions of physical phenomena.
  • Experimental Techniques. Students will be able to take physical measurements in an experimental laboratory setting and analyze these results to draw conclusions about the physical system under investigation, including whether their data supports or refutes a given physical model.
  • Communication and Teamwork Skills. Students will be able to clearly explain their mathematical and physical reasoning, both orally and in writing, and will be able to communicate and work effectively in groups on a common project.
  • Research Proficiency. Students will be able to formulate personal research questions that expand their knowledge of physics. Students will be able to apply sound scientific research methods to address these questions, either by researching the current literature or developing independent results.

Can Program Learning Outcomes be revised?

Yes, ideally! Program Learning Outcome (PLO) revision is among the many changes that might occur from the assessment of student learning outcomes, as refining intended learning outcomes is part of the continuous improvement of learning.

In addition to potentially revising a PLO, assessment results could be used to stimulate faculty discussion, improve curriculum, re-consider curriculum content, examine skill development, change pedagogy, or improve the overall assessment process.

Your team might discover that given the way a PLO is written, the learning outcome cannot be “demonstrably” or measurably be achieved. For example, a PLO might state “Students will be able to value collaboration.” A faculty team may discover that being able to ‘value collaboration' is something that can be measured indirectly with surveys but cannot be measured directly with student work. In that case, the PLO might be revised to read “students will collaborate successfully”. Described this way, student collaboration can be measured once the team elaborates a rubric identifying criteria and standards for these criteria that together define “successful collaboration”.

Another possible revision includes choosing a more accurate active verb to describe a skill level. For example, a PLO might originally read, “Students will be able to synthesize complex issues within the profession to define innovative strategies for change.” While the synthesis of complex issues might become an important learning priority, it may be that the ability “to define innovative strategies for change” is an expectation more appropriate for graduate-level work.

PLO revision might also result from curriculum alignment. For example, a PLO might originally state that students will be able to demonstrate comprehension of a wide range of sub-fields within a discipline. As the major develops and curriculum becomes more aligned, those sub-fields may become more specific, so the PLO needs to be revised.
To summarize, revision of Program Learning Outcomes is part of the assessment process. Revisions may occur to refine descriptions of skill levels, reflect curriculum alignment, or revise according to what is measurable.

C. Evidence

What is the difference between direct and indirect evidence?

Direct evidence is actual student work showing student knowledge and skills. Examples might include embedded assignments, portfolios, senior thesis projects, or observations of presentations.
Indirect evidence is student, faculty or employer perceptions of student performance. Examples might include surveys, exit interviews, focus groups, or student self-assessment.

These types of evidence are meant to be combined to more fully understand a learning outcome; for more information, see triangulation of data

What is the difference between grading and assessing?

Assessment is an evaluative process distinguished from grading. Grades are given by individual faculty to individual students; assessment is concerned with whether a population of students achieves learning outcomes developed by a department for a course or program. While grades may not be used as assessment data, certain projects, exams or assignments that were given a grade may be used as long as evaluated by other faculty with a rubric.

Why not grades?

The use of grades in program assessment is highly discouraged for several reasons:

Grades indirectly represent the quality of student work as they often reflect more than just the students’ intellectual skills and abilities. For example, does a course have attendance requirements or strict due date policies?

Grades represent the perspective of a single member of the faculty based on work in a single course. As such, they do not provide a programmatic perspective on the attributes of student achievement. Part of the goal of program level assessment is to develop agreement among faculty about what proficiency in a given skill or knowledge area looks like. Program learning outcomes and the levels of proficiency graduates will exhibit should represent shared goals of the faculty working toward an exceptional program that produces successful graduates.

Grades are an inefficient way to gather useful program assessment data. If grades are used in combination with faculty review of actual student work (which is required, because it is the heart of the matter) then basically both lines of evidence - the direct line, the actual student work and the indirect line, grades - both offer faculty perspectives on the quality of student work. Since faculty must review actual student work as part of their assessment plan for a given outcome, why not combine it with a line of evidence that provides insight into the perceived quality of the work from a different perspective - either that of the graduates themselves, as part of an exit survey for example, or from their employers? Using one or both of these alternative lines of indirect evidence, instead of grades, can provide meaningful insights.

Embedded Questions and Assignments

Embedded assessment, in its broadest sense, provides opportunity to evaluate student progress and performance from outcomes integrated into instructional materials, indistinguishable from day-to-day classroom activities. These embedded questions or assignments are often designed by faculty committees, providing assessment beyond what was planned for the particular course by its instructor(s). If reviewing existing exam questions, select multiple questions for reliability in terms of the conclusions one draws (Wilson & Sloane 2000).

To gain perspective on stages of learning, faculty could consider the progression of a student through a series of courses (program-specific required courses, in particular). With this goal, an embedded assignment is tracked through various levels of coursework to review progressions. The following is a strong example of how this works; the Classics department at the University of Colorado embedded questions in the following way: “Faculty review students' translations of selected passages in the final exams of classes at various levels, from beginning courses to advanced ones. The students have not seen these passages before. The translations are evaluated by at least two faculty members, one of whom is the course instructor.” To motivate students to perform authentically on the assignment, embedded materials are graded within the course as well.

Works Cited:
Wilson, M., & Sloane, K. (2000). From principles to practice: An embedded assessment system. Applied Measurement in Education, 13(2), 181-208. [Full-Text Article]

D. Process

Using rubrics to evaluate and communicate standards

Previous to the development of an embedded assignment or exam questions, it is useful to develop a rubric that articulates criteria and standards. As a quick matter of definition, criteria are “the qualities we look for in student evidence” (Driscoll and Wood, 2007) and, therefore, the means by which you will judge how well a student has achieved a learning outcome. Standards are the actual levels of performance a student might achieve with respect to a criterion, for example, satisfactory, commendable, exemplary.

To offer a working example, imagine that in a given year you will be evaluating this Program Learning Outcome from the Spanish Major: “Possess Spanish listening and speaking skills equivalent at least to the advanced level of the American Council on the Teaching of Foreign Languages Proficiency Guidelines: Understand the main ideas of most speech in a standard dialect and use oral Spanish to speak about a variety of everyday activities, school, and work situations, but also to support opinions, explain in detail and hypothesize.” One example of direct evidence of this outcome will be oral presentations. Fortunately, the ACTFL offers standards, including superior, advanced (high, mid, low), intermediate (high, mid, low), and novice (high, mid, low). Criteria should be determined by faculty to describe skills or attributes, while standards for a criterion differentiate novice from superior. During those conversations, it helps to discuss national disciplinary standards (when available) and examples of student work.

A recommended rubric development tool is Rubistar. At Winona State University's Assessment and Research site, sample rubrics for every imaginable skill or activity are available. Open Education Practices is a comprehensive teaching and learning resource center as well.

Why is it important to triangulate data?

A well-balanced and reliable assessment plan includes a carefully selected but diverse set of methods, each of which provides a different perspective on the question at hand “Are the students as a whole achieving the particular learning outcome at the desired level(s) of performance?” Much as a location in space cannot be determined without three reference points, drawing a conclusion about student learning based on a single line of evidence does not fully nail down the conclusion. Using multiple lines of evidence, usually direct and indirect, to enhance the reliability of conclusions is also in keeping with good, scholarly practice.

To provide an example of how this works, the Physics Department at Kansas State University evaluated a PLO that is focused on fluency in professional discourse, in particular the extent to which students can distinguish between everyday and specialized usage. Students were asked, in an exam or workshop activity, to write definitions for terms like “momentum,” “force,” etc. A focus group was conducted for further elaboration. Relevant embedded questions were evaluated to assess current fluency and its impact on success more broadly. The combination of these approaches results in enhanced reliability and information-rich data.

What kind of financial support is currently available to fund these efforts?

The Center for Research in Teaching Excellence is currently offering expedited mini-grants to fund assessment plans. Financing can be used to compensate for workload, staffing needs, national exams, etc.

E. Participants

Can students (undergraduate and graduate) be included in the assessment process?

SATAL (Student Assessment on Teaching and Learning) is a program designed to provide feedback to professors interested in enhancing student learning. Carefully selected and trained UC Merced students respond to professors’ invitations to gather data on classroom activities and give them confidential feedback. The program is open to all instructors and can serve as a resource 1) to provide faculty members with feedback that helps them gain a better sense of the student experience in their classes; 2) to foster closer relationships between faculty and students; 3) to allow students to reflect on teaching and learning; and 4) to improve the quality of education at UC Merced. There are different options for faculty to participate in the program. Instructors may require the student consultant to be a recorder/ observer, filmmaker, faux student, class interviewer, or another role, depending on the evaluative feedback on the classroom activities or particular learning issues instructors envision to obtain. SATAL at UC Merced values the voice of the students in their educational development while providing instructors with valuable insights to supplement student evaluations and peer reviews. For more information or to sign up for the program, contact Adriana Signorini in COB 341, or asignorini@ucmerced.edu

F. Minor Program

Tips for how to focus assessment with minor programs

How can minor and major programs be distinguished, particularly in terms of levels of proficiency?
Example:: “Because of their more limited exposure to Sociology coursework, we expect minors to achieve three of the five learning goals we have outlined above. Specifically, we expect PLOs #1 (Think critically about the causes and consequences of social inequality) and #4 (Communicate verbally and in writing about social concepts) to fully apply to our minors. We also expect PLO #2 (Design and evaluate empirical sociological research) to partially apply to our minors. This is because minors are not required to take courses in statistics, so we only expect them to gain knowledge related to this outcome from our sociological research methods course. We assess these three goals for our minors using the same methods and timeline outlined for the major, except that PLO #2 will only be assessed in research methods (but not statistics) for our minors. All of our assessment tools will collect information about student class standing and major / minor status so that we can think about our pedagogy and course requirements as they may apply differently to each group of students.” – Sociology Minor Program