BEST ASSESSMENT PRACTICES AT ST. THOMAS

The Bush Foundation Grant Program Seminar, January 24-25, 2006

Tuesday, January 24 – morning session

Michael Livingston:

1. Connect assessment to mission and objectives.
2. Assessment is yours!
3. It is doable – use existing data, instruments; use successful assessments.
4. Data must be useful and lead to improvements in education.

Tom Rochon:

We are doing our work in a larger, national context. UST is proactive.

1. Commission on Learning in Higher Education – set up to develop an assessment idea how higher education can be held more accountable – “Collegiate Learning Assessment”
2. NSSE – National Survey of Student Engagement

Angie Barretta-Herman:

• Assessment at UST - Big shift from input to effect/output – are the students learning what you expect them to?
• 1996 – an assessment plan was submitted to North Central and approved – assessment for each department.
• 2003 – over 100 assessment plans were in place.

Plan – every 4 years each department submits a summary of their assessment.

There are 7 externally assessed units at UST. Two parallel plans: internal and external.

UST’s Academic Assessment Coordinating Committee (AACC) suggests a change is how we structure institutional assessment.

• Proposal – to decentralize assessment – it is closer to actual activity – eliminate university-wide committee; faculty-led assessment committee for each unit – internal assessment committee; University-wide coordinating body consisting of assessment coordinator from each unit; subcommittee for assessment of core curriculum.
• Need for more use of existing data – disseminate through deans.

Groups identified good and useful aspects of departments’ assessment reports:

What works:

• Focused reports, beyond self-report and use of measurements
• Focused objectives (at what level? Program/department/major)
• Use tools that imply actual changes
• Share results or its usefulness – include students in assessment process (e.g. Department newsletter); have transparent process
• Make assessment educational opportunity (M. Livingston)
• Use external exams when available
• Track placement
• Use quantifiable/qualitative measures, some open-ended
• Portfolios that include pre- and post-
• Assessment of examples of writing at multiple points in time
• Use feedback from graduates and communicate that to current students

What I can use:

• Different rubrics and criteria
• Use a rubric for student papers and presentations
• Embed periodic assessment in curriculum
• Use multiple assessments
• Develop a matrix of courses/skills – e.g. Presentation, writing
• Alumni interviews

Question: Are we assessing student outcome or programs?

Answer: Both students and programs (M. Livingston). Faculty productivity & vitality.

Closing question: What will you implement in your department?

• Start with department mission statements and identify objectives – what we want our students to learn. Develop matrixes and measurement.
• Include faculty to get ownership. Include students.
• Emphasize program not people
• Develop rubrics for assessment
• Have baseline data – evaluate learning and retention
• Proper sequence: mission – outcomes – strategic planning

Presentations by department chairs who have good assessment plans:

Tom Hickson, Geology

· Department’s main interest was growth in majors and total course enrollment – quantifiable statistics in that area; evolving document

· Summarized students’ evaluations

· Developed goals matrix – every course has goals

· Yearly meeting of faculty – ask questions about assessment/goals

· Try to respond to local needs

· Small department – can respond dynamically, quickly

· Define core content – a small range of things

Chris Vye, Professional Psychology

• Competitive program with other local programs
• Forms used to evaluate comprehensive exams – evaluated by faculty
• Doctoral programs – competency evaluation
• Program evaluation every 5 years – external evaluation by APA – evaluated on multiple dimensions of the program
• Alumni surveys – maintain their own database on satisfaction and professional accomplishments
• Other data from licensure exam scores, employers’ surveys, internship competition
• Internal evaluation of retention and satisfaction
• Experimentation with focus groups of Master’s students conducted by Ph.D. students – not a lot of participation
• Feedback loop – retreats about mission

Mark Stansbury-O’Donnell, Art History

• There are no standardized tests in art history
• Developed 4 year plan for assessment – 5-6 majors per year; looked at what all students were doing – identified 4 areas (in the report)
• Identified basic skills in writing, presentation, methodology, knowledge of the discipline
• Identified standards – 10-20 quality scales; identified matrices; kept portfolios
• Presentations were videotaped; senior papers were collected; student surveys and exit interviews based on matrices; multiple-choice tests to test knowledge
• All materials were put in a box every year and assessed every 4^th year; the chair looks at the box every year
• Department is developing problem-based courses instead of survey/intro courses – now needs a new assessment tool to assess problem-based/problem-solving skills
• External revue of department every 8 years

Joe Brom, Chemistry (Lynn Hartshorn is assessment coordinator)

· For over 50 years undergoing external evaluation by American Chemical Society

· Major field test in chemistry – standardized test, quantifiable, reasonable expectations – every senior must take it

· Alumni surveys

· Senior attitudinal survey

· Four goals/academic achievement

· Annual faculty summer retreat – discuss assessment results

· We take a hard look at measurement procedures and take action

· Students are informed about the objectives of the standard test

Bob Farlow, Political Science

• Varied enrollment pattern – up and down
• We asked a question “What we don’t now about majors?” – We don’t know how they compare with majors at other institutions.
• In 1994 we adopted national standardized test – administered at the end of senior seminar – students concerned that it goes on their record
• 1994-2000 results were always the same
• There was a need for collective decision on standards for seminar papers – faculty read everybody’s papers. Faculty learned a lot about each other and had better sense what the students were doing.
• Developed quantitative methodology – for that we selected public opinion polls and asked students to examine their nature and intentions – we tested each major on that. Results tend to be the same every year

Chester Wilson, Biology

• How to deal with an assessment program in a department that does not want it?
• Assessment needs to be internally grown not imposed. It has to be doable – not good if nobody wants to do it.
• We try to look for trends and we compare yearly results
• Questions about allied courses – there is disconnect between how students like the courses and how useful they think they were
• Serious problem is how to utilize this data in a serious way
• Will start to implement measurements for writing ability

Terry Langan, Economic

• Alumni survey every summer
• Evaluation of senior papers

Mark Neuzil, Journalism

• Lots of time is spent to develop journalism major community – a large number of majors and minor – 400+ this year
• Already had some measurement tools and baseline before 1996
• Three main areas of mission statement – what we say we want to do and how we are doing it
• Assessment tools: employers’ entry exams, writing test (mechanics and content), senior projects, oral ethics exam
• Representative sample is tested; alumni are not useful – response is problem

Wednesday, January 25

Caitlin McLaughlin, Institutional Research - Overview of institutional data research

3 Major areas of data:

1. Banner – a) most available on ad hoc basis – connect by RFS system

b) Data Dashboard – coming soon (e.g. GPA of students by departments)

c) Persistence rates

2. Student surveys

· CIRP (Cooperative Institutional Research) – entering freshmen survey, detailed information – most useful to profile a class of entering freshmen, their entering characteristics

· ACT – senior student opinion survey; 41% in 2004 – on web basis, does not collect ID number but collects majors; demographic data; data on satisfaction with college services and college academic environment; data on student outcomes; has proprietary questions devised by UST

· NSSE – first time done in 2005 – in alternating years with AC

3. Alumni surveys
• Graduate survey – annual, 100% students – 45% response last time
• UG survey – one year after graduating, offered by Career development Office, on line “Post Graduation Survey” - there are department-specific questions
• NCHEMS - 3-year survey, long-term alumni survey – will not be used any longer

Tom Rochon - Overview of National Survey of Student Engagement (NSSE)

• Survey of freshmen and seniors – UST chose on-line option
• Tells us some very specific things, but there are also challenges
• NSSE Benchmarks (handout)
• Value of NSSE – what students have done, what skills they acquired
• All ACT schools participated in 2005 – it would be useful to get together with coordinators at all those schools
• See Power Point presentation - handout

January 25, afternoon – Student Outcome Assessment – recommendations

• Describe the objectives of your program/department relative to student outcome assessment and the evidence that shows you are achieving these objectives.
• Identify actions you are taking to address the objectives not yet achieved.

Bob Werner’s re-drafted plan for assessment (Geog. Dept)

Plan Intended Learning Outcome Assessment

Start with department Teach majors to: Test, senior survey

Mission statement a) synthesize info from a NSSE 2a-e, 11a-h

variety of sources

Deliver good LA ed, b) analyze geographic problems

Teach well

Deliver career preparation Prepare student for career Student evaluations,

Senior surveys, track

jobs/grad school

Faculty-student research In-depth research experience Track projects and

with a faculty member majors who made

conference presentations;

calculate % of majors who

had internships; NSSE 7d

Community Students share learning Number of faculty/student

Meetings; exit interviews

Advising well Develop career plan NSSE 129; ACT 9, 10:

Discussion

Exit interviews

• What departments do them? Answer: several
• Mari Heltne, QMCS, asked seniors to write questions that would assess department/program and then administered those questions to her students.
• Michael Livingston told about a colleague at St. John’s who developed a survey about advising

Use of assessment data

• Advertise your program
• Faculty annual report – relate to department mission and tell how you advanced that mission in relation to student learning

Michael Livingston’s comments

• Develop baseline data: pre-test data; at the beginning of student career data; trend data
• Use multiple measures
• You can strengthen the assessment plan by adding other measures