Assessing Learning Outcomes Online: A longitudinal, collaborative, inter-institutional case study

Assessing Learning Outcomes Online: A longitudinal, collaborative, inter-institutional case study

Ryan Fowler (Center for Hellenic Studies) and Amy Singer (Franklin and Marshall College)

1. Overview of Paper and Introductions

a. Sunoikisis, an inter-institutional initiative to supplement the curriculum for small
classics programs (, offered its first course for advanced students of Latin
in the fall of 2000. This presentation will (1) describe the survey instruments and
interview protocols and tools the program has developed for measuring outcomes, (2)
describe the data from the fall semesters of 2013 and 2014, and (3) discuss how the
design of the courses evolved in response to the evaluation process.

b. We are Ryan Fowler and Amy Singer. Ryan has been a participating faculty member
since June 2009, and been the director of both Greek and Latin Sunoikisis courses
since June 2012. His research interests center primarily on Greek philosophy, rhetoric,
and religion during the Roman Empire and Late Antiquity… Amy has been involved
with Sunoikisis assessment projects, and has coordinated data collection efforts, since
June 2012.

2. History of Sunoikisis and Assessment Context

a. Sunoikisis is a national consortium of Classics programs. Since 1999, Sunoikisis has
yielded new collaborative and interdisciplinary paradigms of learning in the liberal arts
for the 21st century.“Sunoikisis” comes from Thucydides (3.3.1) in reference to the
alliance formed by the cities of Lesbos (Methymna excluded) in their revolt against the
Athenian empire in 428 B.C.E. Likewise, this collaborative program seeks to develop a
set of common goals and achieve a degree of success and prominence that goes beyond
the capacity of a single program. Sunoikisis enables students and faculty at participating
institutions to benefit from opportunities normally available only at large research
institutions, while maintaining the advantages of a small liberal arts learning

b. The curricular elements within Sunoikisis include inter-institutional collaborative
courses, excavations, internships, travel study, undergraduate research symposia, and
faculty development seminars. The curricular elements of Sunoikisis expose our
students to a wider range of subject material and faculty than would be possible
otherwise. Indeed, the president of an elite northeastern college commented in October
2004 that the Sunoikisis program surpasses programs offered by large institutions in
that the collaborative nature unusually enriches it in terms of content and
methodological approach. The program, by providing a range and quality of
opportunities for majors, prepares students who choose to continue their training in
graduate school to compete with graduates from the leading research universities in the

c. Currently, Sunoikisis offers two upper-level seminars each fall semester: each fall
semester, Sunoikisis offers one Greek course and one Latin course, offered in cycles.
The Sunoikisis faculty develop these courses each June, working with a course
consultant who is an expert in the fall seminar’s topic.

d. Each course consists of several elements: readings, online discussion, online common
session, and face-to-face meetings with participating faculty on the students’ home
campus. Readings and asynchronous online discussion precede and prepare for the
weekly online common session.

e. A different faculty member leads the common sessions each week, in order to expose
students to a variety of approaches and expertise. Generally, the sessions consist of an
audio lecture and discussion via a chat room. Shortly before the common session begins,
students will log into a virtual classroom with an associated chat room, which allow the
director of the course (Ryan) to monitor the reception of the audio lecture, ensure that
students receive credit for attending, and permit students and faculty members to direct
questions to the lecturer. During the lecture, faculty members can display slides on the
whiteboard of the virtual classroom.

f. Because these Sunoikisis courses can be used asynchronously, during later semesters in
the same academic year or within future academic semesters or projects, it is also
possible to engage with these course materials after the “live” courses have concluded.
Participating faculty and departments have ongoing access to Sunoikisis syllabuses,
recorded common sessions, and even archived online discussions.

i. This presentation will discuss assessment projects that relate to these Sunoikisis
courses, which take shape every fall semester and which focus on either Greek or

g. In addition, Sunoikisis has just begun to develop a new course for the spring semester:
Sunoikisis faculty are offering an inter-institutional course in which the participants will
read the Odyssey in a focused way, averaging two books each week, and in turn
respond to writing prompts in working groups made up of participants from all
collaborating institutions. Each week a synchronous common session meeting is held in
which all students are in a chat room together, and all professors, along with a few
student presenters, discuss the conversation thread for their group that week. In short,
the purpose of this course is to read the Odyssey while at the same time developing
methods and approaches to close reading and interpretation.

i. This presentation will not discuss assessment projects relating to this new Sunoikisis
undertaking, although current assessment models and approaches might soon be
applied to newer programs such as this one.

3. Assessment Projects and Outcomes

a. In 2002 the Andrew W. Mellon Foundation funded a multi-year study of the Sunoikisis
project by outside evaluators, who developed additional survey instruments and
interview protocols. Following the completion of that study in 2005 and the transition of
the sponsorship of the project from the Associated Colleges of the South to the National
Institute for Technology in Liberal Education in 2006, however, the program stopped
collecting assessment data from the faculty members and students involved in the
courses, leaving the evaluation of the courses to individual participating institutions.

b. Since the summer of 2012, Sunoikisis has been taking a new approach to the assessment
of its courses, thinking critically and seriously about how to evaluate course goals and
learning outcomes given the particular challenges and strengths of an inter-institutional,
high-tech, high-touch program. We decided on a three-year plan.

c. Year 1: During the 2013-14 academic year, we began working with instructors to
determine their impressions of the course outcomes and learning goals for the students.
Through surveys and interviews, the program collected data about faculty members’
understanding of the learning goals for the courses. These interactions with the director
of the assessment program, in turn, focused the participants’ attention on the
relationship between the desired outcomes and the individual components of the course,
the content, and approach.

d. [Amy will show a few slides here, based on last summer’s data report.]

e. Data collected during Year 1 significantly affected the way that Sunoikisis courses for
2014-15 were designed, shaped the way that syllabuses were developed, and deeply
altered the way that course goals were articulated to students. For example, the faculty
development and course planning seminars during the intervening summer months
featured a discussion of assessment data collected so far, with the aim of identifying and
describing a revised set of measurable, meaningful learning goals for the courses offered
in the fall semester (2014).

f. Year 2: In our ongoing effort to assess Sunoikisis educational goals in their revised form,
during the 2014-15 academic year we decided to survey both faculty members (as we
had been doing) and students (this piece is new).

g. Our faculty assessment work again focused on determining their engagement with
ongoing course outcomes and learning goals for the students. Through surveys and
interviews, the program again collected data about faculty members’ understanding of
the learning goals for the courses. Faculty members were asked to reflect on how they
prepared for their participation in the course, given their understanding of course goals
and student learning outcomes.

h. This year, we distributed evaluations to participating students for the first time, asking
them to reflect on their engagement with two specific learning outcomes. In assessment
parlance, we collected “indirect measures” of those learning outcomes from the group of
participating students by asking them questions about what they feel they achieved and
what elements of the course supported their learning. As a result, we will be able to
report on students’ strengths and weaknesses, with regard to our two focal learning
outcomes, based on faculty members’ analysis of students’ work.

i. This year, we are also in the process of collecting “direct measures” of their learning by
examining a sample of students’ written and online work at the end of the semester.

j. Together, these direct and indirect measures will allow us to think more carefully about
whether students are learning what we hope they are, and why or why not they are
achieving the learning outcomes we have in mind for them.

k. Year 3: Moving forward, we plan to review the next wave of data— which will include
student assessments in addition to faculty data—before next June’s planning seminars,
and make further refinements in the service of future learning outcomes and
pedagogical clarity (as well as pedagogical coherence across participating faculty
members, which is an additional challenge related to the structure of the Sunoikisis