Browse Database - Assessment for Learning at Boston College

Assessment Details

Academic Year: 2021-2022 Level: Undergraduate

Campus Department: Morrissey College of Arts & Sciences [UG and Grad]

Program Type: Major [UG] / Program [Grad]

Program Name: Theology BA (Link)

Description of Data Collection:

The undergraduate director has successfully fulfilled the core assessment goals for AY2021-22 (which were submitted in the Core E1A from 2021), namely:
“1. Deliver a proposal to the department in the Fall of 2021 to create a Theology Undergraduate Program Assessment and Advisory Committee (hereafter TUPAAC), and, if approved, to begin assessment work with this group in the Spring of 2022. The committee should be comprised of full time faculty members who teach regularly within the standard theology core. Its purpose is: a) to establish means and rubrics for assessing both the core and the major, b) to implement that assessment, and c) to conduct initial interpretation of the resulting data and to make potential recommendations for curriculum modifications if necessary. Additionally, the TUPAAC will assist the undergraduate director with a handful of advisory tasks as they come up through the year.
2. Establish a schedule for assessment of each of the core’s and major’s learning goals to be executed over the next few academic years.” The theology department now has a standing assessment committee (now: TUPAC) with a rotating slate of elected members the first group of which began their terms in the late Fall of 2021. In the Fall of 2021, the TUPAC established a working schedule of both theology core and major assessment. The assessment of undergraduate Theology learning outcomes is based on a cycle where select learning outcomes serve as the assessment focus for an academic year. During academic year 2021-22, this assessment focus was on the Major learning outcome “mount cogent, constructive theological arguments in dialogue with other disciplines”.

In the Spring of 2022, the TUPAC began assessing learning goals in the theology major, specifically the 3rd major learning outcomes, that graduating theology majors should be able to: Mount cogent, constructive theological arguments in dialogue with other disciplines. (Learning Outcome #3 – Graduating theology majors should be able to: Mount cogent, constructive theological arguments in dialogue with other disciplines.)

Currently, the theology major does not have a seniors-only major course. Through the end of AY2021-2022, the only specific courses required by majors are THEO3330 Majors Seminar and THEO3577 Conciliar Traditions. The TUPAC decided to collect direct evidence by assessing the major writing assignment from THEO3577 (Spring 2021), since, in lieu of a seniors-only major course, this was one of the courses likely to be populated with majors primarily, and specifically majors who had already completed some portion of the theology major overall. Nine examples of student papers were collected (of a potential 17 who had enrolled in the course). Of these nine, four were sophomores, four were juniors, and one was a senior. The five members of the TUPAC (inclusive of the DUS) scored the writing assignment across six categories using a scoring rubric that had been developed for a general research paper (see appended document), with additional categories added, it was hoped, to better address the learning outcome. The scoring range of this particular rubric is intended to identify competency/inadequacy, with a score of 1 = Inadequate, 2 = Moving Toward Competency, and 3 = Competent.

The aggregate results of the TUPAC’s scoring is as follows:
Table 1: Assessment Scores Summary
A. Introduction 2.2
B. Research 2.2
C. Interdisciplinary Engagement 0.5*
D. Theological Interdisciplinary Engagement 1.3*
E. Conclusions 2.2
F. Writing 2.2
Two notable conclusions can be drawn from these results. The first is that the average score of 2.2 across four (A, B, E, and F) of the six categories shows that students are likely moving towards competency in these categories, which align with the portion of the learning goal which seeks to “mount cogent, constructive theological arguments.” We consider this movement towards competency to be an outcome that is positively aligned with the stated learning outcome, especially since eight of the nine students whose work was evaluated were sophomores and juniors, rather than seniors. This conclusion is underscored by indirect evidence pulled from 2020 NSSE survey data: [see attachment]

Senior theology majors considered their experience overall at BC to have contributed to their critical and analytical thinking on par with their colleagues across the university. Indeed, they were far more likely than their cohort to indicate that their time at BC added to critical and analytical thinking “very much” (67% for theology majors vs. 46% for non-theology majors). The second conclusion regards the two categories that focus on interdisciplinary engagement (C and D, general and inner-theological). While ostensibly the scores are rather low, they are the result of the fact that several of the members of the TUPAC felt they could not score the papers in these categories or, if they did, it could only be done with an exceptionally generous hermeneutic. This does not indicate an alarming inadequacy either on the instructor’s or students’ part, but rather a misalignment of the scoring rubric with the assignment chosen to evaluate. This prompted discussion as to the process by which to select future assignments for the purpose of assessment (see also #5 below); the portion of the learning outcome that focusses on “dialogue with other disciplines” was not gauged with direct assessment methods in this round of assessment. Nonetheless, indirect evidence from the 2020 NSSE survey would indicate that senior theology majors consider themselves to operate in a far more interdisciplinary manner in their courses than their fellow students in other majors: [see attachment]

While, given the vague terms used by the question, it is impossible to determine the sources of the ideas that theology majors combine when completing their assignments (are they overall interdisciplinary or interdisciplinary within the theological enterprise?), it is clear that theology majors, in percentages considerably higher than their peers (“often” or “very often”: theology majors = 100%, non-majors = 63%) understand their studies as curricularly integrative.

Review Process:

The Theology Undergraduate Program Assessment Committee (TUPAC), led by the director of undergraduate studies constructs a process for assessing the individual learning goals, then interprets the data collected. The TUPAC constructs recommendations based on these data and analysis, and the undergraduate director will present this to the chair and department executive committee, before presenting them to the department as a whole. This collation of data, analysis, and recommendation from the previous Spring and Fall semesters, is currently scheduled to take place in the Spring Semester every year. For example, during AY2021-2022 TUPAC: 1) formulated the assessment schedule for core and majors’ learning outcomes; 2) generated an associated rubric; 3) scored a “signature assignment” paper based on the selected learning outcome of interest; 4) discussed the findings and formulated next steps for academic year 2022-23. The final meeting of the spring semester addressed the assessment plan for the upcoming year; during the summer, this plan is refined by the TUPAC Chair, possibly in consultation with Provost Office personnel. TUPAC activities and recommendations are communicated to the broader faculty at Theology department meetings.
The current TUPAC members are:
Jeffrey L. Cooley (DUS, ex officio chair)
Amey Victoria Adkins-Jones (1-year term ending F2022)
Fr. Liam Bergin (1-year term ending F2022)
Yonder Gillihan (2-year term ending F2023)
Dieter Roth (2-year term ending F2023)

Resulting Program Changes:

The TUPAC’s schedule is such that assessments on core and major learning goals conducted in the Spring and Fall semesters, are presented to the department for consideration in the following Spring semester. Thus, major assessments conducted in the Spring of 2022 and core assessments conducted in the Fall of 2022 will be presented to the department for their consideration in the Spring of 2023. Thus, no changes have yet been made. Nonetheless, the assessment process during AY 2021-22 yielded several potential action items: Learning Outcomes: The design of the assessment schedule required a close review of all undergraduate learning outcomes, particularly those related to theology majors, the assessment focus for AY2021-2022. This review affirmed the continued appropriateness of the learning outcomes in that they remain reflective of both the practical and aspirational goals of the department. However, the TUPAC recognized that some refinement in select language could serve to clarify, and make more measurable, some of the learning outcomes. For example, it was noted that the major learning outcome “engage articulately in some aspect of both ecumenical and inter-religious dialogue and learning” may warrant some discussion around the meaning of “engage articulately” – this topic will be introduced at F2022 TUPAC and S2023 Department meetings. Curricular Outcomes: Assessment of the major learning outcome #3 “mount cogent, constructive theological arguments in dialogue with other disciplines” presented substantive and positive evidence that Theology Majors are competent in this area and so no needed curricular change was identified. Assessment Process: Acknowledging that it is difficult to align an assignment with a rubric, the TUPAC relied on a specific, single course to supply assessment evidence and noted that, for future assessments, using multiple forms of evidence (triangulation) would help to round out findings and enrich discussion. To this end, the use of survey data was explored; for example, Class of 2020 results from the National Survey of Student Engagement helped to support Theology majors’ learning, as outlined in Tables 2 and 3. Moving forward, survey data (indirect evidence) will be used to supplement direct evidence, as appropriate. Sources of Assessment Evidence: Serving as a curriculum mapping exercise, enrollment in select courses were also reviewed in order to assess if theology majors were engaging in learning experiences that would expose them to the skills/knowledge as outlined in the learning outcome. For example, courses which foster the interfaith component of the learning outcome #4 “engage articulately in some aspect of both ecumenical and inter-religious dialogue and learning” were reviewed for all theology majors graduating from 2018 through 2021. During this span, 94.6% of theology majors enrolled in one (or more) of these interfaith courses, helping to validate the alignment of the curriculum and the learning outcomes. Due to EagleApps’ challenges, and the variety of ways in which this type of data can be viewed, this assessment source is still in an exploratory stage, however it looks to be a solid complement to the other forms of assessment evidence

Date of Most Recent Program Review:

Attachments (if available)