Browse Database - Assessment for Learning at Boston College

Assessment Details

Academic Year: 2021-2022 Level: Graduate

Campus Department: Morrissey College of Arts & Sciences [UG and Grad]

Program Type: Major [UG] / Program [Grad]

Program Name: Earth and Environmental Sciences PhD (Link)

Description of Data Collection:

a. All MS and PhD students are required to submit a preliminary draft thesis proposal as the final paper for the Earth Systems Seminar (EESC6691). This class is required in the fall semester for all 1st-year MS and PhD students who have not completed an MS degree in a geoscience-related field.
b. All MS and PhD students who have not completed an MS degree in a geoscience-related field must submit a final thesis proposal during the third semester of the program.
c. All MS and PhD students who have not completed an MS degree in a geoscience-related field must take an oral qualifying exam during their third semester. A faculty committee of three administers the exam. The student’s academic advisor serves as the chair of the committee. At the conclusion of the exam, the chair provides a summary, including an assessment of how well they have achieved the learning outcomes described above at this stage of the program. A written copy of this assessment is provided to the department graduate committee and is placed in the student’s file.
d. All PhD students must complete an oral comprehensive exam within 18 months of passing the qualifying exam. The comprehensive exam covers the completed draft of the first chapter of the PhD thesis and the research proposed for the remaining thesis chapters. The exam consists of a public presentation, followed by questioning from the student’s thesis advisory committee.
e. All PhD students must complete a final thesis defense, consisting of a public presentation, followed by a private examination by the student’s thesis advisory committee.
f. Placement and publication data for graduate students are recorded annually in the department office and reviewed by the Graduate Program Committee.
g. The Graduate Program Director reviews student evaluations of Teaching Assistants.
h. A Graduate Student Research Colloquium is held every spring at the end of the semester. Most graduate students make presentations and the faculty evaluates the students on a scale of 1 to 5 with respect to Outcome #5 above (i.e., speak effectively to professional and lay audiences about issues in the field).

Review Process:

The Graduate Program Committee meets each year to review the above data and make recommendations to the whole department for improvement. The department discusses these recommendations at our annual faculty retreat in late May

Resulting Program Changes:

Oral Exams: The oral qualifying exam assessment form was used for the first time during AY15-16. The questions were modified slightly in the fall of 2021 to better reflect the difference between the expected level of preparation at the MS/PhD oral qualifying exam and the PhD oral comprehensive exam (see attached). Specifically, at the qualifying exam both PhD and MS students are expected to present proposed publishable research, while at the comprehensive exam PhD students are expected to present completed publishable research. These changes were approved by the EES faculty in Fall 2021 and were explained to the students prior to the fall 2021 examination season.
Overall, the faculty is pleased with graduate student performance on the exams (typically between very good and excellent). In general, we see improvement (or no difference, but very good/excellent) for all learning outcomes. We also evaluate student performance on learning outcome 5 based on presentations at the annual Graduate Student Research Colloquium and outcome 6 based on TA evaluations.
Students typically score lowest on outcome 2 (broad knowledge of Earth processes). This may indicate a need to modify our current course distribution requirement slightly in order to ensure students are getting proper preparation for this aspect of their graduate education. Over the next few years our department plans to perform a full curriculum review. As part of this review, we plan to re-evaluate our breadth requirement for the graduate program to (1) ensure that it aligns with our current faculty and graduate students, and (2) helps to better prepare students for outcome 2.
One recent change our department has made to better prepare students for their oral exams, is that the graduate committee now meets with all 2nd-year (and most 1st-year) students in the early fall to go over what to expect on the oral qualifying exam. We have found that this demystifies the process and reduces stress associated with the exam. We plan to continue these informational meetings in the future.

Results from Oral Qualifying exam assessment, student colloquium, and teaching evaluations are summarized in the tables below. In AY21-22 all 9 2nd-year students passed their oral qualifying exams, and we had our first PhD student successfully pass their oral comprehensive exam.

Table 1a. Assessment of learning outcomes from faculty committee comments after oral examinations. The number of students is in ( ).
2016 (9) 2017 (6)* 2018 May (3)^ 2018 Nov
(4) 2019 Nov (7)+ 2020 Nov (5) 2021
(9/1)#
1. Conduct original, publishable research in the field. 4.3 4.4 5.0 4.6 4.5 4.6 4.6/5
2. Demonstrate a broad knowledge of earth processes. 3.6 3.6 4.2 4.0 4.0 3.7 3.8/4.5
3. Demonstrate in-depth knowledge of a sub-discipline in the earth sciences. 3.9 4.1 4.0 4.0 4.7 4.4 4.1/4.5
4. Demonstrate acquired skill in field or laboratory methods and application of appropriate quantitative methods. 4.2 4.2 4.5 4.9 4.6 4.3 4.4/4
5. Write and speak effectively to professional and lay audiences about issues in the field. 3.7 4.5 N/A N/A
N/A N/A 4.4/5
6. Teach effectively at the undergraduate level. 3.9 4.6 4.0 4.2 4.1 4.6 4.7/4
Average 3.9 4.2 4.5 4.4 4.3 4.3 4.3/4.5
* 6/7 (one not reporting)
^ 3/5 (two not reporting)
+ Two students retook oral exam; final scores reflect averages of the two attempts.
# Includes 9 MS oral qualifying exams and 1 PhD comprehensive exam
Table 1b. Assessment of learning outcomes 5 and 6 from faculty evaluations of colloquium presentations (1 to 5 with 1=poor and 5=excellent) and student evaluations of Teaching Assistants (composite of responses to attached form, 1 to to 5 with 1=poor and 5 = excellent).
Colloquium
2017:
10 evals; 15 presenters 2018;
9 evals; 17 presenters 2019;
12 evals; 14 presenters 2020; no colloquium held due to COVID-19 2021
Online colloquium 2022
5. Write and speak effectively to professional and lay audiences about issues in the field. 4.4 all
4.3 1st yr
4.31 2nd yr 4.35 all
3.98 1st yr
4.57 2nd yr 4.22 all
4.0 1st yr
4.36 2nd yr No data 4.55 all
4.49 MS
4.80 PhD
4.40 all
4.38 MS
4.41 PhD
Student TA Evaluations
6. Teach effectively at the undergraduate level. 5s and 4s;
<10% 3s and lower 5s and 4s;
<10% 3s and lower 5s and 4s;
<10% 3s and lower 75% 5s and 4s
25% 3s and lower 76% 5s and 4s
25% 3 and lower 85% 5s and 4s; 15% 3 and lower

Evaluation of TAs: We implemented a new online TA evaluation form in Spring 2020 that can be more easily analyzed to collect statistics (see example form below). Overall, our TAs are receiving ‘excellent’ and ‘very good’ for the vast majority of questions. In 2020 and 2021 the scores were slightly lower than before we went to the online evaluation form; however, in 2022 the scores rebounded.
We attribute the lower scores in 2020 and 2021 to the fact that our TAs were often working remotely due to the Covid-19 pandemic and did not have the same opportunity to interact face-to-face with the students. It is also possible that the online nature of the evaluations led to a difference in scoring. We plan to continue to monitor these evaluations over the next few years as we shift back to more traditional in person teaching.
In addition, based on feedback from the TAs, the Graduate Program Committee compiled a short Roles & Responsibilities document that is circulated to all TAs and instructors at the beginning of each semester. The purpose of this document is to help manage expectations and ensure consistency in TA workloads across the department.
Placement: Of the three students who completed their MS degrees from summer 2021 to May 2022, two are now enrolled in geoscience PhD programs at Princeton and U.C. Berkeley, respectively, and the third works for InterSystems, a software development company in Cambridge, MA. Our impression is that the vast majority of students (if not all) who have sought employment in the field or continuation to a higher degree have been successful. We have taken a more formalized approach to evaluate this recently by asking students what their plans are when they complete the MS degree and following up on an annual basis with all our former students. We now have fairly good information for the last five years.
New PhD Program: The EES department began a new PhD program in Earth and Environmental Sciences in Fall 2020. Leading up to the inaugural PhD class, the EES faculty developed a new curriculum, evaluation metrics, and learning outcomes for the PhD program. These include the addition of an oral comprehensive exam taken during the 3rd year of the PhD program (described above) and a new Broader Impacts course required for all PhD students. Our first oral comprehensive exam was completed successfully in Fall 2022 and the Broader Impacts course was taught for the first time in Spring 2022 by Prof. Jeremy Shakun to four Ph.D. students. The course explored numerous ways to connect science to society to enhance public impact, such as through K-12 education, museums, citizen science, communication and media. The students all reported that the course is a great addition to our curriculum, increasing their understanding of why science matters, exposure to careers beyond academia, competitiveness in grant writing, and will help them write a required dissertation chapter on broader impacts related to their research.
We were very pleased with the performance of our PhD students at the 2022 spring colloquium, where they received an average rating of 4.4/5 for their research presentations (Table 1b).

Date of Most Recent Program Review:

Attachments (if available)