Category: professional development

Building a Culture of Integrity, Part 2: What we learned from students

Over the last 9 months, I’ve had the privilege of working with a group of dedicated educators in the Dalhousie Faculty of Engineering to explore building a culture of integrity. In Part 1, I describe our faculty learning community where we read James Lang’s Cheating Lessons together. Part 2 is about a survey we ran with Engineering students and how we analyzed their responses. Part 3 brings it all together with a resource we created for course instructors. Our analysis of responses and the resource for course instructors are still drafts. The co-authors are not ready to have their names appear in public so while I can’t give their names, I want to acknowledge these were collaborative projects and I wouldn’t have anything to write about without these colleagues.

Guided by recommendations in James Lang’s Cheating Lessons, we followed up our conversations about academic integrity with faculty by continuing the conversation into the community, namely, the Engineering students.

The Associate Dean and I invited students to join an online focus group about academic integrity in Engineering at Dalhousie. We did not want these discussions to be about cheating but rather, the culture and environment. The discussion (and the survey for students who were unable to join the meeting) revolved around these three questions:

  1. What would make the homework, lab reports, quizzes, exams, etc. meaningful to you, so that
    you’d want to use your time and attention to do it yourself?
  2. What are some factors making it difficult for students to do their own work?
  3. When we have a strategy for building this culture, how do we connect with students so they
    want to be a part of it?

We spoke with 7 students and 15 completed the survey. Between the notes I took during the meeting and the survey, we ended up with 109 responses.

Categorizing and analyzing the responses

Ever since my colleagues Beth Simon and Jared Taylor wrote their terrific article, “What is the Value of Course-Specific Learning Goals?” in 2009, I’ve been waiting for a project where I get to analyze student comments. It was important to me (and the Associate Dean) that we learn what the students are actually telling us, and not just seek confirmation of what we wanted to hear.

I recruited two members of the Cheating Lessons book club to help me sort and analyze the responses. My colleagues aren’t ready to have their names in public – this work and report is still a draft – so I’ll refer to them as F and P. Here’s what we did:

  1. I created an Excel spreadsheet with three worksheets, one for each question. Each worksheet had one student response per row, with a unique identifier (so we could later talk about Response #16, etc.). I sent the spreadsheet to F and P.
  2. Each of us independently read through the responses and created 3 or 4 “buckets” or categories of responses for each question. The idea is to let the categories emerge from the responses, rather than creating categories ahead of time (and risking simply confirming what we wanted to hear.)
  3. We met to compare categories. Some were identical, some were different words for the same idea, and a few were different. We reached consensus on three categories for each question.
  4. I reformatted the Excel spreadsheet, still with one student response per row and now 3 new columns, one for each category. I sent the spreadsheet to F and P.
  5. Each of us independently re-read all the responses, assigning each response to one of the categories. F and P sent me their spreadsheets.
  6. I merged all three spreadsheets together to show how each of us categorized each student response. I used Fleiss’ kappa to calculate the inter-rater reliability. It’s an index between 0 (no agreement) and 1 (total agreement) that normalizes for random agreement (3 monkeys would agree occasionally). That Wikipedia page, btw, is very good and I followed the step-by-step example.
  7. We met to compare how we categorized the responses. We completely agreed on the majority, were split on some, and completely disagreed on a handful of responses. We discussed that handful of responses and how we interpreted the what the student said. A few “Ohh!” and “Ahh!” later, we’d reached consensus. We also agreed to categorize any split response in the majority category.
  8. With every response now categorized, I repeated the inter-reliability calculation:
    Question Number of responses Fleiss’ kappa Interpretation
    What would make the homework, lab reports, quizzes, exams, etc. meaningful to you, so that you’d want to use your time and attention to do it yourself? 48 0.745 Substantial agreement
    What are some factors making it difficult for students to do their own work? 37 0.725 Substantial agreement
    When we have a strategy for building this culture, how do we connect with students so they want to be a part of it? 24 0.483 Moderate agreement

    I’m glad we had good agreement on question 1 (“meaningful”) and question 2 (“factors”). I’m not too worried about our lower agreement on question 3 (“how to connect”) as that was asking for their suggestions rather than learning about their experiences.

What did the students tell us?

When investigating inter-rater reliability and analysis of responses, I learned it’s good to present a “code book” along with the data that names the categories, defines the categories, and gives sample responses.

What would make the homework, lab reports, quizzes, exams, etc. meaningful to you, so that you’d want to use your time and attention to do it yourself?

We asked students, “What would make the homework, lab reports, quizzes, exams, etc. meaningful to you, so that you’d want to use your time and attention to do it yourself?” Twenty of the 48 comments are about relevance, 14 are about assessment and marks, and 14 are about support.
Category Description Sample Responses
Relevance Make the work relevant to: (i) the material in the course, (ii) the material in other courses, (iii) industry, and (iv) career aspirations. • I’m more inclined to do work when I’m also taught how it applies to industry or how it applies to other skills I will apply in the future as an engineer.
• I think they typically mean a lot to me anyway, but to make it more meaningful, I feel like I would be more motivated to do them if it were related to solving real world problems that have had a big impact in the communities I’m a part of (for example, working on engineering projects that deal with the SDGs and so on).
Assessment, Marks Ensure the reward (i.e., marks) justifies the effort expended in completing the assignment (i.e., the assessment). • Sometimes work doesn’t feel like it’s worth the time (long assignment, takes all weekend for 1%, when another assignment is worth 10%).
• Work is meaningful to me if there is a grade attached to it. I have had experiences where homework and labs are not for a grade which discourages students from doing the work since there are other classes that we have.
Support Provide adequate preparation and support mechanisms so students feel capable and confident in undertaking the assignment on their own. This could involve ensuring the learning objectives of the assignment are thoroughly explained and understood. • Let 1st years know they have support.
• Having sufficient resources is important – e.g. when class is all theory but then homework is application and students don’t know how to do it, they look elsewhere (like Chegg) for help (e.g. a course with classes involving theory and simple examples, but homework that involves complex problems). Have assignments with a range of questions that build.

What are some factors making it difficult for students to do their own work?

We asked students, “What are some factors making it difficult for students to do their own work?” Of the 37 responses, 15 are about workload and time management, 12 are about (the lack of) support, and 10 are about (the lack of) motivation.
Category Description Sample Responses
Workload, Time Management Workload demands in a given course and throughout the program leading to stress on students to perform well and meet all course expectations. Student and professor time management issues. • Something to note: it takes time to organize your time. Knowing assignment weights for prioritization, knowing deadlines, being able to find assignments easily, being able to find notes easily to solve problems, and even knowing what is expected on exams and how they will be formatted.
• I think for all engineering students, the workload is very heavy which can make it difficult to complete all work individually. It is still possible to do it individually, but I believe that the workload does play a role in making it difficult for students to complete work individually.
(Lack of) Support Issues with required support systems and course resources. • I also think that an overall lack of support from professors and TAs makes it difficult for students to do their own work because they aren’t getting enough help with difficult subjects. The student to TA ratio is often too large for students to be able to comfortably ask meaningful questions. As a result, students are less likely to do their own work because they are confused about what is going on.
• Although professors and faculty are making a valiant effort to provide extra support for students, I think some students still feel a bit overwhelmed and unsure how to get appropriate support online. Overall, online learning seems to be quite a bit more difficult for most students, especially the aspect of not being able to learn in a classroom or be surrounded by peers and professors, as that is the environment a lot of students thrive in most.
(Lack of) Motivation Motivational issues for students. • When instructors aren’t engaged, don’t appear to want to be there, then students are less engaged, more likely to get help from elsewhere.
• When instructors are clear about expectations, students engage more, try to do it themselves.

When we have a strategy for building this culture, how do we connect with students so they want to be a part of it?

We asked students, “When we have a strategy for building this culture, how do we connect with students so they want to be a part of it?” Twelve of the 24 responses mention effective communication, 8 are about ethics and professionalism, and 4 are about collaboration and building community.
Category Description Sample Responses
Effective Communication (the Mode) Engage in effective, well-thought-out communications with individuals and groups. • Communicate through student leaders – students listen to them. Through Dal Eng Society. I think the strategy should be spread via the student representatives.
• Orientation aimed mostly at 1st year, though programs have orientation for 3rd year students coming from Associated Universities.
• Brief updates usually capture students’ attention better than long ones. So, if a strategy is made, whatever mediums it is transmitted over, I guess keep it concise and remind them of its purpose and incentive.
Ethics, Professionalism (the Message) Appeal to the integrity of all involved in the engineering education process; draw on the P.Eng. Code of Ethics. • Throughout the program we get many lectures about integrity (CPST, history of engineering, law and ethics). People understand that if an engineer makes a mistake it can cost lives. We learn about the different historical events and because there is a process in getting the P.Eng. stamp, I believe this process will give you engineers with integrity. I don’t believe that there is much more on the university side that can be done. At some point it is up to the person to decide what type of engineer/person they want to be.
• Phrase it as something that will strengthen their career and not just create obstacles in school.
Collaboration, Community Look for opportunities to encourage collaboration where permitted. Provide opportunities for students to create a sense of community within and outside the classroom. • Make communal spaces where students can work together on assignments and projects, possibly grouped by discipline.
• Round table discussions – especially in person.
• Also hosting fun events so they can socialize with each other and become a greater community. Online classes have made students lose the sense of community within their faculty.

I’m really happy with how we carried out the analysis and also with the results. So are my co-authors, F and B. And so is the Associate Dean who immediately took the next step and asked me, “Okay, so what are the ways I can make my course more relevant to students?” That was the spark for the next step of this project: a campaign based on education. In Part 3 of this series, I’ll share the resource we created for course instructors in Engineering.

Building a Culture of Integrity, Part 1: “Cheating Lessons” book club

Over the last 9 months, I’ve had the privilege of working with a group of dedicated educators in the Dalhousie Faculty of Engineering to explore building a culture of integrity. In this, the first of three posts about our work and its outcomes, I’ll describe our faculty learning community. Part 2 is about a survey we ran with Engineering students and how we analyzed their responses. Part 3 brings it all together with a resource we created for course instructors. Our analysis of responses and the resource for course instructors are still drafts. The co-authors are not ready to have their names appear in public so while I can’t give their names, I want to acknowledge these were collaborative projects and I wouldn’t have anything to write about without these colleagues.

Like every other Department, Faculty, School, and University, course instructors in the Faculty of Engineering at Dalhousie struggled with a rise in academic dishonesty during the Fall 2020 Term, the first full term forced online because of the COVID-19 pandemic. Rather than looking for surveillance solutions, the Associate Dean (“the AD”) was keen to play the long game and find ways to change the culture of the Faculty so that cheating wasn’t an option students considered, and if they did stray, it didn’t help them.

Jim Lang’s Cheating Lessons: Learning from Academic Dishonesty (2013) is the blueprint for what came next.

I pitched the idea to the AD that we should run a “book club” where a group of well-situated members of the Faculty read the book together. The AD liked it and immediately contacted each Department Chair inviting them to join the cohort and to bring 1 or 2 influential course instructors from their Department. By January, 2021, we had a cohort of about 15 people, including me, the AD, a good number of the Department Chairs, and a group of course instructors, every one of us dedicated to teaching and learning. With the support of the Dean, everyone received a copy of Cheating Lessons.

From January – April, 2021, I led four 60-minute, online conversations based directly on three critical questions Lang poses in Cheating Lessons.

Conversation 1: Why should students bother to memorize or learn or connect or do their own work when technology can often provide them with the information they need (more) quickly and efficiently?

Themes that emerged from reading Chapter 9 “On Original Work” and our conversation:

  1. There is certain knowledge — vocabulary, mathematical skills, scientific knowledge — engineers need to recall quickly to begin analyzing, applying, and designing and afterwards, evaluating and checking solutions
  2. The act of memorizing and recalling information helps students learn.
  3. You don’t find novel problems and designs on the internet.

Conversation 2: Why should students bother to do their own work when others can do it better and more easily?

Themes that emerged from reading Chapter 4 “Fostering Intrinsic Motivation” and our conversation:

  1. Intrinsic motivation increases as students more get hands-on experience with authentic engineering problems.
  2. There is ample opportunity in the activities of the courses and program to create opportunities where students are intrinsically motivated to engage and learn.
  3. Assessments, from homework problems to term projects, can be intrinsically motivating when they are grounded in the students’ knowledge, skills, and experiences.

Conversation 3: Why should students bother to complete an assignment on their own when three of them working together may complete it more effectively?

Themes that emerged from reading Chapter 7 “Instilling Self-Efficacy” and our conversation:

  1. We need to convince students to invest the time and energy to learn, not to earn marks.
  2. We need to find ways to boost their self-confidence. This might mean rewarding their progress and incremental successes, without punishing them for getting something wrong. It might be their perception (and experience) that getting the correct answer the only way to be successful.

Conversation 4: What do we do next?

Chapter 8 “Cheating on Campus” is very nearly a step-by-step guide for creating and communicating a program for academically honest education:

  1. Begin the Conversation among the Faculty
  2. Continue It Into the Community
  3. Time It Well
  4. Focus Academic Integrity Campaigns on Education, Not Ethics

I’m happy to say we followed this advice quite well. In Part 2, you can read about the academic integrity survey and focus groups we ran with Engineering students, and how we analyzed their responses. And in Part 3, I’ll share the resource we drafted with strategies for course instructors to promote integrity in the way they teach. The goal is to share this resource with every instructor in Engineering in August while their preparing their Fall courses.

 

Anti-Racism in STEMM

I learned a lot from reading Ibram X. Kendi’s “How to Be an Antiracist“. There were times, though, when I wasn’t able to transfer those ideas and actions into the environment and culture of university and higher education. This new paper, “Responses to 10 common criticisms of anti-racism action in STEMM” (15 July 2021) carefully and clearly makes the connections I was missing. (The second M is STEMM is medicine, btw.)

Gosztyla ML, Kwong L, Murray NA, Williams CE, Behnke N, et al. (2021) Responses to 10 common criticisms of anti-racism action in STEMM. PLOS Computational Biology 17(7): e1009141. doi.org/10.1371/journal.pcbi.1009141

As you can tell from the title, the paper addresses 10 criticisms and suggests actions to counter the criticism. Do these sound familiar to you?

  1. “There is no evidence of racism in STEMM.”
  2. “Don’t politicize STEMM! Stick to the science, not social issues.”
  3. “I’m not racist, so I don’t need to do anything.”
  4. “I only hire/award/cite based on merit; I do not need to consider race.”
  5. “There just aren’t as many BIPOC who want to work in STEMM.”
  6. “Diversity initiatives are unfair to nonminority students/faculty; it’s reverse discrimination.”
  7. “Education is the great equalizer.”
  8. “I don’t agree with racist statements, but people should be allowed to express their opinions and have debates.”
  9. “Focusing on anti-Black racism ignores the experiences of non-Black POC, in addition to sexism, ableism, etc.”
  10. “Improving racial equity and inclusivity does not benefit STEMM as a whole.”

If you’ve heard any one of these and you’re not sure how to respond to counter the misconception, I urge you to read the article.

Navigation