Over the last 9 months, I’ve had the privilege of working with a group of dedicated educators in the Dalhousie Faculty of Engineering to explore building a culture of integrity. In Part 1, I describe our faculty learning community where we read James Lang’s Cheating Lessons together. Part 2 is about a survey we ran with Engineering students and how we analyzed their responses. Part 3 brings it all together with a resource we created for course instructors. Our analysis of responses and the resource for course instructors are still drafts. The co-authors are not ready to have their names appear in public so while I can’t give their names, I want to acknowledge these were collaborative projects and I wouldn’t have anything to write about without these colleagues.
Guided by recommendations in James Lang’s Cheating Lessons, we followed up our conversations about academic integrity with faculty by continuing the conversation into the community, namely, the Engineering students.
The Associate Dean and I invited students to join an online focus group about academic integrity in Engineering at Dalhousie. We did not want these discussions to be about cheating but rather, the culture and environment. The discussion (and the survey for students who were unable to join the meeting) revolved around these three questions:
- What would make the homework, lab reports, quizzes, exams, etc. meaningful to you, so that
you’d want to use your time and attention to do it yourself? - What are some factors making it difficult for students to do their own work?
- When we have a strategy for building this culture, how do we connect with students so they
want to be a part of it?
We spoke with 7 students and 15 completed the survey. Between the notes I took during the meeting and the survey, we ended up with 109 responses.
Categorizing and analyzing the responses
Ever since my colleagues Beth Simon and Jared Taylor wrote their terrific article, “What is the Value of Course-Specific Learning Goals?” in 2009, I’ve been waiting for a project where I get to analyze student comments. It was important to me (and the Associate Dean) that we learn what the students are actually telling us, and not just seek confirmation of what we wanted to hear.
I recruited two members of the Cheating Lessons book club to help me sort and analyze the responses. My colleagues aren’t ready to have their names in public – this work and report is still a draft – so I’ll refer to them as F and P. Here’s what we did:
- I created an Excel spreadsheet with three worksheets, one for each question. Each worksheet had one student response per row, with a unique identifier (so we could later talk about Response #16, etc.). I sent the spreadsheet to F and P.
- Each of us independently read through the responses and created 3 or 4 “buckets” or categories of responses for each question. The idea is to let the categories emerge from the responses, rather than creating categories ahead of time (and risking simply confirming what we wanted to hear.)
- We met to compare categories. Some were identical, some were different words for the same idea, and a few were different. We reached consensus on three categories for each question.
- I reformatted the Excel spreadsheet, still with one student response per row and now 3 new columns, one for each category. I sent the spreadsheet to F and P.
- Each of us independently re-read all the responses, assigning each response to one of the categories. F and P sent me their spreadsheets.
- I merged all three spreadsheets together to show how each of us categorized each student response. I used Fleiss’ kappa to calculate the inter-rater reliability. It’s an index between 0 (no agreement) and 1 (total agreement) that normalizes for random agreement (3 monkeys would agree occasionally). That Wikipedia page, btw, is very good and I followed the step-by-step example.
- We met to compare how we categorized the responses. We completely agreed on the majority, were split on some, and completely disagreed on a handful of responses. We discussed that handful of responses and how we interpreted the what the student said. A few “Ohh!” and “Ahh!” later, we’d reached consensus. We also agreed to categorize any split response in the majority category.
- With every response now categorized, I repeated the inter-reliability calculation:
Question Number of responses Fleiss’ kappa Interpretation What would make the homework, lab reports, quizzes, exams, etc. meaningful to you, so that you’d want to use your time and attention to do it yourself? 48 0.745 Substantial agreement What are some factors making it difficult for students to do their own work? 37 0.725 Substantial agreement When we have a strategy for building this culture, how do we connect with students so they want to be a part of it? 24 0.483 Moderate agreement I’m glad we had good agreement on question 1 (“meaningful”) and question 2 (“factors”). I’m not too worried about our lower agreement on question 3 (“how to connect”) as that was asking for their suggestions rather than learning about their experiences.
What did the students tell us?
When investigating inter-rater reliability and analysis of responses, I learned it’s good to present a “code book” along with the data that names the categories, defines the categories, and gives sample responses.
What would make the homework, lab reports, quizzes, exams, etc. meaningful to you, so that you’d want to use your time and attention to do it yourself?
| Category | Description | Sample Responses |
|---|---|---|
| Relevance | Make the work relevant to: (i) the material in the course, (ii) the material in other courses, (iii) industry, and (iv) career aspirations. | • I’m more inclined to do work when I’m also taught how it applies to industry or how it applies to other skills I will apply in the future as an engineer. • I think they typically mean a lot to me anyway, but to make it more meaningful, I feel like I would be more motivated to do them if it were related to solving real world problems that have had a big impact in the communities I’m a part of (for example, working on engineering projects that deal with the SDGs and so on). |
| Assessment, Marks | Ensure the reward (i.e., marks) justifies the effort expended in completing the assignment (i.e., the assessment). | • Sometimes work doesn’t feel like it’s worth the time (long assignment, takes all weekend for 1%, when another assignment is worth 10%). • Work is meaningful to me if there is a grade attached to it. I have had experiences where homework and labs are not for a grade which discourages students from doing the work since there are other classes that we have. |
| Support | Provide adequate preparation and support mechanisms so students feel capable and confident in undertaking the assignment on their own. This could involve ensuring the learning objectives of the assignment are thoroughly explained and understood. | • Let 1st years know they have support. • Having sufficient resources is important – e.g. when class is all theory but then homework is application and students don’t know how to do it, they look elsewhere (like Chegg) for help (e.g. a course with classes involving theory and simple examples, but homework that involves complex problems). Have assignments with a range of questions that build. |
What are some factors making it difficult for students to do their own work?
| Category | Description | Sample Responses |
|---|---|---|
| Workload, Time Management | Workload demands in a given course and throughout the program leading to stress on students to perform well and meet all course expectations. Student and professor time management issues. | • Something to note: it takes time to organize your time. Knowing assignment weights for prioritization, knowing deadlines, being able to find assignments easily, being able to find notes easily to solve problems, and even knowing what is expected on exams and how they will be formatted. • I think for all engineering students, the workload is very heavy which can make it difficult to complete all work individually. It is still possible to do it individually, but I believe that the workload does play a role in making it difficult for students to complete work individually. |
| (Lack of) Support | Issues with required support systems and course resources. | • I also think that an overall lack of support from professors and TAs makes it difficult for students to do their own work because they aren’t getting enough help with difficult subjects. The student to TA ratio is often too large for students to be able to comfortably ask meaningful questions. As a result, students are less likely to do their own work because they are confused about what is going on. • Although professors and faculty are making a valiant effort to provide extra support for students, I think some students still feel a bit overwhelmed and unsure how to get appropriate support online. Overall, online learning seems to be quite a bit more difficult for most students, especially the aspect of not being able to learn in a classroom or be surrounded by peers and professors, as that is the environment a lot of students thrive in most. |
| (Lack of) Motivation | Motivational issues for students. | • When instructors aren’t engaged, don’t appear to want to be there, then students are less engaged, more likely to get help from elsewhere. • When instructors are clear about expectations, students engage more, try to do it themselves. |
When we have a strategy for building this culture, how do we connect with students so they want to be a part of it?
| Category | Description | Sample Responses |
|---|---|---|
| Effective Communication (the Mode) | Engage in effective, well-thought-out communications with individuals and groups. | • Communicate through student leaders – students listen to them. Through Dal Eng Society. I think the strategy should be spread via the student representatives. • Orientation aimed mostly at 1st year, though programs have orientation for 3rd year students coming from Associated Universities. • Brief updates usually capture students’ attention better than long ones. So, if a strategy is made, whatever mediums it is transmitted over, I guess keep it concise and remind them of its purpose and incentive. |
| Ethics, Professionalism (the Message) | Appeal to the integrity of all involved in the engineering education process; draw on the P.Eng. Code of Ethics. | • Throughout the program we get many lectures about integrity (CPST, history of engineering, law and ethics). People understand that if an engineer makes a mistake it can cost lives. We learn about the different historical events and because there is a process in getting the P.Eng. stamp, I believe this process will give you engineers with integrity. I don’t believe that there is much more on the university side that can be done. At some point it is up to the person to decide what type of engineer/person they want to be. • Phrase it as something that will strengthen their career and not just create obstacles in school. |
| Collaboration, Community | Look for opportunities to encourage collaboration where permitted. Provide opportunities for students to create a sense of community within and outside the classroom. | • Make communal spaces where students can work together on assignments and projects, possibly grouped by discipline. • Round table discussions – especially in person. • Also hosting fun events so they can socialize with each other and become a greater community. Online classes have made students lose the sense of community within their faculty. |
I’m really happy with how we carried out the analysis and also with the results. So are my co-authors, F and B. And so is the Associate Dean who immediately took the next step and asked me, “Okay, so what are the ways I can make my course more relevant to students?” That was the spark for the next step of this project: a campaign based on education. In Part 3 of this series, I’ll share the resource we created for course instructors in Engineering.