My Experiment With a Collaborative Exam

As I type this, my students are in the last stage of my grand experiment with a collaborative exam.  I wrote about this desire of mine a year and half ago here.  Despite some negative comments on that post, I wasn't dissuaded.  I still think there's value in having students work collaboratively, in particular because in the real world, much of what we do is in consultation with others (in various forms and to various extents, of course).

In that last post about the topic, I threw around a couple of ideas on how to make this work but wasn't happy with them.  I solved that problem this past winter, when I came up with the idea that I'm currently subjecting my students to.

This experiment is for my first year constitutional law course.  Their exam is today, but yesterday, 24 hours before their exam, I released to them the two big fact-pattern/issue-spotter questions on the exam.  They were told that they could work on those questions before the exam -- alone or with any other current student(s) in the course.  However, when they get to the exam, they can bring in no written materials whatsoever.  The same two questions will be on the exam, so they'll have to produce an answer on their own, albeit after having had the opportunity to work on the questions for the past day.

I did not release the entire exam to them, however.  The two questions that they received yesterday are worth 198 of the exam's 264 points (out of 400 points overall for the course - I have other graded components).  There are also three short answer questions on the exam, each worth 22 points.  They entered the exam this morning without having seen those questions (though they were told in detail the general type of question I'm asking (the same for all three)).

So, in essence, for the bulk of the exam, they will have had the opportunity to work on it beforehand, in groups if they want to.  However, they have to come into the exam and produce on their own.  I'm pretty happy with this solution, as it captures collaboration but with the goal that the students have their own individual understanding of the material.  Because that is what they have to write up, their own individual understanding of the material, that is what I will be assessing.

Of course, as with any exam format, I'm under no illusion that this is a perfect solution.  In fact, I've told them from day one that this is an experiment.  I've laid out my reasons for them so they understand what I'm doing, and I've also told them what I see as the downsides.  As I usually find as a professor, with this level of transparency, I've had virtually no pushback.

What are the downsides I see?  Many:

1) It may flatten the curve.  With many people working together, the quality of the answers should (hopefully!) go up, which could flatten the curve.  But, I'm ok with that for two reasons.  First, it provides more opportunity for learning the material, which is a good thing.  Second, I think there will still (always) be significant variation in understanding the material, parsing the facts and law, and writing it up.

2) Groupwork runs serious risks.  I told them on day 1 about this plan so they knew throughout the semester that they had to think about whether they wanted to work in a group, and if so, with whom.  But even so, there's the risk that someone will be pressured to be in a group with friends that might not be the best people to work with; that someone who knows the material but is less confident will be convinced of the wrong answer by people in the group; that someone might sabotage a group by knowingly giving wrong information; that a student who does little work will free-ride, etc.  I acknowledge these are risks, but at the same time, I'm not convinced they're any riskier with this structure than with study groups.  And, ultimately, someone can choose to work alone for the 24 hours beforehand.

3) Closed-book format.  The only way I could think to do it this way was, after the 24 hour collaboration period, to give a closed-book in-class exam, something I've never done before (my exams are always open-everything).  Otherwise, a student could bring in a pre-written answer.  After all, even if I restricted what they could bring into the exam to just the casebook and supplement, what would stop them from writing their answer in the margins?  So, the only way this works is by having a closed book exam, but I don't like what that does.  It rewards those who are better memorizers, not necessarily those who understand the material better.  To me, this is the biggest downside that if I could figure out how to cure in the future, I would be very happy.  If there's a way I could provide them with an electronic copy of the casebook just for the exam that they could use on their computer with the exam software, that would be a step in the right direction.

There are others, but these are the most significant downsides I see.  When I thought and talked it through (and I talked it through with everyone I could get within earshot of in December!), though, I figured it was worth a try.

Next week, I'll have a follow-up post about some ways I'm going to evaluate whether this "worked" or not, but in the meantime, I'd love to hear your thoughts on this experiment.

Comments

[Michael Teter]

I really wanted to create an exam this year that mirrored an important part of the legal practice: being given some limited facts, having an opportunity to think about them and discuss them with colleagues and to start considering what legal issues might be implicated by them, and then having the facts change in such a way that some of the initial thoughts are accurate, but others aren't. If I take this approach next year, I might combine it with your idea. I would give the students a fact pattern that is a bit broad and generalized. They could then collaborate for 24 hours. Then, when they come in for the exam, I'll give them a more complete development of the facts, along with the questions to answer. Some of the original facts will have been slightly incorrect or too vague to really get a full sense of what's going on. I could allow this exam to be open book because I wouldn't have given them the questions and the facts will have changed enough. And I suspect the curve won't be flattened much, if at all.

Just some things to think about over the summer.

Thanks for posting this -- I look forward to hearing your thoughts after-the-fact.

[Joseph Tomain]

Great post. I've used similar approaches for both in-class and take home exams. I believe they were successful, and know, at the very least, they were not disasters.

In a Cyberlaw course of approximately 18 students, I administered an in-class, closed book essay exam. I provided the exam questions more than 24 hours before the exam, but cannot recall exactly how much time. I did not place any limitations on whom they could consult, reasoning that the closed-book nature meant that whatever they wrote down came from their head (regardless of how it got there).

In a Media Law course with over 60 students, I had a two-part exam: (1) 50 multiple choice questions in class on the scheduled exam day, closed book; and (2) a take-home essay question on net neutrality. They had approximately two weeks to do the take-home exam.

They had to bring a hard copy of their take-home answer to the in-class portion of the exam. The take-home portion was anonymous (i.e. exam ID number on the paper). They were allowed to talk to anyone. The only rules were that they could not share their written product with anyone; and they could not receive unpublished written information (such as an email or comments on a draft). They could receive a published work (e.g. here's a law review article that might help you).

A key control on both the in-class and take-home techniques was that a good answer must necessarily draw on required course readings and class discourse. I told the Media Law students: "Even if you write the most brilliant net neutrality essay ever, if it does not incorporate required readings or information from class discussions, it will not be the best paper for this exam."

I am always tweaking exam techniques and open to new ideas, but here is my current thinking on why I provide more than 24 hours: A student cannot complain that they did not have enough time because the day before they had another exam or a flat tire or whatever. In short, I take away any reasonable complaint that they did not have enough time to think about it.

Final point: I agree with your view on transparency with students. The more I am transparent with them, the better for everyone.

[Orin Kerr]

It seems to me that the point of grading is to tell prospective employers information about the relative knowledge and skills of different students to aid them in deciding who should get a job. If I'm right, I would think the most serious risk is that when people work together and report the conclusions of the group, the grades of individuals largely reflect their luck or skill in picking who to work with, rather than their individual knowledge or skills.

[Joan Heminway]

Last fall, I offered a collaborative oral mid-term in a large class (maximum enrollment: 72). Students worked in groups of three (with two groups of two, given the class size) based on a fact pattern given to them a week in advance. The questions were not given to them until the scheduled exam time. Each team of three was examined separately. The exam comprised three multi-layered questions. Each team member was the primary respondent for one question and a secondary or tertiary respondent for the other two.

The curve for that examination was somewhat flat, but most of the grade in the course was the individually written final exam. I found that offering the collaborative mid-term not only was a great peer-to-peer teaching moment, but also a super combination of summative and formative assessment at a critical juncture in the overall learning experience for the course. And for me, it was just such a joy to see students become lawyers in front of my eyes.

I have to make some tweaks in the procedures, but I will offer the exam again in the fall.

[oakley sunglass wholesale]

I tell myself I must have faith. Rushangshenglongba

[Law Schooled]

Your post inspired a student author to write about collaborative exams on Law Schooled (http://lawschooled.org/?p=1569). As always, Law Schooled would appreciate your thoughts and comments. Thank you.

[Mike Madison]

I permit collaborative work on assessed exercises, but not on exams. I have my students write practice-style memos during the term. (I distinguish "practice-style" memos from the often more formal, stylized research memos that students learn to produce during their first year. Michael Teter's comment captures much of my method in giving students fact patterns.) I do not give any exams. Each student turns in his / her own paper.

I've been doing this in upper-level courses (Copyright, Trademark, Cyberlaw, E-Commerce) for a little over 10 years. I wrote up my method and my reasoning at Writing to Learn Law and Writing in Law: An Intellectual Property Illustration,
52 St. Louis Univ. L.J. 823 (2008), for anyone who is interested in a lot of detail.

Two brief observations here:

Relatively few of my students collaborate with others, although the number of collaborators goes up modestly over the course of the term. (I give three assignments per term.) I have never studied the reasons for this pattern. I surmise that students are quite unsure about how to proceed once they are released from the standard expectation that they work alone. And I also surmise that their competitive juices reinforce the idea that solo work is the preferred path. I have never seen a race to work with students perceived to be stronger. I could, conceivably, introduce some training on collaborative work. That would probably be a good idea. But because I grade each paper in great depth, I have already doubled the amount of time that I put into each course. I have almost no additional time to give.

I have not lost the ability to discern which students are stronger and which are weaker at an individual level. Might this change if more students collaborated with each other? Perhaps. And might this change if I were working with exams rather than with papers? Perhaps again. Much depends on what you are trying to teach (substantive knowledge, "skills" (I put that in quotation marks because I don't like the word or the concept much), professionalism, other things) and therefore on how you are looking to differentiate students. Also, I practiced law for a long time. I draw a lot on my earlier career in evaluating student work, in the sense that I tend to look at student writing from the perspective of a supervising lawyer in practice, or from the perspective of a client (I have been both), or both, rather than solely from the perspective of a law teacher. Of course, I tell the students this up front, in connection with each assignment, and we spend some in-class time both talking about what that means and using the concept in some hypotheticals.