How Do You Grade 100,000 Papers?

I am currently taking another massive open online course. It is the "Crash Course on Creativity" course being offered by Stanford.

I am planning to teach a MOOC in 2013. I am taking notes and rethinking the way I have taught online in the past ten years. I know that I will have to change some things about the way I offer content. I will need to think about presentation and perhaps create some video lectures. And I will really have to rethink how to assess the student work.

As a recent NY Times article pointed out, assessing these massive courses is a massive problem. Some courses use automated graders, but most have had to move to alternatives. Particularly difficult are courses that require writing and analysis.

Coursera uses peer grading and five peers should grade your submission and you need to grade five assignments. If you are a teacher and have tried peer grading in your regular classes, you know that it has lots of problems. Not every student can assess accurately or fairly. Certainly we shouldn't expect that they could just step into that role.

Some course hubs are experimenting with software to grade work and to flag students who assign very inaccurate grades much like holistic norming sessions have done for years with multiple readers.

The article points to Mitchell Duneier, a Princeton professor, who is studying the peer grading on the final exam from his summer MOOC on Coursera. But considering the diversity of MOOC learners - something that is part of the mission of the courses - you have learners of all ages, experiences, training and native languages.

“We desperately need crowdsourcing,” says Cathy N. Davidson, a Duke professor of English and interdisciplinary studies. “We need a MOOCE — massive open online course evaluation.”


Unlearning Being a Sage on the Stage

drawingboardIf I use the acronym MOOC much more in posts, some readers will probably click off this blog. Yeah, in this year of the MOOC, there has been some saturation. But some of the issues connected to those courses are issues that have been relevant to online education for decades and some literacy and pedagogical concerns are important to all versions of the classroom.

One of the phrases that I heard when I first entered higher education (from the world of secondary education) was that teachers needed to shift from being the "sage on the stage" to being a "guide on the side."  This pedagogical shift was from a teacher-centered classroom to a learner-centered space.

That shift did occur - to a degree - and online learning and the arrival of the Internet made it occur faster.  Even in the college lecture hall of 300 students, the teacher was still the sage on the stage at the center of the learning experience. In the online class of 30 or 300, the teacher still controlled the content and, to a lesser degree, the interaction.

Remember "lifelong learning?"  As with the sage and guide learning shift, lifelong learning got some traction, but didn't move as fast or as far as expected. I expect that it will again be part of the conversation.  Many of the learners in these college MOOCs are lifelong learners who are seeking knowledge and not necessarily degrees.

Unlearning is harder than learning. That is not only true with bad habits, but with good ones. I have spent a number of years working with faculty to create or redesign courses to be used online. It was not always that their face-to-face course was bad and needed to be fixed. It was more often true that you simply could not do the same things you did in a classroom online and make it work. You just had to be less the sage because you had lost the stage. You have to rethink not only what you teach, but how you teach.

Now, these MOOCs are forcing us to rethink what we teach, how we teach and also why we teach. If you want to be a guide on the side, teach a course with 25,000 students. If you are a learner and want to experience sages on stages, go to YouTube.


Universal Design for Learning

cartoon

I read that in Maryland they have decided to incorporate Universal Design for Learning into the state educational systems.  Universal Design for Learning (UDL) is a set of principles for curriculum development that seeks to give all individuals equal opportunities to learn. It also supports test design and instructional material selection for all learners.

UDL is based on the learning styles research that shows that people learn best in different ways such as via visual representations or text or through engaging activities etc.

The UDL approach requires curricular flexibility so that teachers don't have to design multiple lesson plans to accommodate individual students.

The main tenets of UDL all are concerned with providing options (via http://www.cast.org)
Multiple means of representation - give diverse learners options for acquiring information and knowledge
Multiple means of action and expression -provide learners options for demonstrating what they know
Multiple means of engagement - to tap into learners' interests, offer appropriate challenges, and increase motivation

Many students can benefit from UDL. I think a lot of people associate this design with students who have disabilities, but that is not really the reason for its acceptance. Although any student can benefit from having options, it may particularly help students who speak English as a second language, international students, and older adult students.

Many of the techniques are simple: putting course content online so that students can review material missed in class. Making peer mentoring, group discussions, and cooperative learning situations rather than just lecture part of the course. Teachers can use "guided notes" so that students listen for essential concepts without just being handed notes or simply copying notes from a board or projected slide.

One way to evaluate your own course and methodologies is to monitor what options you do offer and how willing you are to change your instruction.

How often do you change course materials based on current events and student demands?
Do you vary your instructional methods?
Do you provide illustrations, handouts, auditory and visual aids?
How do you get student feedback, provide instructions, ask questions, and connect new topics to prior learning or real-life situations?

The full report and recommendations of the Maryland Statewide Task Force to Explore the Incorporation of Universal Design for Learning UDL Principles is available as a pdf at http://www.marylandpublicschools.org




Web Site Credibility

validStudents at all grade levels need to be forced to evaluate Web site credibility for the sources they use. What is taught or required of elementary school students and what we expect from graduate students is clearly very different. But it seems to me that too many teachers at all the higher levels (high school and above) make dangerous assumptions that students "should have learned how to do that" at the preceding level.

There are many online resources to use, but most require students to consider questions of credibility.  Here are a few typical questions students need to be taught to ask when using a site as a source.

1. Who is the author(s)? What are their credentials?

2. Note the site domain (such as .com, .org or .edu) which may provide useful information. While .ac and .edu sites are regulated educational sites, that doesn't validate everything found there. For example, my own web space at NJIT http://web.njit.edu/~ronkowit/  contains my own content. That tilde ~ indicates that this is not part of the main njit.edu site. The domains .com and .biz sites are for commercial purposes and .gov sites are U.S. government sites. Is information credible because it is on a government site? Is information biased if it is on a commercial site? There is no standard answer and students need to be able to distinguish the differences. Other URL endings indicate the country of origin of the site and some are sponsored and regulated (for example, .jobs, .museum and .travel)

3. Especially when there is no "author" available by name, who is making the information available?

4. How is the site being funded? Are they trying to sell you something?

5. Does the site appear to have any social or political biases? You might find clues in an "About Us” section but you probably need to dig deeper to get an unbiased view about their bias.

6. When was the information first published? Has it been updated recently? Some pages indicate when they were created/revised but finding the equivalent of a copyright date isn't as easy as with a print publication.