Sunday, March 15, 2015

@ONE Designing Effective Online Assessments Course Week 3

Key Take-Aways from this week: Performance, Alternative and Authentic Assessment

Performance Assessment
The following three videos provide a good introduction to Authentic Assessment:
Assessment Overview: Beyond Standardized Testing
Authentic Assessment
Challenge Based Learning: Authentic Assessment

Standardized tests debate:
  • measuring the important aspects of student learning?

  • test recognition and recall of information and not higher order cognitive functions such as analysis, synthesis, and evaluation

  • only test knowledge and not skill or outcome

  • do not help educators understand how to improve their instruction.
As a reaction to these issues and concerns alternative approaches to assessment have developed in the last few decades. This movement has been labeled variously Alternative Assessment, Authentic Assessment or Performance Assessment.

All of these labels refer to the same philosophical approach to assessing student learning:

  • Assess student learning by examination of student products

  • Make the assessment situation as realistic as possible

  • Ensure that the assessment informs improvement in instruction

  • Assess higher order thinking and performance skills

  • Assess process as well as outcome
Technologically-mediated instruction environment aids the Performance Assessment movement:

  • computer and course management systems routinely capture communication and interactions between students as well as instructors

  • communications and interactions can be put on a timeline, categorized on their subject or content, examined for who participated in the interaction or communication, etc.

  • as students access external digital resources and produce new digital products, the products can be evaluated in a variety of ways (videos, audios, text, graphics, artwork, or a mixture of media)
Instructor can use own creativity and ingenuity to identify measures that inform:

  • Frequency of a student behavior

  • Timing of the behavior

  • Directionality of communication or interaction

  • Tone of a communication

  • Emotional content of a communication

  • Etiquette

  • Listening Behavior and skill
Many of these measures are directed at measuring student communication or interactions. This is consistent to the move to group and collaborative projects, problem-solving and higher order skills needed for the workplace.

Discussion Boards: Tips for Effective Discussions
MSJC Regular Effective Contact Policy

Rubrics are an essential component and should be provided.

Discussion prompts:

  • open-ended

  • requires students to demonstrate that they understand a concept

  • required to reply to each other’s posts, encouraging exposure to a variety of examples, and support or debate each other’s selections.

  • clear due dates for the post and the reply portion of the activity. Staggering due dates for posts and replies
Assume the role of a coach, helping guide the student(s) back to the right path, opposed to simply offering a judgment of correct/incorrect:

  • If a student is struggling with a concept and other students are leading them down the wrong path.

  • Not only is this a wonderful opportunity to help coach the students back to the right path, this could also be an indicator that there is a knowledge gap in the course that will need to be corrected.

  • Even if you have created a “student forum” to discuss class topics, if a student is struggling with a concept and not receiving help from other students.

  • Jump in and offer assistance, and remind the class to help one another out. If this happens to be a component of their participation grade, now might be a great time to remind them of that.

  • If a student is using the discussion board for a purpose other than that in which is created.
Social Media (Blogs, Wikis, FB, Twitter, etc.)
"Creating Content and Designing Learning Activities" from @One's Introduction to Online Teaching and Learning by Michelle Pacansky-Brock and her blog
Kathryn Damm's seminar entitled "Interactive Online Assignments to Engage All Learning Styles" YouTube video
Journal Writing
May take many forms ranging from structured responses to specific questions, to a general review of the events in a lesson or day. Furthermore, journals provide a setting in which students not only display their knowledge, but reflect on the learning activity and ask questions or indicate a need for additional help.

Reflective feature - metacognition, students can become more aware of how they learn. Developing control over their learning helps students use their strengths to improve achievement.

To guide the writing process, students should be asked to respond to prompts, usually in the form of questions. These prompts can be general or specific, and should be written clearly to ensure that student responses contain the assessment information the teacher is seeking. For the problem-solving activity contained in this overview, students could explain their concept of volume or the strategies they use to solve the problems, or analyze their own strengths and weaknesses in doing such problems. Sample Journal Writing Questions:

  • In your own words, what was the goal, or goals, of this activity?

  • What strategies did you use in this activity?

  • Did you have an “Aha” experience during this activity? What was it?

  • What are you confused about after completing this activity?

  • How did you feel in class during this activity?
Other possible, more open-ended writing prompts include...

  • The most important thing I learned today is . . .

  • Write a letter to a classmate who could not attend class today, explaining what we did so that she will understand and learn as much as you did. Be as complete as possible.

  • How do you know your solution is correct?

  • How would trying harder have helped you learn better during this activity?
The key in using students' activities, projects, and assignments for assessment purposes is to be clear about the following:

  • what the student is to produce

  • how you are going to evaluate it

  • providing feedback to the student about his/her performance and how to improve it

  • guide your students to the desired level of performance or behavior

  • project based learning model -  guided in a step-by-step process in completing the necessary elements  of an assignment
  • incremental deliverables
Project Based Learning Story: Collaborating Successfully
Rena Palloff and Keith Pratt's Collaborating Online (2005) provides an effective step-by-step method for developing successful collaborative activities in online classes

Setting the Stage
Palloff and Pratt's first step in creating a successful collaboration environment is "setting the stage" and this consists of 1)
explaining the importance of the collaborative work, and 2)providing clear guidelines for completing it

In this stage, the instructor must also be sure students are comfortable with the technology they will be using in the project; therefore, using the technology as a tool in an ice breaker the week prior (for example) is a good way to introduce it prior to the collaborative activity. Be sure to always explain the relevance of technology to the collaboration -- why are you asking them to learn yet another new technology?

Create the Environment
Palloff and Pratt also stress the importance of clarifying the "rules of engagement
  • Will the project/activity collaboration occur completely within one area (a specific VoiceThread, in Ning, in the discussion forums in Blackboard?

  • Will students be expected or encouraged to have a dialogue outside of this online space (through email, meeting in person, etc.)?

  • What role will the instructor play (provide weekly input, regular and direct participation, observer only)?

Model the Process
Instructors need to model collaboration in order for students to learn it effectively and follow. I believe modeling the process also contributes to the integrity of a learning community. The more involved and participatory an online instructor is in the collaborative activities s/he designs for students, the more valid and pertinent those activities feel to the learners.

Evaluate the Process and Listen to the Feedback
Asking for feedback during and at the end of a collaborative activity is essential to improving the quality of your students' learning and making necessary adjustments to the design of your course.

As Dan Wilton in his ePortfolio: a portal points out there are three commonly accepted types of ePortfolios:

Developmental ePortfolios. Demonstrate the advancement and development of student skills over a period of time. Developmental portfolios are considered works-in-progress and include both self-assessment and reflection/feedback elements. The primary purpose is to provide communication between students and faculty.

Assessment ePortfolios. Demonstrate student competence and skill for well-defined areas. These may be end-of-course or program assessments primarily for evaluating student performance. The primary purpose is to evaluate student competency as defined by program standards and outcomes.

Showcase ePortfolios. Demonstrate exemplary work and student skills. This type of eportfolio is created at the end of a program to highlight the quality of student work. Students typically show this portfolio to potential employers to gain employment at the end of a degree program.

Hybrids. Most eportfolios are hybrids of the three types of eportfolios listed above. Rarely will you find an eportfolio that is strictly used for assessment, development, or showcase purposes. Occasionally, you may come across showcase eportfolios that do not show evidence of self-reflection, rubrics for assessment, or feedback; however, as Helen Barrett, an expert in the field of eportfolios, would say, "A portfolio without standards, goals and/or reflection is just a fancy résumé, not an electronic portfolio" (Barrett, 1999, p.56).

Linda Suskie in her “Assessing Student Learning” notes six critical characteristics for portfolios:

Clear educational purpose: what is it that you want the students to do and learn from creating and using the portfolio

Student participation in selecting contents using faculty selection criteria: while the instructor must define the criteria to be used by the student, the student also is involved in selecting the artifacts that go into the portfolio

Evaluation criteria: the portfolio is assessed using criteria in the form of a rubric to evaluate the portfolio

Illustration of Growth: Portfolios can be used to illustrate growth in the student learning by including work samples at various points in the instructional process.

Continual Updating: students can and should continually update an refine their portfolios so they represent the current state of their competency

Student Reflection: this is one of the greatest aspects of using portfolios the ability to cause and capture student reflections on their work and their learning process.

CVC Seminar Series on ePortfolios

eFolio Getting Started, Part 2: Collaborating with Feedback, RSS Feeds, & Web 2.0 Object Integration (YouTube)

Using ePortfolios:

  • Portfolios are especially good for assessing programs or courses with small number of students

  • Portfolios can take a great deal of time and thus require careful planning and gradual implementation

  • Portfolios are especially powerful in assessing growth in students learning and reflection
Providing Feedback
Assessment can be more than just a mechanism for grading and benchmarking our students, that it can truly be an agent for learning. Providing feedback via a rubric is a powerful tool that will help guide your students to increased performance and learning.

A powerful context that you may consider when providing feedback is to assume the role of a coach, helping guide the student(s) back to the right path, opposed to simply offering a judgment of correct/incorrect.

Non-Traditional Assessment Feedback

Rubrics are an essential component of an online course. They set expectations for your students, offer the student a clear path on how to succeed, and provide you as the instructor with a mechanism to assess your students in the most bias way possible.  By their very nature, rubrics will facilitate learning by distinguishing levels of performance in which the student can deuce, by the rubric criteria, what they did well or not well. However, by selecting a rubric that allows a space for you to provide targeted feedback each student, the opportunity to leverage assessment for improved learning increases.

In addition to rubrics, try to get in the habit of looking for opportunities to build in feedback loops into assignments. With f2f courses, this comes naturally as you interact with your students. In the online class, these opportunities have to be intentionally created. The goal of course is to provide targeted, stimulating and supportive feedback that will support your students in their learning process.

A good rubric tells the student and instructor
  • By what criteria will the work be judged?
  • What is the difference between good work and weaker work?
  • How can we make sure our judgments (or scores) are valid and reliable?
  • How can both performers and judges focus their preparation on excellence?
Rubrics are one of the cornerstones of effective online assessments. For each assessment administered, a rubric should be provided. As we move into the next sections of the course and discuss the various assessment tools at your disposal, keep a constant eye on how rubrics will play a part in each assessment you create.

The use of performance measures, even in traditional situations such as open-ended items or narratives responses, all require a systematic approach to scoring the student work product. The purpose of a systematic mechanism for scoring the products is to help insure that the score derived is not dependent on who is doing the scoring. In ideal situations, scorers are trained on the mechanism until they reach a set criterion for reliability. This rarely happens in classroom situations but the goal is still the same.

Parts and Levels
A rubric can be thought of as having four parts:

Task Description: this is a description of what the student is expected to do.

Scale: this is a description of different levels of performance by student on a specific dimension being scored

Dimensions: these are the aspects of the performance or work product that are being scored, for example, quality, effectiveness, quantity, etc.

Description of the dimensions: this is a definition of the scoring dimensions
A rubric usually also includes levels of potential achievement for each criterion, and sometimes also includes work or performance samples that typify each of those levels. Levels of achievement are normally given numerical scores. A summary score for the work being assessed may be produced by adding the scores for each criterion. The rubric may also include space for the judge to describe the reasons for each judgment or to make suggestions for the author.
PPT slideshow excerpted from a workshop on "Constructing Rubrics for Open-ended Activities" conducted by Susan Haag and Ann Kenimer. These slides illustrate the process of constructing rubrics and some important considerations to think about.

Types and Examples

Analytic: scores are derived on multiple dimensions or criteria for a single work product

Holistic: a single score is derived to represent the entirety of how well the work product satisfied the work assignment.

Analytic rubrics are most common in the classroom and instruction. Holistic rubrics are applied if one is interested in either a quick gross assessment or the situation requires a single value to represent the performance (this often happens in a judgment situation such as Pass/No Pass). 
Holistic Rubric for Essay Questions
Clarity of thought, Complete. Shows understanding of all processes, reasonable hypothesis or thoughtful questions, conclusions supportable by data, shows creativity, some graphic representation of data or concepts.
Clarity of thought, shows understanding of major processes, includes good hypothesis or questions, draws acceptable inferences and conclusions, may have graphic representations.
Minor Flaws
Completes the assignment, but explanations may be slightly ambiguous or unclear, may contain some incompleteness, inappropriateness, or unclearness in representation, hypothesis, understanding of processes, or conclusions.
Nearly Satisfactory
Begins successfully, but omits significant parts or fails to complete, may misuse scientific terms, representations may be incorrect or omitted, incorrect or incomplete in analysis, inferences and conclusions.
Fails to complete
Assignment and explanation is unclear, or major flaws in concept mastery, incorrect use of scientific terms, inappropriate or omitted hypothesis.
Unable to begin effectively
Product does not reflect the assignment, does not distinguish what information is needed, restates the question without making an attempt at a solution.
No attempt
Does not begin assignment.

Analytical Rubric for logs and journal writing
Area of Product
daily entries
regular daily entries
entries 90% of the time
entries 80% of the time
entries less than 80% of the time
use of scientific language
consistent, accurate usage of terms
adequate usage of scientific terms
occasional use with few errors
no terms or frequent errors in usage
application to the real world
able to apply learning
usually finds practical application
occasionally relates to real life skills
no practical application
concept understanding
shows understanding of key concepts
usually demonstrates understanding
inadequately demonstrates understanding
poor understanding of concepts
clarity of thought
well organized
adequate organization
limited organization
poor organization

Examples are from a site at Georgia State University that provides examples and information on rubrics and their use.

Other Rubrics
Rubrics can also be developed for digital artifacts created by students. The link below is an example from Michelle Pacansky-Brock for one of her Voice Thread Assignments: Voice Thread Rubric PDF.
The Enberg Grading Forms PDF is another good example of grading rubrics.

Supplemental Materials for Rubrics

A repository of rubrics from College of Charleston with a matrix of rubrics across many disciplines 

Example Rubric of Civic Engagement for Values from the Association of American Colleges and Universities CIVIC RUBRIC

Handout for 'How to Create Rubrics' with best practices outlined. CREATING RUBRICS

Harvard Resource for Essays: Essay Rubric

Giving Feedback to Students
Assessment can be more than just a mechanism for grading and benchmarking our students, that it can truly be an agent for learning. A powerful context that you may consider when providing feedback is to assume the role of a coach, helping guide the student(s) back to the right path, opposed to simply offering a judgment of correct/incorrect or a numbered score.

Rubrics are an essential component of an online course. They set expectations for your students, offer the student a clear path on how to succeed, and provide you as the instructor with a mechanism to assess your students in the most bias way possible. As you will see, or know from your own experience, rubrics can be implemented in a variety of ways. By their very nature, rubrics will facilitate learning by distinguishing levels of performance in which the student can deuce, by the rubric criteria, what they did well or not well. However, by selecting a rubric that allows a space for you to provide targeted feedback for each student, the opportunity to leverage assessment for improved learning increases. In particular, it helps to identify these three points:

1.     What is the student is doing correctly

2.     What areas still need improvement

3.     Suggestions and ideas for making these improvements

Materials and Resources for non-Traditional Assessments
Performance Assessment:
A Practical Guide to Assessment (PDF)



My Submitted Assignment:
Performance Assessment and Rubric Assignment
Noncredit ESL Level 7 Performance Assessment

SLO: Students will be able to compose a well-organized, coherent paragraph with appropriate detail and sentence variety.

Prompt: Write a paragraph comparing and contrasting some of the key features of two education systems, that of the United States and that of your native country or other country in which you have lived or worked, to explain how the systems are similar and different to someone from your native country.  The paragraph needs the following:

  • An effective topic sentence that clearly identifies the two education systems to be compared and contrasted and states the purpose of the paragraph
  • Specific support provided in a variety of sentence types explaining at least two comparable features of the two education systems and at least two differences; correct vocabulary use; a minimum of one comparative or superlative grammatical structure; and a consistent order with a variety of comparison and contrast transition words
  • An effective concluding sentence
  • Standard format (heading, centered title, indented first line, standard font and margins, double-spaced) and no major errors in mechanics (grammar, spelling, punctuation) that distract the reader or interfere with meaning

Comparison Contrast Paragraph Rubric                                               
Total points possible = 20       Pass = 16 points
Content & Organization
12 points
Topic Sentence
Effective topic sentence that clearly identifies the two education systems to be compared and contrasted and the purpose
Topic sentence included but the topic and/or purpose unclear
No topic sentence
Supporting Sentences
At least two clearly-explained points of comparison and at least two clearly-explained points of difference with specific examples; only relevant information
Only three points of comparison/contrast explained with examples and/or one piece of irrelevant information
Features only compared or contrasted, not both, or only two features compared/contrasted; some support general or incomplete; some information not relevant
Only one feature compared or contrasted; support too general / not sufficiently explained and/or irrelevant
No features compared/contrasted; details not provided or not explained
Consistent order when discussing the comparisons and differences
Organization difficult to follow because some information is in the wrong order, causing some distraction or confusion for reader
Many details not in a logical or expected order, causing incoherency in paragraph
Variety of comparison and contrast transition words to move smoothly from one idea to the next
Transition words used but some repetition/lack of variety
One or no transitions used
Concluding Sentence
Effective concluding sentence
Concluding sentence does not effectively sum up details/direct reader to main idea
No concluding sentence
4 points
Sentence Structure
Variety of sentences (simple, complex, compound)
Lack of variety in sentence structure or only simple sentences
Excessive sentence errors (word order, run-ons, fragments) that interfere with meaning
Correct use of vocabulary; variety in word choices
Some word errors that do not seriously interfere with meaning; some repetition
Numerous errors in word choice that interfere with meaning and/or excessive repetition
2 points
No errors in grammar, spelling, or punctuation; at least one comparative and/or superlative grammar form
Minor errors in grammar, spelling, or punctuation; attempt at using a comparative and/or superlative form
Multiple errors in mechanics that distract the reader from the content or interfere with meaning; no use of a comparative or superlative form
2 points
Heading correct, title is centered, first line is indented, standard font and margins, double-spaced
Minor errors in format
Format is incorrect/not standard


No comments:

Post a Comment