TO: Dr. Maria Harper-Marinick, Vice Chancellor of Academic and Student Affairs, Maricopa Community Colleges
FROM: Jeffrey Andelora, Chair, and Shelley Rodrigo, Professor, MCC English Department
DATE: May 4, 2010
SUBJECT: Proposal Concerning New Placement Procedure for First-Year Composition
We write to propose running a pilot version of a feasibility study to be conducted during Fall 2010 concerning the MCC English Department's recently mandatory placement procedure for Developmental Writing (ENG 071, 081, and 091) and First-Year Composition (ENG 101).
Need/Concern to be Addressed
As you are well aware, MCCCD has been moving toward a mandatory placement policy for the last couple of years, and the MCCCD Strategic Plan 2009-2010 indicates that better student retention is a district-wide strategic direction as well as a strategic goal specific to MCC. It has been hoped that the district-wide Student Success Initiative (SSI) would contribute to reaching this strategic direction/goal, and one of the requirements of the SSI is that a majority of students must take a battery of placement tests if they plan to enroll in English, reading, math, or ESL courses throughout the district. As stated in the MCC Catalog 09-10, under the current policy placement testing is strongly urged, but it is not yet mandatory for students to enroll in the English, reading, or math course(s) indicated by their placement testing outcomes. It is our understanding, however, that the MCC Catalog 10-11 will reflect changes to the placement policy making it mandatory for students to enroll in the courses indicated by their placement testing outcomes. It is in this context that the district's Course Placement Council (CIC) recently asked the English Instructional Council (EIC) to investigate a more valid and reliable placement instrument that would place students more accurately and successfully than the current English placement instruments, which are strictly multiple choice tests and thus indirect measures of writing competence/ability. In addition to this, there has been for some time a strong desire from within the MCC English Department itself to investigate a more theoretically and practically valid and reliable placement instrument with which to place students more accurately and, thus, promote their greater academic success.
MCC's Current English Placement Practices
Currently, the MCC Testing Center's official practice is to use one of three multiple-choice tests for English Placement: ACT's ASSET, ACT's COMPASS, and The College Board's ACCUPLACER. District cut scores are set by the English Instructional Council. Every three years, the EIC reviews these scores based on statistical data regarding retention and pass rates. ASSET is rarely used today because of a variety of reasons including the fact that it isn't adaptive (e.g., has a set number of questions, 34-36 for English) and is a pencil-and-paper examination. The Testing Center had been using COMPASS for at least the last ten years, but about one year ago the director began using ACCUPLACER primarily due to it being adaptive to student performance, web-based, and easier to set up in computer labs across campus when the large volume of students testing exceeds the physical confines of the Testing Center's limited space. Although placement testing occurs year-round, the Testing Center administers the majority of tests, approximately 9,000, right before the Fall semester begins. As you are aware, for a long time students have been able to re-test in one subject area of their choice (e.g., English, reading, Math, or ESL), and this is still the case. It is also true that for a long time students have had the option to appeal the placement outcome based on their scores. While there is still an option for students to appeal their placement outcome, the wording in the district guidelines has recently been changed. The old language implied that all the student had to do was ask for a placement waiver from the appropriate department chair. The updated language of the district guidelines indicates that the decision of whether to sign the placement waiver belongs to the chair of the requisite department and that additional testing may be required. Without clear guidelines, some chairs would waive the score and allow students to enroll in a course above where they tested. This practice has rightly raised concerns among college and district administrators, however, that bumping students to a course above placement sets them up for academic failure.
Proposed Action
So as to not impact registration and enrollment practices that are already in place, we propose that students in our pilot study take the established ACCUPLACER placement test before the start of the Fall 2010 semester. Prior to students taking ACCUPLACER, eight full-time faculty with experience teaching all four relevant courses (ENG 071, 081, 091, and 101) will be identified and briefed on the pilot study during the summer of 2010.
Faculty who agree to participate will allow two ASU graduate student researchers, Emily Hooper and Yazmin Lazcano, to give a brief introduction/overview of the study to their courses during the first week of classes, which begin on August 21, 2010. Participating faculty members will also agree to a 45-minute individual interview with either Emily or Yazmin. For their time, students will be given a ten-dollar gift card to iTunes. In addition to this, self-selecting students who agree to an additional 20-minute interview with either Emily or Yazmin will be given an opportunity to receive some sort of extra credit in their course.
Students in each course (ENG 071, 081, 091, 101) will be divided into two groups, A and B, for the purpose of staggering which test--either WritePlacer or e-Write--they will take in weeks two and three of the semester. During week two, students in A groups will take e-Write and students B groups will take WritePlacer in the MCC Testing Center at their individual convenience. The following week, A groups will take WritePlacer, while B groups will take e-Write.
Having the same group of students take all three placement tests will yield data offering a comparative view of performance across an indirect and two direct assessments, as well as across two different computer-scored essay placement tests. To supplement this data set, qualitative data will be collected in the form of a brief survey for all participating students and interviews with both faculty and a smaller subset of student participants. This qualitative data will provide insight into the perceptions of the fairness and reliability of each placement test taken. Finally, researchers will collect quantitative data regarding passing and retention rates of students in the pilot for comparison to a control group of students outside the pilot.
Resources Needed
At the beginning of March, we submitted an application for a 2010-2011 Learning Grant with which to run the proposed pilot study. We should receive news of a decision regarding this grant within the next couple of weeks. The table below accounts for how we plan to allocate the funds from the Learning Grant.
Resources
Description and Amount/Qty/Duration
Money
Student stipend: iTunes gift card ($10/student X 176 students) $1,760
The College Board’s ACCUPLACER WritePlacer essay component ($4/unit) $704
ACT’s COMPASS e-Write essay component ($5/unit) $880
People
Full-time MCC English Department faculty -- 8
MCC students from 2 sections each of ENG 071, 081, 091, and 101 (22/class) -- 176
Self-selecting student population for interviews -- TBD (ideally 16+)
ASU PhD student researchers (Emily Hooper & Yazmin Lazcano) -- 2
Time
Class time (20 minutes/class on First Day of Classes) 20 minutes X 8 classes = 2.5 hours total
Student testing (Weeks 2 and 3) 2 hours/student X 176 students = 352 hours total
Self-selecting student interviews (Weeks 4) 20 minutes/interview X 1 interview/student X 16 students = ~6 hours total
Faculty interviews (Week 4) 45 minutes/interview X 1 interview/faculty member = 6 hours total
Facility
MCC Testing Center computer lab Aug. 28-Sept. 10
Calendar
Summer Semester 2010
• June 2010: Identify 8 full-time faculty with experience teaching ENG 071-101. Contact and meet with them to discuss placement study design plan.
Fall Semester 2010
• August 21, 2010 (Saturday) – First day of classes
• Week 1: August 21-27
o Emily and Yazmin visit participating courses and introduce the study to students
• Week 2: August 28-September 3
o All group A students take e-Write and all group B students take WritePlacer in the Testing Center at their convenience. All students answer brief online survey (see Appendix A) directly following testing while they are still at the Testing Center. Emily and Yazmin provide faculty with interview protocol questions via e-mail.
• Week 3: September 6-10
o All group A students take WritePlacer and all group B students take e-Write in the Testing Center at their convenience. All students answer brief online survey (see Appendix A) directly following testing while they are still at the Testing Center.
• Week 4: September 11 -17
o Conduct individual 45-minute interviews with 8 faculty members after their students have taken both placement tests. (Emily conducts 4. Yazmin conducts 4. Both use audio recordings.)
o Conduct individual 20-minute interviews with self-selecting student population based on extra credit offer. (Hopefully 16+, approximately 2 from each of the 8 classes.)
• Weeks 5-16: September 18 - December 11
o Preliminary data analysis
• Last Day of Class: December 12 (Sunday)
• Last Day of Fall Semester 2010: December 17 (Friday)
Spring Semester 2011:
• Continued data analysis and reporting of findings.
• May 1, 2011: Final Report due to Office of Vice Chancellor of Academic and Student Affairs.
Appendix A
Interview protocols include adapted questions from:
Herrington, A. & Moran, C. (2006). WritePlacer Plus in place: An exploratory case study. In F.Ericsson & R. Haswell (Eds.), Machine scoring of student essays (114-129). Logan: Utah State University Press.
Interview Protocol for Faculty
Week 4 Faculty Interview
1. Have you tried the current placement system, ACCUPLACER, yourself?
2. How satisfied are you with using ACCUPLACER as a placement system? Do you believe the students in your class were placed accurately by ACCUPLACER?
3. Do you feel there is a connection between the placement system with ACCUPLACER and the curriculum? If yes, what is it? If no, why not?
4. What is the best thing about ACCUPLACER?
5. What is the worst thing about ACCUPLACER?
6. Have you tried WritePlacer and e-Write for yourself? If so, what was it like to write an essay online to be evaluated by a computer program?
7.Are you generally satisfied with the writing-placement process that includes WritePlacer and e-Write? (Do you think it is a fair way of evaluating students' writing for placement? If given a choice, would you prefer to have a person or the computer program evaluate your students' placement essays, or doesn't it matter? Why?)
8. Do you think a computer looks for different things when evaluating your writing for placement than a person would?
9. What is the best thing about WritePlacer/e-Write?
10. What is the worst thing about WritePlacer/e-Write?
11. Do you feel there is a connection between the placement system with WritePlacer/e-Write and the curriculum? If yes, what is it? If no, why not? (Explicate with course competencies)
Interview Protocol for Students
Weeks 2 and 3 Student Survey: WritePlacer
Background Information
• Gender: Female ___ Male ___ Transgender ___
• Age: under 18___ 18-22___ 23-35___ over 35___
• When were you last in school prior to enrolling at MCC?
• When did you last take an English class? Where?
Short Answer Questions
1. What was it like to write your placement essay online with WritePlacer?
2. What do you think WritePlacer is reading for when it evaluates your writing? That is, what aspects of your writing do you think it's considering when evaluating it? (Do you think a computer looks for different things when evaluating your writing for placement than a person would?)
3. Does it matter to you whether your teacher or a program reads your writing? Why?
4. Do you think that the computer program will be fair to you in evaluating your essay for placement? (Do you think a person would be more fair?) (If given a choice, would you prefer to have a person or the computer program evaluate your writing, or doesn't it matter? Why?)
Weeks 2 and 3 Student Survey: e-Write
Background Information
• Gender: Female ___ Male ___ Transgender ___
• Age: under 18___ 18-22___ 23-35___ over 35___
• When were you last in school prior to enrolling at MCC?
• When did you last take an English class? Where?
Short Answer Questions
1. What was it like to write your placement essay online with e-Write?
2. What do you think e-Write is reading for when it evaluates your writing? That is, what aspects of your writing do you think it's considering when evaluating it? (Do you think a computer looks for different things when evaluating your writing for placement than a person would?)
3. Does it matter to you whether your teacher or a program reads your writing? Why?
4. Do you think that the computer program will be fair to you in evaluating your essay for placement? (Do you think a person would be more fair?) (If given a choice, would you prefer to have a person or the computer program evaluate your writing, or doesn't it matter? Why?)
Week 4 Student Interview
1. At this point in the semester, what is your perception of how you are meeting the number X course competency for ENG 071/081/091/101? (At this point, the interviewer will share a list of the appropriate MCC course competencies with the student and question the student about each course competency individually. For ENG 071 and 101, there are seven course competencies each. For ENG 081 and 091, there are ten course competencies each.)
2. At this point in the semester, what is your overall perception of how you are performing in ENG 071/081/091/101?
3. You were placed into your current course by ACCUPLACER. Do you believe this placement decision was accurate? Did this placement decision set you up for academic success?
4. Your WritePlacer score would have placed you into ENG 071/081/091/101. Do you believe this placement decision would have been accurate? More accurate/less accurate/or about the same level of accuracy as the ACCUPLACER placement decision that was actually used?
5. Would the WritePlacer placement decision have set you up for academic success?
6. Your e-Write score would have placed you into ENG 071/081/091/101. Do you believe this placement decision would have been accurate? More accurate/less accurate/or about the same level of accuracy as the ACCUPLACER placement decision that was actually used?
7.Would the e-Write placement decision have set you up for academic success?
Appendix B
MCCCD Official Course Competencies ENG 071: Language Skills: Speaking and Writing Standard English
1. Generate grammatically correct simple, compound, and complex sentences.
2. Brainstorm, develop coherent sentences that can be organized into thought groups.
3. Revise sentence fragments into complete sentences.
4. Proofread and edit written work to correct errors in punctuation, spelling, and usage.
5. Organize and present an oral report based on library skills.
6. Use library skills to locate and gather information to organize and present in an oral report.
7. Use a computer to generate written text.
http://www.maricopa.edu/curriculum/D-L/096eng071.html
MCCCD Official Course Competencies ENG 081: Basic Writing Skills
1. Describe the contextual nature of writing, including the importance of circumstance, purpose, topic, audience and writer.
2. Organize writing to support a central idea through unity, coherence, and logical development.
3. Use conventions in writing complete sentences, using appropriate grammar, and using mechanics.
4. Use conventions in writing, including consistent voice, tone, and diction.
5. Recognize effective and appropriate ideas.
6. Craft a variety of sentence types.
7. Recognize and implement steps in the writing process for sentence and paragraph projects, including prewriting, drafting, and editing for unity and coherence.
8. Use feedback obtained from peer review, instructor comments and/or other resources to revise writing.
9. Assess one's own writing strengths and identify strategies for improvement through instructor conference, portfolio review, written evaluation, and/or other methods.
10. Generate, format, edit, and deliver writing using appropriate technology.
http://www.maricopa.edu/curriculum/D-L/096eng081.html
MCCCD Official Course Competencies ENG 091: Fundamentals of Writing
1. Recognize how rhetorical contexts (including circumstance, purpose, topic, audience and writer) affect writing.
2. Organize writing to support a central idea through unity, coherence, and logical development.
3. Use conventions in writing complete sentences, using appropriate grammar, and using mechanics.
4. Use conventions in writing, including consistent voice, tone, and diction.
5. Generate and support effective and appropriate ideas.
6. Integrate a ariety of sentence types.
7. Recognize and implement steps in the writing process for paragraphs and multi-paragraph projects, including prewriting, drafting, and editing for unity and coherence.
8. Use feedback obtained from peer review, instructor comments and/or other resources to revise writing.
9. Assess one's own writing strengths and identify strategies for improvment through instructor conference, portfolio review, written evaluation, and/or other methods.
10. Generate, format, edit, and deliver writing using approriate technology.
http://www.maricopa.edu/curriculum/D-L/096eng091.html
MCCCD Official Course Competencies ENG 101: First-Year Composition
1. Analyze specific rhetorical contexts, including circumstance, purpose, topic, audience, and writer, as well as the writing's ethical, political, and cultural implications.
2. Organize writing to support a central idea through unity, coherence, and logical development appropriate to a specific writing context.
3. Use appropriate conventions in writing, including consisten voice, tone, diction, grammar, and mechanics.
4. Summarize, paraphrase and quote from sources to maintain academic integrity and to develop and support one's own ideas.
5. Use feedback obtained from peer review, instructor comments and/or other resources to revise writing.
6. Assess one's own writing strengths and identify strategies for improvement through instructor conference, portfolio review, written evaluation, and/or other methods.
7. Generate, format, and edit writing using appropriate technologies.
http://www.maricopa.edu/curriculum/D-L/096eng101.html
