Blog #5: Dissertation– Predictive Validity of MC Test for CC Placement

Verbout, Mary F. Predictive Validity of a Multiple-choice Test for Placement in a Community College. Diss. Indiana University of Pennsylvania, 2013. Ann Arbor, 2013. 3592228. Web. 18 Mar. 2016.

Verbout studied scores from Compass to prove that the cut-off scores used for placement did not correlate with success in the courses. The program in the study was optimal because while students were required to take the test, they were not required to use the results when selecting their courses.

 

According to the test creators, “The test is efficient; an algorithm recalculates the students’ overall score with each answer, and as soon as the internal calculation achieves certainty, the test is over, and the score is calculated (ACT Inc., 2006)” (3). It should be noted that “The eight domains addressed by the writing diagnostics are: punctuation, spelling, capitalization, usage, verb formation/agreement, relationships of clauses, shifts in construction, organization” (75-76), so the test does not coincide with the overall goals of the courses.

 

Research questions: does the placement based on Compass scores predict success in FYC1 and 2; is there a significant difference between the mean scores for White,  Hispanic, and Native American students; is there a significant difference between success rates for White, Hispanic and Native American students in FYC 1 and 2 (67).

 

SPSS and Excel were the tools used to process data. The tests were ANOVA and Chi-Square. Of the students who scored in the 13-37 range on Compass, 23% of the students who enrolled in BW1 passed FYC, 39% of BW2 passed, and 76% of the students who chose to enroll in FYC1 against recommendation passed it!

 

The group of test scores for direct BW2 enrollment ended up with pass rates below the students who chose to skip BW2 and enroll directly into FYC1 (46% to 81%) (82).

The results about race were: the test scores placed Hispanic and Native American students into BW at a higher rate than White students. The pass rates for students in FYC were not significantly different. What was of interest is that the Hispanic and Native American students were more likely to enroll in the course that was recommended for them based on test scores than the White students, who were more likely to choose to take FYC against test score recommendation (84).  “With one exception (Hispanic students scoring 13 -37, first  course BW2 students in each score range and ethnic category completed FYC1 and FYC2 at higher rates when they began in a more advanced course” (85).

The researcher’s conclusions were to discontinue use of Compass and consider creation of studio-model BW courses similar to CCBS’s Accelerated Learning Program (ALP).
The results and methodology sections were not as full as the excellent literature review that combines educational and composition theorists. Verbout does not stipulate the year range for this study and how long students were followed nor include whether the students in BW1 and BW2 also had additional remedial courses to take that could slow their entrance into FYC. More information would be needed in order for the study to be replicable, but the statistics were clear, direct, and surprising.

Advertisements

#4 Dissertation about Predictive Factors associated with Need for Remediation

Whiton, John C. Predictive Factors Associated with Newly Graduated High-School Students’ Enrollment in a Remedial Course at a Community College.  Diss. Liberty University, 2015.  Ann Arbor, 2016. 10027104. Web. 14Mar. 2016.

Whiton administered a questionnaire at a community college in Maryland. His research question was which factors best predicted student need for remediation. His research focused on recent high-school graduates and analyzed factors such as age, gender, race, high-school GPA and social and cultural capital. The questionnaire was an already established survey.

The survey itself was administered as a convenience sampling. He needed at least 150 survey responses from students who were enrolled in remedial classes (reading, writing, math) and 150 from students who were enrolled in credit-bearing courses.

Many of the questions asked were coded into binary and a logistic regression analysis was used for data analysis. The surveys were analyzed based on student response to the question of whether or not they were enrolled in a remedial course.

Students were surveyed in a variety of class types (math, English, performing arts, etc.) as convenience allowed. The researcher needed to garner instructor permission to administer the survey in class. The survey took 10 minutes of class-time, and students placed their completed surveys into an envelope that was then placed with other enveloped-surveys into a larger envelope. Students were eligible for a drawing for a $200 gift card if they filled out a separate sheet with their contact information. Surveys were kept anonymous, and data was entered into an Excel spreadsheet that was coded in binary terms whenever possible.

In the methodology section, Whiton describes how he eliminated one question category as all but 3% answered in the affirmative. For other questions, he regrouped some of the answers. For instance, in one question about how often parents spoke to the student about college, two answers were combined into a category: “never/sometimes” were grouped as one while “often” was used as the reference group. Whiton describes other instances of this regrouping, most notably when parental income was grouped at below $35,000, $35,001-$50,000, and above $50,000.

His findings were that students were 73% less likely to enter a remedial course if they had taken math above Algebra 2, 68.9% less likely if the annual family income was above $50,000, and 53.5% less likely if students discuss community, national, and world events with parents/guardians often.

ANALYSIS:

As a composition researcher, I am troubled by the focus on math. Whiton provides a literature review for why he focuses on the highest level of math taken in high school as a predictive quality, but he also quotes the Scott-Clayton piece I referred to in an earlier blog post as showing that math placement exams are traditionally more reliable for placement than the English placement tests. If anything, he has only proven that the math scores are predictive of who will place into remedial math or not based on prior math experience.

The surveys were given in a variety of courses, so students could have been in a credit-bearing course such as Intro to Fine Arts but also co-enrolled in a remedial course. Additionally, the students who responded “yes” to being enrolled in a remedial course could have been enrolled strictly in math or in an English course or in a total of three remedial courses. As some of the questions focus on composition skills such as language, it seems like a missed opportunity to not group the students by which courses they were enrolled in.