Rajah E. Smart, Ed.D. & Phillip Caldwell, II, Ph.D.
Journal of Effective Schools Project, 2018, Volume XXV
Purpose and Significance of Study
In Michigan, where charter and traditional public schools receive approximately the same operational funding, according to a recent analysis which found that the school resource allocation average for charter school spending represented nearly $800 more per pupil annually on administration and $1100 less on instruction (Arsen and Ni, 2011). The purpose of this study was to analyze whether education management organizations (EMO) and its schools are more efficacious than similar self-managed (hereafter, non-EMO) schools using student performance on the Michigan Student Test of Educational Progress (M-STEP) in grades three through eight. To determine if relationships exist between schools that have EMO and non-EMO schools in Michigan, this study investigated differences in student performance between locale and school types. The study used the data from grades three through five on the math and English language arts (ELA) M-STEP from 2014-2016. A factorial analysis of variance (ANOVA) using a 3 x 2 factor design examining the effects of locale and differences between EMO and non-EMO schools. The outcome of the investigation yielded no statistical significance between EMO and non-EMO school's student performance on the M-STEP; and no cause and effect relationship between locale and student performance on the M-STEP.
Problem Statement and Research Questions
Does the variance of resource allocation by school type and locale yield statistical significance relative to student performance in Michigan public schools? The research questions for this study were the following:
1. Is there a significant difference between the EMO and non-EMO schools M-STEP scores in English language arts (ELA) and math in grades three through five?
2. Is there a difference in M-STEP scores for grades three through five ELA scores based on the locale and school type (EMO and non-EMO)?
3. Is there a difference in M-STEP scores for grades three through five math scores based on the locale and school type (EMO and non-EMO)?
For this study, the following hypothesis was developed:
H0: There is a no significant difference in performance on M-STEP based on locale between the EMO and non-EMO schools.
H1: There is a significant difference in performance on M-STEP based on the locale for the EMO and non-EMO schools.
After obtaining the means from the analysis, an independent-samples t-test was used to determine the difference in the mean of scores for math and ELA, grades three through five, and whether the scores differed between the EMO and non-EMO schools. The locale of the school was included to compare the EMO and non-EMO performance. The data was used to perform a factorial analysis of variance to determine whether there is an interaction effect between the school’s locale and the school type (EMO and non-EMO), and its influence on the M-STEP scores.
Across the research questions, M-STEP data from grades three through five were collected to discover differences in the scores for math and ELA from years 2014 through 2016. The sample includes n=96, and elementary schools ranging from grades three through five. The EMO and non-EMO schools in Michigan are distinguished by urban (n=40), suburban (n=44), and rural (n=12). The raw scores are collected for EMO schools (48) and a sample of non-EMO schools (48) with similar characteristics. The M-STEP raw scores for only two years of data (2014-2016) are combined for math at grades three through five, as well as ELA at grades three through five. There are only two years of scores because the M-STEP has only been in implementation since 2014-2016. The 2016-2017 scores were not available at the time of the study. The score means for non-EMO and the EMO schools in the areas of math (MG) and ELA (EG) were compared at each grade level. The data collected from the descriptive analysis is used to perform a 3 x 2 factorial analysis of variance (ANOVA). The ANOVA compares the effects of the groups to determine whether there is an interaction effect between the locale and school type, and its influence on the math and ELA M-STEP scores.
Data Collection Sources and Techniques
The study is solely quantitative and uses various collection databases. The quantitative research sources include student data collected from the Michigan Department of Education’s, Center for Educational Performance and Information (CEPI), the Michigan Student Data System (MSDS), and the Educational Entity Master (EEM). These databases contain all student data relevant to the study (i.e., locale and M-STEP data).
Along with the M-STEP data, the EEM was used to verify a sample of similar non-EMO to compare based on locale. The query for the locale was initially conducted using a 10-mile radius around each the EMO school to find similar non-EMO schools. Unfortunately, the query was changed to the intermediate school district (ISD) area or a wider area because schools near the EMO schools are different in configuration. The EEM helped map the locale and run the query for similar schools in the ISD area.
Data Analysis Techniques
Using the scores from grades three through five on the math and English language arts (ELA) M-STEP from 2014-2016, the study used the data to understand the difference between the EMO and non-EMO schools. The scores were combined over the two-year span to provide an average score for grades three through five performances on the math and ELA assessments. Once the Mathematics and ELA grade level means were calculated for the EMO and non-EMO schools, an independent samples t-test was conducted to determine if a difference exists between the means of the two independent groups on the M-STEP.
The factorial analysis of variance (ANOVA) was used to understand the interaction effect between the independent and dependent variables. The factor of locale included urban, suburban, and rural. The factor of school type included the EMO and non-EMO. The analysis used a 3 x 2 factor design to examine the effects of locale between EMO and non-EMO schools. Also, the analysis determined if the effect of locale on the M-STEP depends on the school type. The study sought to determine whether there is a significant interaction. The obtained value was set at .05 (i.e., p < .05) to determine the statistical significance of between-subjects’ effects.
Analysis and Findings
Research Question 1: Is there a significant difference between the EMO and non-EMO schools M-STEP scores in English language arts and math at grades 3-5?
After performing the t-test for M-STEP ELA and math grades three through five, it was found that there is no statistical significance and small practical significance for ELA and math assessments. Specifically, there was no statistically significant difference in mean scores between EMO and non-EMO schools EG3 through EG5, even though the scores for EMO were slightly higher. EG3 result is, M = 4.35, 95% CI [-4.16 to 12.86], t(94) = 1.016, p = .312 or P > .05, d = .21. EG4 result is, M = 7.6, 95% CI [-.87 to 16.07], t(94) = 1.783, p = .078 or P > .05, d = .36. Lastly, EG5 result is, M = 3.94, 95% CI [-.87 to 16.07], t(94) = 1.783, p = .078 or P > .05, d = 0.18.
There was no statistically significant difference in mean scores between EMO and non-EMO schools MG3 through MG5. The result for MG3 is, M = 4.94, 95% CI [-3.91 to 13.79], t(94) = 1.108, p = .271 or P > .05, d = 23. MG4 result is, M = 1.13, 95% CI [-8.48 to 10.73], t(94) = .233, p = .817 or P > .05, d = .05. MG5 result is, M = 4.44, 95% CI [-3.95 to 12.82], t(94) = 1.051, p = .296 or P > .05, d = .21.
Research Question 2: Is there a difference in M-STEP scores in grades three through five ELA based on the locale and school type (EMO and non-EMO)?
A two-way analysis of variance (ANOVA) was conducted to examine the effects of school type, and locale on M-STEP ELA grades three through five mean scores. The M-STEP scores for EG3 through EG5 showed no significance between EMO and non-EMO schools (type) and locale using the 3 x 2 factorial analysis. Based on the results, grades three through five show no significant difference, which indicates the EMO schools are no better or no worse than non-EMO schools.
Research Question 3: Is there a difference in M-STEP scores for grades three through five math scores based on the locale and school type (EMO and non-EMO)?
The ANOVA was conducted to examine the effects of school type and locale on M-STEP math grades three through five mean scores. The M-STEP scores for MG3 through MG5 showed no significance between EMO and non-EMO schools (type) and locale using the 2 x 3 factorial analysis. Based on the results, grades three through five show no significant difference, which indicated the EMO schools are no better or no worse than non-EMO schools.
First, the study yielded no significant difference between EMO and non-EMO schools. One school type (EMO and non-EMO) did not outperform the other in student performance. These findings are similar to Zachary (2014) who examined two different types of grade level configurations to determine if students performed differently and the degree of growth on reading and math tests (Zachary, 2014). No significant interaction could be found between school type and year using the 3 x 2 factorial analysis. While the outcome was not statistically significant, the study did discover a common trend between both school types of scores dropping at the sixth-grade level. This study shows math scores gradually decreased from grade three to grade five for both EMO and non-EMO schools.
However, urban locale individually had a significant effect on students' performance for both EMO and non-EMO schools. Of course, urban children are more than twice as likely to be living in poverty than those in suburban locales. Likewise, urban students are more likely than suburban or rural students to receive free and reduced lunch, which explains a common phenomenon in education (U.S. Department of Education, 2017). Suburban and rural locales had no statistical significance compared to one another, but urban compared to suburban and rural yielded a statistically significant result. The combined effect of type and locale has no statistical significance on overall student performance. This finding corroborates a previous study (Joseph, 2014) that reported no significant effect of locale. School proprietorship was a greater factor in students’ performance rather than locale and school type.
Urban locale yielded a significant statistical outcome, whereas as suburban and rural did not. The result showed a limitation in the study. Examining the population variables not limited to special education, English language learners, race, and economically disadvantaged would provide greater context to variables that influence student performance. While the sample between the EMO and non-EMO schools were equal, the populations of special education and English language learners are higher in non-EMO schools and low in the EMO schools after analysis of school-level data. This factor may affect the outcome scores on the M-STEP test.
Data collection was a challenge. The M-STEP test is difficult to use for determining growth because Michigan may alter the assessment again. Because of this issue, many authorizers suggest that their respective charter schools concentrate on the Northwest Evaluation Association (NWEA) data versus the M-STEP because it has been changed and will more than likely be updated. The contention of charters is there is no ability to establish growth. Education management organizations and non-EMO schools use the NWEA. The data between those two tests could provide another source of performance data. Unfortunately, the researcher could not access the data for this study.
Recommendations for Further Study
Further research can compare the types of schools, but examine the salient variables in a school that may influence student performance or predicts student performance based on multiple independent variables. Those variables are not limited to funding, special education percentages, English language learners, economically disadvantaged, staffing, truancy, etc. This study examined the type of school and locale. A deeper consideration of the population specifics can expose more influences or relationships within the EMO and non-EMO schools and inequities in each system.
Arsen, D. & Ni, Y. (2011). Is administration leaner in charter schools? Resource
Allocation in charter and traditional public schools. Education Policy Analysis Archives, [S.I.] 20, p. 3. Retrieved from doi: http://dx.doi.org/10.14507/epaa.v20n31.2012
Joseph, E.U. (2014). School variables and mathematics performance among students in
Akwa Ibom State. International Journal of Development and Sustainability, 3(7), pp. 1558-1568.
U.S. Department of Education, Office of Educational Research and Improvement.
National Center for Education Statistics (2017). The condition of education. Retrieved from https://nces.ed.gov/pubs2017/2017144.pdf
Zachary, M. (2014). A study of the relationship between public K-8 and middle schools
and student achievement in one South Carolina school district. Retrieved from