Home PublicationsJRTEPast IssuesVolume 33 Number 5

	Edited by Dr. David J. Ayersman, Mary Washington College, and Dr. W. Michael Reed, New York University
formerly Journal of Research on Computing in Education

Volume 33 Number 5 Summer 2001

The Effectiveness of Mathematics Software for Ohio Proficiency Test Preparation

Patricia Deubel
The Ohio State University–Mansfield

Abstract
This study, which took place during 1999–2000, examined the use and effectiveness of software to help students pass the mathematics portion of the Ohio Ninth Grade Proficiency Test. Grade 8 mathematics, special education, and proficiency intervention teachers (N =113) in 35 middle schools across 13 midsize districts in the Ohio Department of Education Urban Schools Initiative were surveyed. Results indicated that administrative support, teachers’ instructional style, teachers’ perceived priority of learning about computers and software, computer availability and access, technical assistance, and software quality were significant factors affecting teachers’ decisions to use technology in mathematics instruction.

With the exception of some students with disabilities, students in Ohio must pass all five parts of the Ohio Ninth Grade Proficiency Test (ONGPT) to receive a high school diploma. (Since its first administration in the fall of 1990, the test has included writing, reading, mathematics, and citizenship. A requirement to pass science was added to the test and implemented with the freshman class of 1997.) The mathematics portion of the test has proven to be the most difficult for students to pass. Although the ONGPT measures achievement of basic skills that should have been acquired during Grades K–8, for the most part, students have not mastered tested mathematics objectives to the degree needed to pass. In an effort to correct this problem, the National Council of Teachers of Mathematics (NCTM, 1989) and the Ohio Schools Technology Implementation Task Force (1999) recommended greater use of technology products in mathematics classrooms.

Relevance and Significance

This study, which received support from the Ohio Department of Education Urban Schools Initiative, was conducted during 1999–2000 and addressed the use and effectiveness of software to help students pass the mathematics portion of the ONGPT. Two questions were posed:

1. What effect has technology in the form of software made on the preparedness of students for this standardized test?
2. Is the software that has been specifically developed to address the mathematics objectives for the ninth-grade test effective?

Ohio’s Senate Bill 55, the academic accountability package passed by the legislature in 1997, requires that proficiency tests measure student knowledge of core academic areas through a 10th-grade level beginning in 2003. The first class to be affected by this law is the Class of 2005 (Ohio Department of Education [ODE], 1998), the 1999–2000 seventh graders. The new mathematics exam increases proficiency requirements by placing a greater emphasis on mastering algebraic and geometric concepts.

This study is not the first to investigate the effects of using technology/software on academic achievement. Christmann, Badgett, and Lucking’s (1997) meta-analysis examined effectiveness of microcomputer-based software on academic achievement of students in Grades 6–12. Twenty-six studies conducted between 1984 and 1995, involving 3,694 students from all educational settings and subject areas, were included in their research. They found a small but positive overall mean effect size (0.187) that indicated, on average, students exposed to computer-assisted instruction showed higher achievement than 57.2% of students exposed to traditional instruction.

After an extensive survey of single studies in achievement at elementary and secondary levels, meta-analyses, and reviews that criticized both kinds of research, Kirkpatrick and Cuban (1998) concluded there is sufficient research to support any view about advantages and disadvantages expressed by policymakers, practitioners, and parents. Middleton and Murray (1999) examined teachers’ perceptions of their levels of technology implementation and the impact of technology use on fourth- and fifth-grade achievement in mathematics and reading. Their findings revealed that teachers’ level of technology use had a significant effect on mathematics and reading achievement of fifth-grade students but not on fourth-grade students in both areas.

Wenglinsky’s (1998) national study included data drawn from samples of 6,227 fourth graders and 7,146 eighth graders and the standardized 1996 National Assessment of Educational Progress (NAEP) in mathematics. He found technology can be effective in increasing mathematics achievement and other educational outcomes, but its effectiveness depends on how it is used. Findings revealed how computers were used with eighth graders.

• Teachers’ professional development in technology and computer use to teach higher-order thinking skills were positively related to their students’ academic achievement in mathematics, as was the frequency of the students’ home computer use
• Computer use to teach lower-order thinking skills was negatively related to academic achievement, as was the frequency of school computer use. The extremely high level of use might have been associated with students using computers in unproductive ways.
• Professional development and using computers for higher-order thinking skills were each associated with a substantial increase of more than a one-third of a grade level in academic achievement.

Wenglinsky (1998) suggested it is important for states to collect more data to know which policies and types of software are effective and for which subject areas computers should be used. The primary focus of technology initiatives should be on middle schools rather than on elementary schools because the typical mathematics curriculum begins to focus on higher-order concepts at this level rather than on the computational skills typical of mathematics in elementary schools. According to Hiebert (1999), few states regularly collect information on what is happening inside classrooms. This absence of data collection is unfortunate because without such information it is hard to prevent mistakes from being made.

Methodology

A 50-question survey was developed to examine the effect of software use on preparedness of students for the ONGPT. Gay (1996), Frary (1996), Krosnick (1999), and Roden (1998) provided guidelines for designing questions, increasing response rate, and ensuring reliability and validity of questions and responses.

The survey addressed teacher beliefs, technology use, and the instructional and technical merits of the software used. Organizational factors addressed included administrative support, time, staff development and training, computer availability and access, software quality, funding, and technical assistance, all of which had been identified in the literature (e.g., Chiero, 1997; Dexter, Anderson, & Becker, 1999; Dusick, 1998; Hope, 1996, 1997a, 1997b; Jaber & Moore, 1999). Individual factors included teaching experience, teaching style, source of computer learning, perceived relevance of computers (Chiero; Hope, 1996, 1997a, 1997b), and anxiety (Dusick).

Projected outcomes of the study were anticipated from findings in Berg, Benz, and Lasley (1998), Chiero (1997), Education Week’s (1999) National Survey of Teachers’ Use of Digital Content (Fatemi, 1999), Ginsberg and McCormick (1998), Jaber and Moore (1999), and Mergendoller (1997), for example.

Analysis

The participants were 128 eighth-grade math teachers, special education teachers of non-exempt eighth-grade students, and proficiency intervention teachers in 13 midsize districts in the ODE Urban Schools Initiative. Administrators identified teachers who would be preparing Grade 8 students for the March 2000 math test. One hundred thirteen surveys were received (88%). Respondents included 74 math teachers (return rate 90%), 35 special areas teachers (return rate 83%), and 4 proficiency intervention teachers (return rate 100%).

Percentages were calculated for each response to 50 survey questions. Chi-square tests at a .01 probability level determined significant differences in beliefs between software users (52%) and nonusers (48%) and between teachers from those districts (N = 7) in which more than 50% of teachers used software and those (N = 6) in which 50% or fewer of teachers used software.

Chi-square tests determined significance on passing the test for software use during math class time and on passing the test when students received extra proficiency intervention using software. District and/or building administrators reported building-level test results. Teachers reported results for those who used software during class time and had extra proficiency intervention using software. Judgments regarding some data were made because teachers had indicated on the survey if software had been used.

Summary of Findings

The results showed that software users and nonusers held significantly different beliefs regarding the following:

• district and administrative support of computers and software use in instruction,
• computer access,
• availability of high-quality math software that was relevant for instructional needs,
• availability of technical assistance when needed, and perceptions about the priority of learning about computers and software.

In all those categories, software users had higher percentages of agreement than nonusers. The only significant difference in beliefs among teachers from those districts in which more than 50% of teachers used software and those in which 50% or fewer used software was in regard to availability of high-quality software that was relevant for instructional needs (Table 1). Other findings support much, but not all of the previous literature.

Table 1. Teacher Beliefs—Percentages of Those Who Strongly Agreed and Agreed with the Given Statements
	Statement	All (100%, N = 113)	Users (52%, N = 59)	Nonusers (48%, N = 54)	Teachers in the 7 Districts with > 50% Users (N = 51)	Teachers in the 6 Districts with 50% Users (N = 62)
*1.	District-level administration is supportive of computer and software use in instruction.	88%	97%	80%	96%	82%
*2.	Building-level administration is supportive of computer and software use in instruction.	89%	97%	81%	96%	84%
3.	Software use will make a difference in achievement of my students on the math test of the ONGPT.	50%	61%	39%	49%	52%
*4.	Using computers and software agrees with my instructional style.	58%	76%	37%	67%	50%
*5.	Computers are easily accessible when I want to use them.	54%	69%	37%	57%	52%
6.	Available computers are well maintained.	69%	76%	61%	76%	63%
**7.	High-quality math software is available and relevant for my instructional needs.	39%	58%	19%	51%	29%
*8.	Technical assistance is available when I need it.	60%	76%	43%	69%	53%
9.	Staff development and training is necessary for me to learn to use technology in the classroom.	64%	71%	56%	69%	60%
10.	What I learned from staff development and training has met my needs to use technology in the classroom.	40%	49%	30%	39%	40%
11.	What I know about computers and using software, I primarily learned on my own.	58%	54%	63%	59%	58%
*12.	Learning about computers and using software is a priority.	54%	71%	35%	65%	45%
13.	I have adequate time to learn about computers and new software.	19%	19%	19%	22%	16%
14.	I have anxiety when I use computers and software.	12%	10%	13%	12%	11%
Note. Chi-square tests were made at the probability level of .01. * An asterisk implies a significant difference in beliefs between software users and nonusers. ** Statement 7 shows the only significant difference in beliefs between the two groups of districts.

Issue: Administrative Support

Teachers believed that district- (88%) and building-level (89%) administration was supportive of computer and software use in instruction, but among those who used software this perception was significantly higher (97%) compared to those who did not use software (80%). Administration played a greater role in software selection (44% vs. 22%) in districts where more than half the teachers used software compared to other districts; teachers in those districts had a greater perception of their influence in software selection (60% vs. 38%). The combination led to their significant overall satisfaction in the quality and relevance of available software (51% vs. 29%).

Most software was paid for from district and school/department budgets (41%). Only 3% of teachers indicated purchase of software from their own funds, which was in sharp contrast to the 20% of teachers who reported doing so in Education Week’s (1999) national survey (Fatemi, 1999). Eight of 13 districts, however, sought funding from outside sources, such as grants, parent associations, and Ohio School Net.

Issue: Perceived Relevance of Computers and Software

Teachers were not convinced that software use would make a difference in student achievement, as 50% strongly agreed or agreed with the statement and 42% remained neutral. The difference among software users and nonusers was not significant. This belief affected their decisions to use software. Only about half did so (52%). Consequently, among those who used software, there was a significant difference and greater agreement (71%) that learning about computers and using software was a priority compared to those who did not (35%) use software.

Software use during math class time was not an integral part of curriculum in most district settings. Computer use ranked among the top test-improvement strategies among software users, but among all teachers, the top four test-improvement strategies included small-group tutoring, whole-class instruction, individual tutoring, and communicating high expectations (Tables 2 & 3). Communicating high expectations ranks high on the pedagogical list for narrowing the achievement gap between urban students and all other students (Williams, 1996). Even among software users (52%), the majority used it only occasionally all year (45%), which meant at most a few times a month (50%), with 12% of those teachers adding regular use just before the test. Most students worked individually at the computer (73%) from 30–45 minutes per session (57%).

Table 2. Top Test-Improvement Strategies Among All Teachers (N = 113)
Strategy	Selected in Top Four		Most Frequent Rank
Small-group tutoring	81%		2		45%
Whole-class instruction	75%		3		31%
Individual tutoring	71%		1		41%
Communicating high expectations	56%		4		43%
Using computer software	50%		3		53%

Table 3. Top Four Test-Improvement Strategies among Software Users and Nonusers
	Strategy Selected		Most Frequent Rank
Users (N = 59)
Using computer software	80%		3		60%
Small-group tutoring	76%		2		56%
Individual tutoring	75%		1		48%
Whole-class instruction	71%		4		26%
Nonusers (N = 54)
Small-group tutoring	87%		3		30%
Whole-class instruction	80%		1		37%
Individual tutoring	67%		2		67%
Communicating high expectations	52%		4		39%

Nearly all teachers (94%) who did not use software during 1999–2000 commented on their decision not to do so. Fifty-two responses indicated that barriers to using technology (including lack of access to available computers in labs, no computers or not enough computers in classrooms, old or outdated computers, and poor or nonexistent software) were the primary reasons for not using software. Special education teachers commented that available software was not suitable for their students’ ability levels. Other reasons included lack of training and knowledge about computers and their potential to aid learning, lack of funding, and lack of time. Technology problems related to licensing of software or wiring problems were mentioned. Some did not believe in the value of computers as a tool for learning, while others, particularly those new to teaching, wanted to use computers and would be willing to try if given appropriate training and access. In one district, software was available for use with sixth and seventh graders but not eighth graders.

Issue: Computer Availability and Access

Only 54% of teachers indicated that computers were easily accessible when they wanted to use them; beliefs between software users and nonusers were significantly different (69% vs. 37%). Computers were available in classrooms and labs, but most teachers (38%) indicated having one or two computers available for use during scheduled class time, with at most two computers with Internet access (61%).

More computers seemed to be available for class time use in districts where more than half the teachers used software; percentages were more than double, indicating more than 11 computers available for class time use (41% vs. 18%).

There was an increase in the percentages of teachers who used software for each school year since 1994. There were, however, noticeable differences in the trends between the seven districts (group 1) where software was used by more than 50% of teachers and the six districts (group 2) where software was used by 50% or fewer teachers (Table 4). Software use sharply increased from 2% in 1995–1996 to 15% in 1996–1997 among group 2 teachers. Since the 1997–1998 school year, however, the growth rate in software use among group 1 teachers was noticeably higher in comparison (Figure 1). Data must be interpreted with caution, however, because teaching staff and teaching assignments vary from year to year. Greater or lesser actual use of software may have been present in these districts. Group 2 districts appeared to make greater use of proficiency intervention classes using software that met in addition to regular math classes, which might be one explanation for their lower class time use of software during 1999–2000. For March 2000 test results, 7 of 11 schools in which teachers reported students using software in an extra proficiency intervention class were in group 2.

Table 4. Software Use 1994–2000 for Districts Divided into Two Groups
	Replies		No Answer		94–95		95–96		96–97		97–98		98–99		99–00		Never Used
All	113		2%		3%		4%		12%		20%		35%		52%		40%
> 50% users (group 1)	51		0%		4%		6%		10%		18%		41%		76%		22%
50% users (group 2)	62		3%		2%		2%		15%		23%		31%		34%		55%

Figure 1. Software use for districts, divided into two groups by level of teacher use of software.

Issue: Technical Assistance

Most teachers indicated that technical assistance was available when needed (60%), but beliefs among software users were significantly different from nonusers (76% vs. 43%). Software users were about equally divided in who fixes technical problems: themselves (30%), a knowledgeable colleague (25%), or an on-site technology coordinator or technician (30%). No group 1 teacher indicated the need to consult a software company’s technical support team, compared to 19% of group 2 software users.

Issue: Instructional Style

Instructional style is a personal issue. Using computers and software agreed with teachers’ instructional styles among those who used technology (76%). Among nonusers, only 37% strongly agreed or agreed that computer and software use aligned with their instructional styles. This difference was significant.

Issue: Staff Development and Training

Staff development and training opportunities did not meet the needs of teachers to use technology in classrooms. Regardless of how comparisons were made, there was no significant difference in beliefs between groups. Only 49% of users and 30% of nonusers strongly agreed or agreed that their needs were met. Likewise, the percentage of agreement among teachers in the top software-using districts (39%) was comparable to that in the lower software-using districts (40%). Yet among all teachers, 64% believed staff development and training was necessary to learn to use technology in classrooms, particularly among those who used it (71% users vs. 56% nonusers).

Only 34% of teachers had more than five hours of staff development during the last 12 months, integrating computers and software into curriculum. An exceedingly high percentage (82%) of teachers who did not use software indicated that they had five or fewer hours of staff development during the past year, compared to approximately half (48%) of teachers who did use software.

Issue: Teaching Experience

Teaching experience was not significantly related to software use, a finding that agreed with Fatemi (1999). The highest percentage of software use was among teachers with more than 20 years of experience (64%), but the percentage was comparable to that for teachers with less than 1 year of experience (60%). Only 36% of teachers with 1 to 5 years experience used software, which might become a future concern for districts if staff development continues not to meet teachers’ needs to use technology in classrooms.

Issue: Source of Computer Learning

What teachers know about computers and using software had been learned primarily on their own (58%), with no significant difference in agreement between the two groups of districts (59% vs. 58%), nor between users and nonusers (54% vs. 63%).

Issue: Time

Neither software users nor nonusers had adequate time to learn about computers and new software (19% vs. 19%), but time was not a factor that distinguished software users from nonusers. The difference in beliefs between groups was not significant.

Issue: Anxiety

Previous studies found that computer anxiety may lead to lack of technology use in classrooms (Bradley & Russell, 1997; Dusick, 1998; Rosen & Weil, 1995), but anxiety was not a factor in this study. Only 12% of all teachers expressed anxiety when using computers and software. The difference in anxiety between software users and nonusers was not significant (10% vs. 13%).

Issue: Software Used

Math Blaster (1997), Passing the Ohio Ninth Grade Proficiency Test (1997), and Optimum (1997) dominated the list of more than 50 commercial software titles and Internet resources used. Respectively, 43%, 37%, and 25% of teachers used that software. Most software was used for remediation and review (72%) and drill and practice (72%). Copyright dates among additional titles used dates as far back as 1991; some were on 3.5" floppy disks; software developers had discontinued some.

Issue: Software Quality

Ten of the top 12 software packages teachers used correlate with NCTM standards, other state and national standards, or Ohio’s proficiency tests (Table 5). The most frequently used software, however, was not among those. On a scale from 1 to 5 (excellent to fails), teachers (50%) rated overall product quality as good (average 2.3) for preparing students for the math test of the ONGPT. Software’s ability to foster higher level thinking skills received the lowest average rating (2.5).

Table 5. Top 12 Software Titles Used During Class Time
	All Districts (13, N = 60)				Districts in which > 50% of the Teachers Use Software (7, N = 39)				Districts in which 50% of the Teachers Use Software (6, N = 21)
Title	Used		Best		Used		Best		Used		Best
Math Blaster (1997)	43%		3%		28%		0%		71%		10%
Passing the Ohio Ninth Grade Proficiency Test (1997)	37%		8%		33%		10%		43%		5%
Optimum (1997)	25%		5%		23%		5%		29%		5%
Math Skills and Concepts SuccessMaker (1999)	18%		5%		10%		5%		33%		5%
Hot Dog Stand (1996)	13%		0%		15%		0%		10%		0%
Math for the Real World (1996)	13%		2%		8%		3%		24%		0%
GoFigure (1997)	12%		5%		10%		8%		14%		0%
Math Munchers Deluxe (1997)	12%		0%		13%		0%		10%		0%
OPT for Success (1998)	12%		3%		13%		5%		10%		0%
Desktop Tutor (1996)	10%		2%		10%		0%		10%		5%
Mystery Math Island (1995)	10%		0%		13%		0%		5%		0%
SkillsBank (1998)	10%		0%		8%		0%		14%		0%
Note. All products at the time of the study, except Math Blaster and Hot Dog Stand, had been correlated to either national or state standards. The columns headed as Best indicate the percentages of teachers who chose the product as best for addressing the 16 learning objectives on the math test of the ONGPT.

Read more...

Copyright © 2001, ISTE (International Society for Technology in Education). All rights reserved.