Home PublicationsJRTEPast IssuesVolume 33 Number 5

Edited by Dr. David J. Ayersman, Mary Washington College, and Dr. W. Michael Reed, New York University

Volume 33 Number 5 Summer 2001

The Effectiveness of Mathematics Software for Ohio Proficiency Test Preparation, Part II

Patricia Deubel
The Ohio State University–Mansfield

Discussion

What effect has technology in the form of software made on the preparedness of students for this standardized test?

Software use during math class time appears to have increased from 1994 to 2000 among all 13 participating school districts. Teacher-reported use showed a rise from 3% in 1994–1995 to 52% in 1999–2000. Software use during class time, however, was not an integral part of the mathematics curriculum in most district settings in this study. Forty percent of participants had never used software in instruction, which does not necessarily mean teachers were resistant, incompetent, lacked expertise, or were technophobes, according to Cuban (O’Neil, 2000). Computer use did not rank among the top four-test improvement strategies that all teachers used. Even among the 52% of teachers who used software during 1999–2000, most only used it occasionally all year (43%), which meant at most a few times a month (50%).

Analysis of test results revealed that occasional use of software during class time appeared to have a detrimental effect on students’ performance on the math test. A significantly greater number of students who had not used software during class time passed compared to those who did use software during class time (Table 6). Software use, however, made a significant difference on passing for students who did not use software during class time, but had used software in proficiency intervention classes that met in addition to regular math classes (Table 7). The implication was that regular and focused effort on using software to combat weaknesses is required for skill fluency that leads to achievement gains.

Table 6. Test Results for Class Time Software Use
	Actual								Expected
	Passed		Did Not		Total		% Passed in Group		To Pass		Not Pass
Software	1,129		1,706		2,835		40	%	1,264		1,571
No software	1,991		2,172		4,163		48	%	1,856		2,307
Total			3,120		3,878		6,998		45	%
Chi-Square:	43.714		Significant at .01 level df = 1
Probability:	3.8E-11
Note. Data for one school (N = 294 Grade 8) could not be included because results were reported for Grades 8 and 9 combined. The 45% who passed the test included 16% who used software and 29% who did not use software.

Table 7. Test Results for Class Time Software Use and Extra Proficiency Intervention Using Software
	Actual								Expected
Software Use	Passed		Did Not		Total		% Passed in Group		To Pass		Not Pass
Class use, no extra	682		962		1,644		41	%	761		883
Class use, with extraa	27		84		111		24	%	51		60
No class use, no extra	1,150		1,428		2,578		45	%	1,193		1,385
No class use, with extra	753		559		1,312		57	%	607		705
Total	2,612		3,033		5,645		46	%
Chi-Square:	104.808		Significant at .01 level df = 3
Probability:	1.4E-22
Note. This table includes data from 27 of 35 participating schools, which provided test results with a breakdown for students who had extra proficiency intervention using software. a This category includes results from three teachers in two schools. In school 1, 9 of 18 passed (50%). In school 2, 18 of 93 passed (19%). Caution is needed in generalizing the category result because the sample size is too small in comparison to other groups.

Analyses of beliefs and percentage differences between software users and nonusers revealed factors relating to teachers’ decisions to use technology. These factors included administrative support, teacher’s instructional style, teacher’s perceived priority of learning about computers and software, computer availability and access, availability of technical assistance when needed, and software quality.

Source of computer learning, whether by staff development or learning on one’s own, and lack of time to learn about computers and software were issues of concern in meeting the needs to use technology, but decisions not to use software would not be based on those factors. Comparisons revealed no significant differences between source of learning and time to do so among software users and nonusers nor between districts where more than half the teachers used software and those in which half or fewer used software. Anxiety was not an issue of concern and, likewise, was not a factor relating to technology-use decisions. Regardless of how comparisons were made, no significant differences were found. Percentages for those who believed they had anxiety when using computers and software were very low and comparable.

Is the software that has been specifically developed to address the mathematics objectives for the ninth-grade test of value for learning?

Because software use significantly affected performance on the test for students who had not used software during class time but had used software in extra proficiency intervention classes, it did have value for learning. Ten of the top 12 software packages teachers used correlate with NCTM, national, and state standards or with the mathematics learning objectives of the ONGPT. Most software was used for remediation and drill and practice but did not necessarily have tutorial features or teacher management systems.

Teachers rated overall software quality as good for preparing students for the ONGPT math test. It was not surprising, however, in light of previous research by Gurney (1996), that teachers observed differences in software quality and in some cases discontinued use of certain software because of its quality. Gurney’s findings on Grade 9 mathematics taught in a computer lab revealed that, when approximately 60% of the course material had been completed, the remaining topics for the course could not effectively be taught using computers because available software did not match the curriculum at that point. The software appeared suitable only for review of previously learned material. Classes were moved back into regular classrooms and taught more traditionally.

This study addressed only software use to prepare students for the ONGPT in mathematics, which is just one of several approaches to proficiency intervention. Mann, Shakeshaft, Becker, and Kottkamp (1999) indicated that 70% of factors relating to achievement are out of schools’ control. Such factors include students’ mental ability, motivation, and attitude; influence of family, home, neighborhood background, and peer groups; and low social capital. Gibson (1997) noted that attendance is the number one useful factor in predicting performance on the math and reading portions of the ODE test, with 15 or more days absence being the key figure. No strategy can positively affect achievement if students are not in school.

The exact percentage of Grade 8 math teachers who had used software in the years 1995–1999 to prepare students for the ONGPT and their perceptions of available software during those years could not be determined from this study. Faculties change from year to year, as did many teaching assignments.

Data from the current study could be used only to measure the effect of software use on preparedness of students for the March 2000 test. Results relied on accurate reporting of test results from buildings and teachers who responded to the survey. It required their efforts twice—once for the survey and once for the follow-up test data report.

The study did not seek to discover instructional and technical merit of individual software packages, although some determination of value was possible for several packages, based on additional comments teachers made about software they used. Fifteen teachers had each used only one piece of software, with a total of six different packages. An assumption was made that survey responses reflected their views of those packages. Moersch (1999) asked how researchers could quantify teachers’ using technology in classrooms and the general academic achievement that results from their instructional technology practices. It is impractical to try to determine the effectiveness on learners of every conceivable application.

Educators can rely on data from recent studies (e.g., Mann et al., 1999; Middleton & Murray, 1999; Wenglinsky, 1998), which found strong links among technology use, academic achievement, staff development, and classroom instructional practices. This study contributed to that body of knowledge. Results regarding the perceptions of the value of mathematics software may generalize to all other districts in Ohio and districts elsewhere that require students to demonstrate mathematics proficiency to receive a high school diploma.

This study helped define one condition under which software use contributed to a significant difference in achievement of students on the mathematics test of the ONGPT. Occasional use of software during class time actually had a detrimental effect on achievement. What is needed for achievement gains is personalized, regular, and focused effort on using software over time to combat individual weaknesses and to build skill fluency, such as found in an extra proficiency intervention class using software. The ideal situation would be for all mathematics classrooms to have regular access to computers because not all schools in this study had the luxury of offering extra proficiency intervention classes using software during the regular instructional day.

Teachers should be aware of software differences before purchases are made. Crandell, assistant director of the ODE’s assessment center, has said she is unaware of Ohio sponsoring a product or recommending products to school districts for proficiency test preparation. School districts are told: Let the buyer beware (Gillespie, 1999). Ohio is clearly different from West Virginia, a state that endorsed two vendors’ products for use within their basic skills/computer education (BS/CE) program, which has demonstrated that technology use leads to achievement gains on a standardized test. The choice of software from a fixed set of vendors departs from the convention of school software selection from among hundreds of vendors. According to Mann et al. (1999), however, part of the explanation for BS/CE’s success was the defined focus of its implementation.

Although policy and political choices always honor local values, Mann et al. (1999) indicated a hope that West Virginia’s BS/CE program would advance consideration of similar initiatives elsewhere. That concern was also consistent with one result from Education Week’s (1999) national survey. Only 12% of teachers reported that their state or district provided lists of software titles that match curriculum standards. Experts agree that the pressure to satisfy curriculum requirements, particularly in states with specific academic standards and high-stakes tests, adds to the difficulty of finding appropriate digital content. Unfortunately, many teachers do not know where to turn to find out which digital content is aligned with their curricula (Fatemi, 1999).

This study contributed to solving that problem. Results yielded a list of more than 50 titles that districts used to prepare students for the math test of the ONGPT. Teacher perceptions of the instructional and technical merit of software they used contributed to a set of guidelines for selecting valuable software. Recommendations for school districts to consider for their own uses were needed, as preliminary discussions with central office administrators, building principals, and teachers in the study indicated. Many said they did not know what software is available and had little time to investigate. The list generated, along with product descriptions and correlations to standards, may prove valuable to districts for the software selection process. Software developers and contact information were included.

Results from teacher beliefs about organizational and individual factors affecting software use might also be valuable to districts that need to form action plans on how technology dollars should be spent in ways that lead to achievement gains. Key words were access, attitude, and training (Hope, 1997b; Mann et al., 1999). Actions plans call for expenditures for high-quality software specific to the purposes of basic skills achievement, computers, teacher staff development and training, and release time for teachers to build positive attitudes toward computers and to spend time using software.

As in Education Week’s (1997) national survey (Fatemi, 1999), results add needed data to educators’ and policymakers’ understanding of issues that affect teachers’ use and beliefs about digital content, which Fatemi said have been dominated by anecdotes and observations. Although only approximately half (52%) of teachers used software during the 1999-2000 school year, the increasing number of software users for mathematics classroom instruction since 1994 lends support to the realization of Ohio Schools Technology Implementation Task Force (1999) goals. The goals include that teachers use electronic products in their classrooms to improve student learning; students have access to electronic products and resources that will help them meet high standards of achievement. The challenge remains, however, for mathematics teachers to use technology during class time to the point where it will make a difference in achievement of their students.

Current Practice

Teachers clearly pointed out need for drill-and-practice software for students who were failing the math test because they lacked such skills. Using drill-and-practice software is important at the K–8 level. As Hirsch (1999) stressed, expertness in a skill depends on automation, through a great deal of practice, of the repeated, formal elements of the skill to free the conscious mind for critical thought. Once basic underlying skills have been automated, the next step leading to development of higher-order thinking skills is acquisition of a broad, well-integrated base of background knowledge relevant to the subject. The best way to develop a skill or a domain of knowledge in students is to focus on that skill or domain and to monitor whether the skill or knowledge has been gained. Unfortunately, as Hirsch also noted, drill and practice has a disparaging connotation as a tool to teach skills and runs contrary to the progressive movement of discovery learning and the project method. The method should not be slighted as low level, however, because it is just as essential to complex intellectual performance as drill and practice are to virtuoso violinists or athletes on the playing field.

Future Studies

Future studies might investigate teachers’ perceptions of software use to prepare students for the state science, writing, citizenship, and reading proficiency tests. Technology studies, including a longitudinal study, with students in Grades 9–12 are also recommended because beginning in 2003 Ohio’s proficiency tests shall measure student knowledge of core academic areas through a 10th-grade level. Tracking results over time of percentages of students who pass the test early when teachers use software in instruction would lend additional support for technology expenditures for that purpose.

Changes in Academic, Professional, and Organizational Practice

To resolve the issue of computer access, school districts might consider wireless solutions. According to Greaves (2000), student achievement cannot be built on the old-fashioned model of five computers in the back of the classroom or visits to the computer lab once a week. This means continuous, personal technology access for every student, regardless of socioeconomic status. Until this happens, education will view technology use as an event rather than something fully integrated into curriculum. As one teacher in this study stated: It’s a pleasant break for kids. For each student to have a computer, the computer must be portable, durable, and wireless.

Staff development and training opportunities should play a greater focus on meeting the needs of teachers to learn to use technology in classrooms. To help form action plans, districts might begin with a needs analysis of the state of knowledge that teachers have about computers and software/Internet instructional resources in their area of expertise and how to use them in instruction. This might be essential in light of the fact that only approximately one third of teachers in this study with one to five years of teaching experience used software during class time. As Gullickson (2000) pointed out, the appropriate training method can be determined depending on the needs of the individual. A just-in-time approach is usually preferable and may be the preferred way to support an ever-changing and sometimes unpredictable environment.

Discussions with administrators revealed a lack of consistency between middle school buildings even within the same district in terms of available technology and instructional programs, which might be remedied. Hirsch (1999) points out the need for uniform grade-by-grade standards and a common core curriculum at each grade level, mastery of which should be monitored yearly. A common core curriculum appears to be the only practical means for achieving universal readiness at each grade level and is essential in a highly mobile society, such as found in the United States.

Concluding Remarks

Soloway (1998) noted that technology and national standards testing are the two big ideas energizing public education today. Schools are not buying software in many cases because they are still building a computer infrastructure. As this study demonstrated, the high price of software is often associated with curriculum content that may be useful for only a few days out of the total school year. State standards, state tests, curriculum, and textbooks have a firm lock on the way schools operate. Teacher evaluations often rest on how students perform on standardized tests. Testing results are often used to rank schools and allocate state, local, and federal funds. The bottom line is that schools want curriculum, not software.

As this study revealed, even when software is tied to curriculum, some schools do not purchase software that is available to accompany their textbooks. Money is an issue for many urban schools caught in the battle between the haves and have-nots. In describing their school district, one middle school principal stated wealthy districts can afford technology and many urban districts qualify for a lot of federal aid. Their district needed computers, but fell in the middle and, like so many others, must purchase their own technology. Although district and school budgets paid for the majority of software, teachers in 8 of 13 districts indicated some use of outside funding sources.

Moursund (1999) addressed the question that people have posed on why education’s large investment on information technology (IT) has not produced significant improvement in education. Annual expenditures on information technology in K–12 schools are now approximately 2% of the entire school budget. Since the 1960s, the amount of money U.S. businesses have spent from their IT equipment funds has risen from an average of 3% to the current 45%. Data do not provide clear-cut evidence, however, that IT as a tool has significantly increased productivity in non-IT businesses.

Integration of IT into education would mean that IT was thoroughly integrated into curriculum, instruction, and assessment, which this study revealed is not yet the case in mathematics instruction for participating districts. It would mean that students and teachers had routine access to these facilities, which also was not the case, and that a technical support staff was in place to provide high quality and timely support. As this study revealed, perceptions of availability of technical support when needed also significantly differed among software users and nonusers. If business is still questioning the effectiveness of their investment, Moursund (1999) says, it is not surprising that in education, where far less money has been spent on IT, solid evidence is lacking that this aspect of IT improves productivity of students and educators.

As Cuban (as cited in O’Neil, 2000) reflected on difficulties of sustaining change, the structure of school (e.g., grades organized by age and departments) works against a lot of the changes that have to be made for technology to be used in more imaginative and creative ways. Technology has flaws; schools cannot keep up with purchases of the latest software to run on older computers. Teachers cannot be expected to have contingency lesson plans for technology that fails when they need it. Therefore, they continue to use textbooks, overhead projectors, and chalk because they are reliable and flexible, as this study demonstrated.

Contributor

Patricia Deubel earned a PhD in computing technology in education from Nova Southeastern University in Florida. She has 28 years of experience in mathematics and computer education teaching, teacher training, staff development, and curriculum development, and she has presented computer workshops at the state and local levels. Her interests lie in K–12 educational consulting with mathematics/computer education emphasis; courseware development for learning with computers, including online learning environments; and educational research. Dr. Deubel currently teaches mathematics at The Ohio State University in Mansfield and is an adjunct professor in the doctoral program for the School of Computer and Information Sciences at Nova Southeastern University. Recently, her writing has appeared in the Journal of Instruction Delivery Systems and HyperNexus: Journal of Hypermedia and Multimedia Studies.

Contact:

Dr. Patricia Deubel
1310 Cedarlawn Ct.
Mansfield, OH 44906
pdeubel@aol.com or deubelp@nova.edu

References

Berg, S., Benz, C. R., & Lasley, T. L. (1998). Exemplary technology use in elementary classrooms. Journal of Research on Computing in Education, 31(2), 111–122.

Bradley, G., & Russell, G. (1997). Computer experience, school support, and computer anxieties. Educational Psychology, 17(3), 267–284.

Chiero, R. T. (1997). Teachers’ perspectives on factors that affect computer use. Journal of Research on Computing in Education, 30(2), 133–145.

Christmann, E., Badgett, J., & Lucking, R. (1997). Progressive comparison of the effects of computer-assisted instruction on the academic achievement of secondary students. Journal of Research on Computing in Education, 29(4), 325–337.

Desktop Tutor (Version 1.0) [Computer software]. (1996). Clayton, OH: Hollandays Publishing Corp.

Dexter, S. L., Anderson, R. E., & Becker, H. J. (1999). Teachers’ views of computers as catalysts for changes in their teaching practice. Journal of Research in Computing in Education, 31(3), 221–239.

Dusick, D. M. (1998). What social cognitive factors influence faculty members’ use of computers for teaching? A literature review. Journal of Research on Computing in Education, 31(2), 123–137.

Education Week. (1999). National survey of software/CD-ROM and Internet usage. Available: www.edweek.org/sreports/tc99/articles/survey.htm.

Fatemi, E. (1999, September 23). Building the digital curriculum: Summary. Education Week. Available: www.edweek.org/sreports/tc99/articles/summary.htm.

Frary, R. B. (1996). Hints for designing effective questionnaires. ERIC/AE Digest. (ERIC No. ED 410233). Available: http://ericae.net/edo/ED410233.htm

Gay, L. (1996). Educational research: Competencies for analysis and application (5th ed.). New York: Merrill.

Gibson, S. (1997, September). A comparative study of previous achievement indices for two groups of ninth-grade students: Those who passed and those who failed sections of the Ohio ninth grade proficiency test. American Secondary Education, 26, 2–6.

Gillespie, C. (1999, September 24). Firm offers a trial Ohio proficiency test on Net. The Plain Dealer, p. 5B.

Ginsberg, R., & McCormick, V. (1998, Winter). Computer use in effective schools. Journal of Staff Development, 19, 22–25.

Go Figure [Video and computer software]. (1997). (Available from WVIZ Channel 25 and Ohio University Public TV, 4300 Brookpark Road, Cleveland, OH 44134; www.wviz.org/itv/math/gofigure/index.htm.)

Greaves, T. (2000). One-to-one computing tools for life. T.H.E. Journal, 27(10), 54–56.

Gullickson, P. (2000). The promise and challenge of a connected world. T.H.E. Journal, 27(8), 50–54.

Gurney, P. (1996). Grade 9 mathematics in the computer lab: An introductory study. McGill Journal of Education, 31(1), 83–101.

Hiebert, J. (1999). Relationships between research and the NCTM standards. Journal of Research in Mathematics Education, 30(1), 3–19.

Hirsch, E. D., Jr. (1999). The schools we need and why we don’t have them. New York: Doubleday.

Hope, W. C. (1996, November/December). Factors facilitating teachers’ use of computer technology. The Clearing House, 70, 106–107.

Hope, W. C. (1997a, March/April). Teachers, computer technology, and the change process. The Clearing House, 70, 191–193.

Hope, W. C. (1997b, June). Why technology has not realized its potential in schools: A perspective. American Secondary Education, 25, 2–29.

Hot Dog Stand: The Works [Computer software]. (1996) Pleasantville, NY: Sunburst.

Jaber, W. E., & Moore, D. M. (1999). A survey of factors which influence teachers’ use of computer-based technology. International Journal of Instructional Media, 26(3), 253–266.

Kirkpatrick, H., & Cuban, L. (1998, Summer). Computers make kids smarter—Right? Technos Quarterly, 7(2) 26–31. Available: www.technos.net/journal/volume7/2cuban.htm.

Krosnick, J. A. (1999). Survey research. Annual Review of Psychology, 50, 537–567.

Mann, D., Shakeshaft, C., Becker, J., & Kottkamp, R. (1999). West Virginia’s basic skills/computer education program: An analysis of student achievement [Online document]. Santa Monica, CA: Milken Family Foundation. Available: www.mff.org/publications/publications.taf?page=155.

Math Blaster [Computer software]. (1997). Torrance, CA: Knowledge Adventure.

Math Concepts and Skills SuccessMaker (Version 5.2) [Computer software]. (1999). Sunnyvale, CA: Computer Curriculum Corporation. Available: www.ccclearn.com/products/successmaker.

Math for the Real World [Computer software]. (1996). Torrance, CA: Knowledge Adventure

Math Munchers Deluxe [Computer software]. (1997). Novato, CA: The Learning Company.

Mergendoller, J. R. (1997, January). Sifting the hype: What research says about technology and learning. Principal, 76, 12–14.

Middleton, B. M., & Murray, R. K. (1999). The impact of instructional technology on student academic achievement in reading and mathematics. International Journal of Instructional Media, 26(1), 109–116.

Moersch, C. (1999). Assessing current technology use in the classroom. Learning & Leading with Technology, 26(8), 41–49.

Moursund, D. G. (1999). Is information technology improving education? Learning & Leading with Technology, 26(4), 4–5.

Mystery Math Island [Computer software]. (1995). Galesburg, MI: Lawrence Productions, Inc. (Available) www.voyager.net/lawrence/software/mathematics/software_mystery.html.

National Council of Teachers of Mathematics. (1989) Curriculum and evaluation standards. Reston, VA: Author.

Ohio Department of Education. (1998). News release: Ninth-graders break overall, individual records on ninth-grade tests. Columbus, OH: Author. Available: http://ode000.ode.state.oh.us/www/ims/proficiency/9th_prof_9_oct97.txt.

Ohio Department of Education. (2000, June). March 2000 Ohio Ninth Grade Proficiency Test results. Columbus, OH: Author. Available: www.ode.state.oh.us/proficiency/spring2000/RPT_808S00.TXT.

Ohio Schools Technology Implementation Task Force. (1999) Report of the Ohio Schools Technology Implementation Task Force [Online document]. Available: www.ohioschoolnet.k12.oh.us/extlink/frameset.asp?exturl=/resources/
document/documents/doclib_490.pdf&returl=/.

O’Neil, J. (2000). Fads and fireflies: The difficulties of sustaining change. Educational Leadership, 57(7), 6–9.

OPT for Success (Version 2.0) [Computer software]. (1998). Austin, TX: Pearson Publishing, Inc.

Optimum (Version 1.0) [Computer software]. (1997). Cincinnati, OH: Photonics Graphics.

Passing the Ohio Ninth Grade Proficiency Test (Version 2.0) [Computer software]. (1997). Columbus, OH: Englefield & Arnold, Inc.

Roden, J. K. (1998). Four ways to make a survey slip and fall. Journal of Staff Development, 19(4), 28–32.

Rosen, L. D., & Weil, M. M. (1995). Computer availability, computer experience and technophobia among public school teachers. Computers in Human Behavior, 11(1), 9–31.

SkillsBank (Version 4) [Computer software]. (1998). Novato, CA: SkillsBank Corporation.

Soloway, E. (1998). No one is making money in educational software. Association for Computing Machinery, Communications of the ACM, 41(2), 11–15.

Wenglinsky, H. (1998). Does it compute? The relationship between educational technology and student achievement in mathematics [Online document]. Princeton, NJ: ETS Policy Information Center. Available: www.ets.org/research/pic/technolog.html.

Williams, B. (Ed.). (1996). Closing the achievement gap: A vision for changing beliefs and practices. Alexandria, VA: Association for Supervision and Curriculum Development.

A PDF file of the full article is available. Contact: jrte@iste.org. Please specifiy Volume and Issue number.

Back to page 1

Copyright © 2001, ISTE (International Society for Technology in Education). All rights reserved.