Home ResearchReportsResearch on Technology in Education (2000) Achievement

Accelerating and Tracking Student Achievement

Executive Summary: Information technology is a powerful aid to the organization and delivery of instruction, and to student assessment. Extensive research on computer-assisted learning (CAL) indicates that, on average and over a wide range of subject areas and grade levels, CAL helps students learn both significantly faster and significantly better as compared to traditional teaching methods. Part of the reason for this is that CAL provides a method of implementing some of the results from educational research in curriculum, instruction, and assessment.

In addition, CAL fits in well with an "any topic, any time, any place" approach to learning. Thus, CAL is likely to be increasingly used in our formal and informal educational systems.

Accountability is a driving force behind many of the national, state, and local educational reform movements. These initiatives strive to set high standards, assess how well these standards are being met, and hold both students and the educational system accountable for results. This research report focuses mainly on use of information technology (IT) as an aid to meeting high standards in curriculum, instruction, and assessment.

National Educational Technology Standards

The ISTE National Educational Technology Standards (NETS) profiles describe learning expectations of students completing various grade levels (International Society for Technology in Education, 1998)). Here are a few examples related to student learning:

(Pre K-2). Use a variety of media and technology resources for directed and independent learning activities.

(Pre K-2). Use developmentally appropriate multimedia resources (e.g., interactive books, educational software, elementary multimedia encyclopedias) to support learning.

(Grades 3-5). Use general purpose productivity tools and peripherals to support personal productivity, remediate skill deficits, and facilitate learning throughout the curriculum.

(Grades 3-5). Use telecommunications efficiently and effectively to access remote information, communicate with others in support of direct and independent learning, and pursue personal interests.

(Grades 6-8). Apply productivity/multimedia tools and peripherals to support personal productivity, group collaboration, and learning throughout the curriculum.

(Grades 9-12). Evaluate technology-based options, including distance and distributed education, for lifelong learning.

The ISTE NETS recognize that IT provides a variety of aids to learning. An overriding goal is for students to become independent, self sufficient, lifelong learners.

School Reform

A number of the ideas in current school reform movements can be traced back to A Nation at Risk (The National Commission on Excellence in Education, 1983), which detailed shortcomings of our educational system and proposed possible solutions. The proposed solutions focused on increasing requirements and accountability. It included a specific IT-oriented recommendation that the teaching of computer science in high school should equip graduates to: (a) understand the computer as an information, computation, and communication device; (b) use the computer in the study of the other basics and for personal and work-related purposes; and (c) understand the world of computers, electronics, and related technologies.

When A Nation at Risk was published in 1983, our schools had approximately one microcomputer per 125 students. By 1998, this ratio had improved to one microcomputer per six students (Becker & Anderson, 1998). Many of the IT in education goals that were initially envisioned as only appropriate at the high school level have now be moved down to lower grades.

A Nation at Risk emphasized the need to strengthen course content, set high standards for students, and encourage students to take a rigorous program of study. Many school reform models developed in the past decade follow these same ideas.

Curriculum content can be loosely divided into lower-order skills and higher-order skills. The various current school reform models discussed in Northwest Regional Education Laboratory (1998) vary in their emphasis on lower-order versus higher-order skills. Some of the reform models might best be described as "back to basics" while others have a major emphasis on higher-order thinking skills. In recent years, however, there has been growing agreement that problem solving and other higher-order skills should be given increased emphasis (Moursund, 1996; Presidents Committee of Advisors on Science and Technology, 1997). This has increased public debate about the nature of testing methodologies, and whether they place an undue emphasis on lower-order skills.

Teaching to the Test

The following is quoted from the introduction to a report from Educational Testing Service (1999).

There is just too much standardized testing going on in our schools, lamented an unlikely source, the late Gregory Anrig, then-president of the world's largest testing organization, Educational Testing Service. (Before that, he was Chief State School Officer for Massachusetts, and had been an educator throughout his career.)

The critics of such massive testing, including many in educational measurement, offer the following complaints: Tests have been composed mostly of multiple-choice questions, which cannot assess a student's ability to come up with his or her own answers. Commercial or state tests may not test what local schools are actually teaching. Some critics argue that teachers are pushed in the direction of narrowing instruction to what they think is on the test. Further, test preparation sometimes becomes the instruction, with instructional materials mimicking the formats and exercises that appear on such tests.

Educational reformers note that curriculum content, instructional methods, and assessment should be closely aligned. We should test what we teach, and teach what we test. In recent years, such ideas have formed the basis for an increasing emphasis on authentic assessment. Authentic assessments are performance-based, realistic, and instructionally appropriate. Wiggins (1997, 1998) argues that assessment should be aligned with curriculum content and teaching methodologies. Wiggins and others argue that if the assessment methodologies are appropriate, then teaching to test is not only appropriate, it is also highly desirable.

The authentic assessment movement has stressed the value of alternatives to multiple-choice questions, such as performance evaluation, open-ended questions, student writing, students explaining the reasoning behind their problem-solving work, rubrics, and portfolio assessment.

Many schools, school districts, and states are now making use of portfolio assessment. Details on use of computers in the creation and storage of such portfolios are given in Lankes (1995). IT makes it possible to store a wide variety of portfolio items in a computer, and then to develop a specific portfolio to fit the particular needs of a given situation.

Computer-Assisted Learning

Computer-assisted learning (CAL) is a broad term that includes computer-assisted instruction, computer-based instruction, computer-based learning, and so on. Within most CAL software there is considerable alignment among curriculum content, instructional methodology, and assessment.

Most CAL software can be roughly placed into one of two categories: limited-scope and broad-scope. Substantial CAL software is limited-scope and might focus on a specific topic at a specific grade level. Broad-scope CAL software might cover many different subject areas at a number of grade levels. Such software is often called an integrated learning system (ILS).

Regardless of its category, CAL software may include a student assessment system and provide reports to be used by the student and/or the teacher.

One of the key features of CAL is that it is a vehicle for translating educational research into practice. A CAL system can be based on one or more learning theories-accurately implementing the research ideas from those theories. Park (1993) analyzes CAL from this point of view. His research found evidence that CAL developers employ a variety of cognitive learning theories in addition to the commonly used behavioral learning theory.

A different approach to analyzing the research-into-practice issue is to evaluate the alignment of the computer-based learning materials with state and national curriculum standards. The California Learning Resource Network (CLRN) provides a statewide example of such software evaluation. A nationwide approach to this task has been developed by a commercial company (MediaSeek).

There has been extensive research, including a number of meta-studies, in CAL.  Indeed, Kulik (1994) is a study of the meta-studies. There are a wide range of types of CAL, such as drill and practice, tutorial, and simulations (including virtual realities). In brief summary, the research on CAL indicates that it works with a broad range of students, grade levels, and subject areas. On average, over a huge number of studies, Kulik found that students learn about 30 percent faster and .35 standard deviations better (moving from the 50th percentile to the 64th percentile).

An ILS may be adopted by a school, school district, or even an entire state. Mann, et al. (1999) reports on long term use of two different ILS's in West Virginia. Substantial gains in test scores were noted.

Online Assessment

Online assessment is a common component of CAL systems. Each CAL system receives student input, evaluates it, and provides feedback to the students. In essence, this is a continuous assessment paradigm, and is less intrusive that summative evaluations given at the end of a unit of study. More complex CAL systems also include pretests and posttests. Such online assessments often make use of multimedia such as audio, video, animation, and color.

Online testing, which has been common in CAL for many years, has proven to be a forerunner of innovations in statewide and national assessment. Educational Testing Service (1998) provides a vision of the future in which testing is available on demand (any time, anywhere). In this vision, computer-adaptive testing significantly decreases the time needed to make an accurate assessment. In computer-adaptive testing, the test questions are adapted to the performance level of the test taker, quickly narrowing in on questions that are appropriate to the level of knowledge of the test taker. Rudner (1998a) provides an excellent tutorial/example of computer adaptive testing.

In some cases, an online testing system can generate the test questions at the time they are to be delivered. For example, arithmetic computation problems can be generated by use of a random number generator. More common, however, is for the testing system to contain a databank of test items. Items in such a databank can be coded by subject area, instructional level, instructional objective measured, item difficulty, discriminating power, and so on. There is significant research on the development of appropriate test items for such databanks (Rudner, 1998b). Although software exists for the storage and use of such item banks, the development of a good set of items is a major task. Thus, one tends to find most online testing systems have been developed commercially and/or through substantial outside grant support, rather than by individual teachers as part of their ongoing job.

Final Remarks

Although CAL has a long history and is broadly used, in some sense CAL is still in its infancy. We are living at a time where significant research progress is occurring in learning theory and brain theory. We are beginning to develop useful levels of knowledge in the science of learning. The National Science Foundation is currently funding a variety of research projects in this area (National Science Foundation), especially in its Knowledge and Distributed Intelligence programs. Research in these areas will provide the underpinnings for continued improvements in CAL systems. Such improvements will likely lead to increasing use of CAL.

References

Becker, H. and Anderson, R. (1998 National Survey). Teaching, learning, and computing: A national survey of schools and teachers [Online]. Available: http://www.crito.uci.edu/tlc/html/tlc_home.html

The California Learning Resource Network [Online]. Available: http://www.clrn.org/

Educational Testing Service (1998, June). Reinventing assessment: Speculations on the future of large-scale educational testing [Online]. Available: ftp://etsis1.ets.org/pub/res/reinvent.pdf

Educational Testing Service (1999). Too much testing of the wrong kind; too little of the right kind in k-12 education [Online]. Available: http://www.ets.org/research/pic/testing/204928tmt.pdf

International Society for Technology in Education (1998). National Educational Technology Standards. Eugene, OR: Author.

Kulik, J.A. (1994). Meta-analytic studies of findings on computer-based instruction. In E.L. Baker and H.F. O'Neil, Jr. (Eds.). Technology assessment in education and training. Hillsdale, NJ: Lawrence Erlbaum.

Lankes, A. (1995). Electronic portfolios: A new idea in assessment. ERIC Digest [Online]. Available: http://www.ed.gov/databases/ERIC_Digests/ed390377.html

Mann, D., Shakeshaft, C., Becker, J., and Kotthamp, R. (1999). West Virginia's Basic Skills/Computer Education Program: An analysis of student achievement. Santa Monica, CA: Milken Family Foundation. [Online]. Available: http://www.mff.org/edtech/

MediaSeek [Online]. Available: http://www.mediaseek.com/

Moursund, D. (1996). Increasing your expertise as a problem solver: Some roles of computers. Eugene, OR: International Society for Technology in Education.

National Science Foundation [Online]. Available: http://www.nsf.gov/

Northwest Regional Educational Laboratory (1998). Catalog of school reform models: First edition. [Online]. Washington, DC: NCES. Available: http://www.nwrel.org/scpd/natspec/catalog/

Park, S. (1993, June). Cognitive psychology in education: Some implications of learning strategies for designing computer-assisted instruction. Ph.D. Dissertation: University of Oregon.

Presidents Committee of Advisors on Science and Technology (1997, March). President's Committee of Advisors on Science and Technology (Mrach, 1997). Report to the president on the use of technology to strengthen K-12 education in the United States. Washington DC: Author.

Rudner, L. (1998a). An on-line, interactive, computer adaptive testing mini-tutorial [Online]. Available: http://ericae.net/scripts/cat/catdemo.htm

Rudner, L. (1998b). Item banking. ERIC/AE Digest [Online]. Available: http://www.ed.gov/databases/ERIC_Digests/ed423310.html

National Commission on Excellence in Education (April 1983). A nation at risk: The imperative for educational reform [Online]. Available: http://www.ed.gov/pubs/NatAtRisk/title.html

Wiggins, G. (1998). Educative assessment: designing assessments to inform and improve student performance San Francisco, CA: Jossey-Bass.

Wiggins, G.P. (1996- 1997, December-January). Practicing what we preach in designing authentic assessments. Educational Leadership. pp. 18-25.