Home PublicationsL&LIssuesVolume 27 (1999-2000) April (No. 7)

Download the full article (PDF, 256 KB, PDF Instructions)

Various online assessment tools (as well as those you create) can help you see exactly how technology affects student learning

Currently, the political solution to underachieving schools is to legislate performance standards and standardized tests, whether these tests really measure all the skills the SCANs report (U.S. Department of Labor Secretary’s Commission on Achieving Necessary Skills, 1991) says students will need to succeed as adults in coming years.

In addition, school board members, policy makers, and the media are saying that educators have spent millions of dollars on technology for classrooms without being able to measure its effects on student learning. The media is also raising concerns about the role and expense of technology while other programs such as music and art are being cut. Answering these concerns has not been easy for the educators who have championed the use of technology.

In this article, I attempt to put the question of assessing technology effects on students in perspective and offer some strategies for administrators and technology-using teachers that will produce data to show that the use of technology is affecting student learning in ways that can’t be measured by standardized tests. First, I discuss the technology infrastructure that must be in place. Then I help you identify your data needs. Finally, I provide assessment ideas and tools.

Preconditions

A solid technology infrastructure is the first step. To create this, you must have (1) easy access to technology, (2) effective and ongoing staff development, and (3) routine technology integration.

Access has four major components.

A. Hardware ratio. How much hardware is there compared to the number of students, and what is the capability of the hardware? A student to computer ratio of 5:1 may sound excellent until you find out that more than half of the available hardware are Apple IIe or 286 machines; these machines have value, but they will not run today’s more sophisticated software.

B. Software. What software is available for student use, and to what degree is it correlated to the district’s instructional goals? If the software is not tightly correlated to the curriculum, then little academic improvement can be expected.

C. Connectivity. Are the computers connected to the network, and what is the network’s bandwidth? If the network doesn’t reach every classroom and is not high speed, then the chances that students will use it routinely are greatly diminished.

D. Deployment. Where are the computers located? Are they in labs or in classrooms? You can’t expect computers to affect instruction if they are used only once a week. A recent study in West Virginia (Lemke, Quinn, Zucker, & Cahill, 1998) demonstrated that students who used computers in classrooms did significantly better on standardized tests than students who used them in a lab. The study also found that teachers were more likely to integrate their instructional programs with computers when they were in their classrooms.

The second target area is staff development. If teachers are to use technology effectively to enhance instruction, then they must have continuous access to staff development. The “spray and pray” series of workshops may get them started, but it will never result in them moving much from productivity (e.g., using technology for e-mail, word processing, and electronic grading) to integration, where technology becomes another learning tool and supports the instructional program.

Curriculum integration is the third target area. Technology use must be fully correlated to the district’s standards and tests. If teachers use technology to teach butterflies and the test measures caterpillars, then there is little chance of demonstrating that technology is making a difference.

Once your infrastructure is strong, you can begin assessing your data needs to measure outcomes.

Data Needs

What’s important is to determine the assessment data you need to collect. There are always trade-offs. You have to balance the cost of collecting the data with the richness of the evidence you can collect. The data you collect should be customized to your local requirements and needs yet still reflect national or state norms to allow comparison. Start by reviewing your technology plan. It should provide guidance about the data you need. Then decide what types of data you will need (e.g., teacher observations, student grades, etc.). Then you can begin collecting it.

Collecting Data

Now that the three preconditions of access, staff development, and curriculum integration are in place and you have a handle on the data you need to collect, let’s look at some assessment tools you can use to collect your data.

Teacher Data
As teachers are busy and reporting technology use may add more to their plates, try to maximize the data you already have so you can minimize the extra data you ask teachers to gather.

Teacher surveys work well to gather user demographics, including current teaching practices, level of student, and student use as well as identifying training needs and improvement strategies. Look at www.wested.org/tie/techplan/pdfs/TSURVEY.PDF for a sample teacher survey.

Student Data
Student data provide another way of describing technology’s effects on student learning. Collect student data from classrooms where technology is routinely used to support the instructional program and from classrooms where technology is seldom used. Differences in achievement tests, attendance, discipline referrals, and report card marks for behavior are a few of the data points you can use.

Interview students to learn what they say about their learning. Ask how they feel about their successes. Ask if they recall important concepts and relationships and if they feel more empowered to develop their own learning rather than wait for the teacher to tell them what to do.

Rubrics are one of the multiple measures you can use to assess student work. A study of high school students by Apple Classroom of Tomorrow (ACOT) determined that students who use multimedia well were only so-so performers judged by more traditional forms of testing. Student multimedia projects lend themselves to assessment by rubrics. A rubric for assessing multimedia projects can be found at www.wested.org/tie/techplan/curplan.shtml.

Rubrics also help students know, while they are planning their work, what components are required in a superior project. Rubrics have even greater power when teachers lead students to develop them as a class. The George Lucas Educational Foundation (www.glef.org) Learn and Live video has a section that demonstrates the power of the project process when students create their own rubrics.

Student products, Web pages, multimedia projects, computer-generated art, desktop published items, or video projects provide powerful examples of the impact of technology on student learning. Collect them and show them off at every opportunity. They will provide parents, administrators, and policy makers with evidence that the dollars they are spending on technology are giving a good return on investment.

Parent Data
Parent surveys are a good way to gather data about the level of technology use at home. Ask questions about the type and number of computers at home, whether they have Internet access, how often the child uses the computer, and what parents think about the use of technology at school.

You can also measure parent demographics, which may help explain student differences.

Technology Data
Develop a technology scope and sequence that serves a variety of purposes. A scope and sequence defines what students are expected to know about the use of technology and serves as a basis for teachers to define and assess student levels of technology use. Teachers can also use the scope and sequence to assess their own level of technology skills and to decide what further staff development they will need to help their students meet the technology standards for their level.

The Milpitas (California) Unified School District’s technology scope and sequence (www.milpitas.k12.ca.us) gives an excellent example of technology skills students are expected to acquire as well as examples of software students can use and activities students can do to learn technology skills. The Milpitas Scope and Sequence also provides performance indicators for teachers to assess student achievement.

A replacement unit can serve as an assessment tool for both technology skills and subject matter content. A sample unit can be found at www.ctap5.k12.ca.us. Replacement units are designed to take the place of regular activities a teacher would use to teach a concept or theme. Replacement units might include technology-based teacher and student activities, assessments, and the templates for students. When students have completed the unit, they will have demonstrated both their mastery of the subject matter and their technology skills.

A tool that teachers can use to develop a technology-enhanced lesson is the unit of study (www.wested.org/tie/techplan/curplan.shtml). The unit of study describes the curriculum objectives the students will meet, the tools they will use, the classroom organization (teams, individual, or whole class), and how students’ work will be assessed. Completed units of study and samples of student work provide rich and telling evidence about the level of student attainment of technology skills.

Other more easily collected measures of technology’s effect in the classroom are the amount of time students use technology, how many hours of training teachers have, and a review of lesson plans with a count of how often technology was a part of each lesson.

Online Tools
Additional resources for assessing technology use are:

• North Central Regional Technology in Education Consortium’s Learning with technology profile tool: www.ncrtec.org/capacity/profile/profile.htm
• Western Carolina University’s competency list for students: www.ceap.wcu.edu/Martin/Compdef.htm#10
• International Society for Technology in Education’s recommended foundations in technology for teachers and NETS standards: www.iste.org
• Utah Educational Awareness Project’s (UTAP) rubrics and an online teacher skill survey: www.uen.org/utap

Sharing Your Findings

By using these assessment tools and resources and others you develop locally, you can demonstrate that you are getting a return on your investments in classroom technology. Collecting and analyzing your technology-use data is only half the work that has to be done. The other half is to communicate your findings to as many audiences as possible including service clubs, the chamber of commerce, the media, parents, and the school board. Develop a team to present the principal who paints the school’s vision, the teacher who describes the educational objectives, and the students who demonstrate their learning. The more specific information parents and decision makers have, the less likely they are to be concerned the next time the local media questions the role of technology in schools.

Implementing these ideas will mean that your assessment program will consist of performance standards + standardized tests + other measures of student achievement.

References

Lemke, C., Quinn, B., Zucker, A., & Cahill, S. (1998). Report to the Commonwealth of Virginia: An analysis of education technology availability and usage in the public schools of Virginia. Santa Monica, CA: Milken Family Foundation. Available: http://www.mff.org/pubterms.taf?file=http://www.mff.org/pubs/ME156.pdf.

U.S. Department of Labor, Secretary’s Commission on Achieving Necessary Skills. (1991). What work requires of schools. Washington, DC: U.S. Government Printing Office. Available: www.ttrc.doleta.gov/SCANS/work.html.

Copyright © 2000, ISTE (International Society for Technology in Education). All rights reserved.