Home PublicationsL&LIssuesVolume 19 (1992-1991) September (No. 1)

To quell concerns that instructional technology is just a costly fad, districts need to develop meaningful, data-driven assessments of technology’s effects. U.S. school districts have tackled this challenge using the six-step process described here to develop authentic and meaningful assessments for instructional technology.

How often have you heard it? You know, the question that comes from parents, board members, and increasingly from educators themselves: “We’ve spent millions on technology, but how do we know it’s having any effect on student achievement?” To avoid the pessimistic forecast that information technology (IT) will simply become another educational fad, attention must be paid to defining goals for technology use and then measuring progress made. In the absence of any assessment criteria, technology will not be funded solely on faith. The challenge is to create realistic evaluations of IT’s effect on students and their academic achievement.

Districts across the United States are beginning to meet this challenge as they move beyond counting computers and student–teacher “contact hours” spent in labs and workshops. Sun Associates has had the pleasure of working with some of these districts to develop and implement meaningful processes for assessing technology. In this article, I present methods and findings from this work.

When considering technology evaluation, we are predominantly concerned with formative versus summative methods. The evaluations we review are of the “how are we doing?” versus the “how have we done?” variety. We have found this to be an important point to make, as the general public usually expects evaluations to be “tests” that summarize the success or failure of a particular event or occurrence. As most teachers know, technology implementation—and certainly the larger educational enterprise of which technology is just a small part—cannot be treated as a “been there, done that” event. Meaningful assessments take a variety of factors into consideration and transpire over time. Exemplary technology evaluation work incorporates this philosophy by developing broad-based indicators that are measured using quantitative and qualitative data.

Evaluation is a data-driven process. The basic evaluative process is to collect data that tell a story. The framework, or plot, of this story is determined by the evaluation questions created, but the story itself comes from data. Therefore, in educational technology evaluation, the evaluator’s task is to gather data that tell the stories of how technology has affected student achievement. (For an excellent discussion of this basic point, see Bingham [1999].)

Data can be quantitative (counts of things) or qualitative (descriptions of things). A well-rounded evaluation will make use of both. It is no more possible to tell the story of technology’s effect solely through reporting test scores and computer counts than it is by simply laying out a string of anecdotal stories. Rather, data need to be deeply descriptive and logically supportive of the questions they answer.

The Evaluation Process

Most teachers will be familiar with our basic evaluation process, as it is the model for an authentic, performance-based assessment. Although such assessments have traditionally been used for student work, we apply this same methodology to assessing the performance of a system—in this case, IT. Developing and deploying a technology evaluation is a considerably larger task than developing a rubric for assessing student projects, but the underlying logic is the same. Furthermore, if we are increasingly able to accept authentic assessments for student learning—the true “product” of our educational system—then isn’t it equally appropriate to employ such assessments for various aspects of that system? Our basic technology evaluation process has six steps:

1. Create an evaluation committee
2. Develop evaluation priorities and related questions
3. Create performance indicators for each question
4. Organize indicators into assessment rubrics
5. Collect data and score using rubrics
6. Create an evaluation report on the results of the scoring process, auxiliary findings, and directions for future efforts.

Assessment in Action

Effect on Students

Sun Associates has worked with Fayette County (Kentucky) Public Schools (www.fayette.k12.ky.us) to frame and implement an evaluation process that intends to assess technology’s impact on student achievement. The district’s evaluation centers on three critical questions:

• Has technology affected student achievement positively?
• Are teachers fluent with technology tools to the extent that they can use these tools effectively with students?
• Has the district allocated technology resources to best support all teachers and students?

As an initial step, the district created a technology evaluation committee composed of stakeholders from throughout the district and local community. This committee closely paralleled the structure of the district’s technology planning committee. Just as with the planning committee, the evaluation committee included teachers and administrators from all grade levels, district-level administrators, technology staff, parents, and school committee members. Once established, the committee developed a set of indicator rubrics (Table 1). Rubrics help identify evidence and define successful mastery of the performance indicators associated with each evaluation question or goal.

Significantly, these indicators are relevant to the actual behaviors and attributes appropriate for Fayette County teachers and students to have if the district is meeting its IT goals.

One of the most useful aspects of Fayette County’s work has focused on defining success. The district’s committee has spent considerable time debating and refining their position on what behaviors will be exhibited by students and teachers who have been positively affected by technology. In addition to their own knowledge base, their work has also been informed by a review of the Milken Exchange’s Seven Dimensions for Gauging Progress of Technology and other similar tools such as the CEO Forum’s StaRChart (www.ceoforum.org). The committee decided that a connection between the district’s indicators and some of the elements of the Seven Dimensions framework was useful. This provided their rubrics with external validity as well as a potential connection to other districts and future work. Nevertheless, Fayette felt strongly that their indicators be specific to the actual practices, configurations, and technology directions in place in the district. In short, these “directions” are the goals and actions originally articulated in the district’s strategic technology plan.

It is certainly possible to create evaluation questions that do not adhere directly to planning goals, but this is done at the risk of having the evaluation drive a plan’s actions instead of guiding and focusing its progress. This leads to what I feel is one of the underlying principles of technology evaluation: base evaluation questions on existing strategic goals. Avoid creating new goals for technology as a result of the evaluation process.

Certainly, it is appropriate to use evaluation work to fine-tune existing planning goals. In this formative way, some districts choose to use the evaluation process as the kick off to a strategic plan update.

Once again, it is critical to design indicators that identify actions you expect of individuals in your own environment. In other words, do not allow the indicators—and in turn, the assessment—to be based in a generic or arbitrary set of idealized teacher or student behaviors. People from your district should create an assessment for their peers.

Data Collection and Reporting

Beyond developing indicator rubrics, there are two steps remaining for the typical technology evaluation project—data collection and reporting. Data collection makes use of tools and techniques such as surveys, observations, interviews, focus groups, reviews of teacher and student work, and public meetings. The point is to collect data that relate to the developed indicators.

For example, if an indicator of high achievement in teacher use of technology is that teachers will use e-mail to communicate with peers outside of the district, then data are needed that show the amount as well as qualitative substance of teacher e-mail communications. This might include technical logs (e.g., how often do teachers access their e-mail accounts); teacher surveys to determine how often e-mail is used and for what; and teacher interviews to determine the value placed on e-mail communication. All of this quantitative and qualitative data are used to determine a level of overall achievement in the indicator rubric. A similar logic would be used to measure achievement with any set of indicators. Sometimes an external consultant might play a role in evaluation data analysis and scoring. Sometimes the committee itself will handle this step.

Fayette County’s evaluation effort used a variety of data collection strategies. We designed an extensive online survey of all teaching staff that provided teachers with the opportunity to give us detailed text comments. This survey had a response rate of more than 70%. Classroom observations and focus group interviews were conducted over the course of a week-long site visit by our staff. Find samples of Fayette County’s data collection instruments (and links to sites where you can find the code for making your own online survey) at www.sun-associates.com/eval/sample.

Here, it is worth mentioning that data collection might take place at the individual level of performance, but individual data should never be reported. The mission of a districtwide evaluation is to determine the progress of the district as a group of individuals in meeting its goals. Nothing will undermine an evaluation project faster than the perception that it is measuring or ranking individuals. If individual assessments are important, these should be developed and administered separately from your district technology evaluation.

Reporting, the final process step, is important because it reinforces the basic point of the evaluation—to provide a structured assessment for fine-tuning and improving progress toward meeting the district’s technology implementation goals. If findings are not reported, then the assessment has no value. Evaluation’s benefit really occurs when results are published widely throughout the district. Such publishing expands and informs the discussion and reflection processes that started in the evaluation committee meetings.

Yes, it sometimes seems risky to open district technology planning and implementation work to this level of public discussion; but in the end, this full disclosure of success, failure, and ongoing challenge is necessary to quell concerns that technology is just an expensive and nonproductive activity. Quite simply, for benefit to be recognized, there must be an open forum for discussing costs and benefits. A successful evaluation project will inspire this discussion.

Results

Fayette County views its technology evaluation projects within the broad context of educational change and improvement. The greatest value in this evaluation process, even at the classroom teacher and department levels, is the reflection that the process inspires. The development of indicator rubrics requires that teachers and administrators spend considerable time examining and defining what constitutes “success” in their efforts to integrate technology into teaching and learning.

Fayette County’s evaluation effort has just recently come to the conclusion of its initial cycle, but already there are plans to use the developed rubrics in an ongoing, formative, basis. In its first evaluation year, the district did not score beyond the midrange of its rubrics; but in keeping with the point of developing the descriptive indicators, our evaluation has shown where the district is now and provides a concrete picture of where they want to be. In other words, the district has an ordered set of qualitative indicators that graphically show where they have come from and where they can go. In practice, we describe this as “quantifying the qualitative.” Though indicators such as those developed by Fayette County are largely qualitative, they are ordered and ranked to show evidence of a quantitative progression (Level 1, Level 2, etc.) over time.

Conclusion

The biggest challenge confronting technology planners and implementers is not a lack of evidence that technology has positive effects. Rather, it is that currently most districts and individual teachers are not evaluating their existing efforts in a systematic and compelling way. At best, many teachers can only offer anecdotes, which though meaningful to the profession are largely unconvincing to a lay public accustomed to “hard data.” Evaluation efforts such as that in Fayette County prove that districts can focus their investigations around concrete questions, develop visual indicators of performance, collect data that support a performance-based assessment, and, finally, use these evaluations in productive, formative improvement efforts. Read more about Fayette County and other evaluation projects in the online supplement. Though these efforts are certainly time-consuming and somewhat frightening in their capacity (and design) to open technology integration to the scrutiny of a public debate, the end results are a stronger commitment to ensuring the benefits of technology to teaching and learning.

References

Bingham, M. (1999). Stories with data and data with stories. THE Journal, 26(9), 20.

Copyright © 1991, ISTE (International Society for Technology in Education). All rights reserved.