Home PublicationsL&LIssuesVolume 27 (1999-2000) April (No. 7)

Download the full article (PDF, 933 KB, PDF Instructions)

To quell concerns that instructional technology is just a costly fad, districts need to develop meaningful, data-driven assessments of technology’s effects. U.S. school districts have tackled this challenge using the six-step process described here to develop authentic and meaningful assessments for instructional technology.

How often have you heard it? You know, the question that comes from parents, board members, and increasingly from educators themselves: “We’ve spent millions on technology, but how do we know it’s having any effect on student achievement?” To avoid the pessimistic forecast that information technology (IT) will simply become another educational fad, attention must be paid to defining goals for technology use and then measuring progress made. In the absence of any assessment criteria, technology will not be funded solely on faith. The challenge is to create realistic evaluations of IT’s effect on students and their academic achievement.

Districts across the United States are beginning to meet this challenge as they move beyond counting computers and student–teacher “contact hours” spent in labs and workshops. Sun Associates has had the pleasure of working with some of these districts to develop and implement meaningful processes for assessing technology. In this article, I present methods and findings from this work.

When considering technology evaluation, we are predominantly concerned with formative versus summative methods. The evaluations we review are of the “how are we doing?” versus the “how have we done?” variety. We have found this to be an important point to make, as the general public usually expects evaluations to be “tests” that summarize the success or failure of a particular event or occurrence. As most teachers know, technology implementation—and certainly the larger educational enterprise of which technology is just a small part—cannot be treated as a “been there, done that” event. Meaningful assessments take a variety of factors into consideration and transpire over time. Exemplary technology evaluation work incorporates this philosophy by developing broad-based indicators that are measured using quantitative and qualitative data.

Evaluation is a data-driven process. The basic evaluative process is to collect data that tell a story. The framework, or plot, of this story is determined by the evaluation questions created, but the story itself comes from data. Therefore, in educational technology evaluation, the evaluator’s task is to gather data that tell the stories of how technology has affected student achievement. (For an excellent discussion of this basic point, see Bingham [1999].)

Data can be quantitative (counts of things) or qualitative (descriptions of things). A well-rounded evaluation will make use of both. It is no more possible to tell the story of technology’s effect solely through reporting test scores and computer counts than it is by simply laying out a string of anecdotal stories. Rather, data need to be deeply descriptive and logically supportive of the questions they answer.

The Evaluation Process

Most teachers will be familiar with our basic evaluation process, as it is the model for an authentic, performance-based assessment. Although such assessments have traditionally been used for student work, we apply this same methodology to assessing the performance of a system—in this case, IT. Developing and deploying a technology evaluation is a considerably larger task than developing a rubric for assessing student projects, but the underlying logic is the same. Furthermore, if we are increasingly able to accept authentic assessments for student learning—the true “product” of our educational system—then isn’t it equally appropriate to employ such assessments for various aspects of that system? Our basic technology evaluation process has six steps:

1. Create an evaluation committee
2. Develop evaluation priorities and related questions
3. Create performance indicators for each question
4. Organize indicators into assessment rubrics
5. Collect data and score using rubrics
6. Create an evaluation report on the results of the scoring process, auxiliary findings, and directions for future efforts.

Assessment in Action

Effect on Students

Sun Associates has worked with Fayette County (Kentucky) Public Schools (www.fayette.k12.ky.us) to frame and implement an evaluation process that intends to assess technology’s impact on student achievement. The district’s evaluation centers on three critical questions:

• Has technology affected student achievement positively?
• Are teachers fluent with technology tools to the extent that they can use these tools effectively with students?
• Has the district allocated technology resources to best support all teachers and students?

As an initial step, the district created a technology evaluation committee composed of stakeholders from throughout the district and local community. This committee closely paralleled the structure of the district’s technology planning committee. Just as with the planning committee, the evaluation committee included teachers and administrators from all grade levels, district-level administrators, technology staff, parents, and school committee members. Once established, the committee developed a set of indicator rubrics (Table 1). Rubrics help identify evidence and define successful mastery of the performance indicators associated with each evaluation question or goal.

Significantly, these indicators are relevant to the actual behaviors and attributes appropriate for Fayette County teachers and students to have if the district is meeting its IT goals.

One of the most useful aspects of Fayette County’s work has focused on defining success. The district’s committee has spent considerable time debating and refining their position on what behaviors will be exhibited by students and teachers who have been positively affected by technology. In addition to their own knowledge base, their work has also been informed by a review of the Milken Exchange’s Seven Dimensions for Gauging Progress of Technology (www.milkenexchange.org) and other similar tools such as the CEO Forum’s StaRChart (www.ceoforum.org). The committee decided that a connection between the district’s indicators and some of the elements of the Seven Dimensions framework was useful. This provided their rubrics with external validity as well as a potential connection to other districts and future work. Nevertheless, Fayette felt strongly that their indicators be specific to the actual practices, configurations, and technology directions in place in the district. In short, these “directions” are the goals and actions originally articulated in the district’s strategic technology plan.

It is certainly possible to create evaluation questions that do not adhere directly to planning goals, but this is done at the risk of having the evaluation drive a plan’s actions instead of guiding and focusing its progress. This leads to what I feel is one of the underlying principles of technology evaluation: base evaluation questions on existing strategic goals. Avoid creating new goals for technology as a result of the evaluation process.

Certainly, it is appropriate to use evaluation work to fine-tune existing planning goals. In this formative way, some districts choose to use the evaluation process as the kick off to a strategic plan update.

Once again, it is critical to design indicators that identify actions you expect of individuals in your own environment. In other words, do not allow the indicators—and in turn, the assessment—to be based in a generic or arbitrary set of idealized teacher or student behaviors. People from your district should create an assessment for their peers.

Data Collection and Reporting

Beyond developing indicator rubrics, there are two steps remaining for the typical technology evaluation project—data collection and reporting. Data collection makes use of tools and techniques such as surveys, observations, interviews, focus groups, reviews of teacher and student work, and public meetings. The point is to collect data that relate to the developed indicators.

For example, if an indicator of high achievement in teacher use of technology is that teachers will use e-mail to communicate with peers outside of the district, then data are needed that show the amount as well as qualitative substance of teacher e-mail communications. This might include technical logs (e.g., how often do teachers access their e-mail accounts); teacher surveys to determine how often e-mail is used and for what; and teacher interviews to determine the value placed on e-mail communication. All of this quantitative and qualitative data are used to determine a level of overall achievement in the indicator rubric. A similar logic would be used to measure achievement with any set of indicators. Sometimes an external consultant might play a role in evaluation data analysis and scoring. Sometimes the committee itself will handle this step.

Fayette County’s evaluation effort used a variety of data collection strategies. We designed an extensive online survey of all teaching staff that provided teachers with the opportunity to give us detailed text comments. This survey had a response rate of more than 70%. Classroom observations and focus group interviews were conducted over the course of a week-long site visit by our staff. Find samples of Fayette County’s data collection instruments (and links to sites where you can find the code for making your own online survey) at www.sun-associates.com/eval/sample.

Here, it is worth mentioning that data collection might take place at the individual level of performance, but individual data should never be reported. The mission of a districtwide evaluation is to determine the progress of the district as a group of individuals in meeting its goals. Nothing will undermine an evaluation project faster than the perception that it is measuring or ranking individuals. If individual assessments are important, these should be developed and administered separately from your district technology evaluation.

Reporting, the final process step, is important because it reinforces the basic point of the evaluation—to provide a structured assessment for fine-tuning and improving progress toward meeting the district’s technology implementation goals. If findings are not reported, then the assessment has no value. Evaluation’s benefit really occurs when results are published widely throughout the district. Such publishing expands and informs the discussion and reflection processes that started in the evaluation committee meetings.

Yes, it sometimes seems risky to open district technology planning and implementation work to this level of public discussion; but in the end, this full disclosure of success, failure, and ongoing challenge is necessary to quell concerns that technology is just an expensive and nonproductive activity. Quite simply, for benefit to be recognized, there must be an open forum for discussing costs and benefits. A successful evaluation project will inspire this discussion.

Results

Fayette County views its technology evaluation projects within the broad context of educational change and improvement. The greatest value in this evaluation process, even at the classroom teacher and department levels, is the reflection that the process inspires. The development of indicator rubrics requires that teachers and administrators spend considerable time examining and defining what constitutes “success” in their efforts to integrate technology into teaching and learning.

Fayette County’s evaluation effort has just recently come to the conclusion of its initial cycle, but already there are plans to use the developed rubrics in an ongoing, formative, basis. In its first evaluation year, the district did not score beyond the midrange of its rubrics; but in keeping with the point of developing the descriptive indicators, our evaluation has shown where the district is now and provides a concrete picture of where they want to be. In other words, the district has an ordered set of qualitative indicators that graphically show where they have come from and where they can go. In practice, we describe this as “quantifying the qualitative.” Though indicators such as those developed by Fayette County are largely qualitative, they are ordered and ranked to show evidence of a quantitative progression (Level 1, Level 2, etc.) over time.

Conclusion

The biggest challenge confronting technology planners and implementers is not a lack of evidence that technology has positive effects. Rather, it is that currently most districts and individual teachers are not evaluating their existing efforts in a systematic and compelling way. At best, many teachers can only offer anecdotes, which though meaningful to the profession are largely unconvincing to a lay public accustomed to “hard data.” Evaluation efforts such as that in Fayette County prove that districts can focus their investigations around concrete questions, develop visual indicators of performance, collect data that support a performance-based assessment, and, finally, use these evaluations in productive, formative improvement efforts. Read more about Fayette County and other evaluation projects in the online supplement. Though these efforts are certainly time-consuming and somewhat frightening in their capacity (and design) to open technology integration to the scrutiny of a public debate, the end results are a stronger commitment to ensuring the benefits of technology to teaching and learning.

References

Bingham, M. (1999). Stories with data and data with stories. THE Journal, 26(9), 20. Available: http://serve-line.serve.org/seir-tec/publications/storiesdata.html.

Tools: Fayette County Public Schools
Note. The Web sites listed on this page were valid when this issue of L&L went to press. We have no control over these sites, and the Web is very volatile. Please let us know if you find a broken link, and we’ll do our best to fix it.

Student Impact Rubric

Data Collection Instruments: www.sun-associates.com/eval/sample

Fayette County Public Schools Update

On February 14, 2000, Sun Associates presented the final report on Fayette County’s formative evaluation to the district school committee. Though the district is only just beginning to digest the report and its implications, the project has already generated considerable discussion around a variety of technology integration issues. Thus far, the discussion has centered around two areas: (1) the link between technology and pedagogy and (2) the link between our evaluation work and existing, traditional assessments.

Technology and Pedagogy
Our evaluation found that although technology was widely used by Fayette County teachers as a productivity tool, considerable room for growth existed in terms of how teachers used technology to enrich and reform their teaching practice. In other words, the majority of the district’s teachers were using technology to support existing instructional models and practices. Readers familiar with the research base relating to technology and educational change will quickly recognize that Fayette County’s teachers are for the most part at the adoption and appropriation levels of the Apple Classrooms of Tomorrow (ACOT) framework. At these stages, teachers are using basic technology tools to augment, reinforce, and support existing practices.

In Fayette County’s indicator rubrics, teacher behaviors associated with adoption or appropriation are indicative of performance at the midrange on the district’s four-level scale. To attain higher levels of achievement, teachers would need to exhibit behaviors more in line with that ACOT research terms appropriation or invention. Furthermore, the indicators themselves provide a range of examples of just what sorts of practices and outcomes one would expect to find/observe among teachers performing at these higher levels.

Fayette County did not set out to confirm the ACOT research through its formative evaluation process. Rather, the district evaluation committee simply identified what it wanted to find in terms of teacher use of technology and the effects that different degrees of technology integration could have on student learning. As the various degrees of achievement were determined, it soon became obvious that these levels mapped to and confirmed those documented by ACOT and other research related to change. A review of this research is suggested for any district considering an evaluation of where its teachers are in terms of technology integration. Still, it is extremely important that the district create its own performance indicators which interpret levels of teacher adoption as actually observable and realistic behaviors within the context of the evaluating district. As we point out in the main article, all districts have unique expectations for teachers and students. These need to be reflected in the developed indicators if the evaluation is to be taken seriously, or considered valid, by the district’s teachers, administrators, and community. Validity is required for the evaluation to reach its potential as a formative assessment tool.

The Link Between Formative Technology Evaluation and Traditional Student Assessments

After considering the indicators, rubrics, and findings developed for Fayette County’s educational technology evaluation, some teachers and administrators in the district asked if there wasn’t another “bottom line” that we had missed; that is, what is the connection—if any—between student and teacher technology use and traditional student assessments such as standardized test scores. Given the amount of attention provided to these traditional assessments, it is certainly expected that one should attempt to tie all important educational initiatives, such as technology, to performance gains or losses on these scores.

We at Sun Associates believe that a link exists between technology and student performance as measured by traditional assessments, but this link is not direct. In other words, technology alone is not responsible for increases in student achievement. Rather, the integration of technology is one of a number of changes and improvements made to the teaching and learning environment. When these changes are made systemically, then student performance increases.

Perhaps the strongest point we made in the district’s full evaluation report is that technology alone has little positive effect on student achievement or teacher behavior. In fact, we found that as a whole, Fayette County schools are reasonably rich in technology tools and devices. For the most part, teachers actively use this technology for personal productivity. But, we found that the simple existence of the devices and the fact that teachers know how to use them does not ensure that technology will be used in ways that affect student achievement.

Educational research is absolute in its findings that student achievement increases when learning activities are engaging and student-centered. Learning needs to be standards based, relevant, attuned to the individual student’s style of learning, and holistic—that is, tied to a student’s prior knowledge, experience, and interests. Another way of saying all of this is to shorthand these descriptions and state that high performing student learning is engaged and constructivist in nature and guided by strong and meaningful curriculum frameworks. Students who are able to work in environments that encourage this type of learning will achieve their maximum potential. Although existing standardized tests do at best only an adequate job of truly measuring student achievement and knowledge, students who are the products of supportive learning environments and reformed teaching practices will generally score higher than students who are not.

When we examine the various indicators developed by the Fayette County Technology Evaluation Committee, we see that the indicators have been written to “reward” those uses of technology that are products of and supports to high-performance learning environments. The committee’s work and the indicators they developed were informed by a knowledge of what the research says about technology and learning. High-performing schools use technology in ways that score high on the committee’s indicators. What the evaluation shows is that most schools could do better in terms of how technology is being used to support student achievement. In other words, we are certain that as the district scores higher on its technology rubrics, it would also as a whole have higher standardized test scores.

It all comes back to the same basic point: technology is only useful—and, in this case, we can say impactful—when used by a skilled teacher. The issue is pedagogy, not computer skills. Subsequently, schools where teachers are employing reformed instructional practices, where administrators support and expect such practices, where students are actively and excitedly involved in the process of learning—these schools will be “high performing schools.” And when we visited those schools in Fayette County, we found technology being used by students and teachers in ways that would score very highly on our technology rubrics.

Lexington Public Schools: Assessing Technology Staff Development

Since 1997, we have worked with teachers and administrators in the Lexington (Massachusetts) Public Schools (http://lps.lexingtonma.org) to construct and implement a variety of assessments relating to Lexington’s technology plan. Lexington’s five-year technology plan calls for each year to be devoted to a particular strategic theme. Year One (1996–97) emphasized completing the district’s network infrastructure. Year Two’s theme, “teacher immersion,” emphasized developing teacher technology skills. That year, we helped the district assess their technology staff development efforts. This afforded a more limited and tighter focus than an overall evaluation of technology’s effect on student achievement. The findings from Year Two provided valuable background for subsequent years in which the emphasis would be placed on developing and implementing technology-supported curricula.

In fall 1997, Lexington convened a districtwide technology evaluation committee with a very similar stakeholder composition to that of Fayette County. Over the course of a day-long meeting, the committee developed evaluation questions rooted in that year’s specific technology plan goals. This method of developing questions rooted in planning goals addresses the district’s basic need to know whether they are meeting their goals for IT and the various implementation process steps.

After setting their basic evaluation questions, the district created a mechanism for assessing their achievement in meeting the Year Two goals. This was accomplished using a multilevel rubric that organized indicators of success toward meeting an overall indicator of what it “looked like” to achieve the performance goal associated with each evaluation question. These rubrics can be viewed online at www.sun-associates.com/eval/sample. As in Fayette County, Lexington’s evaluation committee worked to develop indicators relevant to actual behaviors and attributes appropriate for Lexington Public School teachers to when meeting the Year Two goal of “teacher immersion.”

Just as in Fayette County, Lexington’s evaluation committee took time to debate exactly what teacher behaviors were exemplary of meeting the district’s technology professional development goals. The time spent doing this represented a critical period of reflection. In my experience, reflection is one of the major benefits of conducting a technology evaluation effort. Reflection on anticipated and desired outcomes helps deepen a district’s collective understanding of technology integration. This deeper understanding among stakeholders will translate into greater support for technology and the benefits it brings to the classroom.

Resources

Apple Computer Corporation. (1996). Changing the conversation about teaching, learning, and technology: A report on 10 years of ACOT research. Cupertino, CA: Author. Available: www.apple.com/education/k12/leadership/acot/library.html.

Fayette County Public Schools: www.fayette.k12.ky.us

Loucks-Horsley, S., Hewson, P.W., Love, N., & Stiles, K.E. (1998). Designing professional development for teachers of science and mathematics. Thousand Oaks, CA: Corwin Press.

Milken Exchange’s Seven Dimensions for Gauging Progress of Technology: www.milkenexchange.org

Table 1. Fayette County Public Schools’ Student Impact Rubric
Student Achievement

Back to Supplement
Back to top

Copyright © 2000, ISTE (International Society for Technology in Education). All rights reserved.