Home PublicationsL&LIssuesVolume 27 (1999-2000) November (No. 3)

Assessing the Assessment

How do you know using instruments based on self-assessment are really effective if teachers don't know what they don't know? Peter Pappas and Jon Pearson of Pittsford Central Schools in New York asked their teachers to use an online survey based on the Mankato Scale and then used the results to design a technology development course for them (Pappas, 1999). This process created a more effective course.

I did a similar study with my preservice teachers at St. Mary's College in Moraga, California, in 1996. Students took a survey similar to the Mankato Scale before instruction began. The results from the survey drove the design of the course. Yet, what resulted was that the students tended to rate themselves at a higher level than they really were. After I had demonstrated a project, several of the teachers asked if I could go a little slower and review some of the basics. After 25 hours of instruction were completed, I gave the survey again, and many of the students rated themselves closer to their actual levels.

How do you develop an assessment tool that provides a clearer picture of where teachers are? Using a combination of tools that assesses proficiency and comfort levels, attitudes, and levels of integration will result in better assessment of teachers' technology use than using a single tool would. A good assessment tool provides feedback that not only helps develop what courses to offer, but also helps the individual teachers know what courses or other learning opportunities to take.

Examples of these instruments are popping up all over the Internet, yet any survey or rubric needs to be customized to meet the needs of the teachers at your site. When designing a survey, questionnaire, or rubric, decide what you want to ask and try to keep the instruments short and to the point.

Include the following in any survey:

1. name
2. grade levels and subject areas taught and for how long
3. computer at home and what type
4. e-mail address
5. what technology, including software used
6. what technology to learn
7. any dreams or hopes for technology use in the classroom, the lab, or home
8. concerns or obstacles that keep dreams from coming true

For more information on assessment tools, check Computer Strategies' Web site (www.compstrategies.com).

Reference

Pappas, P. & Pearson, J. (1999, May/June). Take a skills snapshot: Employing online self-assessments. MultiMedia Schools [Online serial].

Assessment Example

I developed a rubric that uses nonthreatening language to assess comfort level and attitude, proficiency levels for different technology, and level of technology integration (Computer Strategies, 1999). The section on basic computer use is a six-point scale based on the stages of concern. Management of files and resources is a four-point scale that reflects attitude and organizational skills. Productivity tools includes four-point scales for technology proficiency levels based on ISTE standards. Level One is non-use; Level Two is personal; Level Three is instructional; and Level Four is mentorship. Level of Integration is a six-point scale that reflects use of technology with students as part of the curriculum. Level of Integration ranges from Level One at which the teacher does not use technology with students to Level Six at which students create projects that expand the curriculum and teachers collaborate with other teachers in teams.

Fictitious ninth-grade English teacher Bill Smith filled out a rubric indicating that he is comfortable with technology and uses it for all of his written and oral communications and personal use. Yet he has trouble finding files that he saved on his computer (File Management level). Most of the work Bill does on the computer is word processing, e-mail, and Internet surfing. But he wants to learn how to import graphics, create a slideshow using PowerPoint, import data into a spreadsheet to create a chart, and design a Web site to use with his students.

Figure 2 depicts Bill’s current level of technology use, entered in a database created in FileMaker Pro. Bill was also asked where he would like to be by the end of the school year, which was recorded under “Desired Level.” At the end of the year, Bill was asked to input the actual levels he reached. See Figure 3 for Bill’s Individual Learning Plan.

The responses Bill gave to the rubric were processed using a formula in a FileMaker Pro database that combined several of the levels to determine the learning opportunities for him. These opportunities were drawn from the list developed for the school (see the section on learning opportunities). He used this form as a guide for workshops and meetings with his department or with a mentor to help him reach his goals. Because Bill used a PC for productivity, but he had trouble finding and organizing files, one suggestion was for him to meet with a mentor to review Windows 98 and answer questions on troubleshooting. Another teacher who is more advanced may have an ILP that points to online courses and collaborative planning time.

Reference

Computer Strategies. (1999). Staff development resources [Online document]. San Leandro, CA: Author. Available: www.compstrategies.com.

Back to the top

Rubrics and Surveys

Note. The Web sites listed in this page were valid when this issue of  L&L went to press. We have no control over these sites, though, and the Web is very volatile. Please let us know if you find a broken link, and we'll do our best to update it.

Several surveys and rubrics based on what teachers need to know about technology use for productivity, integration with students, and as leaders are available online. This list represents only a fraction of what's available.

One rubric is the Mankato Scale, which uses a four-point scale based on proficiency levels of productivity tools (see Bellingham, Washington Public Schools' Web site: http://www.bham.wednet.edu/technology/techself.htm).

The Milken Exchange on Educational Technology has a Professional Competency Continuum that you can refer to when designing an assessment tool (www.milkenexchange.org/projects.taf).

The North Central Regional Technology in Education Consortium (NCRTEC) has developed the Learning with Profile Technology Tool, a self-assessment survey to help teachers compare their current instructional practices with a set of indicators for engaged learning and high-performance technology (www.ncrtec.org/capacity/profile/profwww.htm).

The Utah Technology Awareness Project (wwwj1.uen.org/UTAP/) is an online comprehensive survey that provides multiple categories of use with four subcategories. When the user completes the survey, a chart with growth analysis and examples of classes appears.

Back to the top

Online Resources for Staff Development
and Training Opportunities

Note. The Web sites listed in this page were valid when this issue of  L&L went to press. We have no control over these sites, though, and the Web is very volatile. Please let us know if you find a broken link, and we'll do our best to update it.

Back to the top

Reference

Pappas, P. & Pearson, J. (1999, May/June). Take a skills snapshot: Employing online self-assessments. MultiMedia Schools [Online serial].

Figure 2. Bill Smith’s self-assessed levels of technology use and desired areas for growth. © 1999–2000, Computer Strategies, LLC. All rights reserved. Reprinted with permission.

Figure 3. Bill Smith’s Individual Learning Plan. © 1999–2000, Computer Strategies, LLC. All rights reserved. Reprinted with permission.

Back to the top

Copyright © 1999, ISTE (International Society for Technology in Education). All rights reserved.