Classroom
Observation Protocols:
Potential
Tools For Measuring The Impact Of Technology In The Classroom
by
Peter J. Dirr, Ph.D.
For the past decade or more, there has been a major national effort in the United States to introduce instructional technology to elementary and secondary school classrooms. By some estimates, federal, state, and local governments have invested about $8 billion per year (with half coming from the federal government) with a primary goal of providing access to instructional technology (computers, educational software, and instructional materials) to ALL students. As a result of this concentrated effort, today virtually all schools and most classrooms have computers and access to the Internet.
In recent years, the dynamics of government investment in educational technology have begun to change from a focus on access to looking at the impact that technology is having on classroom practices and how and how much students learn. Some policy makers have begun to question whether the investment made over the past decade is making a noticeable contribution to school improvement and student academic achievement. At the same time, the 2001 No Child Left Behind legislation requires that schools prove that the technologies being used in the classroom are contributing to improved teaching and learning. Throughout the legislation, emphasis is placed on “scientifically-based” educational practices, i.e., schools and districts are expected to employ practices and technologies that have been proven by research to be effective and, in turn, are expected to use sound research methods to document the impact that those practices and the technologies are having on teaching and learning within their classrooms.
This paper responds to critical issues raised by the policy makers and the 2001 legislation by exploring the potential of one class of measurement tools, classroom observation protocols, to scientifically document the impact that technology is having on how teachers teach and students learn (i.e., classroom practices).
Classroom observation protocols are not new. In the 1970s, there was a flurry of research on the topic and many measurement instruments were developed. Most of that research focused on tracking classroom management and the interactions that took place between teacher and students. That research was so vast that it filled approximately 15 hard-bound volumes that were published as an encyclopedia of classroom observation research, “Mirrors On Behavior.”
Today’s expectations for documentation are more complex than those of the 1970s. While the pioneering efforts of Ned Flanders, Egon Guba, and Hilda Taba, and others provide a useful foundation, researchers today are finding that even the best of the protocols and instruments developed in the 1970s require significant revisions to make them appropriate for the demands and expectations of the 21st Century classroom.
The purpose of this paper is to identify some of the more promising work on classroom observation protocols that is taking place in the United States today, especially those protocols that might be helpful in documenting changes in classroom practices as a result of using instructional technologies. The paper is based largely on a survey of classroom observation literature on the Internet and from telephone follow-up with researchers in selected cases. This paper is intended to give the leadership of the Appalachian Technology in Education Consortium (ATEC) and the other Regional Technology in Education Consortia (RTECs) a sense of the promise of classroom observation protocols, some of the key players in the field, and the prospects for this approach to contribute convincing documentation of the impact of the use of technology on teaching and learning.
Over the past several years, prodded particularly by the expectations and requirements of federally funded research projects (especially funding programs of the National Science Foundation and the U.S. Department of Education), many educators and researchers have developed various approaches to observing and documenting changes in classroom practices. These efforts have taken place at the elementary/secondary and higher education levels. This paper will focus on the former.
The Importance Of Purpose
One piece of advice that surfaces often in the literature is the importance of aligning the classroom observation protocol to the instructional context and the objectives of the evaluation. A school district might take different observation approaches for different programs within the district. Observation protocols might be helpful tools to:
Types Of Classroom Observation Protocols
The world of classroom observation protocols is complex and largely uncoordinated. Different researchers have taken different approaches. Some have built on the prior work of others, but there is little evidence of any attempt to bring cohesion to their efforts.
Some researchers have developed self-administered, reflective approaches for teachers to use as they introduce classroom reforms; others have developed third-party observation approaches. Some protocols focus on changes in attitudes; many address changes in classroom behaviors, especially the interactions among teachers and students.
Classroom observation methods include a wide variety of approaches: checklists, inventories, timed interval ratings, holistic ratings, narrative descriptions, logs, questionnaires, rubrics, matrices, models, conceptual grids, and open-ended questions.
One important consideration for researchers, addressed directly by some and implicitly by others, is what unit of instruction they will study. This is especially an important consideration for third-party observers. Will they observe an entire class period? A segment? An entire day? An entire course or program? Different observation protocols have taken different paths with this consideration.
Potential Pitfalls
Constructing a classroom observation protocol, especially third-party observer protocols, presents many challenges and potential pitfalls, including:
A Schema
While the variety of approaches to classroom observation protocols defies easy categorization, one might think of protocols that are self-administered vs. those that require third-party observers. A further split might be those that are primarily quantitative (e.g., checklists and inventories) vs. those that are primarily qualitative (i.e., requiring observers to make judgments and provide narrative descriptions) vs. those that have both quantitative and qualitative elements (e.g., rubrics that require the observer to convert judgments to ratings for different “levels” of behavior). Many of the protocols that have taken the quantitative or combined approaches to third-party classroom observation have also focused on training observers to a high confidence level in inter-rater reliability and, therefore, in the protocol itself.
Table 1
Exemplary Classroom Observation Protocols
|
|
|
|
|
LTPT
(NCREL/NCRTEC)(1) ATRL
(SCRTEC)(2) |
RTOP
(ACEPT)(5) CLASSROOM
OBSERVATION (Littleton)(6) SOM (U of
Memphis)(7) TOI
(WestEd)(8) TELAR (Penn
State)(9) TIC
(WestEd)(10) |
|
|
Qualitative (incl. Narrative descriptions) |
|
CLASSROOM
OBSERVATION (Sun Assoc.)(11) SFO
(SRI/MSU)(12) |
|
Combination (incl. Rubrics) |
UTAP (Utah
Education Network)(3) TX STaR (TX Ed. Agency)(4) |
COP
(Horizon Research, Inc.)(13) COP
(CETP/U. of MN)(14) TUOT
(WestEd)(15) VCOT (VT
Instit. of M/S)(16) TTSC (KY
Dept. of Ed.) (17) |
This paper is not a comprehensive listing of all the work being done in classroom observation protocols. Rather, the efforts shown in Table 1 have been selected because they meet the following criteria:
The selected protocols are described in more detail in the following section of this paper, noting the categories of classroom activities they include and whether or not they specifically address the question of the use of technology in the classroom.
Leading Classroom Observation Protocols
1. Instrument Name: LEARNING WITH TECHNOLOGY PROFILE TOOL (LTPT)
Developed By: North Central Regional Educational Laboratory/North Central Regional Technology in Education Consortium
Notes: A technology-specific, self-administered tool intended primarily for professional growth and development. Uses a 4-point scale to rate indicators of engaged learning (including: vision of learning, tasks, assessment, instructional model, learning context, grouping, teacher roles, and student roles) and indicators of high-performance technology (including: access, operability, organization, engageability, ease of use, and functionality). (Available: http://www.ncrtec.org/capacity/profile/profwww.htm)
2. Instrument Name: APPLYING TECHNOLOGY TO RESTRUCTURING AND LEARNING (ATRL)
Developed By: South Central Regional Technology in Education Consortium (SCRTEC) (K. Victoria Dimock)
Notes: An extensive study of how teachers use computers and technology in constructivist learning environments. Includes self-reporting instrument (the Teaching, Learning and Computing Teacher Survey) as well as a third-party Classroom Observation instrument. Looks at the use of technology in the context of constructivist learning environments. (Program description but not the instrument available: http://www.sedl.org/work/historical/tap.html)
3. Instrument Name: UTAH TECHNOLOGY AWARENESS PROJECT RUBRICS (UTAP)
Developed By: Utah Education Network
Notes: An online, self-administered instrument with a total focus on technology. Uses a 4-point scale to help teachers focus on their technology skills in seven areas: basic concepts, productivity, communication/information, classroom instruction, educational leadership, technology implementation, and technology troubleshooting. (Available: http://wwwj1.uen.org/UTAP, “Log-in As Guest”)
Developed By: Texas Education Agency(based on the CEO Forum/ISTE STaR Chart, also adapted by the Tennessee Department of Education)
Notes: A self-administered instrument that uses a 4-point rubric in four key areas: teaching and learning, educator preparation and development, administration and support services, and infrastructure for technology. This is a technology-specific instrument. (Available: http://www.tea.state.tx.us/technology/etac/campus_txstar and http://www.state.tn.us/education/tennesseestarchart.doc)
5. Instrument Name: REFORMED TEACHING OBSERVATION PROTOCOL (RTOP)
Developed By: Arizona Collaborative for Excellence in the Preparation of Teachers (ACEPT), Arizona State University (Michael Piburn and Daiyo Sawada)
Notes: No items are specifically “technology” oriented. Uses 5-point scale to rate lesson design, content, and classroom culture. Also requires narrative evaluations from observer. (Available: http://purcell.phy.nau.edu/AZTEC/rtop/RTOP_full, click on “Using RTOP”)
6. Instrument Name: CLASSROOM OBSERVATION
Developed By: Littleton Academy (Colorado)
Notes: Modified and simplified the Horizon protocol (see #13 below), boiling it down to a single page. Uses 4-point scale to rate instructional skills, knowledge of content and use of materials, focus on students, and classroom environment. Contains only two items on instructional materials and no items on technology. (Available: http://www.cde.state.co.us/cdechart/guidebook/adm)
7. Instrument Name: SCHOOL OBSERVATION MEASURE (SOM)
Developed By: Center for Research in Educational Policy (CREP), University of Memphis (S. M. Ross, L. J. Smith, and M. J. Alberg)
Notes: A single page instrument for assessing quality of an entire school. Requires extensive rater training. Uses timed observations (ten 15-minute observations in a single day) and a 5-point rating scale of 24 variables in six categories: instructional orientation, classroom organization, instructional strategies, student activities, technology use, and assessment. Two of the 24 variables deal with technology (use of computer for instructional delivery, and, technology as a learning tool or resource). This protocol also has a companion instrument, Survey of Computer Use (SCU) that records information about student use of computers over the same 15-minute intervals. Information includes computer availability, configurations, student abilities, and student activities while using computers. (Description of work, but not instruments, available: http://www.people.memphis.edu/~coe_crep/instruments/som.htm)
8. Instrument Name: TECHNOLOGY OBSERVATION INSTRUMENT (TOI)
Developed By: WestEd (Mike Timms) (for the Nebraska PT3 Catalyst Project)
Notes: Part of a broader assessment process that includes a pre-observation conference. Uses timed intervals (5 minutes) and a 4-point scale to rate: class organization, cognitive ability, classroom interaction, student role, student engagement, technology integration, teacher’s technology use, and students’ technology use. (Available: http://www.necatalyst.org/MTimms.tech.observ.instrmnt%20final1.pdf)
9. Instrument Name: TECHNOLOGY-ENHANCED LESSON ASSESSMENT RUBRIC (TELAR)
Developed By: Penn State University (Kyle Peck)
Notes: Designed to capture data by PDA, this application requires FileMaker V. Uses 3 to 5-point scales to rate: appropriateness of technology use, student readiness for tech use, student proficiency with the technologies, teacher proficiency with the technologies, student knowledge of lesson purpose, on-task behavior, quality of student interaction, teacher-student interaction, transitions to-and-from technology, teacher as facilitator, teacher as instructor, meeting diverse student needs, higher order thinking, tech problems, and student participation. The entire focus of this instrument is on technology use. (Description, but not the instrument, available: http://www.iu5.org/imts/ptla_web/pdf/KylesReadMe.pdf)
10. Instrument Name: TECHNOLOGY IN THE CLASSROOM TEACHER RUBRIC (TIC)
Developed By: WestEd (Harvey Barnett)
Notes: Developed as an aid for principals, the focus of this instrument is entirely on technology. Uses 4-point scale to rate many variables in five broad areas of technology use: productivity, multimedia and communication, research and problem solving, specific content/subjects, and teaching practice. (Available: http://www.westedrtec.org)
11. Instrument Name: CLASSROOM OBSERVATION
Developed By: Sun Associates
Notes: A single page instrument that uses a combination of narrative and 3-point scale to rate: classroom activity/lesson, student groupings and interactions, and technology and/or instructional materials used. Captures the number of computers in the classroom and other notes on the teaching environment. (Available: http://www.sun-associates.com/eval/clsobsv.html)
12. Instrument Name: STEP FACULTY OBSERVATION (SFO)
Developed By: SRI/Montana State University
Notes: A largely open-ended instrument that gathers assessment information on: instructional setting, teaching methods employed, instructional goals and objectives, evidence of student learning, student engagement, relevance to the real world, student and teacher questioning, and use of technology. Used mainly in teacher education programs at the university level. (Available: http://oerl.sri.com/instruments/te/obsvclassrm/instr76.html)
13. Instrument Name: CLASSROOM OBSERVATION PROTOCOL (COP)
Developed By: Horizon Research, Inc. (Refined with several educational organizations. See, for example, #6 above and #14 below.)
Notes: Part of a larger assessment process that includes a pre-classroom observation interview. Originally focused on math and science. Uses combination of checklists and 5-point scales to describe and rate: classroom demographics, lesson description, lesson purpose, instructional materials, classroom instruction, design, implementation, (math/science) content, and classroom culture. Also requires observer to provide overall rating and a “quality capsule.” Few items deal directly with technology or instructional materials. (Available: http://www.horizon-research.com/LSC/manual/0102/tab6/cop0102.pdf)
14. Instrument Name: CLASSROOM OBSERVATION PROTOCOL (COP)
Developed By: The Collaborative for Excellence in Teacher Preparation (CETP) in conjunction with Horizon Research, Inc. (used by the Texas CETP and also by the Univ. of Minnesota CETP)
Notes: Uses timed intervals (5 minutes) and 5-point scale to rate: type of instruction, student engagement, cognitive activity, and key indicators. Also requires observer to provide a “capsule description” of the quality of the lesson. Originally designed for math and science. Only reference to technology is the category “utilizes technology” in the “type of instruction.” (Available: http://www.sci.tamucc.edu/txcetp/admin/documents/ClassObProt2002.pdf)
15. Instrument Name: TECHNOLOGY USE: OBSERVATION TOOL (TUOT)
Developed By: WestEd (Harvey Barnett and Susan Brooks)
Notes: Uses a 5-point scale and narrative (“stages of use”) to rate thirty variables in three areas: learning environment, student technology use, and lesson implementation. The focus is entirely on technology use. (Available: http://www.westedrtec.org)
16. Instrument Name: VERMONT CLASSROOM OBSERVATION TOOL (VCOT)
Developed By: VT Institute for Science, Math, & Technology (VISMT), in conjunction with Education Development Center (EDC)/ Northeast and Islands Regional Technology in Education Consortium (NEIRTEC), and VITA-Learn (modeled after work done by the Science and Math Program Improvement (SAMPI) at Western Michigan University and Horizon Research, Inc.)
Notes: Part of a broader assessment process that includes a pre-observation interview and a post-observation interview. Focuses on four areas: planning and organization of a lesson, implementation of a lesson, content of a lesson, and classroom culture. Imbeds technology criteria within those four areas. (Instrument is closely held by the developers and released only as part of the required extensive training. It was not available for review to determine how extensively technology criteria are covered.) (Description of the program, but not the instrument, available: http://www.vismt.org/programs/leadership/vcot.html)
17. Instrument Name: TEACHER TECHNOLOGY STANDARD CONTINUUM (TTSC)
Developed By: Kentucky Department of Education, Division of School Instructional Technology
Notes: While not developed strictly as a classroom observation protocol, this scoring guide (Rubric) provides a continuum of technology skills which administrators (e.g., principals) can use to track teachers’ accomplishment of the Kentucky Teacher Technology Standard. Uses a 4-point scale plus commentary in sixteen areas of technology knowledge and application (e.g., knows technology terminations; knows how technology is used in business and the community; knows how to use the computer and peripherals; uses word processing, databases, e-mail, and presentation software to enhance productivity and support learning; creates multimedia presentations; uses assistive technologies for students with special needs; encourages lifelong learning through technology; uses technology for individual, small group, and large group instruction; uses technology to assess student learning; instructs students in ethical and legal uses of technology. The KY Department of Education also makes available a Technology Impact Review Tool, a set of interview questions for all stakeholders to determine how effective a district’s or school’s use of technology is. (Available: http://www.kde.state.ky.us/oet/customer/evaluation.asp)
The purpose of this paper has been to identify several classroom observation protocols that can be used to build a body of research-based evidence, consistent with the expectations of the No Child Left Behind legislation, on what effect the use of technology has on classroom practices.
A review of literature revealed several dozen observation protocols that have been developed in recent years. A few of these are rooted in the extensive work done by researchers in the 1970s. Some are self-administered and might be used by teachers to reflect on their classroom practices, including the use of technology. Others are third-party observer protocols that have been developed to provide an objective assessment of the quality of classroom practices.
While the self-administered instruments might be valuable tools for teachers and might, in fact, lead to improved classroom practices, they are not particularly helpful for the purposes of this paper. They are not likely to lead to scientifically-based research evidence that would withstand professional scrutiny.
Third-party observer protocols hold the greatest promise for building a body of research evidence on the effectiveness of the use of technology in the classroom. They are more likely than self-administered instruments to:
Some of the instruments reviewed in this study are well developed and carefully tested and are effective for documenting the classroom environment in general. They include:
These might be particularly appropriate for assessing changes in classroom dynamics when constructivist approaches are used by teachers. However, the attention they pay to the role of technology is so small that they are likely to overlook its contribution to the classroom.
Other instruments are focused more on technology but might lack the necessary research base to be ready for widespread use. These include:
They deserve further consideration and might be developed into useful instruments with further testing and validation.
Two other efforts are quite far along in the development process and deserve careful consideration. They are:
This paper responds to critical issues raised by policy makers about the effectiveness of the government’s investment in educational technology over the past decade. It suggests that the development and systematic use of a classroom observation protocol focused on the role of technology in the classroom could contribute significantly to building a body of evidence of changes in classroom practices (i.e., how teachers teach and students learn) as a result of technology in the classroom. That would be the first of a two-step process for measuring the impact of technology in education. The second step would be to measure the effect that technology has on student academic achievement. That is an ambitious undertaking that will require a concentrated effort and at least five years of funding. It would necessarily include serious consideration of what student achievements should be measured, taking into account the nascent work that is now being done to articulate a body of 21st Century Skills.
Both the tracking of changes in classroom practices and testing the impact on student academic achievement are consistent with the expectations of the No Child Left Behind legislation for research-based educational practices. They are also consistent with the Education Sciences Reform Act of 2002 which established the Institute of Education Sciences, including the National Center for Education Research (NCER). One of the missions of NCER is to “improve student academic achievement through the use of educational technology.” The Center will be funding field-initiated research that follows “scientifically-based research standards.” Assessing the impact of technology on classroom practices and student academic achievement should be of great interest to the new Center.
Appendix
Contacts:
1. LEARNING WITH TECHNOLOGY PROFILE TOOL
Kristin Ciesemier, Director
North Central Regional Technology in Education Consortium
1120 East Diehl Road, Suite 200
Naperville, IL 60563-1486
(630) 649-6500 (P)
(630) 649-6700 (F)
2. TECHNOLOGY INTEGRATION PHASES
K. Victoria Dimock
Director, South Central Regional Technology in Education Consortium
Southwest Educational Development Laboratory
211 E 7th St.
Austin, TX 78701
(512) 476-6861, x 219 (P)
3. UTAH TECHNOLOGY AWARENESS PROJECT RUBRICS
Steve Hess
Utah Education Network
101 Wasatch Dr.
Salt Lake City, UT
(801) 581-2310 (P)
4. TEXAS STaR CHART
Anita Givens, Senior Director
Educational Technology Advisory Committee
Texas Education Agency
Educational Technology Division
1701 North Congress Avenue
Austin, TX 78701
(512) 463-9400 (P)
(512) 463-9090
TENNESSEE STaR CHART
Jerry Bates, Director of Applied School Technology
710 James Robertson Parkway
Andrew Johnson Tower, 6th Floor
Nashville, TN 37243-0382
(615) 532-6287 (P)
(615) 532-7510 (F)
5. REFORMED TEACHING OBSERVATION PROTOCOL
Mike Piburn
AZ Collaborative for Excellence in the Preparation of Teachers (ACEPT)
College of Education
Arizona State University
Tempe, AZ 85287-0911
602-965-0261 (P)
602-965-9144 (F)
6. CLASSROOM OBSERVATION
Littleton Academy
Mary Ann Campbell, Principal
1200 West Mineral Ave.
Littleton, CO 80120
(303) 798-5252 (P)
(303) 798-0298 (F)
7. SCHOOL OBSERVATION MEASURE
Steven M. Ross
Center for Research in Educational Policy
University of Memphis
325 Browning Hall
Memphis, TN 38152-3340
(901) 678-2310 (P), or,
(866) 670-6147 (F)
8. TECHNOLOGY OBSERVATION INSTRUMENT
Mike Timms
WestEd
730 Harrison St.
San Francisco, CA 94107-1242
(415) 615-3210 (P)
(415) 615-3426 (F)
9. TECHNOLOGY-ENHANCED LESSON ASSESSMENT RUBRIC
Kyle Peck
Education Department
Pennsylvania State University
411D Keller Building
University Park, PA 16802
(814) 863-4316 (P)
(814) 865-0128
10. TECHNOLOGY IN THE CLASSROOM TEACHER RUBRIC
Harvey Barnett
WestEd
730 Harrison St.
San Francisco, CA 94107-1242
(415) 615-3116 (P)
(415) 615-3177 (F)
11. CLASSROOM OBSERVATION
Jeff Sun
Sun Associates
55 Middlesex Street
Suite 216
North Chelmsford, MA 01863
(978) 251-1600 (P)
(978) 251-8700 (F)
12. STEP FACULTY OBSERVATION
Elisabeth Swanson
Montana STEP Project Administrative Office
Montana State University
401 Linfield Hall
Bozeman, MT 59717
(406) 994-6768 (P)
13. CLASSROOM OBSERVATION PROTOCOL
Iris R. Weiss
Horizon Research Inc.
326 Cloister Ct.
Chapel Hill, NC 27514
(919) 489-1725 (P)
(919) 493-7589 (F)
14. CLASSROOM OBSERVATION PROTOCOL
Frances Lawrenz
University of Minnesota
Burton Hall
178 Pillsbury Dr. SE
Minneapolis, MN 55455
(612) 625-2046 (P)
15. TECHNOLOGY USE: OBSERVATION TOOL
Harvey Barnett
WestEd
730 Harrison St.
San Francisco, CA 94107-1242
(415) 615-3116 (P)
(415) 615-3177 (F)
16. VERMONT CLASSROOM OBSERVATION TOOL
Nicole Saginor
Associate Executive Director
Vermont Institute for Science, Math, & Technology
7 West Street
Montpelier, VT 05602
(802) 828-0068 (P)
(802) 828-0076 (F)
nsaginor@vermontinstitutes.org
17. TEACHER TECHNOLOGY STANDARD CONTINUUM
Lydia Wells-Sledge
Director
Division of School Instructional Technology
Kentucky Department of Education
500 Mero Street, 19th Floor
Frankfort, KY 40601
(502) 564-7168 (P)
(502) 564-6470 (F)