GUIDE BOOK TO EXAMINE SCHOOL CURRICULA
The Project 2061 Curriculum-Analysis Procedure
(Reprinted here from the TIMSS Resource Kit)
Introduction
Deciding which curriculum materials to use is one of the most important professional
judgments that educators make. Textbook adoption committees make recommendations
that influence instruction for years to come, and the daily decisions teachers
make about which teaching units or chapters to useand how to use themlargely
determine what and how students will be expected to learn.
Such important decisions require a valid and reliable method for evaluating
the quality of curriculum materials. Even an in-depth review of the topics
covered by a textbook or a teaching unit may not be sufficient to determine
whether the material will actually help students learn that content. What
is needed is a manageable process for examining curriculum materials that
gets below the surface by focusing intensely on the appropriateness of
content and the utility of instructional design.
With funding from the National Science Foundation and in collaboration with
hundreds of K-12 teachers, curriculum specialists, teacher educators, scientists,
and materials developers, Project 2061 of the American Association for the
Advancement of Science (AAAS) has been developing a process for analyzing
curriculum materials. Field tests suggest that Project 2061's curriculum-analysis
procedure will not only serve the materials adoption needs of the schools
but also help teachers revise existing materials to increase their effectiveness,
guide developers in the creation of new materials, and contribute to the professional
development of those who use it.
Specific Learning Goals Are Key
Until recently, there was nothing against which to judge appropriateness of
content and utility of instructional design. Now, as a result of the standards-based
reform movement in education, these judgments can be made with a high degree
of confidence. In science and mathematics, for example, the appearance of
Science for All Americans (AAAS, 1989), Curriculum and Evaluation
Standards for School Mathematics (National Council of Teachers of Mathematics,
1989), Benchmarks for Science Literacy (AAAS, 1993), and National
Science Education Standards (National Research Council, 1996) has made
it possible to make more thoughtful decisions about curriculum materials than
ever before.
Although the Project 2061 curriculum-analysis procedure was developed using
the learning goals in Benchmarks and the mathematics and science standards,
subsequent work has indicated that some state education frameworks also can
be used. Indeed, the process would seem to apply to any K-12 school subject
for which specific learning goals have been agreed upon. These goals must
be explicit statements of what knowledge and skills students are expected
to learn, and they must be precise. Vague statements such as "students
should understand energy" are not adequate. Instead, consider this benchmark
dealing with energy-related concepts that students should know by the end
of the eighth grade:
Most of what goes on in the universefrom exploding stars and biological
growth to the operation of machines and the motion of peopleinvolves some
form of energy being transformed into another. Energy in the form of heat
is almost always one of the products of an energy transformation.
Similar explicit statements can be found in the fundamental concepts of the
National Research Council's National Science Education Standards (NSES).
At its simplest level, the Project 2061 curriculum-analysis procedure involves
the following five steps:
- Identify specific learning goals to serve as the intellectual
basis for the analysis. This is done before beginning to examine any
curriculum materials. The source for appropriate goals can be national
standards or benchmark documents such as those mentioned above, state
or local standards and curriculum frameworks, or sources like them. To
be useful, the goals must be precise in describing the knowledge or skills
they intend students to have. If the set of goals is large, a representative
sample of them should be selected for purposes of analysis.
- Make a preliminary inspection of the curriculum materials to see whether
they are likely to address the targeted learning goals. If there appears
to be little or no correspondence, the materials can be rejected without
further analysis. If the outlook is more positive, go on to a content
analysis.
- Analyze the curriculum materials for alignment between content and
the selected learning goals. The purpose here is to determine, citing
evidence from the materials, whether the content in the material matches
specific learning goalsnot just whether the topic headings are similar.
At the topic level, alignment is never difficult, since most topicsheredity,
weather, magnetism, and so forthlack specificity, making them easy to
match. If the results of this analysis are positive, then reviewers can
take the next step.
- Analyze the curriculum materials for alignment between instruction
and the selected learning goals. This involves estimating the degree
to which the materials (including their accompanying teacher's guides)
reflect what is known generally about student learning and effective teaching
and, more important, the degree to which they support student learning
of the specific knowledge and skills for which a content match has
been found. Again, evidence from the materials must be shown.
- Summarize the relationship between the curriculum materials being evaluated
and the selected learning goals. The summary can take the form of
a profile of the selected goals in terms of the content and instruction
criteria, or a profile of the criteria in terms of the selected goals.
In either case, a statement of strengths and weaknesses should be included.
With this information in hand, reviewers can make more knowledgeable adoption
decisions and suggest ways for improving the examined materials.
In addition to its careful focus on matching content and instruction to very
specific learning goals, the Project 2061 procedure has other features that
set it apart. For example, its emphasis on collecting explicit evidence (citing
page numbers and other references) of a material's alignment with learning
goals adds rigor and reliability to decisions about curriculum materials.
Similarly, the Project 2061 procedure calls for a team approach to the analytical
task, thus providing opportunities for reviewers to defend their own judgments
about materials and to question those of other reviewers. These and other
characteristics help make participation in the analytical process itself a
powerful professional development experience.
The Project 2061 Curriculum-Analysis Procedure in Detail
To provide a better sense of how the procedure works, the following describes
in more detail each step in the procedure, using learning goals from Project
2061's Benchmarks for Science Literacy as an illustrative frame of
reference. The description pays particular attention to the various criteria
used to evaluate the instructional effectiveness of materials.
Identify specific learning goals to serve as the intellectual basis for the
analysis.
After reviewers have agreed upon a set of learning goals as a framework for
the analysis (in this case, the benchmarks in Benchmarks for Science Literacy),
the task is then to choose specific benchmarks that will serve as the focus
of further study.
When evaluating standalone curriculum units that cover a relatively short period
of time, it might be possible and worthwhile to analyze all of the
benchmarks that appear to be targeted by the material. However, in the evaluation
of year-long courses or multiyear programs, this becomes impractical. Therefore,
a crucial step in the analysis procedure is the sampling of a few benchmarks
that will lead to valid and reliable generalizations about the material.
Sampling of benchmarks should be representative of the whole set of goals specified
in Benchmarks for Science Literacy and should reflect the reviewers'
needs. For example, if the review committee's task is to select a course in
high school Biology that is aligned with Benchmarks, it might identify
a sample of benchmarks from life science sections in Benchmarks (e.g.,
cells, heredity, and evolution) and from other sections (e.g., nature of scientific
inquiry, models, and communication skills). When examining middle-school science
materials, one would probably want to broaden the range of benchmarks examined
to include some from physical and earth science topics (e.g., energy, forces,
and processes that shape the earth).
Make a preliminary inspection of the curriculum materials to see whether they
are likely to address the targeted learning goals.
Once benchmarks have been selected, the next step is to make a first pass at
the materials to identify those whose content appears to correspond reasonably
well to Benchmarks. Materials that do not meet these initial criteria
are not analyzed further.
Reviewers then examine materials on the shortened list more carefully to locate
and record places where each selected benchmark seems to be targeted (e.g.,
particular readings, experiments, discussion questions). If several sightings
are found for some or all of the sample benchmarks in the material, then these
sightings will be looked at more carefully in subsequent steps of the analysis.
If, on the other hand, sightings cannot be found for a significant number
of the sample benchmarks, then the material is dropped from the list.
Analyze the curriculum materials for alignment between content and the selected
learning goals.
This analysis is a more rigorous examination of the link between the subject
material and the selected learning goals and involves giving precise attention
to both ends of the matchthe precise meaning of the benchmark on one end and
the precise intention of the material on the other.
With respect to each of the sampled benchmarks, the material is examined using
such questions as:
- Does the content called for in the material address the substance of a
specific benchmark or only the benchmark's general "topic"?
- Does the content reflect the level of sophistication of the specific benchmark,
or are the activities more appropriate for targeting benchmarks at an
earlier or later grade level?
- Does the content address all parts of a specific benchmark or only some?
(While it is not necessary that any particular unit would address all
of the ideas in a benchmark or standard, the K-12 curriculum as a whole
should do so. The purpose of this question is to provide an account of
precisely what ideas are treated.)
In addition, an attempt is made to estimate the degree of overlap between the
material's content and the set of benchmarks of interest. Thus, this step
in the analysis is designed to answer questions regarding the material's inclusion
of content that is not required for reaching science literacy and the extent
to which the material distinguishes between essential and non-essential content.
(While distinguishing content essential for literacy from non-essential content
in material might seem to be a luxury, it assists teachers in determining
the range of students for which the material can be used. Identifying the
non-essential material makes it easier for the teacher to direct better students
to enrichment activities and allows students themselves to avoid overload
from ideas that go beyond what is vital.)
Analyze the curriculum materials for alignment between instruction and the
selected learning goals.
The purpose here is to estimate how well material addresses targeted benchmarks
from the perspective of what is known about student learning and effective
teaching. The criteria for making the judgments in the instructional analysis
are derived from research on learning and teaching and on the craft knowledge
of experienced educators. In the context of science literacy, these are summarized
in Chapter 13, "Effective Learning and Teaching," of Science
for All Americans; in Chapter 15, "The Research Base," of
Benchmarks for Science Literacy; and in Chapter 3, "Science Teaching
Standards," of National Science Education Standards.
From these sources, seven criteria clusters (shown below) have been identified
to serve as a basis for the instructional analysis (for the specific questions
within each cluster, see Appendix on page 137). The proposition here is that
(1) the analysis would tie the instruction to each one of the sample benchmarks
rather than look at instructional strategies globally and (2) in the ideal,
all questions within each cluster would be well-addressed in any materialthey
are not alternatives.
- Cluster I. Providing a Sense of Purpose: Part of planning a coherent
curriculum involves deciding on its purposes and on which learning experiences
will likely contribute to those purposes. But while coherence from the
curriculum designers' point of view is important, it may not give students
an adequate sense of what they are doing and why. This cluster includes
criteria to determine whether the material attempts to make its purposes
explicit and meaningful to students, either by itself or by instructions
to the teacher.
- Cluster II. Taking Account of Student Ideas: Fostering better understanding
in students requires taking time to attend to the ideas they already have,
both ideas that are incorrect and ideas that can serve as a foundation
for subsequent learning. Such attention requires that teachers be informed
about prerequisite ideas/skills needed for understanding a benchmark and
what their students' initial ideas arein particular, the ideas that may
interfere with learning the scientific information. Moreover, teachers
can help address students' ideas if they know what is likely to work.
This cluster examines whether the material contains specific suggestions
for identifying and relating to student ideas.
- Cluster III. Engaging Students with Phenomena: Much of the point
of science is explaining phenomena in terms of a small number of principles
or ideas. For students to appreciate this explanatory power, they need
to have a sense of the range of phenomena that science can explain. "Students
need to get acquainted with the things around themincluding devices, organisms,
materials, shapes, and numbersand to observe them, collect them, handle
them, describe them, become puzzled by them, ask questions about them,
argue about them, and then try to find answers to their questions."
(Science for All Americans, p. 201) Furthermore, students should
see that the need to explain comes up in a variety of contexts.
- Cluster IV. Developing and Using Scientific Ideas: Science for
All Americans includes in its definition of science literacy a number
of important yet quite abstract ideas (e.g., atomic structure, natural
selection, modifiability of science, interacting systems, common laws
of motion for earth and heavens). Such ideas cannot be inferred directly
from phenomena, and the ideas themselves were developed over many hundreds
of years as a result of considerable discussion and debate about the cogency
of theory and its relationship to collected evidence. Science literacy
requires that students see the link between phenomena and ideas and see
the ideas themselves as useful. This cluster includes criteria to determine
whether the material attempts to provide links between phenomena and ideas
and to demonstrate the usefulness of the ideas in varied contexts.
- Cluster V. Promoting Student Reflection: No matter how clearly
materials may present ideas, students (like all people) will assign their
own meanings to them. Constructing meaning well is aided by having students
(1) make their ideas and reasoning explicit, (2) hold them up to scrutiny,
and (3) recast them as needed. This cluster includes criteria for determining
whether the material suggests how to help students express, think about,
and reshape their ideas to make better sense of the world.
- Cluster VI. Assessing Progress: There are several important reasons
for monitoring student progress toward specific learning goals. Having
a collection of alternatives can ease the creative burden on teachers
and increase the time available to analyze student responses and make
adjustments in instruction based on those responses. This cluster includes
criteria for evaluating whether the material includes a variety of goal-relevant
assessments.
- Cluster VII. Enhancing the Learning Environment: Many other important
considerations are involved in the selection of curriculum materialsfor
example, the help they provide to teachers in encouraging student curiosity
and creating a classroom community where all can succeed, or the material's
scientific accuracy or attractiveness. The criteria listed in this cluster
provide reviewers with the opportunity to comment on these and other important
features.
Summarize the relationship between the curriculum materials being evaluated
and the selected learning goals.
In the preliminary inspection, a few benchmarks were selected as representative
of the set of goals that the material appears to target. Having analyzed whether
the content in the material matches these specific benchmarks and how well
the instructional strategies in the material support students learning these
benchmarks, the final step in the process is to provide a profile of the material
based on this analysis.
The analysis makes it possible to produce two sets of profiles. The first illustrates
how well the material treats each benchmark (for which a content
match was found) across all criteria examined in the instructional analysis.
Based on these profiles, conclusions can be made about what the material under
consideration can be expected to accomplish in terms of benchmarks. For
example, the profiles may indicate that the material treats one of the examined
benchmarks well and the rest only moderately or poorly.
The second set of profiles illustrates how well the material meets each criterion
in the instructional analysis tool across all benchmarks examined. These profiles
point to major strengths and weaknesses in the instructional design of the
material. For example, the profiles may indicate that the material consistently
includes appropriate experiences with phenomena relevant to the benchmarks
but only occasionally provides students with opportunities to reflect on these
experiences. Depending on the time available and their interests, a review
committee could decide to produce either one or both sets of profiles. Profiles
of different materials provide the basis for selection decisions.
Support for Users
Project 2061 is in the process of developing "Resources for Science Literacy:
Curriculum Evaluation," a CD-ROM that will offer full instruction
in using the procedure. The CD-ROM and its print companion volume will contain
(1) detailed instructions for evaluating curriculum materials in light of
Benchmarks, national standards, or other learning goals of comparable
specificity; (2) case-study reports illustrating the application of the analysis
procedure to a variety of curriculum materials; (3) a utility for relating
findings in the case-study reports to state and district learning goals; and
(4) a discussion of issues and implications of using the procedure.
Project 2061 also offers introductory workshops and longer training institutes
to groups of educators. Typically three to six days long, the training institutes
can be adapted to suit a variety of needs and time constraints. The project
has offered customized workshops for K-12 science and mathematics teachers,
teacher educators, school and university administrators, developers of curriculum
materials, and policy makers. Depending on the interests of participants,
the workshops can focus on understanding learning goals, selecting materials,
revising existing materials, or evaluating curriculum frameworks, among other
possibilities.
For information on Project 2061's workshops and institutes (or any aspects
of Project 2061's work) contact Mary Koppal, Project 2061, American Association
for the Advancement of Science (see the Guide to Using the Methods of Analysis
section of this Guidebook for contact information).
Putting the Project 2061 Curriculum-Analysis Procedure to Work
Many of the educators involved in developing and field testing the Project
2061 procedure have begun to use it to decide on materials for their classrooms,
school districts, or states; to identify shortcomings in materials they are
using; and to suggest ways to improve them. Here are a few examples of how
educators have adapted the procedure to their local needs and time constraints:
San Antonio.
Faced with the task of selecting a new high-school physical science textbook
from five possible choices, a San Antonio school district committee requested
training in the Project 2061 curriculum-analysis procedure. Already familiar
with Benchmarks for Science Literacy and Science for All Americans,
these 12 educators spent two days studying Project 2061's analytical criteria,
as well as some additional criteria decided locally. Committee members then
evaluated one material apiece, with at least two committee members independently
evaluating each material. When finished with their independent evaluations,
those educators reviewing the same material met to compare their results and
to come to an agreement about the value of the material. Then, about three
weeks after the initial training, the whole group reconvened to share their
results and settle on the material. Because the evaluation procedure requires
reviewers to cite evidence for judgments made, the reviewers were prepared
to justify their recommendations, pointing to specific instructional strategies
for particular learning goals in physical science.
After much discussion, the reviewers reached agreement on one material for
the district. Throughout the process, the reviewers were very reflective and
motivated by the work at hand. In fact, because the evaluation procedure had
revealed some weaknesses even in the material they agreed to select, they
decided to write a supplemental unit on one topic.
Philadelphia.
The Philadelphia school district was already committed to teaching toward specific
learning goals derived from Benchmarks and National Science Education
Standards when it set out to identify materials that are aligned with
those goals. Their list of possibilities included some materials developed
through National Science Foundation funding and some materials that had been
favorably evaluated by the Project 2061 pilot and field test participants.
The district held training institutes to introduce teachers to the evaluation
procedure and to develop evaluation skills that they would use to select materials
from the list of possibilities. More than 200 teachers participated in the
institutes, giving the district a cadre of leaders who could assist in the
school-based selection of curriculum materials.
After employing the procedure to select materials for use in their classrooms,
teachers planned to make a more thorough evaluation of the materials when
they eventually put them to use. Findings from the procedure also will be
used to improve materials currently being implemented in district classrooms.
Such remedies might include developing questions to focus students' reflection
on benchmark ideas, adding activities to address student learning difficulties,
or demonstrating how benchmark ideas are useful for making sense of the students'
world outside the classroom.
Through its work with the Project 2061 procedure, Philadelphia has developed
a group of educators who are becoming more knowledgeable about specific learning
goals in Benchmarks and the National Science Education Standards
and about the analysis criteria used to judge materials in light of these
goals. As new, better aligned materials become available, the district will
have a cadre of informed consumers who can recognize them and put them to
work. Most important, district classrooms will reflect teaching and learning
that engage all students in achieving science literacy goals essential for
the 21st century.
Kentucky.
In the fall of 1996, Project 2061 began to work with the director of the Kentucky
Middle Grades Mathematics Teacher Network to adapt the project's curriculum
analysis procedure to mathematics. The Kentucky Network, which already reaches
some 2,000 teachers, aims to align the state's mathematics content and teaching
practices in fifth through eighth grades with the recommendations of the National
Council of Teachers of Mathematics' Curriculum and Evaluation Standards
for School Mathematics. In particular, the network helps to train teachers
in reviewing and analyzing curriculum materials so that they can (1) discriminate
between materials that only claim to align with the mathematics standards
and those that actually do and (2) recognize standards in the newer, integrated
mathematics curricula.
As the criterion for alignment, Project 2061 has used Kentucky's Mathematics
Core Content for Assessment (which elaborates the national mathematics
standards into more specific goal statements) to analyze five middle-school
curriculum projects funded by the National Science Foundation (NSF). While
developing the analysis procedure and applying it to the materials, Project
2061 received continual feedback from Kentucky teachers and from a national
advisory committee that included the developers of the NSF-funded curricula.
During a 1997 two-week summer workshop, 32 Kentucky teachers used the analysis
procedure and case-study reports to examine middle-grade mathematics materials
and develop workshops for teachers throughout the state. In doing so they
(1) gained a better understanding of integrated, problem-based mathematics
curricula; (2) developed the skills necessary to evaluate mathematics curricula
in light of specific learning goals; and (3) developed skills necessary to
effectively share what they have learned throughout their regions. The workshop
participants worked during the 1997-1998 school year with teachers, schools,
and districts in their regions to assist in analyzing and evaluating mathematics
curriculum materials.
1998. Guide Book To Examine School Curricula: The Project 2061 Curriculum-Analysis
Procedure.
Reprinted from the TIMSS Resource Kit
