Reprinted here with the permission of the Eisenhower National Clearinghouse. No further republication or redistribution is permitted without the written permission of the editor.

Source:
ENC Focus, 2001 - Volume 8, Number 3

Putting Textbooks to the Test

Project 2061's evaluation of textbooks identifies the qualities and features that help all students learn the ideas and skills that make up mathematics and science literacy.

by Jo Ellen Roseman, Gerald Kulm, and Susan Shuttleworth, Project 2061, American Association for the Advancement of Science

In the United States, textbooks have an enormous influence on what is taught in K-12 mathematics and science classes and how it is taught. Research shows that a majority of teachers use textbooks as their principal curriculum guide and source of lessons (St. John, 2001). New and inexperienced teachers, or those who lack adequate time for lesson planning, may actually teach from the first page of the textbook to the last, skipping little or nothing (Tyson, 1997).

Since this is the case, it is imperative that textbooks provide the right content and instructional support. Texts must cover the key mathematics and science ideas that students need for literacy in those areas. Texts also must provide research-based instructional strategies that teachers can use to help students learn those ideas.

In 1998, after developing and field testing a rigorous procedure for analyzing curriculum materials, Project 2061 of the American Association for the Advancement of Science applied the procedure to middle- and high-school textbooks to see how well they align with standards and how well they help students achieve them. This study probed beyond a superficial analysis of alignment by topic heading and examined each text's quality of instruction aimed specifically at key standards and benchmarks, using criteria drawn from the best available research about what helps students learn.

The results were eye opening. Out of 45 texts analyzed (13 middle-grades mathematics texts, 12 algebra texts, 10 middle-grades science texts, and 10 high school biology texts) only five (four middle-grades math texts and one stand-alone physical science unit) were found to be satisfactory, that is, having a high potential for helping students learn ideas that are essential for mathematics and science literacy. Seven of the algebra texts were borderline, considered barely adequate for learning. The rest of the math and science texts were found to be unsatisfactory with little potential for helping students learn important ideas and skills. Visit the Project 2061 web site for a list of the books examined and an explanation of the analysis procedures.

What's Wrong with Today's Texts?

In 1964 the Nobel Prize-winning physicist Richard Feynman reported on his experiences as an advisor to a California textbook selection committee: "...something would look good at first and then turn out to be horrifying.... (the books) said things that were useless, mixed-up, ambiguous, confusing, and partially incorrect. How anybody can learn science from these books, I don't know, because it's not science." Many of these problems remain.

Today's textbooks cover too many topics without developing any of them well. Central concepts are not covered in enough depth to give students a chance to truly understand them. While many textbooks present the key ideas described in national and state standards documents, few books help students learn the ideas or help teachers teach them well. For example, Project 2061's analysis of high school biology texts revealed the following problems:

Research shows that essentially all students-even the best and the brightest-have predictable difficulties grasping many ideas that are covered in the textbooks. Yet most books fail to take these obstacles into account in the activities and questions.
For many biology concepts, the textbooks ignore or obscure the most important ideas by focusing instead on technical terms and superfluous detail- the sorts of material that translate easily into items for multiple choice tests.
While most of the books are lavishly illustrated, these representations are rarely helpful because they are too abstract, needlessly complicated, or inadequately explained.
Even though several activities are included in every chapter, students are given little guidance in interpreting the results in terms of the scientific concepts to be learned.

Project 2061's evaluation of textbooks also helped identify what works-those qualities and features that help all students learn the ideas and skills that make up science and mathematics literacy.

Characteristics of Effective Materials

Project 2061's evaluation organized the instructional characteristics of effective materials into seven broad categories and rated the materials against specific criteria within each category. Examples drawn from the highly rated Connected Mathematics, a series developed for grades 6 through 8, and Matter and Molecules, a stand-alone physical science unit developed at Michigan State University, provide some examples of what effective materials do in a few of the instructional criteria categories:

Taking Account of Student Ideas

To help students gain a better understanding of key concepts and skills, textbooks need to help teachers attend to the ideas that students already have, both ideas that are incorrect and those that can serve as a foundation for subsequent learning.

Every investigation in Connected Mathematics has a section called "Teaching the Investigation," which helps the teacher identify ideas that students may bring to the lesson and offers suggestions on how to address these ideas. The text also reminds teachers of prerequisite knowledge and skills at the beginning of most investigations. For example, before students make parallelograms from triangles, they are reminded of an earlier unit that taught that triangles have rigid structures.

Matter and Molecules lists relevant misconceptions that have been reported in student learning research, explains each one and why it makes sense to many students, and describes how students who hold the misconceptions are likely to respond to probing questions. This helps teachers diagnose their own students' learning difficulties.

While some textbooks have notes in the teacher's guide labeled "Misconceptions" or "Prior Knowledge," these statements often provide little useful information. If teachers (or perhaps even the textbook developers themselves) are unaware of common student misconceptions, then it will be hard to plan effective instruction.

Engaging Students with Relevant Contexts, Experiences, and Phenomena

Much of the point of studying mathematics is to appreciate the range of ideas and applications the ideas can model. To help students gain this kind of appreciation, Connected Mathematics uses a variety of contexts-from visual models to symbolic representations of hands-on activities and firsthand experiences-to build formal ideas and skills. Number experiences include using fraction strips, cooking, using thermometers and number lines, and exploring consumer issues. For algebra, students are engaged in data collection and using graph paper and calculators. In the geometry units, they use area models, grid paper, rulers, and square tiles.

In science, students need opportunities to relate the concepts they are studying to a range of phenomena either directly or vicariously. Matter and Molecules engages students in directly observing physical science phenomena and reading vivid descriptions of other phenomena related to the kinetic molecular theory:

When air is compressed in a syringe, it pushes back on the plunger.
Gases (like perfume) spread out evenly in a room or container.
Liquids (like food coloring or tea) spread out evenly in a glass, rather than falling to the bottom.

These experiences help students see how the idea that "atoms and molecules are perpetually in motion" offers a plausible explanation for the phenomena.

Promoting Student Thinking About Phenomena, Experiences, and Knowledge

For optimal learning to take place, textbooks also need to help students make sense of their experiences and ideas. Textbooks that provide carefully chosen and sequenced questions and tasks can help students reflect on, clarify, and explain their reasoning and ideas.

To help students understand what it means for figures to be mathematically similar, for example, Connected Mathematics engages students in using a rubber band to make enlargements of drawings. As they work on the investigation, students are asked to consider which figures remain the same or change when a figure is enlarged, to compare how the figures differ from one another, and to explain their judgments.

Connected Mathematics routinely emphasizes the need for students to explain their answers, ideas, and solutions with the class or small group. It also provides teachers with suggestions on how to engage students in class discussion. A self-assessment page at the end of each unit encourages students to think about what they've learned.

Matter and Molecules provides question sequences to help students interpret their activities. The questions are structured carefully to lead students step-by-step from one insight to another. Questions frame important issues, help students relate their experiences with phenomena to the scientific ideas presented, or prompt students to contrast common misconceptions with their scientific alternatives. For example, to help students think about how molecules are arranged and move in liquids and gases, Matter and Molecules asks students to consider the following questions as part of an activity in which they push air and water in and out of a syringe:

How far apart are the molecules of gas compared to a liquid?
In which of these two states of matter do you think it would be easier to push the molecules together? Why?
Below is a drawing of a syringe. How would molecules of air be arranged in the syringe when the plunger is all the way out? Draw the air molecules in the syringe.

Students then compare their predictions to what actually happens. They push on a sealed plunger that is filled with water and observe that it does not compress. They replace the water with air and repeat the experiment, this time finding that they can push on the plunger. In each case, questions ask them what happened and why:

Can you push the plunger in when the syringe is filled with water?
What happened when you tried the same experiment but filled the syringe with air?
Why can you push the plunger in when there is air in the syringe, but to when there is water in it?
Whey can't you push the plunger all the way in with air in it?

Students are reminded to talk about molecules in their answers. Then they are asked to observe what happens when they push on and then release the plunder in the syringe filled with air. The question helps them relate the phenomenon to the idea that molecules are constantly moving. Another question relates the phenomenon to the scientific idea:

Explain why the plunger moves back out.

Later in the unit, students are asked a question that anticipates a common misconception-that molecules are not perpetually in motion but only move if the substance appears to move:

If you let this cup (of sweetened tea) stand overnight, would the sugar rise to the top, settle to the bottom, or spread evenly throughout the water? Talk about molecules to explain your answer.

Developing and Using Scientific and Mathematical Ideas

Textbooks need to provide a wide range of problem-solving and practice tasks to help students see the link between concepts and skills.

Connected Mathematics provides a real-world scenario or story line that builds students' understanding. For example, the text challenges students to plan a bicycle touring business. Students develop bar graphs and charts predicting the costs involved, depending on the number of cyclists and the distance of the tour. They write equations that can predict travel times and profits under varying conditions. Students develop and work with their own data, so they can explain the relationship among variables.

Matter and Molecules provides practice tasks for most of the physical science ideas it examines. These include novel tasks that ask students to develop descriptions and explanations of phenomena they see all around them. Their explanations are expected to become increasingly sophisticated as their understanding deepens.

For example, the following questions (taken from several places in the unit) have students move from using the idea that "molecules are in perpetual motion" to combining it with other related ideas:

Draw pictures to show how water molecules are moving.
Can water molecules in ice slow down and stop?
If you want something to dissolve fast, should you mix it with hot water or cold water? Why?
Explain how you can smell an open bottle of vinegar even though you are across the room. What is actually reaching your nose? How did the vinegar molecules get into the air? How did the vinegar molecules reach your nose?
When food is covered with plastic wrap in the refrigerator (or when soup is warming on the stove, but not boiling, with a lid on the pot), water evaporates and then condenses. Where does the water evaporate from? Where does the water condense? How do the water molecules get from the place where water evaporates to the place where water condenses?

In contrast, most science textbooks include few, if any, relevant practice questions. Instead, they rely on multiple choice or short answer questions that simply require students to find the correct answer from similar statements that appear a few pages earlier.

Implications for Literacy in Mathematics and Science

With the wide acceptance of national and state standards and benchmarks in science and mathematics, there is growing optimism about educational improvement:

In a field where fads have ruled, we are seeing something new: a growing commitment to the idea that clear and shared goals for student learning must provide a foundation on which to improve education and achievement. Without clear goals, we cannot succeed, for we cannot know in which direction to move.

- (Stigler & Hiebert, 1999).

But as Project 2061's evaluations have shown, mere alignment is insufficient if students are to actually learn and remember key ideas--materials also must support teaching and learning the ideas. Publishers, authors, funders, and others involved in creating textbooks--along with the educators who use them--must commit themselves now to producing a new generation of textbooks that will help all students achieve these goals. And just as important, teachers need to have carefully targeted professional development that will enable them to appreciate the strengths of highly rated materials and use them well.

In the meantime, schools and school districts can use the evaluation reports to make well-informed adoption decisions. Teachers who have already selected highly rated materials can use the reports to better understand their textbook's strengths. Teachers who have not been able to select highly rated materials can use the reports to identify their textbook's strengths and weaknesses and then focus on supplementing their text with stand-alone units or trade books or with lessons from other textbooks that do a better job. They can use Project 2061's instructional criteria to guide their own classroom instruction and can study the research on student learning that is cited in the reports and use it to revise learning activities and develop new ones. And, finally, they can take advantage of professional development experiences that focus not only on increasing their knowledge of key ideas in science and mathematics, but also on strategies for teaching those ideas more effectively.

References

American Association for the Advancement of Science. (1989). Science for All Americans. New York: Oxford University Press.

American Association for the Advancement of Science. (1993). Benchmarks for Science Literacy. New York: Oxford University Press.

Berkheimer, G., Anderson, C., Lee, O., & Blakeslee, T. (1988). Matter and Molecules. (Occasional paper No. 121). Institute for Research on Teaching: East Lansing, MI.

Feynman, R. P. (1997) 'Surely you're joking, Mr. Feynman!': Adventures of a Curious Character. New York: Norton.

Lappan, G., Fey, J. T., Fitzgerald, W. M., Friel, S. N., & Phillips, E. P. (1998). Connected Mathematics. Menlo Park, CA: Dale Seymour.

National Council of Teachers of Mathematics. (2000). Principles and Standards for School Mathematics. Reston, VA: Author.

St. John, M. (2001). The Status of High School Science Programs and Curricular Decision-Making. Inverness, CA: Inverness Research Associates.

Stigler, J. W. & Hiebert, J. (1999). The Teaching Gap. New York: The Free Press.

Tyson, H. (1997). Overcoming Structural Barriers to Good Textbooks. Washington, DC: National Education Goals Panel.

Jo Ellen Roseman is associate director of Project 2061, Gerald Kulm is Curtis D. Robert professor of mathematics education at Texas A&M University, and Susan Shuttleworth is a writer and editor with Project 2061.

Beginning in 1985, Project 2061 has worked to reform science education in grades K-12 so that all high school graduates become science literate-that is, prepared to live interesting, responsible, and productive lives in a world increasingly shaped by science and technology. The initiative has developed a variety of tools and training for educators to support efforts to translate the established learning goals into classroom activities. Project 2061's textbook evaluations have been funded by the Carnegie Corporation of New York.

Project 2061's Curriculum-Materials Analysis Procedure

Project 2061's analysis procedure was developed over a three-year period in collaboration with more than 100 scientists, mathematicians, educators, and curriculum developers, with funding from the National Science Foundation. The analysts examined the texts' quality of instruction aimed specifically at the key ideas using the criteria drawn from research about how students learn.

For a complete explanation of the criteria and evaluations, visit the project web site (www.project2061.org).

Roseman, J.E., Kulm, G., & Shuttleworth, S. (2001). Putting Textbooks to the Test. ENC Focus (8)3 56-59. Reprinted with permission of Eisenhower National Clearinghouse; visit ENC Online (enc.org).