Adolescent Literacy—Research Informing Practice: A Series of Workshops

Adolescent Literacy: Model Development, Implementation, and Effectiveness

In March and May 2002, two workshops on Adolescent Literacy were held. These workshops were co-sponsored by several federal agencies and professional associations, and brought together researchers, practitioners, administrators, funders and policy-makers. They included presentations, discussion panels, audience questions and comments, and small working groups. In the second workshop, examples of instructional models being implemented in middle and high schools were presented, highlighting the importance of the development, implementation and evaluation of such models, and the value of rigorous design in evaluation. Below is a summary of the comments and issues that were raised about models of instruction for adolescent students.

Conditions of Implementation

As literacy models must operate within the complex instructional and organizational structures of secondary schools, research is needed to identify the characteristics of middle and high schools where implementation is achieved and positively impacts student outcomes. There is also a need to discern the degree to which a range of literacy models can be adapted to accommodate school and student characteristics without diminishing the viability or effectiveness of the approach. Similarly, model developers and evaluators should measure the extent to which model implementation and effectiveness is impacted by inconsistencies in student and teacher behavior and participation.

• Some evidence for and against the benefits or costs of implementing the model is needed: when the model is implemented, does it detract from instruction in other areas? Cost-effectiveness relates to the severity and extent of the literacy problem.
• What theory of knowledge transfer shapes the model?
• Are the models dynamic or static? Can the schools mold it to their needs and their students? Are these product or process approaches?
• How closely linked are the conditions of implementation and success of implementation? Consider the complications of student attendance, absenteeism, turnover, and transience; teacher consistency; and English-language learners.

Evaluation of Models

Evaluations of models should represent partnerships among practitioners, model developers and researchers. Models should be developed to address literacy instruction for English language learners. Research is needed to extract key principles across practice models that can be used to create a menu of strategies for assisting students at different levels of ability. There is also a need for evaluative data on models that appear to interrupt the plateau in achievement often reached by disabled students.

• Models should be compared to models instead of to no treatment groups. This requires some consensus definitions and specific criteria for selecting comparison groups for evaluations of interventions. Also, the theoretical base of the instructional approach in the model must be clearly expressed in order to do the comparisons.
• Conducting evaluations in the setting and reality of a school system is challenging. Effective management of student populations/subjects will be needed to ascertain the effectiveness of the interventions. This will bear on the development of longitudinal studies, clinical trials, and careful selection of outcome measures. Studies must be designed with clear attention to issues of sample size and power, and the unit of analysis (e.g., teacher, class, school, students). The importance of designing solid effectiveness trials depends directly on how extensively the intervention is being implemented.
• To know if practical and permanent change has occurred, long-term follow-up data are needed, and outcomes should be assessed beyond in-school academic achievement. What is the student's performance a year later? Has there been slippage? Do the effects continue and what conditions are necessary for that to continue? What conditions are needed (necessary) for sustained gains over time?
• Model implementation and outcomes measured must be linked, clearly described, and reported. In order to ascertain the fidelity of implementation, changes in daily practice must be measured: what has really happened? Does the actual practice change on a day-to-day basis? Does the model disappear but leave a changed (improved) practice? Some tools for recording implementation are teacher surveys, checklists, observation protocols, classroom observations, anecdotal records (student and teacher). School-level measures should include dropout rates and absenteeism.
• Clarity is needed about the subject population (e.g., how students are selected, assessed, and placed) for any model. This will be important in attempting to generalize the use of the model to other groups of students.
• What is the role of standardized tests in model evaluation? Is there a baseline assessment that ascertains what students will be able to do? Sometimes students are not incapable, just under-practiced or lacking instruction. How is that assessed?

Instructional Approach of Models

Although there is a robust knowledgebase on the development of reading skills during early childhood and the core components of instruction supporting the development of reading proficiency in grades K-3, there is little evidence to suggest whether or not these research findings predict the efficacy of middle and high school literacy models. In constructing new literacy programs for adolescents and scaling up existing models of practice, model developers should clearly define the research-base for and pedagogical approach of these literacy frameworks and identify core instructional strategies. In doing so, model developers should seek to distinguish those practices that have optimal impact on the reading proficiency of students of different ability levels from those with minimal or no impact on student outcomes. The student population- as characterized by ability-level, learning differences, and English language proficiency- should be defined at the outset of model development, in addition to the mode of delivery (e.g. accelerated reading intervention; ninth-grade literacy course; strategies for reading in the content areas.)

• Interventions should build on the existing research base. First, the research and theoretical bases must be clearly described or assessed. How generalizable and applicable to adolescent literacy is the reading research that already exists? How generalizable are the five major components of early reading? Do they apply to adolescence? Examination of extant data would inform the definition of the problem and hypothesis generation for inquiry into effective interventions. Data sources could include: NAEP, NALS, and military entrance exams.
• What are the core instructional strategies in the model? How do they relate to content and instruction across the curriculum? Some of the instructional components common to the practice models are self-selected, reading, teacher modeling, vocabulary development, and cooperative learning. Is there any consensus on which of these components are central and core? The definition of the problem is still anecdotal thus compounding the difficulty of knowing how much and when and what kind of instruction is needed in each of the components of reading in order to develop a proficient reader. For example, how much should be isolated skill instruction vs just practicing reading?
• The model must respond to the level of severity of literacy difficulty. For whom is the specific intervention or model being developed-for special needs students or for all students but developmentally appropriate to subgroups?

Professional Development for Model Implementation

The effectiveness of implementation may depend on the quality of a model's professional development program as measured by its impact on teacher behavior, engagement and consistency in delivery. Professional development programs should be founded in a solid research and theoretical base on adult learning. Model developers should define the recipients of the professional development (e.g. reading specialists; content area teachers; administrators) and identify the most effective modes of training for each group (e.g., in-class coaching; university courses; on-going seminars and teacher working groups.) Model developers should consider the needs of both pre- and in-service teachers and school administrators.

• How much time is required for professional development?
• Is it important to demonstrate the impact of professional development on the effectiveness of implementation? If so, this information would contribute to school-based decisions on how much time and how many resources are to be allotted to the program.
• Is the delivery of professional development static or dynamic--meaning, how does it contribute, or not, to the ongoing improvement of implementation?
• Does the model allow teachers to use what they already know, to be informed decision makers? What decisions can teachers feel free to make that will not alter the effectiveness of the model as tested? If the intervention is complicated, teachers may be reluctant to use it; thus work with teachers to obtain their "buy-in" may be crucially important. How does scripting interact with the teacher's need to be an informed decision-maker?
• Research on persuasion, motivation, conceptual change, and engagement can be used to inform instructional practices for teachers.