Carol Weiss, an influential figure in program evaluation, has written that "only with sensitivity to the politics of evaluation research...can the evaluator be as creative and strategically useful as he should be."1 This is true for all evaluations, but for an evaluation in the extremely controversial area of school-based sexuality education, political factors may become a predominant issue.
In particular, the school environment is likely to be dominated by political constituencies—both liberal and conservative—attempting to foster their own agendas. As a result, school officials must deal with well-organized groups emphatically advocating diametrically opposed views. In such an environment, it is not surprising that conducting a valid evaluation study is often a lesser concern. Other potential stumbling blocks include teachers who are uncomfortable with the curriculum, parents who are uncomfortable in dealing with their adolescents' sexuality and the numerous problems associated with conducting applied research in "real world" settings, where many activities are not under the investigators' control.
We encountered a number of these problems during a 1996 pilot study with 5th-8th-grade students in Memphis, Tennessee. With the aim of increasing parental involvement in school-based family life education, we examined the value of supplementing the curriculum with joint
parent-child homework assignments. In addition, we sought to assess the efficacy of a voluntary parental training program that taught techniques for increasing communication between parents and children.
The purpose of this article is to share some of our experiences in this area, and to provide some insights to others contemplating evaluation work in school-based sexuality education.
Background
In 1988, the Memphis City Schools developed and implemented the Family Life Curriculum, a knowledge- and skills-based sexuality education program designed for students from kindergarten through 12th grade with the stated purpose of reducing the high adolescent pregnancy rate. This initiative was adopted in anticipation of the passage of a 1989 Tennessee state law mandating school-based sex education in counties with adolescent pregnancy rates exceeding 19.5 pregnancies per 1,000 young women aged 15-17.
The program was part of the health education curriculum and progressed from simple concepts of family at lower elementary grade levels to complex family relationships and human sexuality at the junior and senior high school levels. A variety of approaches was used in the presentation of this material, including lectures, discussions, audiovisual presentations and guest speakers.
As with many school-based sexuality education programs, the curriculum was hotly debated before being implemented, and the Tennessee State Board of Education mandated that the program be evaluated five years after its inception. This evaluation, in which we took part, was presented in 1994 to the Memphis City Schools' Board of Commissioners.2
The most prominent finding of our report was that for unknown reasons, the curriculum had been only sporadically implemented. In some schools it had never been presented, whereas in other schools, the teachers assigned to present it had covered only selected portions of the material. Thus, the program's effectiveness with regard to the goal of reducing teenage pregnancy could not be adequately assessed.
The report also called attention to the absence in the program of any initiatives to encourage parents' involvement with their children's sexuality education. While many school-based programs lack such a component, a small body of recent research suggests that the promotion of parental involvement may be an important component of school-based sexuality-education programs.3 Therefore, the report recommended that steps be taken to empower parents of children in the city schools to take part in their children's sexuality education.
Specifically, we recommended that a pilot study be conducted to test the efficacy of adding two supplements to the curriculum: joint parent-child homework assignments to encourage parents to be more involved and to communicate with their adolescents about sexual issues, and a workshop for parents to develop skills for doing so. The Memphis public school administration implemented this pilot study, and we were invited to assist in its planning, execution and evaluation. What follows is an overview of this effort, including our initial strategies, the impediments we encountered, our midcourse corrections and the eventual outcome, which—despite many difficulties—had an unexpected level of impact.
Methods
Initial Strategy
To maximize both the involvement of a variety of stakeholders and the potential for the findings' implementation, the pilot study and evaluation plan were developed in collaboration with Memphis public school administrators and the Family Life Curriculum Council (an advisory group made up of educators, professionals, parents and students). The plan was then reviewed and approved by the Memphis City Schools' Board of Commissioners.
The final plan, which called for an experimental group and a nonequivalent control group, was designed to use the following outcome measures: pretest and posttest data from a survey given to students to assess the quality and extent of their communication with their parents; and a parents' survey addressing their perceptions of the program after the curriculum had been delivered—and, in the case of the experimental parents, their views on the homework and the workshops. Once the plan for the pilot study and evaluation was approved, parents, teachers, school administrators and members of the Family Life Curriculum Council were involved in every step of the pilot study's development and implementation.
Participants
In all, 775 students in grades 5-8 at four Memphis public schools participated in the study conducted during the 1994-1995 school year, as did one or both parents of these students. The four schools chosen for the pilot program—two elementary schools and two junior high schools—were selected by school administrative personnel, who also determined which would be the experimental schools and which would be the controls.
Although school officials were well aware of the needs presented by the research design, we were unable to influence either the selection of schools or their assignment to experimental or control status. The result was that experimental and control schools were decidedly "nonequivalent." Table 1 shows the characteristics of each school and the number of teachers and students involved in the pilot project at each grade level. Experimental and control schools differed in terms of both racial composition and socioeconomic level. In addition, the two groups of schools differed in their levels of experience with the curriculum.
Materials and Instruments
•Homework supplement. The curriculum supplement used in the experimental classes consisted of homework designed to be completed by the student and one or both parents. (Copies of homework assignments are available from the authors upon request.) The homework assignments were developed at a one-day workshop attended by the teachers assigned to present the experimental program, designated school administration personnel and the senior author of this article.
Assignments for each grade level were designed to enhance the curriculum, as well as to promote parent-child discussions of subjects related to family life and human sexuality. The topics were grade-appropriate, were based on the curriculum content, and covered subjects such as family structure and relationships, coping with conflicts, changes that occur during puberty, and dating and sexuality. Each assignment included a parent's sign-off sheet indicating that the parent and child had completed the assignment together. This sheet, rather than the actual homework, was to be returned to the classroom teacher.
•Student surveys. With the assistance of students, teachers, counselors, the Family Life Curriculum Council and the school personnel involved in overseeing the program, two student surveys were developed. These consisted of a 27-item survey for students in grades 5-6 and a 43-item survey for students in grades 7-8. The surveys were developed to assess the students' perceptions of the degree to which they had communicated with their parents on topics related to the program.
The questions were designed in a Likert format and included items such as: "I have talked with my parents about how my body will change as I get older"; "In the last six weeks, how many times have you and your parents discussed how to handle sexual pressure?"; and "How valuable (helpful) were your talks with your parents about abstinence (not having sex)?" The surveys went through several reviews and revisions during the development process and were eventually approved by the superintendent's office.
•Parent workshop curriculum. A workshop, entitled "Communicating with Your Child in the 90s," was conducted by an agent from the University of Tennessee Agriculture Extension Service with teaching experience in this topic. The two-hour workshop consisted of several activities designed to help parents improve their listening and communication skills.
Topics covered in the workshop included parents as teachers, developing listening skills, assessing children's needs, assessing personal strengths and weaknesses, and developing communication skills. In addition, the program included a video entitled "A Family Talks About Sex," which presented several family scenarios demonstrating effective techniques for discussing human sexuality with children at different age levels.
•Parent postintervention survey. We used a 40-item survey to probe parents for their attitudes concerning the family life curriculum and to assess various aspects of parent-child communication. To the surveys sent to parents of children in the experimental group, we added six other items concerning the parent workshops and the homework assignments, neither of which involved parents of the control students. (Copies of the survey are available from the authors upon request.)
Procedures
The original evaluation plan called for pretest and posttest comparisons of nonequivalent experimental and control groups. The principal outcome measure was to be the children's responses on the survey administered in their classes before and after presentation of the curriculum, which consisted of 10 lessons to be taught over 2 weeks. Before the surveys were to be administered, notes were to be sent to parents to inform them of the project, to provide them with instructions as to how they could review the survey at their child's school and to give them an opportunity to withhold consent for their child's participation in the survey.
Teachers in all schools were assigned in the usual way to present the curriculum, and no teachers were given any special training regarding the program. In the experimental schools, instructors were asked to keep a journal indicating which lessons had been covered during the course of the program, when homework assignments had been sent home, the number of assignments sent home and the number of sign-off sheets returned. In control schools, they were simply instructed to teach the standard curriculum as described in the curriculum guide. After the experimental curriculum had been presented, students at both experimental and control schools were asked to take the parents' survey home for their parents to complete, and to return it to their classroom teachers in a sealed envelope.
During the first week of the program, parents of students in the two experimental schools were invited to attend the two-hour parent workshop on communicating with their children. A letter of invitation was sent home with each student. Parents were asked to return a signed form indicating that they would be attending the workshop. Those parents who indicated that they would attend received a telephone reminder on the day of the workshop confirming the time and place.
A major setback in the project occurred when, just prior to the first scheduled student survey, the Board of Commissioners voted, after a brief discussion, not to allow the distribution of any survey instrument to the children in the study. Elimination of this component of the study meant that no data were obtained from the primary targets of the intervention—the children. Thus, the only outcome data available consisted of parents' responses to the survey to be administered after presentation of the curriculum. The result was that all data were postintervention, thereby precluding any assessment of the initial comparability of the participants.
To mitigate the problem created by the elimination of the before-and-after survey, we decided to involve family life curriculum teachers and parents of experimental and control children in separate focus-group discussions. It was hoped that in-depth, qualitative data on perceptions of impact of the homework and parent training would serve to reduce administrators' uncertainty about the value of the curriculum supplements. After family life instruction and the parents' survey, a letter was sent home with each child who participated in the pilot project inviting parents to attend focus-group discussions about the curriculum. Once again, parents who responded that they would participate received a telephone call to remind them to attend.
Discussions with parents of students in the experimental schools focused on overall attitudes toward the curriculum, the impact of the homework assignments, satisfaction with the homework assignments, methods of increasing parental involvement in school-based sex education and suggestions for improving the program. Similar topics were covered with parents of students in the control schools, and the moderator described the use of homework assignments in other classes and invited discussion of that approach.
We also held separate focus-group discussions for teachers of the experimental curriculum and those teaching the standard curriculum in the control schools. Topics discussed by teachers from experimental schools were to include their overall satisfaction with the curriculum, the successes and failures of the pilot program, the teacher's impressions of the impact of the homework assignments and techniques for improving the program. Similar topics were covered in discussions with teachers from control schools. As had been done in discussions with parents, the moderator described the homework assignments and prompted discussion of their use.
All focus-group or interview sessions were audiotaped. The moderator, the assistant to the moderator and the family life coordinator observed and took notes. Verbatim transcripts were prepared from the audiotapes and used to conduct a content analysis of the discussions according to previously established guidelines.4
Results
Teacher and Student Participation
Five of the seven experimental teachers completed their journals. These five indicated that they had covered most or all of the topics in the curriculum. The two remaining teachers failed to complete their journals. One had been absent during much of the time the curriculum was taught, and did not indicate whether the material had been covered in his absence. The other gave no information.
Table 2 shows the percentage of students in each teacher's classroom who returned the parental sign-off sheets. For the seven teachers involved in the experimental group, only one provided students with the entire homework supplement (Teacher 1 in Grade 6), and she had a return rate of 96%. Overall, only 34 of the planned 63 homework assignments (54%) were ever given to the students. For the 34 assignments, 83% of the total number of students who were enrolled in the class returned signed sheets stating that the assignment had been completed. These data suggest, not surprisingly, that gaining the support of the teachers is a key determinant of the success of this type of program.
Parents' Evaluation
•Postintervention survey. In all, 775 students participated in the study: 274 students in the experimental group and 501 in the control group. Each teacher received a parents' survey packet for each student in his or her classroom. A total of 348 parents (45%) returned surveys—172 (63%) from parents of children in the experimental group and 176 (35%) from parents of children in the control group. A factor analysis of the parent survey data, using varimax and oblique rotations, helped us to identify three factors affecting parents' opinion of the curriculum: their "attitude toward family life curriculum," which included five items and accounted for 11% of rotated variance; "recency of communication," which included 12 items and accounted for 13% of rotated variance; and "value of communication," which included 12 items and accounted for 19% of rotated variance.
A t-test of the combined scores of the five items constituting the "parents' attitudes" factor showed significantly higher scores for parents of the experimental students than for parents of the control students, (t[315]=3.87, p=<.05), indicating that parents of students in the experimental group had more favorable opinions about the curriculum than did parents whose children were in the control schools. For the other two factors, t-tests showed no significant differences between parent groups.
Four of the questions added to the survey of parents in the experimental group addressed the homework assignments. Sixty-seven percent of those parents who returned the surveys agreed or strongly agreed that the homework assignments encouraged their children to discuss topics that they had not discussed before, and 77% agreed that the assignments were useful in promoting communication with their children. Forty-eight percent believed that the assignments had provided them with an opportunity to discuss topics that they had previously postponed, although 73% stated that they had discussed most of the topics covered in the assignments before the curriculum had been taught that year.
•Workshops. The parent workshop was very poorly attended; although there were 274 children in the experimental group, only 18 people (representing 14 families) attended. Parents who attended the workshop were asked to rate its value in the survey. Five parents rated the workshop as "very valuable," seven rated it as "valuable" and six parents rated it as "somewhat valuable."
•Focus-group discussions. Twenty-four parents attended one of six focus-group discussions. Comments during these sessions consistently indicated that parents' overall impressions of the curriculum were positive, regardless of whether their children were in the experimental or control schools. Parents in the experimental group expressed a high degree of satisfaction with the homework assignments and indicated that they were an effective method of initiating discussions.
Several parents said that the homework assignments provided a stimulus, or an "excuse to talk," as well as opportunities to move beyond topics covered by the assignments into other areas of concern and interest. These parents said they used the assignments as a "screening process" to monitor what was being discussed in the classroom and to add their beliefs and family values to that instruction. When asked how they would feel if the homework assignments were removed from the curriculum, their response was a resounding "No, don't do that!"
During the discussions with the parents of students in the control schools, examples of the homework were provided, and the concept of the homework assignments was explained. These parents were then asked to discuss their impressions of the potential value of homework assignments. All comments were positive, with most parents emphasizing the importance of knowing what is being taught and having the opportunity to reinforce that information.
Teachers' Evaluation
Of the 13 teachers in the pilot study, only three attended the scheduled discussion groups, one from an experimental school and two from one of the control schools. These three were very positive about the curriculum and believed that it was a productive and useful program. As in the parents' focus group, the teachers believed the homework assignments would foster productive parent-child communication and alleviate the concerns of some parents about the content of the curriculum.
Discussion
It has been asserted that a major function of any evaluation study is to provide information that "reduces the uncertainty of action for specific stakeholders" to assist them in their programmatic decision making.5 Our initial hope had been to provide such information through a carefully designed study. However, because we were not allowed to collect survey data from participating children, the evaluation was limited to posttest measures obtained from parents and teachers who completed surveys and attended focus-group or interview sessions. With all of these measures, selection bias posed a serious threat to the validity of any conclusions. Nevertheless, although the evaluation was far less rigorous than planned, it did provide information that school administrators found useful and that ultimately resulted in a system-wide change in the program.
Our evaluation findings were included in a report that was submitted to the school administration and the Board of Commissioners. For about a year, they took no significant action, and we believed that the report had been "filed." Soon afterward, however, there was renewed interest in the findings (due, in part, to inquiries made of the Board of Commissioners by the League of Women Voters), especially with regard to the joint child-parent homework assignments.
Without contacting us, school administrators designed and initiated a large-scale test of the program, which involved 1,183 kindergarten through ninth grade students in 42 different schools. Outcome measures included short surveys completed by students, parents, teachers and principals. Each survey consisted of Likert-scale items asking the participants to describe their opinions on curriculum content and the effectiveness of the homework assignments. When the school administrators received positive feedback from these surveys, they decided to expand the use of the homework supplement throughout the system and at all grade levels. The senior author of this article was then invited to participate in the development of a final report recommending system-wide implementation of the revised curriculum.
We believe that our evaluation reduced uncertainty among administrators about the acceptability of the program to parents, thereby allowing the administration to act decisively on the family life curriculum. We also learned some valuable lessons regarding strategies for improving the evaluation process and ensuring that adequate outcome measures are obtained in evaluations of sexuality education.
•Bond with those who must deliver the program. Although we attempted to involve family life teachers in the study by meeting with them before the study and by working with several of them when we developed the homework assignments, no structure had been developed to promote their full participation in the curriculum instruction. Thus, only one teacher out of nine in the experimental group distributed all of the homework assignments, and one teacher did not distribute any homework assignments.
Not surprisingly, our data suggest that gaining the support of the teachers is a key determinant of the success of this type of program: There was a high overall return rate for the homework that was assigned. We believe it would have been useful to have held focus groups with the teachers before the program was carried out, to obtain their input on strategies for ensuring adequate implementation. Such a strategy might have increased their motivation to more fully enact the project design.
•Be assertive concerning selection of project participants. A second problem was that the school administrators determined which schools would participate in the study and the experimental condition to which each school was assigned. There were obvious differences between the two sets of schools. In retrospect, we should have been more assertive in communicating to school administrators the importance of the groups' equivalence for the project, especially with respect to the schools' history of instructing the family life curriculum. In other words, we should have set more firmly the conditions under which we were willing to undertake the project.
•Hold fast to the project design. The most serious design problem we encountered was the last-minute decision to prohibit any questionnaires from being administered to students. Although at the time we believed our options were limited, hindsight suggests that we should have made additional efforts to convince school administrators and board members of the importance of this outcome measure. We should also have made a stronger effort to involve board members in the design of the student questionnaire. Although unrelated political issues were at stake, such actions might have resulted in less resistance. The adage "What I'm not up on, I'm down on" certainly has relevance for the type of evaluation we attempted to conduct, and we strongly encourage others contemplating evaluation work in sex education to consider strategies for ensuring intensive involvement of stakeholders.
•Think "qualitative." Our experiences highlight the importance of multiple outcome measures and the value of qualitative methods in program evaluation. There is no doubt that traditional quantitative methods, when they can be applied, maximize internal and external validity and increase the value of the project in reassuring administrators. After the loss of our student survey measure, we decided to conduct focus groups with parents and teachers. Attendance by teachers was low, and attempts to hold additional focus groups for the nonattending teachers were hampered by their loss to follow-up when the project ended with the academic school year. Nevertheless, the discussions that occurred provided us with important insights that were central to the final implementation of the curriculum changes.
•Notice who is watching. Our experience with evaluations in the public sector is that the clients and primary stakeholders often rely on the evaluators to become spokespersons regarding the issues at stake. Especially in controversial areas like sex education, numerous groups, organizations, reporters and others seek information, and the client may direct them to the evaluators for comments, presentations and speeches. As this list of other stakeholders grows and they become familiar with the evaluator's activities, opportunities arise to overcome project blockages caused by bureaucracy, politics and organizational inertia. This was certainly our experience during our project.
Because of the sensitive nature of the subject matter, our study's conduct was hampered by political influences and a degree of reticence on the part of some teachers to implement the curriculum. Although these impediments necessitated "mid-course corrections" in the study's execution and evaluation strategies, the effort did result in some valuable findings, and the child-parent homework supplement to the family life curriculum was eventually adopted throughout the Memphis public school system. Thus, in spite of the many difficulties that were encountered, we believe that it is possible to conduct effective research and evaluation in the area of school-based sexuality education. Such work can yield useful information that can alleviate uncertainty among school administrators about these programs.