How to evaluate educational programs

Based on the American tradition of local control of education, it is thought that increased parental participation on boards developing new educational philosophies and innovative curricula would make school district programs more responsive to local ideological, economic, and cultural values.

The study concluded that citizen judgments must be used judiciously to avoid bias, but that such judgments can be predictive of community responsiveness and receptivity to future collaboration. Program evaluators have paid more attention to political factors in recent years as evaluation has become a stronger force in program design. Hence, attention to public sentiment needs to be a high priority. For principals and superintendents, the purpose of program evaluation is to provide information to help them make decisions regarding programs.

In general, principals feel that the benefits of evaluations are minimal because of their inability to measure program components that are of real importance, or because principals' own proximity to the everyday realities of the educational process gives them what they feel is a better basis for understanding needs and implementing change.

Superintendents tend to be more positive about the value of program evaluation. In particular, evaluations that reported deficiencies and discussed possible solutions were highly rated. Second in importance are personal meetings with evaluation personnel. In small schools, the missing element in evaluations seems to be the attempt to make such studies systematic, purposive, cyclical, comprehensive, and well-communicated James R. Sanders Sanders suggests that a Program Review Committee PRC , composed of the superintendent, principal, grade level chairperson, and an educational specialist, be established.

Each year the committee should conduct a review of one or two programs, so that each program receives careful scrutiny once every five years.

According to Jody L. Fitzpatrick , the job of the evaluator is expanding from technical roles to political and advisory roles. In innovative programs, defined as those still in a research and development phase, evaluators help identify goals and develop a strategy for accomplishing these goals. Another new role for the evaluator is translating policy questions developed by school boards and legislators into the more precise questions of program evaluation.

In this role, the evaluator helps fashion new and innovative programs with features that are readily measurable. Once pilot programs are begun, the evaluator then has the opportunity to determine how fully the program was implemented before evaluating its effectiveness.

According to Fitzpatrick, evaluation questions imply certain design decisions. Besides content, these questions can help determine the parameters of cost, time, and the availability of professional personnel.

The program manager can monitor the innovative program through the oral briefings and written reports of the program evaluator. To be effective, communication should be ongoing and not limited to a final report at the end of the year.

However, developing in-house evaluation expertise and capacity is a beneficial goal for most public health organizations. The lead evaluator should be willing and able to draw out and reconcile differences in values and standards among stakeholders and to work with knowledgeable stakeholder representatives in designing and conducting the evaluation.

Seek additional evaluation expertise in programs within the health department, through external partners e. You can also use outside consultants as volunteers, advisory panel members, or contractors. External consultants can provide high levels of evaluation expertise from an objective point of view. Important factors to consider when selecting consultants are their level of professional training, experience, and ability to meet your needs. Be sure to check all references carefully before you enter into a contract with any consultant.

To generate discussion around evaluation planning and implementation, several states have formed evaluation advisory panels. Advisory panels typically generate input from local, regional, or national experts otherwise difficult to access.

Such an advisory panel will lend credibility to your efforts and prove useful in cultivating widespread support for evaluation activities. Evaluation team members should clearly define their respective roles. Informal consensus may be enough; others prefer a written agreement that describes who will conduct the evaluation and assigns specific roles and responsibilities to individual team members. Either way, the team must clarify and reach consensus on the:.

This manual is organized by the six steps of the CDC Framework. Each chapter will introduce the key questions to be answered in that step, approaches to answering those questions, and how the four evaluation standards might influence your approach. The main points are illustrated with one or more public health examples that are composites inspired by actual work being done by CDC and states and localities.

Together, they build a house over a multi-week period. At the end of the construction period, the home is sold to the family using a no-interest loan. Lead poisoning is the most widespread environmental hazard facing young children, especially in older inner-city areas. Even at low levels, elevated blood lead levels EBLL have been associated with reduced intelligence, medical problems, and developmental problems. The main sources of lead poisoning in children are paint and dust in older homes with lead-based paint.

Public health programs address the problem through a combination of primary and secondary prevention efforts. A typical secondary prevention program at the local level does outreach and screening of high-risk children, identifying those with EBLL, assessing their environments for sources of lead, and case managing both their medical treatment and environmental corrections. However, these programs must rely on others to accomplish the actual medical treatment and the reduction of lead in the home environment.

A common initiative of state immunization programs is comprehensive provider education programs to train and motivate private providers to provide more immunizations. A typical program includes a newsletter distributed three times per year to update private providers on new developments and changes in policy, and provide a brief education on various immunization topics; immunization trainings held around the state conducted by teams of state program staff and physician educators on general immunization topics and the immunization registry; a Provider Tool Kit on how to increase immunization rates in their practice; training of nursing staff in local health departments who then conduct immunization presentations in individual private provider clinics; and presentations on immunization topics by physician peer educators at physician grand rounds and state conferences.

Minimalist theory of evaluation: The least theory that practice requires. American Journal of Evaluation ; Utilization-focused evaluation: The new century text.

Thousand Oaks, CA: Sage, Study of participatory research in health promotion: Review and recommendations for the development of participatory research in health promotion in Canada. Ottawa, Canada : Royal Society of Canada , Health promotion evaluation: Recommendations to policy-makers: Report of the WHO European working group on health promotion evaluation.

Public health in America. Fall January 1, Ten organizational practices of public health: A historical perspective. American Journal of Preventive Medicine ;11 6 Suppl The program evaluation standards: How to assess evaluations of educational programs. The PRC program is a national network of 24 academic research centers committed to prevention research and the ability to translate that research into programs and policies.

The centers work with state health departments and members of their communities to develop and evaluate state and local interventions that address the leading causes of death and disability in the nation. Additional information on the PRCs is available at www. While inspired by real CDC and community programs, they are not intended to reflect the current. Skip directly to site content Skip directly to page options Skip directly to A-Z link.

Section Navigation. The sessions should provide opportunities for each individual to participate in skill-building activities such as methods of handling animals and performing necessary procedures. Adult learners, particularly those in a profession, tend to avoid situations in which they cannot demonstrate competence.

Therefore, it is usually helpful to introduce the lab with a demonstration, slide show, or video presentation to provide background information. Demonstration with models is also highly recommended prior to hands-on experience. However, media is not a substitute for the hands-on experience needed for developing skills. The facilitator must be encouraging, positive, and patient toward learners who have little or no prior experience with a particular species of animal or procedure.

Adult learners appreciate individualized, independent study. A variety of individualized study approaches should be used, including:. Self-assessment tools are a form of individualized independent study. They provide an investigator with an instrument to test his or her knowledge in a confidential way.

This self-assessment tool could be a pencil-and-paper instrument or a computerized program. The essential characteristics are that the results are strictly for the benefit of the person completing the program and that the program identifies areas of weakness. Self-assessment can be combined with self-reporting: a statement that the person has completed the program. Some investigators may resent a requirement for education or give the program a low priority.

Steps must be taken to overcome potential resistance. Some suggestions are as follows:. Involve several key people in planning the educational offerings, for example, people at the institution who represent the needs and views of the researchers, people who have the respect of investigators, or a person from whom resistance is anticipated.

Explain the requirements, available resources, and limitations to those people, and encourage them to problem-solve and incorporate their ideas into an action plan. Name these people in publicity about the courses. Make compliance with institutional goals as personalized and as easy as possible.

Develop packets containing species-specific information relating to requirements and guidelines. Ensure access to information. Develop reading lists and catalog books and reprint files in the resource library by species and subject for easy access.

If a major institutional library will be used as the resource library, arrange for a demonstration on how to locate relevant materials. Develop a "reference bank" of local investigators who have experience with exotic species or are experts in performing advanced techniques. Find out from researchers what obstacles to implementation they perceive and develop a mechanism for reducing difficulties in changing behaviors. Reward and encourage compliance by acknowledging investigators for their cooperation following successful inspections or accreditation visits.

Build a positive image with an active public relations program, such as by displaying articles about research accomplishments. A well-organized educational program will conserve time and resources, be customized to the content needs of the learners, and be flexible enough to encourage enthusiastic participation. Careful planning and preparation are required to provide informative, well-organized courses.

Attention to detail cannot be overemphasized. Up to 6 months should be allowed to organize and implement the first offering of each course. Each course must be designed for a specific audience to encourage active participation and achieve desired results. The audience should be defined on the basis of job responsibilities, educational level, experience, motivation, and training needs.

This audience profile will help the trainer establish program goals, objectives, content, and presentation method. For example, a program for people who support animal research efforts peripherally, such as security, janitorial, or equipment maintenance personnel, will be designed differently from a program for scientific staff. Likewise, a course for newly hired research staff will include introductory information that may be inappropriate or redundant for staff members who have been employed by the institution for several years.

A course budget should be allocated to include honoraria and travel expenses for guest speakers; duplication of handout materials; rental, purchase, or development of audio-visual support materials; room and equipment rental; and costs of publicity.

The goals and learning objectives must be defined clearly during the early phases of course development. As Kemp has stated, "A good goal is a nonambiguous statement.

It means exactly the same thing to all other teachers who use it. From these goals, specific learning objectives can be developed that reflect the institution's mission, the scope of the laboratory animal research projects, and the audience profile. In traditional academic settings, selected learning objectives would become the basis of test questions. In most adult education settings, the learning objectives are shared with course participants, who can use them to structure their learning experience or, after the course, to assess their retention of course content.

The frequency with which training programs are given and their scheduling depends on the total number of people who will receive training, the approximate number who will attend each session, and the availability of facilities and other resources. Mandatory training, which includes the core material required by federal regulations and institutional policy, is likely to be offered more frequently than are training opportunities for special topics or skill development.

Offering multiple options for the dates and time of training will better enable scientists to participate with minimal disruption to their research and teaching efforts. The training facility or facilities should be identified and evaluated before final scheduling is begun. Selection of a location convenient to the participants should be a primary consideration. The size of the room, the acoustics, and the lighting must be appropriate for the teaching format. Once the schedule is established and time is allocated for each content area, speakers for each segment of the program should be identified and contacted.

The choice of speakers might include members of the laboratory animal resource staff; investigators with expertise in a topic area; members of the institutional animal care and use committee; and.

Guest speakers might be desirable for certain topics. The selection criteria for speakers should include not only professional qualifications, but also their level of enthusiasm, oral presentation skills, commitment to the training effort, and ability to speak at the level of the participants. Audiovisuals are effective teaching tools and will help reinforce what is being said. The materials and the equipment should be identified and reserved.

The equipment must, of course, be compatible with the slides, videotapes, or films that are to be used. All audiovisual aids should be previewed for content and technical quality. It may be appropriate to show only part of a film or slide program or to add slides to supplement the program. Slides or overheads should be uncluttered and easy to understand.

It is better to use several slides than to crowd too much information on a single slide. Dark backgrounds and colors are more effective than are black on white. Part IV furnishes more detailed information on ordering and purchasing audiovisual programs.