Up to this point in the book we have alluded to the fact that many evaluators actually perform third-party program evaluations as a primary means of employment. Sometimes you might find yourself identified as the program/project evaluator for your own organization. These “jobs” are few and far between. Usually you are employed in some other aspect of your organization, and program evaluation is specified as one of your duties—probably because someone learned that you have program evaluation skills. Program or project evaluation is stressed here because we are not referring to quality control, employee appraisal, or market analysis activities. These are evaluation-type functions that, though using the skills covered in this book, are not specific to evaluating the efficiency, effectiveness, or impact of a program. This distinction is important as you progress through this chapter because we will be discussing how one might make a living as a third-party evaluator.
Think about the following questions as you read Chapter Ten:
When you finish reading, you should be able to answer these questions based on the information in the chapter.
The immediate answer to this question is “Definitely yes.” Both individuals and organizations that sponsor a program are very interested in knowing whether that program “performed.” They are interested in determining whether the program did what it was supposed to do, did it in an efficient manner, and if it made any difference. These are questions important to sponsors regardless of whether they are part of the public sector (that is, government) or the private sector (corporations, foundations, or private individuals). Based on the answers to any or all of these questions, the sponsor can make a decision to continue funding, expand funding, or to market the program. If you need further convincing, the above topics are elaborated on in Chapter Two, Why Evaluate?
In an effort to provide examples supporting this assertion, let's first look at the opportunities that abound in the public sector. The federal government's myriad funded programs have one common denominator: evaluation requirements. Often, depending on the current climate in the government, these evaluation requirements range from being strongly recommended to being strictly required. Again, this pendulum swings regularly, depending on the need for proof from the executive, legislative, and judicial branches of the government. Currently, there is a culture permeating all funding that requires programs to be scientifically based or data rich, or that decision making be data based. What this means is that funding proposals must have data supporting the assertions made by the proposal author for the need of certain fundable services or activities. It also means that once funded, the recipients will be expected to provide data indicating the performance of program.
For example, the No Child Left Behind Act, which funds the lion's share of educational activities in prekindergarten through secondary education in this country, has specific language that
puts special emphasis on determining what educational programs and practices have been proven effective through rigorous scientific research. Federal funding is targeted to support these programs and teaching methods that work to improve student learning and achievement. (ED Gov., 2003)
A good example of this trend is that the federal government makes funds available to teachers so that they can strengthen current skills or gain new skills in techniques of effective reading instruction. Therefore, all reading programs funded under No Child Left Behind must be supported by scientifically based reading instruction programs for teachers, such as the Reading First program in the early grades, and the new Early Reading First program in preschools. These are tested, evaluated, and validated programs that have shown their effectiveness and transportability.
Specifically stated in the language of all federal funding streams is this requirement of the use or development of data-supported techniques, methods, and activities. For example, the U.S. Department of Education (USDOE) spends upwards of $100 million per year on program evaluation and data collection. Key decision makers in that department, in the Office of Management and Budget, and congressional appropriators on Capitol Hill continually request the information they need to make decisions on maintaining a federal initiative. An interesting confession of the USDOE was that
Evaluation studies are not as helpful as they could be to practitioners at the local level, nor can they answer questions about causation....We propose a significant shift in program evaluation, away from a compliance model and towards a system of research and evaluation focused on results and the effectiveness of specific educational interventions. (ED Gov., 2002)
In 2002 the Office of Management and Business introduced a new performance-based data management process, sporting an electronic system of data collection focused on outcomes. A major goal of the system was to collect higher-quality data on program results:
These data will tell us whether the education system and its components are performing well, and they might help us understand which of our programs are having the greatest impact, but they will not tell us why. How can we supplement this data management system with program evaluations that give decision makers the information they need to allocate funds, make policy changes, and consider new directions? How can we build on the knowledge base so that practitioners know “what works” and can spend their federal dollars wisely? (ED Gov., 2002)
Does all of this sound familiar? It should, as this rationale for evaluation echoes much of what was presented in the Introduction of this book. If you are not convinced of the federal government's commitment to evaluation yet, then consider the following. The Electronic Data Gathering, Analysis, and Retrieval system (EDGAR) process—used to report to the federal government the results of all activities receiving federal money—has specific language relating to the requirements for evaluation. In subpart C (sec. 80.40) of the regulations it stipulates that as part of the monitoring and reporting of program performance:
Grantees must monitor grant and subgrant supported activities to assure compliance with applicable Federal requirements and that performance goals are being achieved. Grantee monitoring must cover each program, function or activity. (ED Gov., 2008)
The regulations also require that performance reports be submitted at least annually and must compare actual accomplishments to the objectives established for the period.
Again, these regulations remain in place continuously, but the extent to which they are enforced varies from administration to administration.
Lest you think that evaluation is required only in the world of public dollar–supported programs, let's briefly discuss the evaluation requirements of the private sector. Private sector support for programming might come from such sources as foundations, corporations, or corporate foundations. As stated earlier, evaluation in the private sector might take the form of quality control, employee appraisal, or market analysis activities. However, a great number of programs are evaluated as well. Most foundations, as part of their application requirements, identify program evaluation activities as one of their required activities. In fact, many foundations employ individuals whose sole or main activity is the monitoring of evaluations. A good example of this requisite by a foundation is found in the materials offered to grantees by The Health Foundation of Greater Cincinnati. They indicate that both process and outcome evaluations are important so that funded projects can
A search of the database of The Foundation Center (http://fdncenter.org), which is the central source of information and the leading authority on philanthropy for thousands of foundations, corporations, public charities, and nonprofit organizations, identified over fifty such organizations that publish informational packets on preparing evaluation designs for proposed projects, and over 3,700 reports or papers on third-party evaluations that were completed through foundation-sponsored activities. Each private sector sponsor will have some form of monitoring or evaluation requirement. Some are quite explicit, like that of The Health Foundation of Greater Cincinnati. Others might be less restrictive, allowing evaluations to be performed by “in-house” personnel. However, they all require some form of evaluation activity—whether process, outcome, or both.
Companies both large and small require evaluations of new and existing programs to determine whether funds should continue to be expended on these activities. Usually the purpose of such evaluations is to assess the value of these programs to the company's bottom line, the extent to which there was value added by these programs, or the return on investment provided by these programs.
We hope readers are convinced that there is a need for third-party or outside evaluations both in the public and private sectors, and that this need will continue over time. In addition, we have identified a number of potential customers for this program evaluation work within public agencies providing educational, human service, health, transportation, correctional, and workforce training services to the general public. Also, a similar need for program evaluation exists in the private sector for programs sponsored by foundations, corporations, public charities, and nonprofit organizations. Now it is time to explore how you find these potential customers.
This is the point in the book where you need to begin thinking creatively as an entrepreneur. If you have a nonprofit mentality, you need to put that aside and begin thinking like a person in business. Also, you need to make a decision whether you want to become an outside evaluator as a full-time occupation or in a part-time consulting capacity.
To make evaluation a full-time activity with a reasonable income, you will probably need to string several evaluation contracts together. Some programs are large enough in scope and duration that one can work full-time on just one evaluation for several years. However, most third-party evaluation contracts are small and last from year to year, so that you need to have several under way at a time to make ends meet. Also, you would need to stagger contracts in long-range plans, so that they do not all end at the same time. Consequently as a third-party program evaluation contractor, you are simultaneously marketing yourself, drumming up new business, and conducting evaluations. Sound enticing? Read on.
If you are looking at performing third-party program evaluations as a part-time supplement to your professional work and income, then you will still be doing the same tasks as mentioned above, but the difference will be in the number of contracts you need or want at any given time. Sometimes people who find themselves in this category are using evaluation work to keep them connected with the field, giving them an opportunity for scholarly activity (such as publishing) or providing a service to the professional community. Of course, the monetary compensation is a welcome addition.
In either case, you can be proactive in seeking evaluation work by convincing program staff that they should include an evaluation component in their planned project and employ you as their third-party evaluator. You can also be reactive by identifying existing programs or responding to requests that you perform a third-party evaluation on a program.
If you are new to the evaluation field, you will probably need to focus on the proactive track. Given your lack of experience and reputation, you are more likely to convince a program director to let you develop an evaluation design and include it as part of a proposal being submitted. If the proposal is successful, you will in all likelihood receive the evaluation contract.
However, if you have been performing third-party evaluations for some time and have a good reputation in the field, program directors who already have funding (but did not have the foresight to identify an evaluator in the proposal) may ask you to provide evaluation services. This, however, is not to say that an experienced evaluator will not, or should not, be proactive. On the contrary, as discussed earlier in this book, it is always preferable to be included in the planning stages of any program so that you can identify the evaluation questions that need to be addressed, have some influence in the development of the program objectives, and integrate the data collection activities into the planned normal operation of the program.
But this is not to discourage a novice third-party evaluator from seeking to perform an evaluation for an existing program. Quite often such evaluations are subject to bidding through a request for proposals (RFP) process. Under such circumstances, the successful bidder is the low bidder, and the lowest-priced bidder might be that novice who will charge less while building a reputation.
In summary, proactively you might look to be written into a proposal submitted to a public or private sponsor, or written into a plan that is being submitted for consideration to your company's decision makers. Reactively you might respond to the evaluation needs of an already funded program or solicit for the evaluation services from a funded program. Sometimes in this reactive mode, however, you are responding to the needs of a program that has not only received funding but has also been in operation for some time—and has been informed or has realized that a third-party evaluation is required. This scenario requires you to decide whether there is sufficient time to perform an adequate evaluation of the program.
If you are looking to identify who potential program grant writers might be, you should probably first find out what sources of funding they might be considering. Before doing any of these things, however, you need to establish your area of expertise. Although it's been said that a good evaluator can evaluate any kind of program, knowing something about the arena in which you plan to evaluate programs is very helpful—and advisable. For instance, will you work in the field of education, workplace training, health, social services, transportation, politics, or corrections? You might specialize even further. For instance, if your field is education, would you prefer to evaluate programs that are in the areas of early childhood, reading, career and technical education, special needs, or adult education?
It is important for you to have some expertise in the content of the programs you will be evaluating—if for no other reason than to communicate with the staff and clients of the program. However, in some instances—a goal-free evaluation, for example—it may be beneficial for the evaluator to be “uncontaminated” by knowledge of the issues and nuances of a particular content area.
Once you have decided to work within a given sector, then you can narrow your search. The federal government publishes its Catalog of Federal and Domestic Assistance (CFDA) on an annual basis. The CDFA is also available online (http://12.46.245.173/cfda/cfda.html). This document lists all areas for which federal dollars will be made available to potential programs. Some of these funds go directly to state governments to distribute in the form of block grants or formula grants. Others are distributed by the various departments of the federal government in the form of federal categorical grants. Still others are solicited by the myriad agencies of the federal government in the form of contracts.
Most of the grants require inclusion of evaluation of the grant activities. Thus once you know the arena in which you will operate, you simply find the appropriate funding streams, identify the regular grant solicitors or recipients, and put out feelers for new agencies who might be submitting such proposals. Then you contact these proposal preparers and ask whether they are in need of a person to develop the evaluation design or plan. Sometimes this person or agency is already identified, but often it is not. This is your foot in the door.
Federal block or formula grants are administered and distributed through the appropriate state department or agency, such as the Department of Education, the Department of Transportation, or the Department of Human Services. Contact the program officer assigned to these particular grants and ask to receive requests for proposals (RFPs) for evaluation services. Often local recipients of these federal and state funds have indicated in their proposals that they will perform an evaluation of program activities, but they have not identified an evaluator as yet.
Federal contracts procure the services, products, and materials that are required by each and every agency, department, or unit of the federal government. Often such services are evaluation services. However, bear in mind that this path may take you out of your geographic area. Such contracts are published daily in the Commerce Business Daily (CBD). This publication can be ordered from the federal government's printing office or can be obtained online (http://cbdnet.gpo.gov). Such contracts can be short term or may extend over several years. However, the turnaround time to bid on such contracts is usually short (two to three weeks), so you must be prepared to react quickly to these requests and to provide services upon award of a contract.
If you wish to operate in the private sector, then again you need to identify sources of funds that might be available to operate programs. Usually, foundations—whether community, corporate, family, national general purpose, or special purpose—or corporations will be the sponsors of programs that you might evaluate. An excellent source of information on the thousands of registered foundations is the Foundation Center located in New York City. You can also research potential sources through the Foundation Center online (http://fdncenter.org/funders). This is an excellent source of information on the myriad foundations and corporations that provide funding to programs in myriad arenas including education, health, social services, and religion. As with the public sector, once you have identified a content area (such as education, health, or social services), you then need to identify the appropriate foundations or corporations that fund programs in that content area and contact them directly to inquire about their own evaluation needs, or the evaluation needs of their funded programs. Sometimes this approach is not possible because so many foundations have only a post office box and no office staff. In that case, obtain a copy of their annual report (or access it online, if available) and review the list of funded agencies. Usually, once an agency has been funded by a foundation or corporation, it will continue to be funded unless there has been a major problem with the agency or a change in funding direction by the foundation or corporation. Contact those funded programs and inquire whether they are in need of program evaluation services.
Often we do not have the luxury of ample planning time to be proactive in our search for evaluation work. This being the case, one could begin by using the same resources mentioned earlier to identify public and private sources of funding, then watch for public solicitations in the form of requests for proposals (RFPs) from federal or state departments or local funding recipients. Respond to these RFPs in the form of a proposal to provide third-party evaluation services on that program. Also look for announcements of funded programs on Web sites of particular foundations or corporations. Often funding agencies will make public announcements in national newsletters or in local newspapers of programs that have been funded for their next cycle.
To reply to these announcements, contact the specific agency to inquire about providing third-party evaluation services. Again, many programs will have already included an evaluation plan and a specific evaluator in their proposal. But often the evaluation has not been solidified or it might even be an afterthought.
Using all the resources discussed above—and even following all the suggested steps—does not guarantee that you will be successful in getting a program to commit to an evaluation contract prior to funding or upon receiving funding. Often it is only after a program is funded and has begun operation that the program director realizes—usually after the first monitoring visit by the funding source—that they need to hire a third-party evaluator. This is where you come in.
The question then becomes whether sufficient time remains in the program cycle to effectively institute and complete a quality evaluation. If this is the first year of a multiyear program cycle, then the answer may be yes. If, however, there are only a couple of months left in the program cycle, then the answer may be no. It really is your call at this point.
In the latter scenario, if a summative evaluation is called for, then you might have enough time to amass the data already collected by program staff, analyze the data, and complete a final report. If, however, a formative evaluation was expected, you really do not have sufficient time to provide feedback to the staff that might help them improve services to clients.
Here is where you enter into an ethical dilemma—where you have to make a decision whether to take the contract (and the money) or to decline the opportunity because you really do not feel you can have an impact as an evaluator. The decision to accept the contract might be easier if you know that there could be further work with that agency on other programs or that you might be asked to prepare the evaluation plan for other proposals that will be developed and submitted. Thus you can establish a track record with that agency. Also, if you are looking for an opportunity to establish some expertise in a particular content area, this could be that opportunity.
This is probably one of the most difficult topics in this book for us to cover and for you to learn. It is difficult because you need to determine what your time, expertise, and work are worth. Perhaps this task is easy for you. You may have determined your hourly, daily, or weekly rate or pay already. You may already understand your total resource needs and how to balance one or several contracts to meet those needs. Perhaps you know how busy you want to be doing third-party evaluation or how much of your professional time you want to spend on evaluations. Finally you may already know which part(s) of program evaluation you like to or want to perform. However, these are all things that the majority of people reading this book have not decided or even thought about. So let's begin with the easy part: negotiating responsibilities.
As you have learned thus far in this book, a number of duties need to be performed in carrying out a program evaluation. Once the evaluation design has been established, certain duties can be performed by you, the evaluator, or by the program staff.
If you are in a situation where you are developing the evaluation plan to be inserted into a proposal that is being submitted to a sponsor for funding, then you have some influence on both the evaluation design and the evaluation budget. The rule of thumb is that a program evaluation should cost approximately 7–20 percent of the overall program budget. This wide range is reflective of the importance placed on the evaluation by the funding source. Funding sources that look to the program as an opportunity to test new techniques, theories, methodologies, or products for subsequent transportation to other programs may place a higher premium on evaluation. Other funding sources that are looking at whether the program is performing to the expectations stipulated in the proposal may put a lower premium on evaluation.
Usually you can get a sense of this cost in the RFP materials provided by the funding source when the costs for evaluation services are noted in the proposal preparation materials. Also, the importance of evaluation services is implied in the formula (provided in the proposal preparation materials) for how the proposal will be evaluated by the sponsor. Usually these formulas total 100 percent and give portions of the proposal different percentages or weights. For example, the management plan might be worth 30 percent; the staff qualifications, 25 percent; the budget, 25 percent; the institutional experience, 15 percent; and the evaluation, 5 percent. One could assume by the weight given to the evaluation plan that the sponsor is interested in a quality (that is, high funded) program evaluation. If the evaluation plan is embedded in the management plan, or not stipulated at all, then one might infer that there is less interest in a quality program evaluation—the evaluation will likely be low funded.
If you find yourself in a situation with few dollars to support the program evaluation, then you might negotiate with the program director that many of the data collection activities—such as record keeping, administering tests, performing client interviews, and mailing surveys—will be performed by program staff. In this way, the evaluator will be responsible for preparing data for analysis, analyzing the data, interpreting, and report writing. Often data collection activities are the most time consuming and therefore most expensive evaluation activities. If there is a larger budget available or a concern that objectivity be maintained, then the evaluator would be responsible for these duties. This will make the cost of the evaluation higher.
This negotiating between the evaluator and the program director is necessary so that duties are clearly delineated and agreed to and data collection operates smoothly once the program is in operation.
If you are an established consultant, you already know your hourly, daily, or weekly cost. If not, then you need to establish these rates prior to developing an evaluation budget. Perhaps the easiest way to do this is to determine your target annual salary. Let's be realistic but not too modest and say $60,000 a year. Dividing this by 52 weeks, your weekly rate is roughly $1,154.00 per week. Considering that there are roughly 250 potential work days in a typical year (364 minus the 104 weekend days and 10 holidays), you would come up with a daily cost of $240.00. Finally, assuming a normal workday of 8 hours, you would have an hourly rate of $30.00 per hour. This might sound pretty good to some of you and not so good to others.
Now you need to begin thinking like a self-employed business owner and compile a list of additional costs to doing business, aside from those specific to the evaluation process. Health insurance, life insurance, FICA tax, professional liability insurance, office expenses—such as supplies, telephone, Internet access, utilities—and professional development costs all need to be factored in as well. When you add them to the annual salary, they might even double the above estimates or more.
Once you have established a cost for your services, you need to determine how much time specific services will require. If you are developing the evaluation design, how long will that take you—one hour, one day, one week? If you are identifying or developing instruments, how long will that take you? How about if you will administer the instruments—give tests to 10 groups of 25 learners, perform interviews of 100 clients, send out 1,000 surveys? How long will these activities take? Analyzing data does not just mean that you look at some numbers or statements and draw some conclusions. You will be preparing the instrument for data entry, entering the data into a formula or computer program, running the analyses, reviewing the results, maybe rerunning some analyses, and then interpreting the analyses to figure out what they mean. Finally, writing the evaluation report takes time if it is to effectively communicate to the program staff or a sponsor useful information that will help them make decisions and improve services. All the preceding does not begin to cover the number of meetings you will have with the program director or project staff to give periodic updates, if you are performing a formative evaluation. Unfortunately, no one can teach you how long these activities will take—you are the only one who can do that. Perhaps you have an excellent analytic mind so that developing the evaluation design comes easily, or you know of instruments already, or your data crunching skills are marvelous, or writing comes easily to you. Then these activities might take less time or more time.
Beyond estimating your costs is determining the actual costs of the program evaluation. These might include
Again, depending on the funds available for program evaluation, you might need to negotiate activities that will be performed by the program staff, as well as materials and services that will be procured by the program in order to decrease evaluation costs.
Regardless of whether you are involved early, are preparing an evaluation plan to be included as part of a proposal, or are involved later after the program has received funding from a sponsor, you will need to prepare an evaluation contract. This becomes the binding agreement between you and the sponsor of the evaluation. It should stipulate what you see as program evaluation activities, your responsibilities as the evaluator, time lines, deliverables, and the costs of the evaluation. Usually, these documents are brief and succinctly outline evaluation activities, duties, and expectations for all parties. An example of such a contract can be found in Appendix C.
The components of such a contract should include a cover page that presents a summary of what you or your evaluation company are proposing to perform, a description of the evaluation design that includes evaluation activities linked to goals/objectives, data collection, and responsibilities of the evaluator and program staff. If more than one report or product is expected, then a time line for specific evaluation activities and deliverables should be presented. The activities and deliverables included in the time line should address those for which the evaluator is responsible and those for which the project staff is responsible. Finally, the budget should be included with explanations of staffing costs (for yourself and any others), supplies, materials, travel, and reproduction costs. Including an evaluation résumé is a nice added touch.
If there are issues regarding who must be responsible for confidentiality in collecting data, maintenance of data collected, and security of data over a long period of time, these points should be spelled out clearly in the contract as well.
The nature of evaluation work is such that many program directors really would rather leave it to someone else to do. They are experts in identifying client needs, designing programs to address those needs, and operating those programs. The function of determining whether those programs were efficient, effective, or had impact on clients is important, but not something that they or their staff wish to do. Therefore if you propose to do this evaluation work and perform it in a manner that addresses the needs of the program and the program sponsor, then you have made your evaluation customer happy. Obviously, the best way to keep your evaluation business alive is to deliver on what you promised.
Perhaps you might think that the information in the preceding paragraph is overly simplistic and clearly common sense. However, as you are not the only person reading this book, you are not the only person who is contacting potential program directors and sponsors to inquire about their third-party evaluation needs. Even if you are successful in obtaining a contract to perform a program evaluation, there is no guarantee that that contract will continue for multiple years or that you will be hired by that agent for other program evaluations.
In the public sector, sometimes the person who submits a lower bid will obtain the work. Sometimes the quality of the evaluation design will make the difference. Sometimes your content expertise—or the expertise of the members of your evaluation team—becomes the deciding factor. The message here is that you need to stay connected with the issues and nuances of the arena in which you select to work. If you have decided to perform program evaluations of career and technical education programs and local school districts, then it would be wise for you to maintain your knowledge of what is happening in the federal funding world around the Perkins Vocational Education Act or what is happening in workforce development or employer training. This approach includes connecting with the state Department of Education official who is responsible for administering federal funds to local programs, maintaining a connection with the national and local professional associations that represent career and technical education professionals, and understanding the cycle of funding for such programs.
When possible, either in developing the evaluation plan to be included as part of a funding proposal, responding to an RFP for evaluation services, or meeting the evaluation needs of a program already in operation, shoot for a long-term contact. If you know that the program is proposing to run for three years, design your evaluation so that different things are examined in each year of the program funding cycle. Of course, there will be specific components that you will examine each year, but propose a follow-up of completers after the first cycle in the second year and a long-term examination of impact the third year.
By establishing a long-term contact at the outset, you will not only cement the relationship between you and the program but you will also ensure that the program director need not put the evaluation contract out to bid every year.
Also, a continuing relationship with either a state department or local recipient means that you will be physically present in those settings, thus making your opportunities to be included in new funding endeavors all the more likely.
The purpose of this chapter is to explain that although program evaluation is primarily concerned with improving services to clients, it is also a means by which an individual can make a living or supplement her income. This is not to say that you become an employee of a program and as a result give up your responsibility to be objective in evaluating that program. On the contrary, the program needs an objective third-party evaluation performed by a professional evaluator, and it is your ability to maintain your objectivity that makes you valuable.
The mechanics of operating one's own business may be uncomfortable to many, but the business of performing third-party program evaluations does not have to be especially complex. Begin with the understanding that we discussed at the beginning of this book: by enabling a program to examine its own performance, you are providing a key service to that program that should lead to program improvement. If you can convey that message to the program director and the program staff, then they should see the value in continuing to contract with you as their evaluator.
Third-party program evaluation: Program evaluation performed by an independent evaluator outside of the organization
RFP: Request for Proposal, a Proposal form issued by a program or grant officer
Public Sector evaluations: Evaluations conducted in the government sector
Private Sector evaluations: Evaluations conducted in corporations and foundations or for individuals