Of every one hundred men, ten shouldn’t even be there. Eighty are nothing but targets. Nine are real fighters … we are lucky to have them … they make the battle. Ah, but the one, one of them is a warrior … and he will bring the others back.
—Heraclitus
Personnel suitability screening for high-reliability positions, commonly referred to as assessment and selection, has become the mainstay of psychologists working in operational settings (Picano, Williams, & Roland, 2006; Picano, Williams, Roland, & Long, 2010; Staal & Stephenson, 2006; Williams, Picano, Roland, & Banks, 2006; Weiss & Inwald, 2010). It is necessary to conduct a psychological assessment of candidates for high-reliability positions due to the “no-fail” nature of their missions and the sensitivity to issues of national security or public safety in their work. Moreover, psychologists working in these sectors have an ethical responsibility to reduce risk for their organizations by avoiding the placement of individuals into positions beyond their physical, cognitive, or emotional capacity, while also maximizing potential performance and organizational effectiveness by identifying candidates who are the best fit for a given position. The research literature on personnel suitability screening is rich and diverse. Excellent reviews and analysis of this practice arena are readily available to readers, and we encourage those interested to explore the personnel and industrial/organizational psychology literature (Corey, 2011; Cuttler, 2011; Fine & Cronshaw, 1999; ITFACG, 2000; Ryan & Ployhart, 2014; SIOP, 2018).
This chapter is intended to act as a primer for operational psychologists supporting national security and public safety assessment and selection (A&S) programs. We will provide a brief history of A&S in these environments, describe one such program, and detail the various elements necessary to design and employ personnel suitability screening with high-reliability organizations. Lastly, ethical considerations when conducting A&S will also be discussed.
The earliest documented efforts to conduct psychological suitability screening for national security occurred during World War I. Corresponding developments in post-industrialized America were in part responsible for these efforts. The expansion of commercial entities and an increased focus on work efficiency and on environmental changes swept through personnel and occupational settings. Psychological science was put to work for the first time to identify aptitude, ability, and optimal occupational design. Furthermore, developments in psychological and mental measurements also enabled an exploration of these new psychological frontiers (Vernon & Parry, 1949). The advent of World War II acted as a further catalyst for the employment of psychology in the assessment and selection of high-reliability personnel. The creation of the Office of Strategic Services (OSS), and its employment of assessment centers, is often identified as a landmark moment in national security or defense-related A&S (Banks, 1995; Fiske, Hanfmann, MacKinnon, Miller, & Murray, 1997; Highhouse, 2002; MacKinnon, 1974; OSS, 1948). A review of the OSS and its formation has already been provided elsewhere; however, it should be noted that its development and successful execution has had far-reaching impacts on the community of operational psychology and particularly in high-reliability personnel screening. Many A&S programs, including the one described in this chapter, claim the OSS and its use of assessment center methodology as a progenitor for contemporary high-reliability organizations and their assessment procedures (Banks, 1995; Highhouse, 2002).
Following the breakup of the OSS, and the establishment of various intelligence agencies and special operations forces, operational psychologists carried forward what had been learned from past efforts, infusing this knowledge into new and emerging initiatives (Warner, 2007). For example, the U.S. Special Operations Command (USSOCOM) commissioned a joint analysis, with the Central Intelligence Agency (CIA) and other organizations, to explore the relationship between previous OSS practices (recruitment, selection, and training) and current-day efforts within the CIA and SOF communities to recruit, select, and train their personnel. They concluded that many contemporary practices had in fact changed very little and were similar to those of the OSS during World War II (USSOCOM, 2011).
Other examples of agencies who have benefited from the past work of OSS assessment centers include NASA and the military. NASA initiated its first astronaut selection in 1959. Astronaut candidates were given an extensive battery of physiological and psychological tests (a total of 15 instruments largely neuropsychological and personality related). This assessment accounted for over 30 hours of testing and interviews (Santy, 1994). Within the Department of Defense (DoD), screening of special operations forces (SOF) was formalized in the 1980s and 1990s with the creation of the army’s Special Forces Selection and Assessment (SFAS) program, the air force’s Commando Look initiative, and the navy’s employment of psychological screening during Basic Underwater Demolition or BUDS training (Banks, 1995, 2006; Saier, 1995; USSOCOM, 2011). Assessment and selection procedures had been employed previously by each organization; however, the inclusion of psychological suitability screening had not been systematically integrated until this time. For instance, in 1994, the army’s SFAS program was further revised, and an extensive job analysis was completed to refine psychological screening efforts (Russel, Crafts, Tagliareni, McCloy, & Barkley, 1994). Due to the increasing complexity of the Special Force’s mission, this effort expanded the army’s A&S program beyond measures of physical and mental toughness. Psychological interviews, field observations, and expert judgments formed the foundation of this initiative. Kilcullen, Mael, Goodwin, and Zazanis (1999) used this data to derive attributes that were “best bet” predictors of performance. Similar efforts were produced by Hartman, Sunde, Kristensen, and Martinussen (2003) in their study of Norwegian Naval Special Forces and Patterson, Brockway, and Greene’s (2004) work with U.S. Air Force SOF. An excellent overview of the assessment and selection principles behind such programs can be found in a technical report commissioned by the North Atlantic Treaty Organization (NATO, 2012).
Currently, operational psychologists are involved in designing, developing, and consulting to various national security–related A&S programs (Picano, Williams, Roland, & Long, 2011). The goal of these suitability screening programs is to evaluate the person-environment fit based on a given organization’s mission and culture. Such programs typically employ (1) multiform procedures (different evaluation techniques such as interviews, psychological and cognitive tests, and situational or assessment center tasks), (2) standardization (processes that are systematic, methodical, and consistent across candidates), and (3) validation (ensuring the tasks are operationally relevant and measure what they purport to measure). Many barriers exist to the construction of well-run A&S programs, including limited time, resources, personnel, ethical considerations, and organizational support. In the following section we provide a description of one organization and its assessment and selection program for high-reliability personnel screening.
Numerous experts in the field of personnel suitability have promoted the importance of hiring the right people for an organization to maximize its effectiveness. To paraphrase Jim Collins, in his book Good to Great, the most important part of an organization is “getting the right people on the bus” (Collins, 2011). In our following example, we provide a review of critical phases found among many different A&S programs used to ensure organizations identify, recruit, assess, and select the right people for the bus.
The first phase of suitability screening is to identify prospective candidates for positions within your organization. In our experience, the recruitment phase of an organization’s A&S program is often conducted by professional recruiters or human resource professionals. Within the DoD, these teams often work as organic elements; however, in civilian sectors they may be external consultants brought in to find applicants from across occupational specialties. Recruiting goals include (1) determining what positions need to be filled; (2) assessing when they need filling based on projected future turnover in the organization; (3) identifying specific job qualifications that are required for each position; and (4) locating, contacting, and processing the applications of prospective candidates. Recruiting teams employ many methods for recruitment, including in-person briefings, mass targeted e-mails, and social media platform exploitation (e.g., Facebook ads, Twitter, YouTube videos) to reach candidates. Once identified, candidates are typically invited to complete an application for an available position within the organization.
In our example, once submitted, candidates enter the first phase of the A&S program. During Phase I, the recruiting team, personnel assigned to evaluate job-specific skills (referred to as the “work center”), operational psychologists, and unit leadership review the candidate’s application. The intent of this pre-screening is to identify any “red flags” that may suggest poor suitability (e.g., large amounts of debt, alcohol-related misconduct, security violations, or readiness vulnerabilities incompatible with the organization’s mission), and either offer or decline an invitation to Phase II of the organization’s A&S program.
While Phase I efforts vary among high-reliability organizations, many include this type of recruitment and pre-screening procedures. We are familiar with others that leverage standardized phone interviews and a consented review of a candidate’s social media activity as well. This phase can be time-consuming and expensive. A common recruitment effort (a mass briefing) may result in 600 prospective candidates in attendance with only 20 completing applications, seven successfully passing the pre-screen resulting in an A&S invitation, and ultimately resulting in two applicants being selected for the organization. Needless to say, “getting the right people on the bus” requires a professional effort that is well resourced and staffed. Once invited, candidates enter Phase II or the assessment phase of our organization’s A&S program.
The assessment phase is built on a foundation of previously completed job analyses, predictive attribute development, and assessment staff training to ensure a professional process. Once this foundation is laid, the invitation to prospective candidates, and the execution of the assessment, can proceed accordingly. This typically includes the use of psychological testing, face-to-face interviews, work center evaluations, and peer assessments. We will briefly describe each element in the assessment phase.
A thorough review of the job analysis literature, as it relates to suitability screening, is beyond the scope of this chapter (see Fine & Cronshaw, 1999; SIOP, 2018); however, we will briefly describe such analyses, how they can be completed, and how predictive attributes are often identified. A job analysis should be conducted for each occupational specialty or position within your organization. This process often requires consultation with subject matter experts (SMEs) in the targeted specialty in question. The initial identification of potential characteristics of success (attributes), a ranking of those attributes by SMEs in order of contribution and weight, and recommended sources or methods for capturing this data are commonly part of a job analysis. To ensure that information is reliably captured, assessment staff are trained on the standardization of procedures and variables that may contribute to assessment bias. Information regarding such biases will be addressed later in the chapter.
SMEs further refine the attributes that have been identified to determine what level of proficiency is required during each stage of the candidate’s career (early, middle, late), what attributes are considered trainable versus non-trainable (e.g., physical fitness vs. intelligence), and what methods or measurements can reasonably assess the attributes in question. Identifying measurable attributes that are associated with success is a critical component to any organization’s assessment process (McCausland, Robson, & Sims, 2017).
The next step in the assessment phase is proper “rater training.” Such training is often led by an operational psychologist with the assistance of other assessment staff. It is intended to ensure the standardization of procedures, the professionalism of the process in general, and the reduction the risk of rater bias. Rater training is intended to help work center representatives who are empowered to observe, rate, and report on candidate performance during their assessment. Training should (1) increase understanding about suitability and how it’s defined by the organization, (2) help mitigate biases related to hiring decisions, (3) assist in the development of standardized questions that assess key attributes, (4) enable the objective recording of behavior, (5) prepare assessment staff to classify behaviors reliably, and (6) improve their understanding of the hiring board and assessment process in general. Rater training is a requirement in our organization for anyone who is involved in the suitability screening process. Making such training an annual requirement ensures that the entire organization understands how to record relevant job-specific behavior, classify the behavior across predictive attributes, and differentiate low from high performers during candidate assessments (McCausland et al., 2017).
For the military candidates our organization employs, physical fitness testing is a requirement. The rigors of the fitness test are directly tied to the fitness requirement for the specific position. For some occupational specialties, physical fitness testing simply provides a basic level of fitness, readiness, and motivation, while for other positions in the organization, a high level of physical fitness is required at all times, and failure to perform well may result in an immediate assessment failure.
Psychological testing is a mechanism to objectively capture key psychological variables that are important to success in the organization. This testing aims to measure candidate ability, stability, and motivation. Ability is often assessed through cognitive aptitude or intelligence testing, while stability and motivation are assessed via personality testing. Personnel suitability research has long since known the value of cognitive ability as a valid measure of occupational performance. A general factor known as “g” has been predictive of job-related success across multiple studies (Hirsh, Northrop, & Schmidt, 1986; Hunter & Hunter, 1984; Levine, Spector, Menon, Narayanon, & Cannon-Bowers, 1996; Salgado & Anderson, 2002; Schmidt & Hunter, 1998). A combination of “g” measures, along with work sample performance, has been previously identified as more predictive than either measure alone (Schmidt & Hunter, 1998). Meta-analyses have also demonstrated the predictive value of personality when it comes to occupational performance (Barrick & Mount, 1991; Costa & McCrae, 1992; Tett, Jackson, & Rothstein, 1991).
In the special operations community, organizations are often looking for people who are highly intelligent, stress hardy or stable under pressure (and stable with others under pressure), people who will not quit, people who will work without external rewards and take initiative, and people who act with integrity (who do the right thing when no one is looking). Psychological testing cannot answer all of these concerns, but it does provide psychologists with added insight into how candidates tend to see themselves, how they wish to be seen by others, and how they tend to approach different types of interpersonal challenges. Psychological testing can help psychologists generate hypotheses about candidates that can be tested during other elements of their assessment.
Operational psychologists often conduct face-to-face psychological interviews with candidates during their assessment phase. Psychological interviews tend to be semi-structured and standardized. They contain questions that have been previously vetted for their relevance to attributes of interest and through legal channels to ensure their appropriateness for occupational suitability screening. Standardized questions ensure every candidate is assessed in the same, reliable manner. In addition, probing or branch questions may be required based on a given candidate’s responses. The combination of standardized questions and probing questions increases the quality of information gathered by the psychologist and aids in testing hypotheses generated by the psychological test data.
At the core of every job suitability screening is an assessment of a candidate’s ability to perform job-related duties. While much can be said about the assessment of ability, stability, and motivation, the knowledge, skills, abilities, and other factors (KSAOs) directly associated with job success are critical. Job knowledge (K) concerns what candidates know in terms of retrievable information, technical facts, and procedures pertaining to performance on the job. Skills (S) describe the ability to perform job tasks (the application of job knowledge in the context of job performance). Abilities (A) refer to trainability or a candidate’s capacity to learn and their proficiency at job-related tasks. Lastly, other characteristics (O) often reflect a candidate’s motivation, temperament, interests, or personality that may contribute to day-to-day job performance and fit.
The assessment of KSAOs is often referred to as a work center evaluation. Work center evaluation criteria are developed by the organization’s operational psychologist and work center personnel. Typically, each work center designs a series of assessment events that evaluate key duty tasks and core attributes believed to be linked with success on the job. Psychologists assist in this development to ensure work center evaluations are valid, reliable, objective, and can be used to collect data for future program validation. In the case of multiple work center evaluators, each assessment event should receive multiple independent ratings, later combined to form an overall work center performance profile used in a final hiring recommendation.
Peer assessments are used by organizations during suitability screening based on the assumption that fellow candidates have useful information about each other. In many instances they see attributes in other candidates that assessment staff are unable to see. Peer assessments usually involve candidates rating themselves, as well as the other candidates, across different attributes and events. Questions used in peer ratings may include overall assessments of fit, performance, and character. For example, some organizations ask fellow candidates to rate other applicants across critical attributes and behavior (e.g., “who is the least likely to take input from others”). Peer ratings can be controversial. Personnel suitability is not a matter of popularity nor a simple measure of how well a given candidate is able to get along with others. Moreover, an analysis of the empirical predictive ability of peer assessments is beyond the scope of this chapter; however, for many A&S programs, the data derived from peer assessments is heavily weighted among other performance data collected.
Beginning with a proper job analysis, followed by the identification of predictive attributes, the application of various psychological methods and measurements, a thorough work center evaluation of KSAOs, and concluding with peer ratings, the assessment phase of personnel suitability screening ensures the presence of valid, reliable, and meaningful data to determine an applicant’s job fit (ability, stability, and motivation) for the organization. Employing a multiform model in suitability screening increases the quantity and quality of information gathered on prospective employees. When constructed thoughtfully, each element to an organization’s A&S program adds incremental validity to their overall assessment and decision making.
The third phase of our organization’s A&S process is candidate selection. While the various events surrounding selection may differ from organization to organization, the example we describe here contains four key components or stages found among many other programs, (1) the formation of the selection board, (2) completion of a “paper” board process, (3) completion of a face-to-face board process, and (4) the provision of candidate performance feedback.
While selection board composition varies, it often includes organizational leaders (e.g., the unit commander and senior enlisted advisor for military boards), human resources personnel, recruiting staff, the lead representative for the work center, and the operational psychologist. For civil servant positions, the board may also consist of the contracting officer or chief of staff. In general, selection boards should consist of a small group of key decision makers able to discuss sensitive topics, receive unbridled recommendations, and culminate their deliberations in a hiring decision. It should be noted that the decision to hire candidates into the organization rests with leadership (its commander or chief of staff). Operational psychologists act as consultants in this process, providing fidelity on relevant information and performance-related data to enable a good hiring decision.
After the hiring board is assembled, the candidate’s performance during the week is briefed to the hiring authorities. Some organizations use the term “paper” board for this process because it involves the review of a candidate’s performance without the candidate being present. Paper boards often include the presentation of information about a prospective candidate by recruiting staff, work center SMEs, and the operational psychologist. The recruiting team may brief the hiring authorities on the candidate’s ability to follow instructions during the application process, his or her demeanor throughout the week, and the candidate’s physical fitness performance (if assessed). The work center provides the hiring authorities information on the candidate’s performance on job-specific skills (KSAOs). The work center presents their candidate performance data and work center staff ratings across events accordingly. Candidates are often stratified against one another using composite work center performance ratings. In our organization, work center SMEs provide a final hiring recommendation (recommend, recommend with reservations/concerns, or do not recommend) regarding the applicant’s suitability for the organization (see Table 4.1).
|
Attribute 1 |
Attribute 2 |
Attribute 3 |
Attribute 4 |
|
|
Event 1 |
Comments: Score (1–5): |
Comments: Score (1–5): |
Comments: Score (1–5): |
Comments: Score (1–5): |
|
Event 2 |
Comments: Score (1–5): |
Comments: Score (1–5): |
Comments: Score (1–5): |
Comments: Score (1–5): |
|
Event 3 |
Comments: Score (1–5): |
Comments: Score (1–5): |
Comments: Score (1–5): |
Comments: Score (1–5): |
|
Recommendation: REC / REC w/RES / Do Not REC |
||||
Note. Work center evaluations are standardized and set against objective grading criteria to improve the quality of ratings and reduce bias.
Following the work center’s evaluation, the operational psychologist provides their own briefing to the hiring authorities concerning the candidate’s suitability for employment. This assessment often includes a stratification compared to other candidates, a brief psychosocial history (e.g., family, relationships, education, occupational experience), and a summary of the candidate’s psychological strengths and vulnerabilities. Once this information is presented, and the hiring authority’s questions are answered, the operational psychologist provides a final hiring recommendation (recommend, recommend with reservations/concerns, or do not recommend).
After the assessment staff have provided their inputs to the hiring authorities, a decision to hire (or not) is rendered. When this happens, the candidate is typically brought into the board and offered a position with the organization or declined. However, in some instances, there is significant disagreement between the assessment staff. There may be conflicting information or critical questions left unanswered. Such instances may trigger a face-to-face board during which time the candidate is brought before the panel of decision makers.
If the hiring authority has decided that more information is required to make a hiring decision, the specific information needed is discussed and the appropriate line of questioning is developed. Once prepared, the candidate is brought in front of the selection board and interviewed on the topics of concern. Once the information is obtained, the candidate is released from the selection board and once removed, the board discusses the new information, reevaluates their final recommendations, and the hiring authorities make a hiring decision. At this juncture, the candidate returns to the selection board, is informed of the hiring decision, and is released to the work center SMEs and recruiting team for administrative support (return travel arrangements, personnel actions, additional feedback, etc.).
Feedback can be powerful, but its timing is critical. In general, feedback is reserved for candidates after they receive the hiring decision and not before. The intent is to separate these two events, to give candidates some time to reconstitute after receiving the positive/negative news, and to prepare them to process critical feedback on their performance from their work center SMEs. Most organizations value leaving candidates better than they find them. There are both altruistic reasons to do so and future recruiting considerations at stake.
The fourth and final phase of the screening process is program validation. The validation phase does not involve the candidate and is solely focused on the organization’s ability to validate its decision making through statistical analyses and prediction models. This phase consists of four processes that are briefly described here.
Records management involves storing each candidate’s application, psychological testing, work center evaluations, psychological summaries, and selection board decisions in one standardized format for easy, but protected and controlled access. Some organizations consider their A&S files to be a form of medical records, protected under the privacy act and other regulatory bodies. Others code their records as personnel files subject to greater inspection but still protected. Regardless, operational psychologists who act as the custodians of their organization’s suitability screening files should ensure the proper disposition, electronic storage, and encryption of these records.
After the records are safely stored, the data can be exported to a statistical software package (e.g., Statistical Package for the Social Sciences or SPSS) and coded for further analyses. Following professional “best practices,” 10 percent of the coded files should be cross-checked for accuracy. Once coded, the records can be saved as “complete” in a secure archival file system. Building a personnel suitability database is a critical but resource-intensive undertaking. Data entry and database management can be a full-time job for organizations with robust or large assessment and selection programs.
Validation includes an empirical examination of the relationships between an organization’s assessment and selection program, its predictive attributes, assessment methodologies, and work center KSAOs and actual performance outcomes (success on the job). Gathering performance outcome data is perhaps the single largest obstacle to program validation. Organizations often default to outcome variables that are readily available such as successful selection or completion of a training course. However, more meaningful variables can be captured (e.g., supervisor ratings of job performance or objective measures of productivity) but simply take more time and resources to do so. In the age of “big data,” many A&S programs have large enough databases with requisite performance outcomes to build predictive analytic decision trees providing actuarial recommendations for hiring consideration. For readers interested in big-data applications to psychology, we refer them to Adjerid and Kelley (2018).
All statistical analyses must be developed into a product that nonpsychologists and nonstatisticians can understand. In general, the simpler the better. Organizational leaders tend to be bright and inquisitive but are often managing large volumes of information. Backed by data, operational psychologists do well to anticipate the hiring authorities need for concise and targeted information in a format that can be easily consumed. For example, the ability to build a simple pictorial psychological and performance profile of top, middle, or bottom performers and provide that feedback to a leader will usually be well received.
Although we have presented the validation phase as the fourth and final phase of an A&S program, it is important to note that we believe program validation should be considered at the beginning, when one is first designing one’s organization’s assessment and selection program. The consideration of validation is important before recruitment, assessment, and selection phases, since what data you need for your analysis will ultimately drive the other three phases (e.g., what is included in the application, what psychological tests are used, and what forms are employed to capture other relevant information?). If done well, program validation will reinforce and refine your organization’s A&S program, and it will aid in defending against any challenges to its validity and hiring decisions.
Very little has been written about how to train individuals involved in personnel screening and yet, it is a critical component to any professional A&S program. Operational psychologists are just one of the many individuals who play an important role in the recruitment, processing, assessment, and selection of job applicant candidates. As a result, investing in the quality of those individuals who interact with candidates is crucial. This section provides an example of assessment and selection team training.
Due to the many misperceptions about personnel screening, providing an overview of the process, the players, and the purpose is advised. Assessment and selection programs are executed on behalf of the organization’s leaders. In the military context, this is the commander’s program and as such, recruiters and assessment staff provide support. The primary purpose is to determine the suitability of candidates as potential future members of the organization; however, there are secondary goals that include (1) reinforcing the values, standards, and priorities of the organization; (2) acting as a guide for the behavior of future employees; and (3) being a recruiting tool for candidates even when they are not hired. Altruistically, it is also an opportunity to make people better than you find them. Providing vicarious training or improving on their professionalism is an added benefit to candidates that costs the organization nothing.
Staff supporting A&S programs may know very little about suitability or may have incorrect assumptions about what makes a good employee and what does not. We define suitability as a combination of ability (can they do the job?), motivation (will they do the job?), and stability (are they fit for the job?). Making this determination isn’t easy, and traditionally, human resources personnel have favored KSAO that can be found on a resume or inferred through letters of recommendation and an interview. There is no doubt that these sources are valuable; however, they must be weighed against factors that make an even more significant contribution to the long-term success or failure of employees in the workplace. We add to this list, a sense of self-concept, personality traits and attributes that are long-standing, cognitive capacity, and the motivation required to fuel initiative and drive. To help best assess the presence or absence of these factors, we focus our training in three areas: (1) rater bias, (2) delivery of behaviorally anchored questions, and (3) performance-linked attributes.
The research literature on human bias is large. The truth is simply this, we are not good at objectively assessing facts, and we tend to apply our own personal judgment to others’ behavior. We have highlighted for readers just a handful of common biases we see regularly during suitability screening. The first is a phenomenon known as “Halos vs. Horns.” This is a tendency to allow for first impressions to wash over the rest of a given candidate’s performance. In other words, the staff’s initial feeling or evaluation of the candidate (good or bad) sets a frame or lens through which all other candidate behavior is viewed. Although most professional assessment staff would deny it, we have heard comments including “I didn’t like him the moment I saw his suit” and “I didn’t like how he came into the room, he seemed like he had an attitude.” Although these observations may reflect some substantive variable, too often they become a broad subjective brush for which negatively appraised candidates can never recover, and positively assessed candidates can now do no future wrong. The next common bias seen during assessment and selection is one referred to as the “Leniency vs. Strictness” effect. This is a tendency of assessment staff to either be “easier” or be “harder” on candidates than their peers. In many cases we see a turn toward the latter. Known by others as “the gatekeeper” these members adopt the role of organizational sentry, ensuring that only the finest are admitted—a standard that if imposed in their own day might have removed them from consideration. There are also individuals who follow the path of “Central Tendency,” a habit of rating most candidates as average, for fear of being wrong or finding themselves too far out on a limb. This is particularly common among less-experienced staff who are still finding their way as evaluators. Staying away from extremes becomes a position of safety and removes their judgments from greater scrutiny.
A tendency to recall whatever is processed first or last reflects another common phenomenon, referred to as the effects of “Primacy vs. Recency.” Candidates who create an initial positive impression or end on a high note are well remembered, despite mediocre performance during their assessment. The inverse is true as well. Those who make an initial positive impression or suffer a poor selection board performance, despite strong performance in the middle, may be remembered as a poor candidate. Another common bias seen during suitability screening is “Stereotyping,” the tendency to allow a prejudice or stereotype about a given candidate characteristic to overly influence the assessment of his or her performance or fitness for a position. One such example heard within SOF units concerns candidates with past SOF experience; “he’s a SOF guy, so he gets it.” Although it may be true that past behavior predicts future behavior, membership within a community says nothing about how well an individual may have performed or how the individual will perform in the future. Lastly, the “Similar to Me” phenomenon reflects the tendency to prefer individuals like ourselves. For example, one assessment staff member, when asked about his evaluation of a given candidate, replied, “This guy reminds me of me … I was just like him at this point, he’ll figure it out if we give him a shot.”
What makes a question a good one? The answer is quite simple, it’s whatever gets you the most accurate answer. More so than perhaps any other profession, psychologists are experienced at asking questions, but more importantly, they are trained to ask the right questions, in the right way, at the right time. That said, most assessment staff don’t have the benefit of psychology training, and many don’t tend to ask thoughtful questions. On the contrary, many tend toward closed-ended questions, ones that fail to tap into underlying issues, and questions that are easy to answer, “the right way.” During many selection boards we have heard the following, “Do you like your job?” “If we hire you, are we going to have to worry about your behavior?” “Would you confront a coworker if he or she was doing something wrong?” “What do you think I am thinking right now?” No one has any idea what these types of questions tell you about a candidate. Furthermore, these questions are likely the result of poorly prepared or untrained assessment staff.
To increase the quality of candidate responses, we recommend using competency-based (behaviorally anchored) questions. Focusing your exploration of a candidate’s recent relevant past performance, in a similar environment, is likely the best predictor of his or her future performance.
Inquiries should also focus on core competencies that have been demonstrated to be essential for success in the job. We recommend interviews and boards be structured and planned in advance, and that questions be extended by appropriate and targeted probes.
Personality traits and attributes are often long-standing and represent underlying individual characteristics. They may include factors such as interpersonal extroversion (a degree of sociability or interpersonal warmth), the degree to which an individual tends to be open to various experiences or agreeable with most people, and how conscientious individuals tend to be when it comes to their work. Other attributes speak less about how an individual is with others, and more about how he or she is within himself or herself—is he or she driven and focused or relaxed, does he or she tend to procrastinate, or does a self-imposed pressure to perform lead him or her to complete tasks quickly? Obviously, there are an endless list of these types of attributes and associated behaviors. The key is to identify which attributes are empirically linked to job-related performance. Once identified (often initially through a job analysis and later validated), these become important variables that inform the very design of the assessment and selection program.
The foundation for ethical practice in the support of assessment and selection of high-reliability personnel is Principle A, Beneficence and Nonmaleficence, of the American Psychological Association’s (APA) Ethics Code (Picano, Williams, & Roland, 2006), “Because psychologists’ scientific and professional judgments and actions may affect the lives of others, they are alert to and guard against personal, financial, social, organizational, or political factors that might lead to misuse of their influence” (APA, 2017). While the APA’s first principle speaks loudly to psychologists involved with suitability screening, other APA Ethics Code principles provide useful guidance as well. The Ethics Code charges psychologists to act with integrity and justice (principle’s C and D), “Psychologists seek to promote accuracy, honesty, and truthfulness in the science, teaching, and practice of psychology” (APA, 2017). It also directs that “psychologists exercise reasonable judgment and take precautions to ensure that their potential biases, the boundaries of their competence, and the limitations of their expertise do not lead to or condone unjust practices” (APA, 2017). This section will briefly highlight legal and ethical issues that should be considered by operational psychologists working in this area of practice.
Legislative acts and case law have contributed greatly to this area in the past few decades due to concerns about discrimination in the workplace and the requirement for equal employment opportunities for all. While Congress and the courts have generally exempted the uniformed services from some federal employment legislation, these legislative acts still apply to civilian applicants to military and government organizations. Psychologists should consider and remain current in their knowledge of applicable federal laws to ensure compliance with the law and regulations. Three of the most important legislative acts include the Americans with Disabilities Act (ADA, 1990) and the Civil Rights Acts (1964 and 1991).
Title VII of the Civil Rights Act of 1964 states,
It shall be unlawful employment practice for an employer … to fail or refuse to hire or to discharge any individual, or otherwise to discriminate against any individual with respect to his compensation, terms, condition, or privileges of employment, because of such individual’s race, color, religion, sex, or national origin.
However, it also makes clear provision for the administration of personnel suitability testing,
Nor shall it be an unlawful practice for an employer to give and to act upon the results of any professionally developed ability test, provided that such test, its administration or action upon the results is not designed, intended, or used to discriminate because of race, color, religion, sex or national origin.
U.S. employment discrimination laws and regulations require that if an employer with 15 or more employees administers a personnel selection procedure that results in a substantially fewer number of individuals in a legally protected class (e.g., gender, ethnicity, age, disability, etc.) “passing,” that the employer must justify its decision by proving that the process is “job related and consistent with business necessity.” It is important to remember that adverse impact is tolerated when an exam is supported by validity evidence, the passing point is job-related, and there are no other equally valid tests that would result in less adverse impact. A thorough and legally defensible job analysis is one way of establishing validity. There are many ways of identifying adverse impact. The most common is the rule known as the “4/5ths” or “80%” rule, which states that a selection rate (passing rate) for any protected class of employee which is less than 4/5th or 80 percent of the rate for the group with the highest selection is evidence of adverse impact (Roth, Bobko, & Switzer, 2006). With respect to ADA (1990), personnel suitability screening programs must be careful to use only ADA-approved or compliant tests and procedures during any preconditional offers of employment. Clinically salient psychological measures may be appropriate for suitability screening, but only after they have been included as required medical standards in a given position’s work description and only during post-offer employment actions. Operational psychologists working in this area would be wise to consult with their organization’s legal counsel. Both the ADA and the Genetic Information Nondiscrimination Act (GINA, 2008) prohibit inquiries into the existence, nature, or severity of a disability before an individual has been given an offer of employment, even if such inquiries are job related (Tippens, 2002). Furthermore, genetic information regarding family history of mental illness or other conditions cannot be used in employment determinations.
Our own experience with these matters has led us to revise our organization’s psychological testing procedures, measurements, interviews, and consent forms. All our organization’s suitability screening processes have been subjected to legal review, and any information that may be considered non-ADA or non-GINA-compliant has been removed from our pre-offer employment screening (when conducted with civilian applicants). We recommend similar reviews for legal fidelity and endorsement.
Standard 2.01: Boundaries of Competence, of the APA’s Ethics Code (APA, 2017), requires that psychologists provide services “with populations and in areas only within the boundaries of their competence based on education, training, supervised experience, consultation, study, or professional experience”. Psychologists supporting assessment and selection programs are expected to demonstrate competence in their application of psychological tests, standardization of instructions, application of procedures, and interpretation of data (Moreland, Eyde, Robertson, Primoff, & Most, 1995). Defining competency in this area can be difficult; however, in recent years several organizations and agencies have created training courses designed to standardize and improve training for operational psychologists. Courses in personnel selection and psychological assessment have emerged providing practitioners with the basic requirements to design and conduct suitability screening at their organizations. The outlines of these courses include a history of assessment and selection in national security and defense sectors, the role of operational psychologists in A&S, empirical foundations, measurement tools, job analysis and validation procedures, program design, cultural, ethical, and legal considerations in personnel screening, and information on selection board consultation. In addition to formal coursework, interested psychologists are encouraged to seek training in personnel or industrial/organizational psychology. Consultation with experienced operational psychologists is also recommended. Within the operational psychology community, there exist several different assessment and selection programs. Less-experienced psychologists are encouraged to seek these opportunities to gain experience in A&S work.
Standard 4.01: Maintaining Confidentiality directs psychologists to disclose to the individuals with whom they work, both the limits of confidentiality, and the foreseeable uses of information generated through their psychological activities (APA, 2017; Picano et al., 2011). In most personnel screening programs, individuals are required to sign a detailed informed consent form. It is also recommended that psychologists administer a verbal reminder of limitations to confidentiality, an explanation of the purpose of the assessment and uses of the information gathered, prior to the initiation of the psychological interview. It is important to note that applicants consenting to personnel suitability screening will not receive unlimited confidentiality. By the very nature of their consent, and the intent behind the screening procedures, information about the applicant will be necessarily shared with members of the assessment staff and the hiring authorities. However, decisions about the disclosure of personal information must be weighed against the needs for individual privacy. Only information that is relevant to their suitability for the position in question is appropriate for disclosure. Personal information that may be interesting, but not relevant to their suitability, should not be disclosed to the assessment staff or hiring authorities.
As mentioned in the previous section, informed consent is an important element when conducting suitability screening. According to the APA’s Ethics Code, Standard 3.10a, psychologists must obtain informed consent when providing research, assessment, therapy, counseling, or consultation services “except when conducting such activities without consent is mandated by law or governmental regulation or as otherwise provided in this Ethics Code” (APA, 2017). In addition, Standard 9.03a states that psychologists obtain informed consent for assessments except when “informed consent is implied because testing is conducted as a routine educational, institutional, or organizational activity (e.g., when participants voluntarily agreed to assessment when applying for a job)” (APA, 2017). It is generally understood, and has been the practice of operational psychologists, to secure informed consent prior to conducting suitability screening in national security settings. Sensitivity toward differentiating between clinical psychological assessments and personnel screening has been highlighted as many job applicants fail to fully appreciate these differences. Candidates should be provided a thorough explanation of the role of the operational psychologist and how information about the candidate will be shared with others.
The ethical dilemmas posed by personnel suitability screening can be diverse and challenging. As A&S programs vary, so do the issues they present to the operational psychologists supporting them. We encourage psychologists conducting A&S work, to reach out to other experienced operational psychologists for peer consultation and support.
Organizations differ in their procedures and methodology. In general, programs that employ multiform methods (those using different kinds of evaluation) to include face-to-face interviews, psychological testing, and situational or assessment center tasks increase the quantity and quality of their data (NATO, 2012). Thoughtful job analyses, the identification of predictive attributes, and early validation database management are critical components (Corey, 2011; Fine & Cronshaw, 1999; SIOP, 2018). Training raters in standardization, objective coding of behavior, and the risks of rater bias are also very important. The investment of leadership, and their involvement in the process, often sets a tone for the rest of the organization about the importance of finding the right individuals for positions.
The roles and responsibilities of operational psychologists have expanded rapidly. So too has their involvement in assessment and selection programs across national security, defense, and public safety sectors. This chapter has provided a primer for those practitioners interested in the design, implementation, and validation of personnel suitability screening. Much has been written about this work and readers are directed to explore the personnel psychology and industrial/organizational psychology literature for more advanced applications (Ryan & Ployhart, 2014). As operational psychologists supporting A&S programs for high-reliability organizations, our highest priority is to put “the right people on the bus” and to do so in a manner that is professional and ethical. In doing so, we optimize both human potential and organizational effectiveness.
Adjerid, I., & Kelley, K. (2018). Big data in psychology: A framework for research advancement. American Psychologist, 73(7), 899–917.
American Psychological Association. (2017). Ethical principles of psychologists and code of conduct (2002, Amended January 1, 2017). Retrieved from http://www.apa.org/ethics/code/ethics-code-2017.pdf
Americans with Disabilities Act of 1990, Pub. L. No. 101-336, §2, 104 Stat. 328 (1991).
Banks, L. M. (1995). The Office of Strategic Services Psychological Selection Program. Fort Leavenworth, KS: Army Command and Staff College.
Banks, L. M. (2006). The history of special operations psychological selection. In A. D. Mangelsdorff (Ed.), Psychology in the service of national security (pp. 83–95). Washington, DC: American Psychological Association.
Barrick, M. R., & Mount, M. K. (1991). The Big-Five personality dimensions in job performance: A meta-analysis. Personnel Psychology, 44, 1–26.
Civil Rights Act of 1964, Title VII, Pub. L. 88-352, 78 Stat. 241.
Civil Rights Act of 1991, Pub. L. 102-166.
Collins, J. (2011). Good to great: Why some companies make the leap and others don’t. New York: HarperCollins.
Corey, D. M. (2012). Core legal knowledge in police & public safety psychology. Paper presented at the American Board of Professional Psychology Summer Workshop Series, Boston, MA, July 11, 2012.
Costa, P. T., & McCrae, R. R. (1992). Revised NEO Personality Inventory (NEO PI-R) and NEO Five-Factor Inventory (FFI) professional manual. Odessa, FL: Psychological Assessment Resources Inc.
Cuttler, M. J. (2011). Pre-employment screening of police officers: Integrating actuarial prediction models with practice. In J. Kitaeff, Handbook of police psychology (pp. 135–163). New York: Routledge.
Fine, S. A., & Cronshaw, S. F. (1999). Functional job analysis: A foundation for human resources management. Mahwah, NJ: Erlbaum.
Fiske, D. W., Hanfmann, E., MacKinnon, D. W., Miller, J. G., & Murray, H. A. (1997). Selection of personnel for clandestine operations: Assessment of men. Laguna Hills, CA: Aegean Park Press. (Original work published 1948).
Genetic Information Nondiscrimination Act. (2008). Public Law 110–233, 122 Statute 881, enacted May 21, 2008. Washington, DC: Executive Office of the President, OMB.
Hartman, E., Sunde, T., Kristensen, W., & Martinussen, M. (2003). Psychological measures as predictors of training performance. Journal of Personality Assessment, 80, 87–98.
Highhouse, S. (2002). Assessing the candidate as a whole: A historical and critical analysis of individual psychological assessment for personnel decision making. Personnel Psychology, 55, 363–396.
Hirsh, H. R., Northrop, L. C., & Schmidt, F. L. (1986). Validity generalization for law enforcement occupations. Personnel Psychology, 39(2), 399–420.
Hunter, J. E., & Hunter, R. F. (1984). Validity and utility of alternate predictors of job performance. Psychological Bulletin, 95(1), 72–98.
International Task Force on Assessment Center Guidelines (2000). Guidelines and ethical considerations for assessment center operations. Public Personnel Management, 29(3), 315–331.
Killcullen, R. N., Mael, F. A., Goodwin, G. F., & Zazanis, M. M. (1999). Predicting U.S. Army Special Forces field performance. Human Performance in Extreme Environments, 4, 53–63.
Levine, E. L., Spector, P. E., Menon, S., Narayanon, L., & Cannon-Bowers, J. (1996). Validity generalization for cognitive, psychomotor, and perceptual tests for craft jobs in the utility industry. Human Performance, 9, 1–22.
MacKinnon, D. W. (1974). How assessment centers were started in the United States: The OSS assessment program. Pittsburgh, PA: Development Dimensions International.
McCausland, T. C., Robson, S. M., & Sims, C. S. (2017). Special Tactics Officer selection program: A review and preliminary assessment. RAND Project Air Force.
Moreland, K., Eyde, L., Robertson, G., Primoff, E., & Most, R. (1995). Assessment of test user qualifications: A research-based measurement procedure. American Psychologist, 50, 14–23.
NATO. (2012). Psychological and physiological selection of military special operations forces personnel, TR-HFM-171. North Atlantic Treaty Organization, Research and Technology Organization Technical Report. Netherlands: NATO.
Office of Strategic Services. (1948). Assessment of men. New York: Rinehart.
Patterson, J. C., Brockway, J., & Greene, C. (2004). Evaluation of an Air Force special duty assessment and selection program. San Antonio, TX: Conceptual MindWorks.
Picano, J. J., Williams, T. J., & Roland, R. R. (2006). Assessment and selection of high-risk operational personnel. In C. H. Kennedy & E. A. Zillmer (Eds.), Military psychology: Clinical and operational applications (pp. 353–370). New York: Guilford Press.
Picano, J. J., Williams, T. J., & Roland, R. R. (2012). Assessment and selection of high-risk operational personnel. In C. H. Kenney & E. A. Zillmer (Eds.), Military psychology: Clinical and operational applications, second edition (pp. 50–72). New York: Guilford Press.
Picano, J. J., Williams, T. J., Roland, R., & Long, C. (2011). Operational psychologists in support of assessment and selection: Ethical considerations. In C. H. Kennedy & T. J. Williams (Eds.), Ethical practice is operational psychology: Military and national intelligence applications (pp. 29–49). Washington, DC: American Psychological Association.
Roth, P. L., Bobko, P. L., & Switzer, F. S. III. (2006). Modeling the behavior of the 4/5ths rule for determining adverse impact: Reasons for caution. Journal of Applied Psychology, 91, 507–522.
Russell, T. L., Crafts, J. L., Tagliareni, F. A., McCloy, R. A., & Barkley, P. (1994). Job analysis of special forces jobs. Alexandria, VA: U.S. Army Research Institute for the Behavioral and Social Sciences.
Ryan, A. M., & Ployhart, R. E. (2014). A century of selection. Annual Review of Psychology, 65, 20.1–20.25.Saier, W. E. (1995). An assessment of assessment: Is selective manning right for USAF special operations aircrew? A research report submitted to the faculty in fulfillment of the curriculum requirement, Maxwell Air Force Base, Alabama.
Salgado, J. F., & Anderson, N. (2002). Cognitive and GMA testing in the European Community: issues and evidence. Human Performance, 15, 75–76.
Santy, P. A. (1994). Human evolution, behavior, and intelligence. Choosing the right stuff: The psychological selection of astronauts and cosmonauts. Westport, CT: Praeger Publishers/Greenwood Publishing Group.
Schmidt, F. L., & Hunter, J. E. (1998). The validity and utility of selection methods in personnel psychology: Practical and theoretical implication of 85 years of research findings. Journal of Applied Psychology, 124, 263–274.
Society for Industrial Organizational Psychology, Inc. (2018). Principles for the validation and use of personnel selection procedures (5th ed.). Bowling Green, OH: Author.
Staal, M. A., & Stephenson, J. A. (2006). Operational psychology: An emerging subdiscipline. Military Psychology, 18, 269–282.
Tett, R. P., Jackson, D. N., & Rothstein, M. (1991). Personality measures as predictors of job performance: A meta-analytic review. Personnel Psychology, 44, 703–742.
Tippins, N. (2002). The Americans with Disabilities Act and employment testing. In R. B. Ekstrom & D. K. Smith (Eds.), Assessing individuals with disabilities in educational, employment, and counseling settings (pp. 221–233). Washington, DC: American Psychological Association.
United States Special Operations Command. (2011). The OSS model and the future SOF warrior. Tampa, FL: Joint Special Operations University.
Vernon, P. E., & Parry, J. B. (1949). Personnel selection in the British forces. London, England: University of London Press.
Warner, M. (2007). An end and a beginning: The Office of Strategic Services: America’s first intelligence agency. Center for the Study of Intelligence, Central Intelligence Agency. Retrieved from https://www.cia.gov/library/center-for-the-study-of-intelligence/csi-publications/books-and-monographs/oss/art10.htm
Weiss, P. A., & Inwald, R. (2010). A brief history of personality assessment in police psychology. In P. A. Weiss (Ed.) Personality assessment in police psychology: A 21st century perspective (pp. 5–28). Springfield, IL: Charles C. Thomas.
Williams, T. J., Picano, J. J., Roland, R. R., & Banks, L. M. (2006). Introduction to operational psychology. In C. H. Kennedy & E. A. Zillmer (Eds.), Military psychology: Clinical and operational applications (pp. 193–214). New York: Guilford Press.