Validity: Basic Concepts
- Construct validity in psychological tests. 'Construct validation was introduced in order to specify types of research required in developing tests for which the conventional views on validation are inappropriate. Personality tests, and some tests of ability, are interpreted in terms of attributes for which there is no adequate criterion. This paper indicates what sorts of evidence can substantiate such an interpretation, and how such evidence is to be interpreted.' 60 references. (PsycINFO Database Record (c) 2019 APA, all rights reserved)
- Convergent and discriminant validation by the multitrait-multimethod matrix. This paper advocates a validational process utilizing a matrix of intercorrelations among tests representing at least two traits, each measured by at least two methods. Measures of the same trait should correlate higher with each other than they do with measures of different traits involving separate methods. Ideally, these validity values should also be higher than the correlations among different traits measure by the same method.' Examples from the literature are described as well as problems in the application of the technique. 36 refs. (PsycINFO Database Record (c) 2016 APA, all rights reserved)
- Analysis of the multitrait-multimethod matrix by confirmatory factor analysis The multitrait-multimethod (MTMM) matrix permits examination of the convergent and discriminant validity of psychological measures. Estimation using confirmatory factor analysis (CFA), the predominant analytical technique, has often resulted in severe difficulties, such as out-of-range estimates and convergence problems. This article shows that an important special case of one of the more frequently advocated CFA models is not identified and is therefore not estimable. Because most MTMM data are likely to conform closely to this special case, resulting analyses suffer from empirical underidentification. Alternative CFA models are discussed.
- Conventional wisdom on measurement: a structural equation perspective The applicability of 5 conventional guidelines for construct measurement is critically examined: (a) Construct indicators should be internally consistent for valid measures, (b) there are optimal magnitudes of correlations between items, (c) the validity of measures depends on the adequacy with which a specified domain is sampled, (d) within-construct correlations must be greater than between-construct correlations, and (e) linear composites of indicators can replace latent variables. A structural equation perspective is used, showing that without an explicit measurement model relating indicators to latent variables and measurement errors, none of these conventional beliefs hold without qualifications. Moreover, a 'causal' indicator model is presented that sometimes better corresponds to the relation of indicators to a construct than does the classical test theory 'effect' indicator model.
- The Incremental Validity of Psychological Testing and Assessment: Conceptual, Methodological, and Statistical Issues. There has been insufficient effort in most areas of applied psychology to evaluate incremental validity. To further this kind of validity research, the authors examined applicable research designs, including those to assess the incremental validity of test instruments, of test-informed clinical inferences, and of newly developed measures. The authors also considered key statistical and measurement issues that can influence incremental validity findings, including the entry order of predictor variables, how to interpret the size of a validity increment, and possible artifactual effects in the criteria selected for incremental validity research. The authors concluded by suggesting steps for building a cumulative research base concerning incremental validity and by describing challenges associated with applying nomothetic research findings to individual clinical cases. (PsycInfo Database Record (c) 2020 APA, all rights reserved)
Test Construction: Overview
- Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychologyA study conducted on model and standardized psychology assessment instruments analyzed the test reliability and validity of these instruments in diagnosing behavior disorders and aiding clinicians and psychologists in their treatment. Simple rules, guidelines and standards were provided to assist them in choosing an appropriate test instrument for psychological assessments.
- Validity issues in clinical assessmentThere are a number of validation issues that arise in designing tools for the measurement of behavior and in regards to the common reasons for applying assessment methods in clinical psychology. It is imperative to clearly categorize and understand the phenomenon, to forecast some of the specific behavioral tendencies of the person being examined, to suggest measures to bring about the changes in behavior and to evaluate the changes over a period of time.
- Constructing validity: Basic issues in objective scale development. This reprinted article originally appeared in Psychological Assessment, 1995 (Sep), Vol 7(3), 309-319. (The following abstract of the original article appeared in record 1996-93318-001.) A primary goal of scale development is to create a valid measure of an underlying construct. We discuss theoretical principles, practical issues, and pragmatic decisions to help developers maximize the construct validity of scales and subscales. First, it is essential to begin with a clear conceptualization of the target construct. Moreover, the content of the initial item pool should be overinclusive and item wording needs careful attention. Next, the item pool should be tested, along with variables that assess closely related constructs, on a heterogeneous sample representing the entire range of the target population. Finally, in selecting scale items, the goal is unidimensionality rather than internal consistency; this means that virtually all interitem correlations should be moderate in magnitude. Factor analysis can play a crucial role in ensuring the unidimensionality and discriminant validity of scales. (PsycINFO Database Record (c) 2019 APA, all rights reserved)
- Factor analysis in the development and refinement of clinical assessment instrumentsFactor analysis is one of the most common and valuable procedures used in the development and evaluation of clinical measures. The exploratory factor analysis differentiates between principal components analysis and common factor analysis based on what the research requires. The dimensional structure of a method is validated by the application of the confirmatory analysis using structural equation modeling. Both exploratory and confirmatory analysis can be used in dealing with common problems.
Test Construction: Content Validity
- Content validity in psychological assessment: a functional approach to concepts and methodsContent validity, a part of construct validity, is the extent to which elements of an assessment instrument are appropriate and the degree to which they represent the hypothesis of an assessment purpose. There are a number of features in construct validity, all of which can have an effect on the obtained scores. The definition, importance and the operational nature of content validity have been analyzed and methods of content validation have been suggested.
- Focus Groups in Psychological Assessment: Enhancing Content Validity by Consulting Members of the Target Population A review of articles in Psychological Assessment reveals that many researchers develop instruments without the benefit of consultation with members of the target population. To the extent that researchers do consult the target population, most fail to bring consultation in early enough to inform the identification and specification of key constructs. Moreover, this consultation typically takes the form of one-to-one interviews. The authors' goal in this article was to elaborate on the importance of population consultation as part of content validation and to critically evaluate the potential of one method of consultation, focus groups, to inform multiple stages of instrument development. The authors suggest that this method holds promise for enhancing the content validity of instruments and, ultimately, the validity of research findings.
Test Construction: Revising Tests
- Methodological considerations in the refinement of clinical assessment instrumentsInstrument refinement, the set of methods fabricated for the betterment of an instrument's description of a construct, is essential in the development of efficient clinical assessment instruments. Five objectives should be utilized in undertaking multisample instrument refinement analysis. Measures should be constantly reexamined relative to new findings and theories.
- Factor analysis and scale revision. This article reviews methodological issues that arise in the application of exploratory factor analysis (EFA) to scale revision and refinement. The authors begin by discussing how the appropriate use of EFA in scale revision is influenced by both the hierarchical nature of psychological constructs and the motivations underlying the revision. Then they specifically address (a) important issues that arise prior to data collection (e.g., selecting an appropriate sample), (b) technical aspects of factor analysis (e.g., determining the number of factors to retain), and (c) procedures used to evaluate the outcome of the scale revision (e.g., determining whether the new measure functions equivalently for different populations). (PsycINFO Database Record (c) 2019 APA, all rights reserved)
- On the sins of short-form development. The empirical short-form literature has been characterized by overly optimistic views of the transfer of validity from parent form to short form and by the weak application of psychometric principles in validating short forms. Reviewers have thus opposed constructing short forms altogether, implying researchers are succumbing to an inappropriate temptation by trying to abbreviate measures. The authors disagree. The authors do not oppose the development of short forms, but they do assert that the validity standards for short forms should be quite high. The authors identify 2 general and 9 specific methodological sins characterizing short-form construction and offer methodological suggestions for the sound development of short forms. They recommend a set of 6 a priori steps researchers should consider and 9 methodological procedures researchers can use to develop valid abbreviated forms of clinical-assessment procedures. (PsycINFO Database Record (c) 2016 APA, all rights reserved)
Issues in Testing
- Hypothetical studies and civil liberties. Criticizes J. D. Matarazzo's (see record 1991-05791-001) discussion of psychological testimony about individuals in legal settings in which he asserts that the research evidence evaluating categorical judgments of professional clinicians yields negative results. He dismisses such judgments in favor of 'valid psychological assessment.' It is argued that the validation of expert opinion is expert opinion. (PsycINFO Database Record (c) 2016 APA, all rights reserved)
- Comment on psychological assessment versus psychological testing. The current author comments on Matarazzo's (see record 1991-05791-001) review of history and philosophy of the assessment process and his exposition of the major issues in that area of applied psychology. The clear distinction drawn among psychological testing, psychological diagnosis, and psychological assessment, and the emphasis placed on individual potential, should make Matarazzo's article required reading not only for novices but also for seasoned clinicians. (PsycINFO Database Record (c) 2016 APA, all rights reserved
- Treatment Utility of Psychological Assessment. This article focuses on treatment utility. A definition of treatment utility was provided by S. C. Hayes, R. O. Nelson, and R. B. Jarrett (1987): 'We propose to use the phrase the treatment utility of assessment to refer to the degree to which assessment is shown to contribute to beneficial treatment outcome' (p. 963). Various methodologies to examine the treatment utility of assessment are summarized. Treatment utility studies using various assessment procedures (i.e., diagnosis and functional analysis) and various disorders (i.e., unipolar depression, social or interpersonal problems, and phobic disorders) are described. Suggestions are made as to when elaborated assessment and/or treatment utility studies are needed. Limitations on the generalizability of results of any particular treatment utility study are presented. Despite progress, for most assessment procedures and devices, the treatment utility question remains: What is the degree to which assessment is shown to contribute to beneficial treatment outcome? (PsycINFO Database Record (c) 2016 APA, all rights reserved)
- Methodological considerations in validating computer-based test interpretations: Controlling for response bias. Although computer-based test interpretation (CBTI) systems have been operational for nearly 25 years, their availability and adoption in routine clinical practice have grown exponentially in recent years. This article addresses methodological considerations in CBTI validation studies, emphasizing those design issues relevant to customer satisfaction studies. Specifically, issues of response bias are addressed as they relate to selection of raters and test respondents, use of random reports as a 'control' for spurious ratings of report validity, and both the format and content of ratings. Deficiencies of various studies from the research literature are reviewed, and advantages and limitations of design alternatives are discussed. (PsycINFO Database Record (c) 2016 APA, all rights reserved)
- Psychological testing on the Internet: new problems, old issues The Internet has significantly changed the way people conduct business, communicate, and live. In this article, the authors' focus is on how the Internet influences the practice of psychology as it relates to testing and assessment. The report includes 5 broad sections: background and context, new problems yet old issues, issues for special populations, ethical and professional issues, and recommendations for the future. Special attention is paid to implications for people with disabling conditions and culturally and linguistically diverse persons. The authors conclude that ethical responsibilities of psychologists and current psychometric standards, particularly those regarding test reliability and validity, apply even though the way in which the tests are developed and used may be quite different.
Foundational Readings
Psychometric Theory by This text considers the measurement problems that arise in areas of psychology, education and areas of business such as management and marketing. This edition includes: statistical estimation; modern test theory and discussion of generalizability theory; factor analysis and the factoring of categorical data; new use of computer simulations while maintaining the use of examples that allow hand calculation; and overviews and summaries which have been added to each chapter, in addition to a greater number of worked examples.
Call Number: BF39 .N8 1994ISBN: 9780070478497Publication Date: 1994-01-01
Reliability and the Standard Error of Measurement
- What is coefficient alpha? An examination of theory and applications Psychological research involving scale construction has been hindered considerably by a widespread lack of understanding of coefficient alpha and reliability theory in general. A discussion of the assumptions and meaning of coefficient alpha is presented. This discussion is followed by a demonstration of the effects of test length and dimensionality on alpha by calculating the statistic for hypothetical tests with varying numbers of items, numbers of orthogonal dimensions, and average item intercorrelations. Recommendations for the proper use of coefficient alpha are offered.
Mediation and Moderation
- The moderator-mediator variable distinction in social psychological research: conceptual, strategic, and statistical considerations. The purpose of this analysis is to distinguish between the properties of moderator and mediator variables in such a way as to clarify the different ways in which conceptual variables may account for differences in peoples' behavior. Specifically, we differentiate between two often-confused functions of third variables: (a) the moderator function of third variables, which partitions a focal independent variable into subgroups that establish its domains of maximal effectiveness in regard to a given dependent variable, and (b) the mediator function of a third variable, which represents the generative mechanism through which the focal independent variable is able to influence the dependent variable of interest.
Alternative Prediction Model
- Antecedent probability and the efficiency of psychometric signs, patterns, or cutting scores. It is difficult or impossible to determine the efficiency of most psychometric devices used in clinical psychology, since (1) base rates are seldom provided in the literature and hence comparisons cannot be made between decisions from psychometric devices and from previous experience, (2) insufficient distribution data are provided and samples are too small, (3) cross-validation data are lacking, (4) type of population to which a psychometric device can be applied is not clearly specified, and (5) results are not reported in terms of the number of correct decisions resulting for individuals within the groups studied. Corrective methods are presented. 19 references. (PsycINFO Database Record (c) 2016 APA, all rights reserved)
Judgmental versus Statistical Prediction
- The robust beauty of improper linear models in decision making. Proper linear models are those in which predictor variables are given weights such that the resulting linear composite optimally predicts some criterion of interest; examples of proper linear models are standard regression analysis, discriminant function analysis, and ridge regression analysis. Research summarized in P. Meehl's (1954) book on clinical vs statistical prediction and research stimulated in part by that book indicate that when a numerical criterion variable (e.g., graduate GPA) is to be predicted from numerical predictor variables, proper linear models outperform clinical intuition. Improper linear models are those in which the weights of the predictor variables are obtained by some nonoptimal method. The present article presents evidence that even such improper linear models are superior to clinical intuition when predicting a numerical criterion from numerical predictors. In fact, unit (i.e., equal) weighting is quite robust for making such predictions. The application of unit weights to decide what bullet the Denver Police Department should use is described; some technical, psychological, and ethical resistances to using linear models in making social decisions are considered; and arguments that could weaken these resistances are presented. (50 ref) (PsycINFO Database Record (c) 2016 APA, all rights reserved)
- Clinical versus mechanical prediction: A meta-analysis. The process of making judgments and decisions requires a method for combining data. To compare the accuracy of clinical and mechanical (formal, statistical) data-combination techniques, we performed a meta-analysis on studies of human health and behavior. On average, mechanical-prediction techniques were about 10% more accurate than clinical predictions. Depending on the specific analysis, mechanical prediction substantially outperformed clinical prediction in 33%–47% of studies examined. Although clinical predictions were often as accurate as mechanical predictions, in only a few studies (6%–16%) were they substantially more accurate. Superiority for mechanical-prediction techniques was consistent, regardless of the judgment task, type of judges, judges' amounts of experience, or the types of data being combined. Clinical predictions performed relatively less well when predictors included clinical interview data. These data indicate that mechanical predictions of human behaviors are equal or superior to clinical prediction methods for a wide range of circumstances. (PsycINFO Database Record (c) 2016 APA, all rights reserved)
Self Reports, Peer Reports, Diary Methods, and Diagnostic Interviews
- On the 'remembrance of things past': a longitudinal evaluation of the retrospective method A longitudinal examination of 1,008 children over the years to study the correlation between retrospective and prospective measures of variables in the seven areas of residence shifts, anthropometrics, injuries, reading capability, family features, behavioral difficulties and delinquency reveals that the validity of retrospective reports on psychosocial variables varies with the situation. The psychosocial variables indicate a low correlation between prospective and retrospective methods. The absolute correlation between the data of retrospective and prospective measures is low even when the individual reports of these two sources agree with each other.
- Self-reports: How the questions shape the answers. Self-reports of behaviors and attitudes are strongly influenced by features of the research instrument, including question wording, format, and context. Recent research has addressed the underlying cognitive and communicative processes, which are systematic and increasingly well-understood. The author reviews what has been learned, focusing on issues of question comprehension, behavioral frequency reports, and the emergence of context effect in attitude measurement. The accumulating knowledge about the processes underlying self-reports promises to improve the questionnaire design and data quality. (PsycINFO Database Record (c) 2016 APA, all rights reserved)