PHE 3025 Assignment Threats to Validity of Research
PHE 3025 Assignment Threats to Validity of Research
Three Experimental Designs
To make things easier, the following will act as representations within particular designs:
X: Treatment
O: Observation or measurement
R: Random assignment
The three experimental designs discussed in this section are:
1) The One Shot Case Study
There is a single group and it is studied only once. A group is introduced to a treatment or condition and then observed for changes which are attributed to the treatment
X O
The problems with this design are:
A total lack of manipulation. Also, the scientific evidence is very weak in terms of making a comparison and recording contrasts.
There is also a tendency to have the fallacy of misplaced precision, where the researcher engages in tedious collection of specific detail, careful observation, testing and etc., and misinterprets this as obtaining solid research. However, a detailed data collection procedure should not be equated with a good design. In the chapter on design, measurement, and analysis, these three components are clearly distinguished from each other.
History, maturation, selection, mortality, and interaction of selection and the experimental variable are potential threats against the internal validity of this design.
2) One Group Pre-Posttest Design
This is a presentation of a pretest, followed by a treatment, and then a posttest where the difference between O1 and O2is explained by X:
O1 X O2
However, there exists threats to the validity of the above assertion:
History: between O1 and O2 many events may have occurred apart from X to produce the differences in outcomes. The longer the time lapse between O1 and O2, the more likely history becomes a threat.
Maturation: between O1 and O2 students may have grown older or internal states may have changed and therefore the differences obtained would be attributable to these changes as opposed to X. For example, if the US government does nothing to the economic depression starting from 2008 and let the crisis runs its course (this is what Mitt Romney said), ten years later the economy may still be improved. In this case, it is problematic to compare the economy in 2021 and that in 2011 to determine whether a particular policy is effective; rather, the right way is to compare the economy in 2021 with the overall (e.g. 2011 to 2021). In SPSS the default pairwise comparison is to contrast each measure with the final measure, but it may be misleading. In SAS the default contrast scheme is Deviation, in which each measure is compared to the grand mean of all measures (overall).
Testing: the effect of giving the pretest itself may effect the outcomes of the second test (i.e., IQ tests taken a second time result in 3-5 point increase than those taking it the first time). In the social sciences, it has been known that the process of measuring may change that which is being measured: the reactive effect occurs when the testing process itself leads to the change in behavior rather than it being a passive record of behavior (reactivity: we want to use non-reactive measures when possible).
Instrumentation: examples are in threats to validity above
Statistical regression: or regression toward the mean. Time-reversed control analysis and direct examination for changes in population variability are proactive counter-measures against such misinterpretations of the result. If the researcher selects a very polarized sample consisting of extremely skillful and extremely poor students, the former group might either show no improvement (ceiling effect) or decrease their scores, and the latter might appear to show some improvement. Needless to say, this result is midleading, and to correct this type of misinterpretation, researchers may want to do a time-reversed (posttest-pretest) analysis to analyze the true treatment effects. Researchers may also exclude outliers from the analysis or to adjust the scores by winsorizing the means (pushing the outliers towards the center of the distribution).
Others: History, maturation, testing, instrumentation interaction of testing and maturation, interaction of testing and the experimental variable and the interaction of selection and the experimental variable are also threats to validity for this design.