ADVERTISEMENTS:
This article throws light upon the two main types of experimental design used in social research. The types are: 1. The ‘After-Only’ Experimental Design 2. The Before-After Experiments.
Type # 1. The ‘After-Only’ Experimental Design:
The After-only experiment is its basic outlines may be represented by the following procedure:
Change = Y2 – V2
ADVERTISEMENTS:
The procedure characteristic of the After-only experiments may be described as follows:
(1) Two equivalent groups are selected. Any one may be used as the experimental group and the other as the control group. As said earlier, the two groups are selected by randomization procedure with or without supplementary ‘matching.’
(2) None of these two groups is measured in respect of the characteristic which is likely to register change, consequent to the effect of the experimental variable. The two groups are assumed to be equal in respect of this characteristic.
ADVERTISEMENTS:
(3) The experimental group is exposed to the experimental variable (X) for a specified period of time.
(4) There are certain events or factors whose effects on the dependent variables are beyond the control of the experimenter. Try as hard as he might, he cannot control them. So these factors may be called uncontrolled events. Needless to say, both the experimental as well control groups are equally subject to their influence.
(5) The experimental and control groups are observed or measured with respect to the dependent variable (Y) after (sometimes, during) the exposure of the experimental group to the assumed causal variable (X).
(6) The conclusion whether the hypothesis, ‘X produces Y is tenable is arrived at simply by comparing the occurrences of Y (or its extent or nature) in the experimental group after exposure to variable X with the occurrence of Y in the control group which has not been exposed to X.
In the tabular representation above, Y2 and Y’2 (after measures) are compared to ascertain whether X and Y vary concomitantly. The evidence that X preceded Y in time, is acquired from the very method of setting up the two groups. The two groups are selected in such a manner that there is reason to assume that they do not differ from each other except by chance in respect of the dependent variable Y.
The final problem of eliminating the effect of other factors, such as contemporaneous events or maturational process is dealt with on the basis of the assumption that both groups are exposed to the same extent and hence undergo similar maturational or natural developmental changes between the time of selection and the time at which Y is measured.
If this assumption is justified, the position of the control group on the dependent variable Y’2 at the close of the experiment includes the influence of external uncontrolled events and natural development processes that have affected both groups.
Thus, the difference between Y2 and Y’2 may be taken as an indication of the effect of the experimental variable. It must be borne in mind, however, that the external events and the developmental processes may interact with the experimental variable to change what would otherwise have been its effect operating singly. For example, the effect of a medicine M may be different when the atmospheric conditions or climate interacts with the medicine.
Thus, babies may register a greater increase of weight when medicine and climate interact with each other as compared to the increase that may be attributed to medicine (M) and climatic conditions (A) operating on the babies independently.
ADVERTISEMENTS:
The major weakness of the After-only experimental design is obvious, viz., ‘before’ measurements are not taken. Both groups are assumed to be similar in respect of the before measure on the dependent variable.
Unless the selection of the experimental and control groups is done in such a meticulous manner that it warrants such assumptions, it is quite likely that the effect the researcher attributes to the experimental variable may really be due to the initial difference between the two groups.
Again, ‘before-measurements’ are desirable or advisable for a variety of reasons. This facility is lacking in the After-only design.
We cannot afford to overlook the possibility that in certain experimental situations, ‘before measurements’ are not feasible owing to certain practical difficulties. Again in certain situations, as we shall have an occasion to appreciate, ‘before measurements’ may not be advisable and the safeguards quite prohibitive in cost.
Under such circumstances the After-only design may be a reasonably good choice provided, of course, that meticulous care is exercised in selecting the groups as equivalents.
Type # 2. The Before-After Experiments:
As their very name would indicate, the ‘before-after’ experiments share common characteristic, namely, the groups are observed or measured before the exposure to the experimental variable.
A ‘before’ measurement of the dependent variable that characterizes the Before-After experiments may be desirable for various reasons such as the following:
(a) A ‘before’ measurement of the dependent variable is necessary for matching the cases in the experimental and the control groups. This measure greatly enhances the sensitivity of the experiment.
(b) A ‘before’ measurement makes it possible to determine the incidences of change in the dependent variable and to take these into consideration in evaluating the effects of the experimental or independent variable.
(c) If the hypothesis of the study specifies the initial position on the dependent variable as one of determining conditions, then obviously, the before measurement is required to test the hypothesis.
For example, the hypothesis may state that an educational programme will have greater effect on persons who have a set of specific characteristics than those who do not have these particular characteristics. In such a case, an initial measure of such characteristics as well as the ‘after’ measure is required by the hypothesis.
(d) If the experimenter is interested in finding out whether the experimental treatment has different effect on cases who were initially at different positions on the dependent variable, he must, understandably have a ‘before’ measure of position on the dependent variable.
(e) In the real life setting, the ideal requirement of selecting the experimental and control groups on a purely random basis is often hard to fulfill and certain compromises are called for.
In such cases, the evidence from a ‘before’ measure that the experimental and control groups were initially equal in respect of the dependent variable helps to increase the confidence that a difference found on the ‘after’ measure is due to the effect of the experimental variable only.
The ‘Before-After experiments may characterize various arrangements and permutations with reference to control groups:
(1) Only one group may be used in the study, with the ‘before’ measure serving as a control, i.e., representing the position of the dependent variable in the absence of the experimental treatment.
(2) The ‘before’ measurement may be on the one group and ‘after’ measure on a different one which is assumed to be an equivalent group.
(3) The ‘before’ and ‘after’ measures may be taken both on experimental groups as well as on one control group.
Whatever the pattern of control groups, the ‘Before-After’ experiment provides evidence of concomitant variations among X and Y, by comparing the occurrence of Y in the group exposed to X with the occurrence of Y in the group not exposed to X.
The second evidence of causality, i.e., that X came before Y in time, is inferred from the assurance provided by randomization that the groups are likely to be equivalent with respect of the referents of Y. This initial equivalence with respect to the referents of Y can be checked by comparison of ‘before’ measures of the two groups.
The ‘before-after’ experiments may involve two or more control groups. The variations in control group arrangements relate to the attempts to take account of and segregate effects of contemporaneous events, maturational or natural development processes and/ or ‘before’ measurement on the experiment.
The possibility of the effects of the ‘before’ measurements on the dependent variable must be reckoned with. The ‘before’ measurement may crystallize the attitudes or views of the subjects or it may exhaust the good will of the subjects.
The subjects may mentally connect the ‘before’ measurement with the experimental treatment as also with the ‘after’ measurement. The ‘before’ measure may thus distort the true effect of the experimental variable. The second (i.e., the ‘after’) measurement may introduce other problems.
The subject may be bored or he may try to give responses which are consistent with his previous responses (elicited during the ‘before’ measurement), may also try to vary the responses just to make them more interesting or just to appear ‘cooperative vis-a-vis the experimenter in his ‘intended’ purpose of being able to show a certain change.
The process of repeated measurement, i.e., ‘before’ and ‘after’ may also affect the measuring instrument, e.g., the observer himself may get fatigued, prejudiced or grow more or less sensitive to the phenomena he is recording. With this general outline of the ‘before-after’ experiments as a backdrop, let us now discuss the specific types of experiments of this class.
The Before-After’ Experiment with a Single Group:
The tabular representation of this type of experiment is given below:
Change = Y2-Y1
It is clear that in this design, the difference between the subject’s positions on the dependent variable before and after the exposure to the independent variable (experimental factor) is taken as a measure of the effect of experimental variable. The subject is made to serve as his own control.
But it is understandable that external factors unrelated to the experimental treatment may have been in operation, leading in turn to a change in the subject’s position on the dependent variable.
Thus, the major weakness of this rudimentary experimental design is that it does not make possible the segregation of such effects (i.e., external, contemporaneous, developmental processes and the effects of ‘before’ measurements) from those of the experimental treatment.
The design may, therefore, be used only when the researcher can assume on just grounds that the ‘before’ measurement does not in any way affect (a) the subjects’ exposure to the experimental variable and (b) the ‘after’ measure.
In addition, the use of this design is justified if the researcher has a sound basis for believing that there are not likely to be any other influences, beside the experimental variable, during the period of experimentation which might have affected the subjects’ response at the time of second measurement.
The ‘Before-After’ Experiment with One Control Group:
The inclusion of one control group in this design is aimed at taking account of the effects both of the initial measurement and of contemporaneous, external factors. In such a design, experimental and the control group are both measured at the beginning and also at the end of the experimental period.
The experimental variable is introduced in the experimental group only. Since both the experimental and control group are subject to the ‘before’ measurement and the uncontrolled factors, the difference between the two groups is taken as the effect of the experimental variable alone.
In view of its typical limitations, this design should be used only in cases where the ‘before’ measure and the uncontrolled events affect the experimental and control groups in the same way. But it is quite possible that the ‘before’ measure or uncontrolled factors may interact with the experimental variable in such a way that its effects is changed.
When such a possibility is present, the ‘Before-After’ study with one control group does not afford a basis for inferring the effects of the experimental variable since it cannot segregate or draw apart the singular effect of experimental variable. R.L. Solomon has devised more elaborate designs to take account of such interactions. These involve use of additional control groups.
The ‘Before-After’ Experiment with Two Control Groups:
The design makes it possible to segregate the influence of the experimental variable from that of the ‘before’ measurement even if there is a likely interaction between them (i.e., experimental factor and ‘before’ measurement). This design may be represented as under:
Interaction = d1 – (d2+d3)
This design involves an addition of one more control group to the previous design, i.e., ‘Before-After’ study with one control- group. This second control group is not pre- measured but is exposed to the experimental variable and subjected, of course, to after measurement.
The ‘before’ measure of the second control group is assumed to be similar to the ‘before’ measures of the experimental and the first control group, i.e., equal to the average of the ‘before’ measure of the experimental group and control group I. Thus, in control group II, there is exposure to experimental variable but no possibility of interaction between the ‘before’ measure and the experimental variable.
If we assume, for a moment, that contemporaneous events or maturational processes are not likely to have significant effect on the dependent variable in this design, then change in control group II, i.e., d3 may be taken as the effect of experimental variable alone.
Again, the change in control group I may be taken as the effect of ‘before’ measurement alone. Further more, the difference between the change in scores of experimental group, i.e., dx and the sum changes in stores of two control groups, i.e., (d2 + d3) may be taken as the effect of interaction between the ‘before’ measurement and the experimental variable.
This interaction may have the effect of enhancing or reducing (in varying degrees) the effects of the experimental variable.
Let us try to understand this by an example. Suppose the researcher wants to test the hypothesis that a new system of instruction (X) has the effect of improving the performance of students at the examination. Should he decide to use the ‘Before-After’ design with two control groups, he would need to follow the procedure shown in the above representation.
He administers a test to two out of the three equivalent groups, i.e., the experimental group and the control group I, to know the ‘before’ measure on the performance of the students.
The ‘before’ measure of the control groups II is assumed to be the average of the ‘before’ measure of the two groups, subjected to ‘before’ measurement. Suppose this measure was 50 marks in both the groups and therefore, control group II is also assumed to measure 50 marks.
Next, the experimental group and the control group II are exposed to the experimental variable, i.e., those groups are exposed to the new method of instruction whereas control group I is taught in the usual way.
Of course, during the time that the groups are subjected to the experimental variable, say for a fortnight, all the groups are equally subject to the effect of factors external to the experiment and beyond the control of the experimenter. Lastly, the ‘After’ measures are taken for all the groups and the changes, i.e., difference between the ‘After’ measures and ‘Before’ measures, are recorded.
It is clear that the change in the control group II (d3) is due to experimental variable, i.e., the new method of instruction, and the uncontrolled events. Now assuming that the uncontrolled contemporaneous events did not have any significant effect on the dependent variable (i.e., performance in terms of marks), this change, let us say (60 – 50 = 10) of 10 marks, may be attributed to the new method of instruction alone.
The change in control group I may be attributed to the effects of ‘before’ measurement, i.e., the awareness in the subjects about the experiment and thus a resultant keenness or extra efforts on their part to do better at the second examination. Let us say, the change amounts to (54 – 50 = 4) four marks.
Thus, the individual effects of the ‘before’ measurement and the experimental variable, assuming the effect of the uncontrolled events as zero, total to fourteen (10 + 4).
Now, the experimental group registers, let us say, a change of (65 – 50 = 15) fifteen marks.
This change is the integrated effect of the ‘before’ measurement, plus the effect of experimental variable, plus the effects of uncontrolled factors, plus the effects of the interactions between:
(a) the ‘before’ measurement and experimental variable,
(b) that between the experimental variable and the uncontrolled factors and
(c) that between the ‘before’ measurement and the uncontrolled factors.
But since there is reason to believe (in our example) that the uncontrolled factors have no or very negligible effect, the interaction of this experiment would really occur only between the ‘before’ measurement and experimental variable differently than they would have, if they were not pre- measured.
Thus, the change, i.e., 15 marks, is the cumulative effect of:
(1) the ‘before’ measurement,
(2) the experimental variable and the interaction between (I) and (II).
From our control groups (I) and (II) it is found that the individual effects of (1), the ‘before’ measurement and (11) the experimental variable, adds up to 14 marks (d2 + d3). But for the interaction the change in the experimental group, i.e., d: would be equal to (d2+d3), i.e., 14 marks. We find however, that (d1=15) exceeds (d2 – d3) by 1 mark.
This means that the interactional effect of (I) and (II) is equal to + 1. (The interactional effect might be negative also). It is clear now that this experimental design is useful and efficient only in situations where there is a sound reason to believe that the uncontrolled contemporaneous events or maturational processes are not likely to have significant effects.
‘How would we proceed in a situation where such uncontrolled factors are quite likely to have important influences on the dependent variable?’
R.L. Solomon has provided an answer to this question by proposing a further elaboration of the earlier two control groups with a view to installing safeguards when contemporaneous events or developmental changes may be expected to influence experimental results. This involves the addition of a third control group.
Before-After Study with Three Control Groups:
Interaction = d-, (d2 + d3 – d4) (y’2 – y’1)
As should be clear from the above representation, the experimental group and control group I are subjected to ‘before’ measurement. As with the previous design (with two control groups), the control groups II and III are not pre-measured and are assumed to have the pre-measure score equal to the average of such scores in the experimental and control group I.
The experimental variable is introduced to the experimental group and control group II. All the four groups are assumed to be equally subject to the effects of external contemporaneous events, let us say. Some national event or some campaign, etc., during the period of experiment. All the four groups are measured after the experiment.
In such a design, the change in control group III, i.e., d4, represents the effect of contemporaneous events beyond the experimenter since this happens to be the only factor operative on this group. The change in control group II, i.e., d3 represents the effect of the experimental variable and of the contemporaneous events.
Change in the control group I, i.e., d2, represents the effects of ‘before’ measurement and of the contemporaneous factors. The effect of the experimental variable alone, i.e., of the new method of instruction, can be assessed by subtracting the change in control group Ii from the change in control group II, i.e., d3 – d4.
The change in the experimental group, i.e., dp reflects the cumulative effects of ‘before’ measurement, of the experimental variable, of uncontrolled events and of interaction between these factors.
Now this design affords us the individual measures of the effects of uncontrolled factor, i.e., d4 (the effect of say, some national campaign which keeps subjects more informed about certain events or things, thus, improving their performance in the examination) and of the effects of experimental variables alone (d3-d4) and finally the effect of ‘before’ measurement (d2– d4).
Hence, we can easily calculate the interactional effect of the three factors, i.e., (a) ‘before’ measurement, (b) experimental variable and (c) the uncontrolled factors, on the dependent variable, i.e., the examination score by subtracting the total of the individual effects of the three factors (a), (b) and (c) from the total change registered in the experimental group. Thus, interactional effect would be equal to dx – (d2 + d3 – d4).
It may be observed that this experimental design with three control groups is tantamount to doing the experiment twice, i.e., once with a ‘before-after’ design with one control group (experimental group and control group I) and the second time, with an ‘after only’ design (control group II and III).
In the context of the discussion on the various types of experimental designs, it must be remembered that these experiments suffer from a general limitation of a practical nature, i.e., the researcher is not always in a position to test a causal hypothesis by assigning subjects to different conditions in which he directly controls the causal – (experimental) variable.
For example, if the hypothesis was concerned with the relation between smoking and cancer, the researcher would hardly be in a position to control the extent of smoking as per the ideal requirement of the experimental procedure by assigning different persons to smoke different number of cigarettes.
All that the researcher can get at is a record of how much an individual has smoked and of whether he has cancer. The correlation between smoking and cancer may be computed. But the existence of a correlation between smoking and cancer does not mean necessarily that one is the cause of the other.
The researcher must contend with the possibility expressed by the correlation that people who smoke heavily are for some as yet unknown reason, also the kind of people who develop cancer, if, therefore, a non- experimental study (since ‘experimental’ control as in this example, is not possible) would provide a test of ‘causal’ hypothesis, it must provide ground for making inferences about causality and safeguards against unwarranted inferences.
But the non-experimental studies cannot provide such safeguards as adequately as the experimental studies do. Certain substitute safeguards are available.
These safeguards involve comparison of people subjected to contrasting experiences in the real life-setting, determination of the time-order of variable (supposed ’cause’ and ‘effect’) and examination of the relationship between the variables in terms of pattern of relationship that might be anticipated if one or the other were to be the causal condition.
Comparison of Groups Exposed to Contrasting Experiences:
If an investigator is not in a position to assign subjects to different groups, one which will be exposed to a given treatment and one of which will not be so exposed, then the only alternate solution is to locate groups of people in the natural setting who are about to be or have been exposed to experiences that differ with respect to the assumed causal variable in which the researcher is interested.
For example, if the researcher was interested in the effect of community development programme, i.e., one of these would be the one exposed to the C.D. programme and the other equivalent community would have to be the one that did get exposed to the C.D. programme.
Such a study approximates an experiment in the sense that the community in which the C.D. programmes have been in operation represents the ‘experimental’ group and other community represents the “control” group.
The difference between the two communities in terms of certain pertinent characteristic may be attributed to the causal variable, i.e., the C.D. programme. Of course, we must be aware of the obvious difficulty involved in selecting those groups (communities) that are equivalent in all respects and differ only in respect of the exposure to the assumed causal variable.
In the real life-setting, it would be a stroke of good luck to come across such comparable groups differing only in respect of the causal variable. The type of design we have just discussed may be called the ‘ex-post facto’ design.
Studies using the ‘ex-post facto’ pattern suffer from a serious limitation, namely, that the subjects cannot be randomly assigned to different conditions and there is no possibility of prior measurements to check whether the two groups were initially similar in their position on the assumed dependent variable due to absence of before measurement or in respect of other characteristics believed to be relevant to it.
As suggested earlier, the researcher may sometimes be in a position to locate two groups of comparable people one of which is about to be exposed to certain experiences (assumed causal variable) and the other, not likely to be so exposed.
Such a study approximates a ‘before-after’ experiment with one control group. The group of subjects about to undergo a particular experience, e.g., those selected to undergo a particular orientation course, represents the ‘experimental’ group; those not selected represents the ‘control’ group.
Let us now discuss how one can get at the second kind of evidence necessary to establish causality, i.e., evidence of the time-order or variables in a non-experimental study design. In some cases, the evidence that X preceded Y and not vice versa, is so clear that no supplementary evidence is needed.
Often, however, the time-relationship between two variables is not so clear. Even though one appears to be prior to the other, this may not really be the case. For example, in a study of the effect of early experiences on the typical response pattern during adulthood, a researcher may have to rely on his adult subjects’ accounts of their childhood.
What he would get out of the adults is actually likely to be statement (about childhood) that have been heavily coloured by the subjects’ personal interpretations based on their personal ‘theories’ and their prospective reflections as adults.