Application

Sample Size and the Precision of Alpha

As we mentioned before, alpha is sample dependent. Thus it might be expected that sample size can affect the precision of our alpha estimate. When a smaller sample is used to estimate alpha, each observation contributes a lot of information to the estimate. An outlier or deviation from expectation in a small sample can have a profound effect on the results and can cause them to vary from the population mean. However, in a larger sample, the outlier or deviation will likely be averaged out by the other observations.
Let’s explore this issue of sample size and precision with the Marsh SDQ data that we’ve been using throughout the book. Let us start by examining the reliability of the parent subscale in the complete data set. The parent subscale is composed of five items:
  • (Par1) My parents treat me fairly.
  • (Par2) I do not like my parents very much.
  • (Par3) I get along well with my parents.
  • (Par4) My parents are usually unhappy or disappointed with what I do.
  • (Par5) My parents understand me.
Obviously Par2 and Par4 are reversed in direction. If you calculate alpha using these five items as is (see Figure 11.2 Output with reverse coded items), you will get a negative alpha (which is impossible) as well as some negative item-total correlations (another sign of trouble).
Figure 11.2 Output with reverse coded items
We must therefore recode our two variables and rerun. Since we likely want to retain a copy of our original data, we will create a new variable with the recoded values in it, Par2_recode and Par4_recode. The syntax for our initial run (summarized above), the recoding, and the revised run are presented below along with comments. Please note we provide two options for recoding—a long way with individual value recoding for each variable and a shorter way using arrays. 
**Try to compute alpha for parenting items;
proc corr data=sdqdata alpha;
   Par:;
run;
**Recode and re-run;
*Recode option 1 - the long way;
data sdqdata;
   set sdqdata;
   *recode Par2;
   if Par2=1 then Par2_recode=6;
   if Par2=2 then Par2_recode=5;
   if Par2=3 then Par2_recode=4;
   if Par2=4 then Par2_recode=3;
   if Par2=5 then Par2_recode=2;
   if Par2=6 then Par2_recode=1;
   *recode Par4;
   if Par4=1 then Par4_recode=6;
   if Par4=2 then Par4_recode=5;
   if Par4=3 then Par4_recode=4;
   if Par4=4 then Par4_recode=3;
   if Par4=5 then Par4_recode=2;
   if Par4=6 then Par4_recode=1;
run;
*Recode option 2 - arrays!;
data sdqdata (drop=i);
   set sdqdata;
   array old(*) Par2 Par4; *old variables to recode;
   array recode(*) Par2_recode Par4_recode; *new versions;
   do i=1 to dim(old);
      if old(i)=1 then recode(i)=6;
      if old(i)=2 then recode(i)=5;
      if old(i)=3 then recode(i)=4;
      if old(i)=4 then recode(i)=3;
      if old(i)=5 then recode(i)=2;
      if old(i)=6 then recode(i)=1;
	end;
run;
**Re-run;
proc corr data=sdqdata alpha;
   var Par1 Par2_recode Par3 Par4_recode Par5;
run;
The updated results from our run with the recoded Par2 and Par4 are displayed in Figure 11.3 Updated output with recoded items. Since we input the raw data and not standardized versions of our variables, we interpret the raw coefficients presented. We now have a normal (and quite decent) looking alpha of 0.834, with an N = 15661. Further, the item-total correlations all range from 0.569 to 0.733, which is reasonable.
Figure 11.3 Updated output with recoded items
Let’s think of this as our gold standard “population” statistic and see what small samples can do to the estimation of alpha. Using PROC SURVEYSELECT, we drew 1000 subsamples of N=50, 1000 subsamples of N=100, and 1000 subsamples of N=250 from the data set using simple random sampling with replacement. The syntax to do this is provided below. 
**Take 1000 subsamples of N=50;
proc surveyselect data=sdqdata method=SRS n=50 out=sdqdata_sub50 
   seed=5092 reps=1000;
run;
**Take 1000 subsamples of N=100;
proc surveyselect data=sdqdata method=SRS n=100 out=sdqdata_sub100 
   seed=821 reps=1000;
run;
**Take 1000 subsamples of N=250;
proc surveyselect data=sdqdata method=SRS n=250 out=sdqdata_sub250 
   seed=291 reps=1000;
run;
We then computed alphas for the parent scale in each subsample and plotted the distribution of alphas by sample size to get an idea of the range of the estimates. Syntax to do this step for the 1000 subsamples of N=50 is presented below. The syntax for the other two sets of subsamples is identical to this, except that the names of the input (and generally the output data sets if you are following best practice and changing names of important output data sets) differ. In this syntax, we use ODS to output a data set containing the alphas, and then we use ODS graphics and the SGPLOT procedure to plot our histograms. 
**Estimate alpha in each subsample;
ods output CronbachAlpha=alpha50;
*noprint option used to suppress printing and outp used to output data set 
 with alphas;
proc corr data=sdqdata_sub50 alpha nosimple nocorr noprob; 
   by Replicate;
   var Par1 Par2_recode Par3 Par4_recode Par5;
run;
ods output close;

**Print histogram of alphas;
ods graphics on /height=3in width=3in;
proc sgplot data = alpha50 noautolegend;	
   title "Sample Size = 50";
   where Variables='Raw';
   histogram Alpha / binwidth=.025 FILLATTRS=(COLOR=cxE5EAF2);
   xaxis label = 'Alpha' grid values = (.55 to 1 by .1);
run;
ods graphics off;
In general, subsample alphas that are closer to our “population” alpha of 0.83 can be considered more precise. However, those farther away are less precise. As you can see in Figure 11.4 Distribution of Cronbach’s alpha among 1000 samples of N=50, N=100, and N=250, alpha became somewhat volatile in the smaller samples. The observed alphas ranged from a low of α = 0.56 to a high of α = 0.94 in the smallest samples with N=50. The range narrowed a little further in the samples of N=100, with a low of α = 0.70 and a high of α = 0.92. Finally, the samples of N=250 had the narrowest alpha range, with a low of α = 0.74 and a high of α = 0.89. Thus, the precision of the alpha improved as sample size increased. This serves to demonstrate that even in a scale with reasonable strong psychometric properties (such as this), small samples tend to have a good deal of volatility.
Figure 11.4 Distribution of Cronbach’s alpha among 1000 samples of N=50, N=100, and N=250

Confidence Intervals for Alpha

Fan & Thompson (2001) point out that few authors provide context in their papers as to the precision of their effect size point estimates. As we saw in the previous example, the precision of alpha can vary dramatically. Small samples or biased samples can greatly affect the alpha estimate. Thus, context about precision could greatly inform researchers about the quality of their results and the likelihood or replication. Confidence intervals are one way of providing information about precision. As we briefly discussed earlier in the chapter, there have been attempts to construct methods to calculate CIs for alpha, but these have not gained traction in routine practice. However, with bootstrapping, we can easily provide empirical estimates that are valuable for readers.
You might remember that the GDS gave us some problems when attempting to perform EFA, as it was unclear whether it was a single factor or multiple factors. Let us assume that we decided it was a single factor (remember that alpha is not a test of unidimensionality!). We can then calculate the coefficient alpha for the variables using the syntax below. 
proc corr data=marshdata alpha;
   var GDS:;
run;
The coefficient alpha and item-total correlations produced by the syntax are presented in Figure 11.5 Alpha and item-total correlations for the GDS data. Since we input the raw data and not standardized versions of our variables, we will interpret the raw coefficients. This provides us with an alpha estimate of 0.885 and a large range of item-total correlations, from 0.252 to 0.615. Some of these correlations are on the lower side, and it might be worthwhile to review the associated items to determine whether the content that they capture is relevant to the construct of interest and whether they should be retained in the scale. Interestingly, none of the revised alphas (in the column labeled Alpha), which represent the alpha if any item was removed, exceed the overall scale alpha (α = 0.885), which indicates that the overall scale reliability would decrease if any items were removed. However, the revised alphas are only .001 to .08 below the scale alpha. Thus, removal of items would not affect the overall reliability very much. Together, these results suggest that if these items are used as a single scale, some items could be deleted with little loss to reliability of the measurement.
Although we would likely spend some more time refining the above scale (e.g., removing or tweaking items) before doing anything else with it, we will proceed with the current version to demonstrate the use of bootstrapped confidence intervals for a scale that is not as stable. We will consider the above results to be our “population” estimates[5] and compare the results produced from subsamples of our “population” to determine how informative confidence intervals might be. Using PROC SURVEYSELECT, we drew a random sample of N=50, N=100, and N=250 from the sample using simple random sampling. We then conducted 2000 bootstrap resamples from each and estimated 95% confidence intervals around the alphas and item-total correlations. (See Chapter 7 for more information about bootstrapping methods and syntax.)
Figure 11.5 Alpha and item-total correlations for the GDS data
Because the steps are somewhat complicated but repetitive for each random sample, we wrote a macro called subNBoot to do these steps for each sample. The syntax is presented below. There are two arguments that are fed into subNBoot: 1) ss, which stands for the numeric sample size value that we would like our subsample to have, and 2) seed, which stands for the numeric seed value that surveyselect will use to identify the sample. Setting the seed allows our analyses to be replicated. subNBoot then uses a series of procedures that you were introduced to in the current chapter (PROC CORR) and Chapter 7 (PROC SURVEYSELECT and PROC UNIVARIATE) to estimate the CI. This macro results in a set of data sets that contain the actual estimates (orig_alpha&ss [6] and orig_item&ss) and the bootstrapped CI (ci_alpha&ss and ci_item&ss). 
%MACRO subNBoot(ss,seed);
   *Subsample;
   proc surveyselect data=marshdata method=SRS n=&ss. 
    out=marsh_sub&ss. seed=&seed;
   run;

   *Estimate Item Stats;
   ods output CronbachAlpha=orig_alpha&ss CronbachAlphaDel=orig_item&ss;
   proc corr data=marsh_sub&ss. alpha;
      var GDS:;
   run;
   ods output close;

   *Take Bootstrap Sample;
   proc surveyselect data = marsh_sub&ss. method = URS samprate = 1 
    outhits out = outboot_marsh&ss. (compress=binary) seed = 5 rep = 
    2000;
   RUN;

   *Estimate Item Stats for each bootstrap sample;
   ods output	CronbachAlpha=boot_alpha&ss. (compress=binary) 
      CronbachAlphaDel=boot_item&ss. (compress=binary); 
   proc corr data=outboot_marsh&ss. alpha nosimple nocorr noprob;
      by replicate;
      var GDS:;
   run;
   ods output close;

   *Estimate CI from bootstrapped results;
   proc univariate data=boot_alpha&ss.;
      where variables='Raw';
      var Alpha;
      output out=ci_alpha&ss. pctlpts=2.5, 97.5 mean=Alpha_mean  
         std=Alpha_std  pctlpre=Alpha_ci ;
   run;
   proc sort data=boot_item&ss. nodupkey; by Variable replicate; run;
   proc univariate data=boot_item&ss.;
      by Variable;
      var RawCorr;
      output out=ci_item&ss. pctlpts=2.5, 97.5 mean=RawCorr_mean 
         std=RawCorr_std  pctlpre=RawCorr_ci ;
   run;
%MEND;
%subNBoot(50,3); *Run analysis with subsample of N=50;
%subNBoot(100,8321); *Run analysis with subsample of N=100;
%subNBoot(250,26); *Run analysis with subsample of N=250;
The estimates and corresponding confidence intervals for each sample are presented in Alphas and 95% CI for three subsamples and Item-total correlations and 95% CI for three subsamples. Ideally, bootstrap analyses of small samples would include the population parameter in the 95% confidence interval, and they would provide information about the precision of the estimate (and thus the potential replicability of the statistic). In the tables below, we can see this pattern is generally true. The 95% CI for the alphas in each subsample contains the “population” alpha. Furthermore, as the sample size increased, the CI became more narrow and precise. This same general pattern is seen in the 95% CI for the item-total correlations. There were only a few instances (highlighted) where the bootstrapped CI did not contain the “population” parameter.
Let’s focus on this idea of CI range and precision for a moment. As we mentioned above, the CI became narrower and more precise as the sample size increased. We can see that the CIs are narrower, but how do we know they are more precise? In the smallest sample of N=50, the range of the CI tended to be rather large, and the sample estimates varied by .12, on average, from the “population” estimates. Some estimates varied substantially more (e.g., GDS06, GDS10). As the subsamples got larger, the estimates came closer to our “population” estimates and the corresponding CIs became narrower.
Table 11.2 Alphas and 95% CI for three subsamples
Subsample
Alpha
95% CI
N=50
0.84
(0.74, 0.89)
N=100
0.88
(0.82, 0.91)
N=250
0.90
(0.87, 0.92)
Table 11.3 Item-total correlations and 95% CI for three subsamples
Var:
N = 50
N = 100
N = 250
Item-total R
95% CI
Item-total R
95% CI
Item-total R
95% CI
GDS01
.51
(.16, .77)
.48
(.24, .67)
.52
(.38, .65)
GDS02
.25
(.00, .55)
.31
(.11, .50)
.41
(.29, .53)
GDS03
.36
(.11, .66)
.53
(.30, .70)
.58
(.44, .69)
GDS04
.36
(-.12, .69)
.42
(.16, .63)
.49
(.35, .61)
GDS05
.32
(.03, .59)
.38
(.09, .61)
.55
(.41, .66)
GDS06
.10
(-.18, .49)
.51
(.29, .69)
.55
(.41, .66)
GDS07
.52
(.33, .77)
.50
(.23, .69)
.47
(.29, .61)
GDS08
.31
(.24, .62)
.36
(.12, .58)
.37
(.19, .52)
GDS09
.56
(.05, .81)
.46
(.23, .66)
.59
(.45, .71)
GDS10
.23
(-.02, .53)
.54
(.32, .70)
.62
(.49, .72)
GDS11
.50
(.15, .74)
.44
(.21, .62)
.44
(.31, .56)
GDS12
.39
(.09, .65)
.43
(.24, .61)
.31
(.18, .44)
GDS13
.38
(-.04, .69)
.51
(.32, .67)
.35
(.20, .49)
GDS14
.26
(.02, .54)
.00
(-.14, .18)
.30
(.16, .43)
GDS15
.23
(-.03, .53)
.63
(.40, .78)
.37
(.20, .52)
GDS16
.78
(.53, .90)
.62
(.43, .77)
.69
(.58, .77)
GDS17
.64
(.33, .86)
.56
(.34, .72)
.69
(.57, .77)
GDS18
.31
(.23, .62)
.34
(.11, .55)
.35
(.18, .51)
GDS19
.46
(.15, .71)
.59
(.46, .71)
.56
(.46, .65)
GDS20
.21
(-.07, .54)
.25
(.03, .47)
.37
(.25, .50)
GDS21
.42
(.12, .68)
.46
(.27, .62)
.43
(.31, .53)
GDS22
--*
(--, --)*
.48
(.21, .66)
.62
(.48, .73)
GDS23
.30
(.07, .60)
.34
(.08, .58)
.56
(.41, .68)
GDS24
.46
(.09, .73)
.59
(.42, .74)
.43
(.29, .57)
GDS25
.38
(.31, .69)
.48
(.22, .67)
.57
(.44, .69)
GDS26
.63
(.33, .84)
.42
(.21, .60)
.49
(.36, .60)
GDS27
.29
(-.03, .61)
.25
(.04, .47)
.37
(.23, .50)
GDS28
.10
(-.17, .40)
.34
(.12, .54)
.40
(.26, .53)
GDS29
.28
(-.01, .60)
.23
(.02, .42)
.37
(.24, .50)
GDS30
.50
(.25, .70)
.30
(.09, .49)
.39
(.27, .50)
*This parameter could not be estimated because of lack of variance among the responses for this item in the reduced sample. Note: Confidence intervals that did not contain the “population” parameter are highlighted.
Thus the bootstrapped CI can serve as a very effective indicator of the precision of an estimate. When CIs are broad, they indicate we are not very confident in our estimates and the true population estimate could vary greatly. They also suggest that we could see substantial variation in our estimate if we were to replicate the analysis in similar samples. When CIs are narrow, they suggest the opposite—our estimates are close to the population parameter and they are likely to replicate. CIs can be a very informative and useful tool to aid in the interpretation of alpha and many other statistics.
..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset