One of the important decisions we need to make when hypothesis testing is determining how much data we need in order to accurately make assumptions about a population. Oftentimes, it is not feasible, practical or worthwhile to test a whole population due to limitations such as time, money, feasibility, etc. and so we need to instead use a smaller sample...but not too small: if the sample is too small the study may produce unreliable results. Taking samples of our populations to test is still a perfectly fine way of drawing conclusions about the population, and precise sampling techniques are required to calculate the ideal sample size in specific research scenarios as one technique may not be as appropriate to use compared to another.
Sampling involves simply taking number of observations from a larger population in order to estimate the characteristics of that whole population. One very important aspect is deciding just how large your sample should be - if it is too small it would not be representative of the whole population, and if it is too big the study comes too complex - and there are two main ways to do this: the precision-based method, and the power-based method.
Sample size calculations can be performed on pretty much all quantitative studies, and should be done before you gather your data for your analyses. Of course, when it comes to actually performing statistical analyses, you have to work with what you have - so make sure what you have is the best to work with!
The tabs of this guide will support you in understanding sample sizes. The sections are organised as follows:
You can use online sample size calculators to calculate the required sample size, however this guide will assist you in understanding these calculations and how to perform them yourself, both by hand and using software.
When we use precision-based sample sizing, we are wanting the estimation to have a certain degree of precision, and for this we make use of the confidence intervals.
Recall that the formula for the 95% confidence interval for normally distributed data is given by
Therefore, if you specify the 95% confidence interval you have, this formula can be rearranged to find the sample size n.
You can use the Cochran formula to calculate sample sizes at the 95% confidence interval when your population is very large, however a modification can be made to the calculation when your population is smaller.
The formula for Cochran is given as:
Be aware that the Cochran formula will sometimes not give a whole number - you will need to round up to find the real minimum sample size you should use.
If the population is smaller we need to use a slightly different formula to calculate the appropriate sample size:
Again, this calculation may not provide an integer answer so would need to be rounded up if this were to happen.
Yamane's formula is another way of calculating the sample size at 95% confidence. It is given by:
Whereas precision-based sample size estimating makes use of confidence intervals, power-based makes use of power, which is calculated by:
It is important to measure the power of your study as a higher power implies a higher likelihood that your results are valid - a study with low power (for example, a value of less than 0.8) implies that your test has a only a small chance of producing valid results, and these may be influenced by random or systematic error.
Since both power and sample size are positively related Power can also be used to determine appropriate sample sizing: if you know the power, significance level, research design and effect size, you can calculate the best size for your sample.
It is very easy to use power to calculate sample size when using quantitative data, however the type of test performed affects the way we need to calculate the sample size. For the example of the unpaired t-test, we can use the formula:
Using software to perform complex calculations is desirable as it tends to save time and reduces the risk of human error. Below are some examples of how to calculate sample sizes for different tests using different softwares and languages.
Minitab uses power-based sample sizing to calculate the right sample size to use. To choose the sample size for a t-test:
To calculate the sample size for an ANOVA:
The important thing with Minitab is to leave the desired field blank - so here, it is sample size. If, for example, you wish to calculate the power instead, you can leave this blank and fill in the other fields as appropriate.
To calculate the required sample size for Pearson's correlation in SPSS:
The value for the sample size will appear in the table SPSS generates under the N column.
You can use PROC Power to calculate the desired sample size. Note that SAS defaults the significance level to 0.05 so if you need to change that you will need to override that default.
To calculate the sample size for a two-sample t-test, write the following and substitute the ____ with the required input as according to your study:
groupmeans = ____
stddev = ____
npergroup = .
power = ____;
Be aware that the value SAS generates will be the number of participants in each group, not the overall total sample size.
To calculate the sample size for a correlation analysis:
pwr -> pwer.r.test
pwr.r.test (r = ___, sig.level = ___, power = ___)
To calculate the sample size for a logistic regression, write the following, substituting the ___ for the values you are using:
wp.logistic(p0 = ___, p1 = ___, alpha = ___, power = ___, alternative = "two.sided", family = "normal"
where p0 is the probability of observing a value of 1 for the dependent variable when the independent variable is 0, and p1 is the probability of observing a value of 1 for the dependent variable when then independent variable is 1.
Bear in mind again that R may not provide integer values for these answers, so it may be the case that you will need to round up in order to find the right sample size.
G*Power is a statistical software used to calculate power, however it can also be used to calculate the sample size of t-, z- and F-tests using power-based calculations.
To calculate the sample size for a multiple regression:
To calculate the sample size for a paired samples t-test: