Skip to Main Content

Reliability, Validity and Reproducibility: Maths and Stats

Guide contents


Fast facts

  • You should always be checking that your data is reliable, valid and reproducible in order to produce trustworthy results.
  • Valid data is reliable, however reliable data is not necessarily valid.

Reliability

In statistics, reliability means the consistency and repeatability of measurements and assessments. In other words, if a measurement is reliable then it will produce similar outcomes under consistent conditions.

For example, a clock which is consistently five minutes slow is reliable. Things do not need to be accurate in order to be reliable!

It is crucial to ensure that your statistics and your data analysis have high levels of reliability because this determines to which extent the method yields stable and consistent results over repeated trials or observations.

There are many different types of reliability in statistics, and therefore many different ways of checking for reliability, such as:

  • intra-rater (within-rater) reliability (also known as test-retest reliability) - the measure of stability of a measurement over time
  • inter-rater (between rater) reliability - the measure of agreement between different observers of the same situation
  • internal consistency - the measure of correlation between different items within a single situation

Testing reliability is a necessary thing when performing data analysis but is not sufficient: you must also be checking for validity. Just because something is reliable, it doesn't mean it is correct! To understand this, see the below on validity.


Validity

Validity in statistics is to do with the accuracy and real-life truthfulness of a measurement or result. Ensuring validity in your research means that you are making sure you are measuring what you intend to measure, and doing it correctly.

For example, if you measure the length of your notebook three times in a row with a thirty centimetre ruler and you get the same result each time, that ruler is called reliable, and since that ruler was made to a standard size so that it gives a true measurement, that ruler is valid. In the example above with the clock, in order for this clock to be valid, it needs not only to be reliable but also to be correct. 

It is important to assess for validity in your data and analysis to ensure that your results can be trustworthy. There are several types of validity to be aware of:

  • Internal validity - the generalised cause-and-effect between the independent variable(s) and dependent variable.
  • External validity - the generalised cause-and-effect between different participants in the scenario.
  • Statistically Conclusive validity - the inference on the degree of correlation between the independent variable(s) and dependent variable. This type of validity is specifically violated when a Type I or Type II error occurs.
  • Construct validity - this type of validity can be assessed with Cronbach's alpha: if the value of Cronbach's alpha is greater than or equal to 0.7, this is a good enough measure of construct validity.

Remember that valid data is reliable, however reliable data is not necessarily valid. However, to make sure you maintain research responsibility, you need to make sure that your research is reproducible.


Reproducibility

Different disciplines will define reproducibility in different ways, however in statistics it is generally understood that reproducibility refers to the concept of a study being replicated with the same methods and obtaining the same results. In other words, reproducibility in studies measures the degree to which different people in different locations with different instruments can obtain the same results with the same methods.

For example, a recipe is reliable as is used to produce the same meal each time, is valid because by following that recipe you obtain the meal you want to make, and is reproducible because anyone can follow it and create the same meal.

There are many ways of ensuring reproducibility in your research. Be aware that many of these should be performed, not just one!

  • Keeping detailed documentation available on each step of your process, including the names and versions of software you used.
  • Making your data available to others (if you have permission to do so).
  • Use standard methods and models for analysis.
  • Pre-register your study.

Having high reproducibility leads to more accurate research, as it means that results can be trusted, other researchers can verify it and can build upon it, and that results were not just a coincidence.