http://blc25.wordpress.com/2011/10/14/why-is-reliabililty-important/#comments
http://chocolateraisons.wordpress.com/2011/10/14/why-is-reliability-important/#comments
http://alanahcounsell.wordpress.com/2011/10/14/why-is-reliability-important/#comment-27
Could you please mark those comments 🙂
Thank You
“How should we ensure our research findings are useful outside psychology and even outside the wider scientific community?”
In order to ensure our findings are useful outside psychology we need to maintain external validity. External validity is used to measure how results are generalizable to real life situations outside the experiment setting. If the sample and time and size is representative of the whole population them it can be said that the results can be generalized outside psychology and wider scientific community. It is not always easy to ensure all the requirements are met and therefore all psychological researchers do lack a part of external validity. Albeit if we do try to maintain external validity it is beneficial to the results as it shows that’s the behavioural measured can be applied to real life situations (researcher aims to research real life behaviour ). Therefore it is really important to maintain external validity. There are certain ways to ensure our research is strong in external validity.
External validity is composed of two elements; ecological validity and population validity. In order to ensure the requirements of ecological validity are met, researcher needs to make sure the design that he/she is using can be generalized to real life situations. There are different design that are used that are consider as being more representative of real life behaviour and less. Lets analyze laboratory experiments. Laboratory experiments are most representative in term of control as , variables are under strict control , as well as confounding variables. However are they genralizable? Certain they not. Laboratory experiments do not usually reflect real life situation. Unless we want to test real life situations that occurs in laboratory? Emm probably not. For example Milligram and his research on obedience (please read the description as I will discuss ithttp://en.wikipedia.org/wiki/Milgram_experiment ) , despite ethical issues that it raises , it cannot be really concluded that people do obey to authority . If the people did blindly obey to the authority, them the crime rate would be going down. In nowadays none of the people would be asked to cause harm to other human (which what they tested on experiment). Although when the research was carried out, the researcher aimed to see why the people blindly obeyed orders during Second World War. Therefore it can be argued that the findings can be generalizable but only to that certain ‘’time ‘’, and not to the outside the study; real life situations. As shown later the results were not consistent and therefore lacked reliability as well ( see my post before on reliability issues). Therefore they were unrepresentative of real life situations.
There are other research designs that are more representative and reflective of the real life situations. For example; observational study. Observational study is where the experimenter observers behaviour in real life situations and them rates for the purposes of the experiment; lets say researcher aims to study children aggressive behaviour between females and males. He or she would just go to the playground and rate the aggressive behaviour between both genders. However as long is representative of real life, is not a’ real experiment’ as it does not manipulate independent variable to see the outcome on dependent variable. That’s why it has got low internal validity due to lack of control of confounding variables. Therefore it seems clear that research will always have to sacrifice one thing upon another. Let’s move on to population validity.
Population validity is maintained by testing the sample that represents the behaviour of the population. Small sample size is unrepresentative as there are different characteristics that everyone in the world share, and as you probably know there are millions of people in the world. However the large sample still can be biased if it’s not allocated into groups in right way. If for example one group consists of males and second is only composed of females and our aim is not to test gender differences. Therefore in order to ensure our sample is representative we need to allocate participants randomly to the groups.
Conclusively we need to maintain that ‘’scientific regime’’ of our study in order for it to be generalized outside scientific community. Obviously still our study will lack something as it impossible to maintain the validity in 100 % but as long is representative we did do a good job.
Replication of the study can be watched at http://www.youtube.com/watch?v=BcvSNg0HZwk.
Can you please mark this comments.
http://blc25.wordpress.com/2011/10/06/do-you-need-statistics-to-understand-data/#comments
http://statsjamps.wordpress.com/2011/10/07/do-you-need-statistics-to-understand-your-data/#comments
http://psuc0f.wordpress.com/2011/10/06/do-you-need-statistics-to-understand-your-data/#comments
Thank You
😉
Hi to all statistics blogers , as reliability is clearly important in both real
life and statistics, in this post I will be aiming to evaluate why is important
in statistics.
Term reliability refers to the extent how results are
consistent over time. Maintain reliability is extremely important when
conducting mainly quantitative study. It shows that methods used to measure in
the study are well designed, and if repeated again, approximately the same
results would be gathered. For example;
if aimed to measure drinking patterns by giving questionnaire, them the participant
answers, should be the same week after (unless they are too drunk to complete
the questionnaire, but them it’s their behaviour that is not reliable). Vallejo
ET all (2006)* conducted questionnaire reliability study on 185 students. His results
show that when students completed the same questionnaire 17days later, the same
results were found. Therefore the questionnaire was reliable.
It can be argued that
if the results are not consistent over the time there is something wrong with research
design, which would threaten internal validly (due to the lack of controls of
variables). If the results are not valid, the whole study is flawed and has to
be carried out again. Therefore it is extremely important to maintain
reliability to not make your supervisor go mad and of course to maintain
reliable results. There are different
measures that aim to asses’ reliability.
Inter rater reliability mainly used in observational field
studies .The data is gathered by more than one observer to ensure there will be
no bias when rating the behaviour. Haskett and Kistner (1991)* research were
based on well defined examples of behaviour that they aimed to measure in
children, (both desirable and undesirable).
However it cannot be concluded with 100 certainties that
this was reliable type of measures. Humans differ from each other in terms how
they perceive and evaluate behaviour. Therefore despite defying specific
behaviours and rate scale, the study cannot be 100 reliable as there still will
be some researcher error conducted. Furthermore it seems clear that reliability
should be not that important when conducting research study as there still be
some bias, although to minimise that error it is important to assess reliability.
Test retest reliability
is used in mainly questionnaires studies when reliability can be measured by repeating
the same experiment week, month later. See example in first paragraph. Parralel
Forms Reliability is when the research is compared to the similar secular
research in order to establish reliability. Finally Internal consistency
reliability when the elements within the test are compared with other
components of the test.
On the other hand as I said before reliability should not be
that important. Let me explain you why. For example when researchers wants to
carry out the sexual preference questionnaire (Buss 1989)* across different
countries then it cannot really test the reliability by test retest. There
would be thousands of the people involved and measuring the same response week
or year later would not be possible, due to the drop out etc (unless the
participants want to check again if they have got the same sexual preferences
as week before, well they might change after one night out). Other example included classroom tests, if repeated
week later then the score might go up as the result of the participant practice
(confounding variable). Therefore when tried to maintain reliability, the
results of the study might be affected and the results of the study would be
under question mark. Which on the other hand its good thing as it spot mistake
and is not misleading to the general public when published.
Taking into account all pros and cons, it is clear that reliability play
important part in research study. The reliability assessments are really
important as they try to minimalist any errors in the study and show how the
results are consistent over the time. Results basically tell, that you can rely
on them and they can be generalized as when repeated the same or similar
results would be shown. Therefore I think reliability should be maintained.
“Reliability and validity
are tools of an essentially positivist epistemology.”
(Watling, as cited in
Winter, 200, p. 7)
This picture clearly illustrates why reliability is
important, because you would not want go to information point, and you answer
would be based on magic. 😉
* http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1794673/
*Developmental psychology by David R. Shaffer , Katherine
Kipp . Page 14
* http://homepage.psy.utexas.edu/homepage/group/busslab/pdffiles/SexDifferencesinHuman.PDF
In order to fully answer the question we need to ask our self, what data are we aiming to collect? Would be quantitative or qualitative type of data? There is a main difference between those two dates that I will try explain later in this post. If we have chosen to collect qualitative data (its descriptive type of data, which does not account for numbers) in our research experiment, then answering the question we would not need statistics to understand our data. Qualitative type of data can be gathered by conducting case studies, where in depth analysis, observation and description about individual can be made. The example includes Freud case studies; little Hans etc, read more on http://www.simplypsychology.org/little-hans. Case studies are subjective as they are based on one person thinking and ideas. They do not involve any systematic measures that can be number based. It can be also argued that when conducting observational study , qualitative data is gathered too, however researcher them aims to number the behaviour and conduct inter ratter observer validity to avoid low internal and external validity or reliability. That type of data them can be explained by statistics. Therefore qualitative type of data does not require statistics, to further the comprehension of the outcome of the study.
On the other hand quantitative type of data is number based and it’s more valid and reliable type of measure. Experimenter manipulate independent variable to see effects on dependent variable ( although there are different research methods such as questionnaire that also involves quantitative type of data but sometimes does not require independent and dependent variable), and he or she end up having lots of numbers that are the results of the study. Those numbers are raw and without statistical test, the sense of what the actual result is cannot be made. For example; Loftus and Palmer conducted study on eyewitness testimony, showing participants car crash and them asking them the same questions but with one word different within the question. They were asked about the speed of the car, and week later questioned if they did noticed any broken glass on the floor (but in reality there was no broken glass). Without statistical analysis their set of data would be just numbers gaved by participants (estimated speed) and conclusion upon their research could not be justified. To read more about their study follow http://www.simplypsychology.org/loftus-palmer.html
Therefore statistics are helpful when it comes to more advance measures that require numbers. However it is important to have strong statistical background as well, as when conducting statistical test to explain the data, tests will come up with calculations, graphs (probabilities level, mean etc). And without that knowledge the conclusion of the experiment cannot be made.
camilia92 
Written
on October 13, 2011