What are some good things to look for when evaluating a psychology method, design and statistics?

Make sure that Type I and Type II statistical error can be accounted for. Also, if multiple statistical tests are being done, consider adjusting for multiple comparisons (e.g. Bonferroni adjustment).

How do I make sure that they can be accounted for? What are signs that they are not? Do you have any examples as I am unsure what you mean.

If all you have to go on is the abstract and results section of a journal article or similarly formatted paper, your evaluation may need to be fairly broad, since you may find that you have insufficient information on which to make particularly detailed evaluations.

You may find some of the following helpful to consider:

- Validity (e.g., does the study measure what it is intending to measure, are any measures used or experiments conducted ecologically valid, etc)
– Reliability (e.g., test-restest, inter-rater, etc)
– Sampling (e.g., what size was the sample, what method was used to recruit participants)
– Strengths, weaknesses and appropriateness of the method (e.g., was it a correlational study, an experimental study, a longitudinal study, etc, and consider the various strengths and weaknesses of these designs, e.g., establishing correlational relationships, establishing causation, good control of extraneous variables, etc)
– Whether or not a control group was used (if an experimental study)
– Use of appropriate statistical tests (e.g., if they used parametric tests, was the data normally distributed)
– Whether any ethical issues were addressed (likely to be yes if an article in a peer-reviewed journal, but think of things such as whether the method involved any deception, whether any distress or harm may have been caused to participants, etc).

There’s loads more you could look at, but I hope that helps :)

Ah sorry I wrote the queston wrong, the only two sections I am getting are the Abstract and Method section

Thanks for the help though some of the pointers still apply but others – such as knowing whether the data is normally distributed – would be impossible to gather from the evidence unless it happens to be highlighted in the abstract. I will still keep it in mind at the back of my head just in case.

Type I error is finding evidence of statistical significance where none exists and type II error is not finding an evidence of significance where it actually does exist. For type II errors, this is often because the sample size is too small (e.g. study is underpowered). For type I errors, this may happen because the study design does not match the question asked.

The Bonferroni correction is one way of dealing with type I errors where multiple comparisons are done. For instance, if you run 6 statistical tests, you’re more likely to get a positive result by chance than if you only run 1 statistical tests. The Bonferroni has you divide the p-value by the number of tests (comparisons) to make it harder to get a positive result by chance alone.

It is too bad you did not ask this question sooner. By now my response will be of no benefit to you.