Firstly, what is meant by the term approaching significance? It is basically a term used by psychologists who’s results fractionally miss out on being significant. For example, if you received a significance of .058 then it is technically un-significant but by such a small margin that it may well have been significant with just one other participant or without a measuring error. But the question is, is using this phrase cheating?
Well there are reasons why this phrase might be used. For example an experiment that receives results this close to the significance line it only takes a few more results your way to be able to state that your experimental results are significant. With such a minimal difference between the significance line and the received significance then the experimental results must surely be effective (in the case of testing a new drug or treatment.)
However the point still stands that it is not significant. If you bend the level given for significance, where do you stop it. This science needs an immobile line that psychologists need to aim for, if they can move that line one way or the other then many more experiments may be given as significant or insignificant, thus leading to type one and two errors on the researchers behalf.
Also, ‘approaching significance’ leads the idea that the more you test the more likely it is that it will be proved as significant, but this can go both ways, it is always possible that this person received a biased sample for some reason (un-be known to the researcher) and that the more people are experimented on the less and less significant this result is going to become.
In conclusion, this phrase should not be deemed as cheating or as not cheating, if this phrase is needed, surely it is vital that this experiment be replicated? Or keep going with more participants in the same experiment until you reach a significant effect, or the significance gets worse.
statssam 
Good blog! I don’t necessarily think that using the term ‘approaching significance’ is cheating as it can be helpful sometimes when describing your results. It would be more descriptive to write that than simply ‘significant’ or ‘not significant’ as it gives a better picture of where your results lie. But we do have P-values for a reason and there does need to be a cut off point. Using the phrase ‘approaching significance’ can seem desperate because it’s like saying ‘I have kind of found an effect’ which can spoil a paper. The results are either significant or non-significant, either way it should be stated as such.
Iv always been under the assumption that is you ever read the term ‘approaching significance’ that it is 95% likely to be significant but would ideally do with being replicated with some variables changed to ultimately decide whether or not it is statically significant.
For example I have had first hand experience of in small groups been shown a set of data that was deemed not significant but instead approach, but by replicating all of the data points (so the data isnt actually being changed but the sample size is increasing) a significance is suddenly present.
However i am dubious about this phrase being used in a psychological journal as it is not full proof that an effect is present, only suggesting that one may be or not maybe there.
You stated your argument well though and i actually enjoyed reading it 🙂
I do not think that using this term is cheating, i think in the case you gave if a figure of 0.58 is achieved that this is so close to actual significance that it should be noted. I liken it a little to when you get a strong or weak correlation, strong correlation would be significance, and a weak correlation would be approaching significance. We state is something is a weak correlation, so i don’s see why it is not okay to point out when something is very close to significance. Obviously there can be some issues with this, such as what are the limits for labelling something as ‘approaching significance’. As there is the limit of 0.5 for actual significance, it seems limits need also be applied for approaching so that it can be something controlled, that we can actually rely on.
I don’t think the term “approaching significance” is cheating however I do see it as taking “the lazy way out “ as through our seminar sessions we have learned that through correlation and increasing the sample size you can effectively show whether or not there is a significance. As another commenter mentioned “it’s like saying ‘I have kind of found an effect’” this doesn’t help prove the argument that psychology should be seen as a science in fact it introduces further doubt into it being a science. As others sciences might rebuttal with “well you can’t prove anything” which is a term in psychology we can’t use as to us nothing can be proven, all we have to rely on is significance. Therefore if a researchers paper is to report “approaching significance” it leaves the true result of their paper in limbo as its not stating in a matter of fact way weather a significance was shown or not. So if your use of the term “cheating” covers areas such as misleading or vague then I support your argument in this well written blog and look forward to reading more of your work. – A
I enjoyed reading your blog and thought it was good that you explained some reasoning why people do this. I agree with you that saying “approaching significance” should not be seen as cheating or as cheating. In all research papers the F statement or other is included which states the p-value. By doing this the reader should be able to interpret there own opinion on whether it is close enough to be classified as pretty much significant or not. The term approaching significance is not incorrect and by doing so they are saying it isn’t therefore they are not to blame for anyone (e.g. media!) misinterpreting this.
Pingback: Comments for 14/03/12 « psud43
Pingback: Comments for Wendy « psud60
The term approaching significance is important as any other level of significance, providing there was a reasonable sample size for example. Although it may not directly tell us any strong results (as in if something has a strong effect or not), but it can show a weak effect, which can lead to bigger discoveries. For example Penicillin was discovered after Dr Alexander Fleming left a lid of a petri dish with staphylococcus on it, and blue mould appeared to be removing the staphylococcus. Although he didn’t think it was significant at the time and there was a very small effect, it was a small idea which led to the invention of penicillin, and saved 15% of amputations during WWII.
It seems like there’s normally a way that researchers can manipulate their results to be significant and cheat anyway, so there’s not a reason to be honest and say results are ‘approaching significance’. Even if researchers determine a universal p value that decides whether results are significant or not (as you’ve discussed), researchers can still easily manipulate their work to become significant no matter what the universal p value is. Syverud (2010) explains that no matter whether your significant level is 0.05 or 0.01, increasing the sample size will make the p value decrease and therefore the significance increase. Therefore, having a specific significance determining p value doesn’t actually show the significance at all because the researcher can just manipulate their data to be significant. Even if the significance level is changed, it seems you can just edit your sample size accordingly too. This is why I believe there’s no reason to have the phrase ‘approaching significance’ and this raises the question, how can we trust the significance of any study?
http://www.rimmkaufman.com/blog/detecting-significant-changes-in-your-data/24022010/#enough_math [increasing sample size]
Good blogging. The previous comment points out one of the flaws of significant testing, and is right there is no way of telling whether the reported significance is actually significant, this is why researchers are urged to report the effect sizes aswell as the p value. The effect size tells us the magnitude of the effect and then perhaps infer about it practicality.
Click to access 55.pdf
Pingback: Final Comments for my TA! « psud74