We make most of our hypothesis test decision criteria with a 95% confidence or a 5% alpha risk.  Not much thought is put into this choice, it is more of a convention than everyone uses.  You may not realize it but DOE generally used a 10% alpha risk not too long ago.  I do not know why, but I expect it is because DOE analysis was generally performed by hand and it made sense at the time, but that is my guess.  Minitab used a 10% alpha risk default in the DOE analysis in V12 and earlier.

If you use the Best Subsets regression tool in Minitab, you may find a case where the tool recommends the use of a factor with a p-value between 0.10 and 0.05.

The factors that occur with a p-value between 0.1 and 0.05 seem to cause a bit of concern for Lean Six Sigma belts.  We are told to set the alpha risk before the analysis and not to change it.  This is still a good practice for hypothesis testing.  Even knowing that rule, I have kept factors with p-values that were above 0.05 in my work and executed and improvement plan that used those factors.  I do not believe I was wrong and you are not wrong if you choose to do the same, under a few conditions.

1. The test size is small (under 100 observations or so).
2. Subject Matter Experts believe the factor should or could be significant.
3. The cost or effort to include the factor is not too high.
4. The risk to the process if you are wrong is low.

The bottom line is that there is knowledge outside of the statistics that provides reasons to keep the term.

#1 is included because there are some analysis efforts that use very large sample sizes that there is little error in the estimates.

#2 is about prior knowledge.  Always include the practical or process knowledge into your statistical decisions.

#3 and #4 are really about the risk and effort in including the factor in the answer.  If it does not cost too much to add the factor to the solution and if it truly turns out to be non-significant in the end, then the business cost impact is minor.

I hope these thoughts help you decide what to do with a marginally significant factor.  I do have an example of someone who made the wrong choice.

At my last employer I found such a case.  When I started working as a statistician, I did not have a lot to do right away so I chose to read a number of technical reports that included statistical analysis provided by my peers.  In one of the published reports (to the US government) the significance decision was made with a 70% confidence level.  Since I had never seen anything like this before, I went to the author to ask him the process that led to this decision.  The answer I heard gave me a chuckle.  He said that they knew that the factor was significant (being an SME) so they just lowered the confidence level until it appeared as significant.  Even though I was new and the author was quite senior to me, I asked if he had verified the significance after implementing his solution and was told that he did check and it turned out to be truly non-significant.  After a long talk, I convinced him to publish an amended report that used a proper confidence level and document the post test results on non-significance.  I did not want people in the future to believe the wrong things.

You might be thinking that I made an enemy so quick into my career, but it was the opposite.  I made a good friend in the end, because I approached it as a learning opportunity not as a “you were wrong” event.  I also did not share my findings with anyone until he published the updated paper.  So he told the world of the issue, not me.  This is another case that you can bring bad news and not get hurt if it is done well.