A man's own conscience is his sole tribunal, and he should care no more for that phantom "opinion" than he should fear meeting a ghost if he crossed the churchyard at dark.
We can never expect to grow in the likeness of our Lord unless we follow His example and give more time to communion with the Father. A revival of real praying would produce a spiritual revolution.