The public scandal is what constitutes the offence: sins sinned in secret are no sins at all.
It's a useful habit to never believe more than half of what people tell you, and not to concern yourself with the rest. Rather keep your mind free and your path your own.