You have to knock on doors. You can't keep your dreams a secret. You have to put them out there on display and people can say whatever they want but keep on pointing your actions towards your dreams.
It is a maxim that no man was ever enslaved by influence while he was fit to be free.