I miss him as soon as he goes. When he isn't with me, I think I made him up.
What you gain in internal knowledge goes from one lifetime to another. It is not wasted. Unlike those stone edifices that will fade, your internal knowledge will stay with you from one incarnation to another.