We had a talk and he backed off.
He started asking about my wife again so I took the opportunity to tell him I thought it was inappropriate that he was taking such an interest in my family and then I told him I didn't want him following me anymore.
He didn't really have much to say but since then he's really stopped being so creepy...at least towards me. We were actually cracking jokes the other day. I'm still trying to maintain that sense of authority while also being a nice guy. I'm keeping him at arms length.