I use to have this problem while downhill skiing in a baseball cap. What I found works great is 2 alligator clips on a piece of string (like a shoelace) clipped to the hat and your shirt. Worst case, it's blows off, but you don't loose it.