Happened to me too.
I realized the IPD adjustment has a bit of slack, so I under adjusted it a bit and nudged it with my nose to bring it up to my IPD.
Now accidental nudges are very unlikely to change the IPD unless I do a large movement.
annoying and feels like a hack but whatever