How to prevent a thrown object from rotating but not effect the physics of its collision?

I have a disc type object (like a frizbee) that that the player throws in zero gravity using HTC Vive controller, but I'm finding in most cases the object when thrown will rotate in an un-frizbee like manner. Apart from the way it looks, I'm happy with the way the disc is deflecting and colliding with the arena walls.

However when I try to constrain/freeze the rotational axis of the disc, the disc doesn't respond the way I expect when it collides or when it is held. In some cases it collides and then gets shot off at crazy velocity, and when held it can't be rotated (for obvious reasons).

I'm working on trial and error here, and I think there's a few ways to resolve this problems. But I'm looking for advice, what is the best way to resolve this problem? Can I somehow separate the visuals of the disc from the physics of the disc, when it's in flight and seamlessly return the state when grabbed again by the player ?