Physic Torque Hinge joint problem

Hi. Trying to make a hockey game. I have this object (player) with a Hinge Joint on a hockey stick and i add Torque on the stick's rigidbody depending on how the mouse moved horizontaly. The problem i have right now is that if you do a really quick gesture the stick doesnt move very much but if you move your mouse the same distance but slowly you'll get move velocity out of the same distance. How would i change this to be the inverse. The move quickly you move in less time the more suddent velocity you would have. I tried doing an average of the mouse.x delta over the last 10 frames but the same happends wierdly.