[VR] Door with Hinge Joint opens when hand moves in the wrong axis

Hi,

 
First of all, I should say that this is the first time I am doing something more complex than "just testing" with Hinge Joints, so it's possible I am missing some setting somewhere that automagically fixes the problem I am having here.

 
So, here is the problem I am having: I have a Hand object, controlled by a VR controller, which can interact with "Interactable" objects. I also have a door (I made a quick door prototype with basic geometry just to develop this functionality) which consists of a parent object, the door frame (scaled cube) and a door handle (cylinder). The handle is an "Interactable", and the parent object has a collision, Rigidbody, and Hinge Joint, which is set at the correct place, with the correct limits (I tested them out with the Motor function, and the door opens and closes within the limits, no problem). The problem arises when I interact with the door, and before I lay out any details, basically, instead of the door opening when I pull in my body's direction (door opens towards me), it opens when I move the hand towards the right. I'm confused. So here is how everything is laid out:

First, a video of the problem itself:
https://youtu.be/Rr-X1pjpi18

 
This is my Door GameObject:

 
This is my Door's (main parent GameObject) Components:

 
My Frame Components:

 
Handle Components:

 
And finally, Hand components:

How it works:

 
Basically, when the hand touches the Handle's trigger collision and the user presses the Trigger button on the controller, it will attach the door's Rigidbody to the hand's Fixed Joint, like so:

void Grab(IGrabbable grabbable)
{
    if (!CanGrab || grabbable == null || grabbable.GetRigidbody() == null) return;
    fixedJoint.connectedBody = grabbable.GetRigidbody();
}

So, the Handle (grabbable here) sends back the door's Rigidbody through GetRigidbody(). This is what I also use to grab objects and move them around, which works flawlessly.

 
Technically that is it. Nothing else is being done with the hand or the door until we let go of the Trigger button on the controller, which just calls a function called "Ungrab()" that sets the fixedJoint's connectedBody to null.

 
This is driving me a bit crazy, I managed to get it to work by turning doors into a special type of IGrabbable, that basically checks the position of the hand in FixedUpdate() and updates the door's rotation accordingly, but I feel like it's extra code that is not needed because I feel like the FixedJoint and HingeJoints should technically work. Why it's opening when I move to the right and not pull it away from its original position is beyond me.

 
I hope this was clear enough. If someone has any more questions I need to clear up please let me know. If you know what's going on here and a way to fix it, please, by all means, give me a hand over here (no pun intended, or maybe it was). Thank you for reading through this!