Gamepad and keyboard act differently in multiplayer game using Input System

I am working on a multiplayer fighting game where players will move in x and y axis (jump) and will shoot towards each other.

I want to get input both from keyboard & mouse and gamepad (I'm using a PS4 controller at the moment).

I assigned Space key and Gamepad south button (X button as I'm using a PS4 controller) for jumping.

Here is my basic configuration for movement actions: To test the jumping, I start the editor as server and I run 2 standalone instances of the game as clients.

When I test the input with keyboard it works as expected. Every client can control their own player (green in the screenshot below). However, when I test it with gamepad each client can control both players.

The screenshots of keyboard and gamepad inputs respectively:

My code:

 [SerializeField] private Rigidbody _rb;
         [SerializeField] private float jumpSpeed = 8f;
 
         private PlayerInputActions playerInputActions;
 
         private PlayerInputActions PlayerInputActions
         {
             get
             {
                 if (playerInputActions != null)
                 {
                     return playerInputActions;
                 }
                 return playerInputActions = new PlayerInputActions();
             }
         }
 
         public override void OnNetworkSpawn()
         {
             if (!IsLocalPlayer) { return; }
             PlayerInputActions.PlayerMovement.Jump.performed += ctx => PerformJump();
         }
 
         private void OnEnable() => PlayerInputActions.Enable();
         private void OnDisable() => PlayerInputActions.Disable();
 
         private void PerformJump()
         {
             if(!IsLocalPlayer) { return; }
             Vector3 jumpVector = Vector3.up;
             HandleJumpServerRpc(jumpVector);
         }
 
         [ServerRpc]
         private void HandleJumpServerRpc(Vector3 jumpVector)
         {
             _rb.AddForce(jumpVector * jumpSpeed, ForceMode.Impulse);
         }

I am using Netcode for GameObjects as multiplayer library, but I tried the same with Mirror but no luck.

Is this a bug with the Input System?