Issue with Character controller and skin width

I just very recently started working with Unity and only today found out about the Character Controller component.

So I immediately wanted to test it and initially liked it very much, however I am experiencing some issues with the skin width.

My issue is that my character has the skin width as an extra layer additional to his radius and height setting, but this alone could be solved by just substracting the skin width from the radius and height.

So the more weird thing is that when I move my char against a wall while just going directly into the direction of the wall it wall have a gap of exactly the amount that the skin width was set to - which I guess makes sense.

However when I move my character not directly into the wall, but while also going up/down (when the wall is e.g. to the left), my character can get closer to the wall.

I will show you an example:

Here you see that my character(which is the sphere) has an x position of ~-0.8, which makes sense, as the wall starts at -0.9 and the skin width is 0.1. (Also note that for the same reason my character is floating in the air by 0.1 units)

The next picture was taken with the same settings, I just pressed left, left+up, left+down alternating multiple times. Which resulted in:

Here you see that now my character has an x position of ~-0.897 which means he would be "inside" the wall if he actually had a skin with a width of 0.1

Also note that this picture was made when I was not pressing any buttons anymore, so no more movement was made, this was the permanent position of the character.

So now my issue is that if i make the radius and height of the character controller smaller (by the amount of the skin width), the character will have 0 distance to the wall when moving directly into its direction, but will get "inside it" if the player presses left+up or left+down. Which is just not acceptable.

It generally seems kind of odd that, you can get closer to the wall by pressing left + up/down that by just pressing left.

Can anyone help me and tell my what I should do to fix this issue?

The only thing that comes to my mind is that normally when objects collide, they push themselves away from each other - and so I could implement something that would push away my character from the wall if he gets "inside" it.

Just for the sake of completeness, here is my code:

 public class GameInit : MonoBehaviour
 {
     private float moveSpeed = 1.0f;
     private GameObject cube;
     private CharacterController characterController;
     // Use this for initialization
     void Start()
     {
         //create characcter
         cube = GameObject.CreatePrimitive(PrimitiveType.Sphere);
         cube.name = "brown_cube";
         cube.GetComponent<Renderer>().material.color = new Color(139f / 255, 69f / 255, 19f / 255);
         characterController = cube.AddComponent<CharacterController>();
 
         //set inital position of character
         Vector3 pos = cube.GetComponent<Transform>().position;
         pos.y = 1;
         cube.GetComponent<Transform>().position = pos;
     }
 
     // Update is called once per frame
     void Update()
     {
         Vector3 moveDirection = new Vector3(Input.GetAxis("Horizontal"), 0, Input.GetAxis("Vertical"));
         characterController.SimpleMove(moveDirection * moveSpeed);
     }
 }

PS: I had the best experience when setting the skin width to 0 or a very small value like 0.001, but the Unity documentation explicitly states that one should not do this.

"It’s good practice to keep your Skin Width at least greater than 0.01 and more than 10% of the Radius."

https://docs.unity3d.com/Manual/class-CharacterController.html