UI Scaler for 4k and lower resolution

Hi, I am currently in the process of creating art assets and adjusting UI scaler for multiple resolutions

basically, the game supports 4k and 1080p

I am drawing my UI assets in 4k. In unity UI's canvas scaler I adjusted my reference resolution to 1920 x 1080.

Subsequently, I made all my UI panel adjustments and position based on 1080p ref resolution.

Problem rises when I switch to 4k in game preview panel, my UI images looks blurry because the canvas scales up (UI Scaler detects a resolution change and scales my canvas to scale 2).

My question is should I be adjusting my UI panels based on reference resolution in 4k instead? In that way, I can prevent a "scale up" causing resolution lost. I am assuming that scaling down maintains the pixels integrity of my sprites.

Are there any potential problems to having 4k as my reference resolution in the canvas scaler? Thank you!