Itvs interesting that you find yellow light to be harsh. Normally, the yellower tones (2700k-3500k) are called warm and soft white. Daylight is 6500k with a notable blue tone and neutral white is somewhere around 4500K. Is your office also filled with brown/dirty surfaces that seem highlighted by the warm light or grays that clash with it? Florescent lights (and cheap LEDs) are especially harsh in general because they have really bad color rendering, meaning certain tones get muted and distort perception. Letting in daylight may just be helping restore color vibrance. Bluer lights also tend to have more UV output, which makes them more painful at night. Yellower lights lean towards the red end and aren't so jarring for the same brightness. Bluer lights get used in hospital, lab, and other high-detail settings for more clarity, while yellower lights get used in more relaxed environments where visual detail is less important.
I wouldn't guess you have a different cone count, but I would guess there's some underlying perceptions about colors and visuals.