Do you OP. I might do a double check if I saw a man wearing uggs but I wouldn't care. My wife uses hers as day slippers even though they are the boot type.
Might as well go full swing and wear leggings/yoga pants with your new pair of uggs. Then we can call you a girl in high school back in the early 2000’s.