My personal "rules" about sperry and loafers is, with trousers wear socks or no socks to your heart content, whatever length. With shorts it's trickier, I mainly avoid white socks and socks higher than the ankle, but I easily go for black or dark blue ankle socks or some brownish in tone with loafers if they are brown, just don't wear them with socks everyday. Most of the times I wear sandals, though, and that's why, no need to bother with socks :D.