Skin color is always relevant. If it isn’t then prove your point and wear black skin for a week or a month. There are whites that have taken the challenge and have been utterly floored by how incredibly different (worse) they were treated in American society.
I am a very light-skinned African-American, and am usually mistaken for something other than black, and so I KNOW, from first hand experience how much better I have been treated in American society, than my darker-skinned relatives. They all know it to. This is disgraceful for a society to behave in this fashion and even more disgraceful for members of the society to act as if this isn’t a fact.
Skin color matters not? Prove it. Wear some black makeup and see how things turn out for you. Then you will have all the facts and proof you need to back up your ridiculous assertions.