Let me share a passage from the dissent in a Supreme court case known as Plessy v Furguson. The majority of the court had just ruled that it was OK to force blacks to use seperate railcars from whites. Not only that, but it was OK for for the government to force railway companies to have such a rule. With this backdrop Justice Harlan spoke in dissent, arguing for true equality under the law. In the screed for justice, he wrote:

There is a race so different from our own that we do not permit those belonging to it to become citizens of the United States. Persons belonging to it are, with few exceptions, absolutely excluded from our country. I allude to the Chinese race. But, by the statute in question, a Chinaman can ride in the same passenger coach with white citizens of the United States, while citizens of the black race in Louisiana, many of whom, perhaps, risked their lives for the preservation of the Union, who are entitled, by law, to participate in the political control of the State and nation, who are not excluded, by law or by reason of their race, from public stations of any kind, and who have all the legal rights that belong to white citizens, are yet declared to be criminals, liable to imprisonment, if they ride in a public coach occupied by citizens of the white race.

Thats right folks. There was a period of us history where even your pro equality arguments were steeped in racism

More to the point. Even if you (for some reason) set asside the hole issue of slavery; there is still the whole Jim Crow era, where we litterally codified rasism into law.