The Lee myth. Lee being above slavery, Lee being in fact anti-slavery, is essential to the neo-Confederate argument that it's not about race, it's no… - Brooks D. Simpson

In 1864, black Union troops were involved in operations against Lee's army outside Richmond and Petersburg, and some of them are taken prisoner. Lee puts them to work on Confederate entrenchments that are in Union free-fire zones. When Grant gets wind of this, he threatens to put Confederate prisoners to work on Union entrenchments under Confederate fire unless Lee pulls out. So Grant was willing to embrace an eye-for-an-eye, tooth-for-a-tooth retaliation policy based upon Confederate treatment of black prisoners. For Grant, it was the color of the uniform, not the skin, that mattered.

In pre-war correspondence, Lee castigated the abolitionists for their political activity, and he never showed any qualms about the social order that he would later defend with arms. He also had a few slaves that he inherited as part of a will agreement, with provisions to emancipate those slaves. But in fact, he dragged his heels in complying with the terms of that will. And he never gave a second thought to the fact that his beloved Arlington mansion was run by slave labor.

[I]if you think that this whole matter can be reduced to whether we should allow the display of the Confederate flag, you really aren't advancing the discussion very far.