When I lived in New York City many years ago, I witnessed a purse snatching on the subway; it was done so quickly and efficiently that none of us in the vicinity was able to do anything to prevent it or to catch the perpetrator. I was left to join the group of bystanders who were attempting to comfort the victim, a frightened and well-dressed white woman who looked to be in her sixties, while we tried to flag down a transit police officer to assist. We murmured the expected comforting words to her and checked to see if she had been injured, but she said nothing for a moment.

Finally, she came back to herself and looked around at our ring of sympathetic faces and asked a single question: “Was he black?”

I presumed for an instant that she was attempting to elicit a description of the suspect from us, but I quickly understood that was not the issue. She instead wanted reassurance that the world as she perceived it was still spinning securely on its axis.

Her world was stable and understandable if the man who robbed her was black. She could go back to her circle of friends and family members, tell her Manhattan horror story of being robbed, and continue to live her life with a single immutable principle at its center: If she continued to avoid daily contact with black people her personal safety was assured.

However, if the man who robbed her was white, her world would likely spin off its axis and crash into the sun because there was no safety and security to be found anywhere. Her ability to wall herself off from danger simply by avoiding a segment of our population would end.

Lest you think I was misinterpreting her stunned reaction to being robbed, allow me to share the serene smile of sheer relief that washed across her face when she found out a black male had robbed her, which was followed immediately thereafter by a series of stories about the troubles caused by “the blacks” intruding upon her lovely (and apparently very, very, white) neighborhood in Queens.

Why did she feel it was appropriate to share these stories with those around her until the police showed up to take her statement? Simply put, it was obvious to her that we would all understand the awful problems she described—because we were all white. It would have shocked her to know that not only did I not agree with her—I also lived in East Harlem. Life gets awfully complex when you get beneath the surface appearances, but that is what drove her—and still drives so many today—to look for simple answers based on the simplest of criteria.

No matter how politely expressed but inexcusably stupid these attitudes and behaviors may be, this type of low-level racism is a national contagion. However, even the mildly afflicted are capable of causing catastrophes when they are police, school teachers, and public officials. Although it is likely true that only a very small faction of Americans are virulent racists, when those in power act on bland impulses rooted in ignorance or unfamiliarity, situations can rapidly metastasize into life-threatening or life-changing disasters. We see this damage all around—mostly notably in a seemingly endless series of fatal confrontations between unarmed African-Americans and heavily armed police; racism leads to a great many problems, but these problems are obviously magnified when a gun and a badge are added to the equation.

A similarly destructive dynamic plays itself out daily in our nation’s classrooms because of the fears, insecurities, and plain idiocy of teachers and administrators who simply cannot seem to understand that students of color require exactly the same levels of respect, patience, and academic challenge as every other student.

Have you ever heard a teacher speak brusquely to an African-American student because they felt it was necessary to demonstrate their authority—by being rude? Are minor episodes of adolescent rebellion or foolish pride by a black student ever escalated because a teacher provokes an unnecessary confrontation rather than demonstrating the least bit of understanding? Why are African-American students disproportionately represented in remedial writing, reading, and math classes where they are babysat with worksheets and television instead of being taught what they need in order to return to grade-level classes with their peers? How many African-American students get stuck in the flypaper of Special Education classes because of “behavior” issues—and never catch up with their classmates academically and socially?

What is the most visible recent reaction to the race relation problems of our nation? It is, sadly enough, yet more separation based on race. Whether this new segregation is voluntary or involuntary, it is eating away at the fabric of our nation and creating more political and social volatility than we have seen since the 1960’s because it produces individuals who are less and less able to thrive in a heterogeneous nation such as ours.

Voluntary segregation, often done in the hopes it will produce either pride or protection, is a disastrously retrograde practice in all its variations. Whether we are talking about the creation of black-only dormitories on college campuses, public schools pushing the recruitment of black teachers to teach their black students, or any of the many other educational initiatives now washing through our progressive schools and colleges—all of which are predicated on the belief that exclusionary racial policies are somehow a benefit because they provide a less stressful or more supportive environment for students of color—I shudder when I consider the long-term implications of these practices for our educational systems and our nation.

The premise of one of the most important Supreme Court decisions of the past century, Brown v. Board of Education, was that separate cannot—by its very nature—ever be equal, and our nation has a clear obligation to strike down any policy that promotes separation based on race. The results have obviously not been all that we hoped they could—or should—be, but I challenge anyone to argue that they would willingly trade the world of 1954 for what we have today.

Involuntary segregation is yet another problem bedeviling both our schools and our country. The residential re-segregation that has taken place over the past couple of decades due to the stunning growth of income inequality in our nation is a slap in the face to the spirit of the Brown decision, leading to the creation of more and more predominately minority neighborhoods that make many of our public schools now look like throwbacks to a Jim Crow era that we thought had been left behind a long time ago. We need to do everything in our power to push back against this pernicious segregation of our nation—and not just shrug our shoulders because it seems a natural outcome of the benign realities of the real estate market. Housing segregation should be front and center of every discussion about providing a high-quality public education to every child in America—and improving the quality of our daily lives.

What can we do about this involuntary segregation caused by the income chasm in our country? Although school vouchers are not fashionable in some quarters at the moment, I firmly believe that a nationwide expansion that will allow all students to take their state and local education funding to the school of their choice is the best—and perhaps only viable—option for combatting school segregation based on the residential patterns now driven by income inequality. Although limited school choices are now locally available in many school districts, we can immediately improve the lives of millions of low-income children by allowing all parents to exercise their informed judgment and break free from the tyranny of public school monopolies that trap many and serve few. The rich always have the option of reaching into their bank accounts and sending their children to a better private or parochial school if they are unhappy with their public school options; this choice should be available to all parents regardless of financial resources.

Nothing will ever be made better by building more walls between ourselves—or by allowing them to be built through our acquiescence. If we do, we will simply end up producing more and more people like that woman mugged on the subway in New York City those many years ago. Casual racism is still all around us, and we deal with its issues every day. However, we cannot eliminate this by allowing our society to drift even further apart because of the fear and frustration that so many of us feel when we hear or read of yet another police shooting, scary street crimes, or the persistent and disheartening achievement gaps between black and white students in our public schools.

If history teaches us anything, it is that retreat in any aspect of life is never a good idea. This is especially important to remember if we hope to ever heal the racial divisions in our society; we will succeed only by pushing forward despite the obstacles posed by those—both black and white—who are ready and willing to judge someone else based on the smallest sliver of difference in our DNA. To exist behind walls of mutual miscomprehension should not even be an option as we strive to create a world where we all see the person inside instead of only the shell on the outside. It is too bad we still need to fight the battles necessary to make this so, but surrender simply cannot be an option: The miserable alternative is yet more anger and misunderstanding driven by increasing distance and disdain.

Rather than allow ourselves to be driven further apart because a boneheaded racist—or perhaps an equally ignorant person with a PhD—believes that more separation and segregation is a great idea, we need to do the hard work of renewing the promise inherent in the Brown decision:

We rise fastest when we rise together.