There's of course no single "right" answer to how policing would happen in an anarchist society but there's a few ways that I've imagined it (all with their drawbacks...but as you pointed out the current system has drawbacks too...I think any system that presupposes bad humans will have drawbacks because bad humans can always work for the good guys...the trick is to keep the bad guys from being able to do too much damage.)
In The Dispossessed, LeGuin describes the anarchist version of jail. If you imagine a society where people can get what they need with less effort than it takes to steal on a regular basis...you can imagine that the only kinds of crimes are the kinds that are committed by the mentally ill. Psychotic crimes. Predator crimes. In an unorganized chaotic society these people would probably be shunned to live in the wilderness or if that proved impossible they would be killed.
In LeGuin's Anarres there is a sanctuary where they can go. They check in voluntarily and as long as they are there they are assured that they will be protected from the wrath of those seeking vengence against them, they will be provided education and psychological care. Those who choose not to enter this sanctuary face whatever happens to them. They may be denied resources by those whose help they need. They may be physically harmed. This is not a threat, it is simply a human reality.
Ostracizing doesn't have to be carried out by "every single person". If some people think that the person hasn't done anything wrong and want to provide services and resources to that person then so be it. Maybe that person really didn't do anything wrong. They retain the right to live with those who will cooperate with them and if they violate the trust of those people they'll have to move on or move out to exile or find sanctuary. It's sort of like everybody is on the jury and the jury can have multiple verdicts.
Various anarchist visions imagine a police force that can be called upon by the Community to protect the weak from the strong. Kind of like a peacekeeper force. Everyone takes a turn on the police force. No one is allowed to serve on the police force for more than a certain length of time. There is a recognition that weilding that kind of power is inherrently corrupting and must be limitted and monitored.
And how do we prevent brutality? How do we prevent people from using the threat of physical violence as a "punishment" for even the slightest annoyance? Well probably the most important thing about an anarchist society is pacifist education. A culture of peace has to be nurtured, celebrated, and maintained. The psyche of our society is fundamentally shaped by the brutality of capitalism, racism, classism, consumerism, fear, and the hoarding of property by force. The fight against capitalism and heirarchy is also the fight for a new way of viewing the world and one's place in it.
The UC-IMC, an organization that is ostensibly Anarchist in structure has a Mediation Policy which is very similar to the court system. Mediators instead of judges and advocates instead of lawyers. The system is wholely voluntary and does not itself have any power but it helps people who have grievances figure out how to communicate and get action taken on their grievances.
Anarchists don't want to throw the whole system out. Just to question it, to strip out the bits that exist to serve the rich and powerful, to strip out the bits that exist to protect the strong and few from the weak and many.
Perhaps all these utopian visions I've provided are naive and incomplete. Perhaps taken to their logical extreme, and left unwatched, these systems would devolve directly into the system that we currently have. But then...they're theoretical ideas about a theoretical future. They're the ideas of one person. The reality is that any true anarchist society would grow organically, would solve problems as they needed to be solved, and would involve the ideas of the entire community. Different communities would solve their problems in different ways. They might learn from each other's successes and failures.
The important thing for me in OUR current situation is the process of questioning power. Questioning the extra-judicial murder of suspects by overzealous, racist cops. Questioning the state-sanctioned murder of convicts who received unfair trials with improper representation. Questioning the use of the court system to intimidate and financially overwhelm those who can not afford to participate. Questioning the criminalization of victimless crimes and the overwhelming bias towards punishing the poor and racial minorities with these laws. Questioning the disenfranchisement of millions.
Something is deeply deeply broken about our justice system. Something has got to change.