How does banning a party work in Germany, and why do we even want to ban a party?

a simple explanation

One of the German intelligence agencies, specifically the "Bundesverfassungsschutz" (Federal Constitution Protection Agency), has produced a more than 1000 pages long report about the AfD, a party you surely know by now.
The report contains a thorough analysis of AfD's ideology and their connection to extremist groups all over the world.

This report and the work the agency does, is something like a warning system. The agency itself can not make any decisions about the legitimacy of a party, and neither can they interfere or disrupt a parties political work. All they are allowed to do is to listen in and collect proof of a parties anti-democratic/unconstitutional tendencies and plans using intelligence gathering techniques like hiring informants, tapping phone lines, analyzing network traffic, etc... This is only allowed, if significant concerns have been raised about a party that is allegedly acting against our constitution or planning serious action against our democratic state. If a preliminary analysis confirms those concerns, the agency gets the power to use said surveillance against the party in question.
In the case of the AfD, they have been a suspected extremist party for several years now, allowing the agency to surveil their communications and inner workings, until they came to the conclusion, that the AfD is indeed fighting our democratic state and are following a right-wing extremist ideology, something that is incompatible with our constitution.

Now that a flag has been raised, there are obvious reasons the party needs to be hindered from successfully eroding or even destroying our democracy. In Germany we have multiple paragraphs in our constitution that allow for a ban of all parties, groups, and organisations that are aggressively attacking the foundation of our free democratic state, are planning to remove the constitution, or to significantly alter it by force or with other antidemocratic means. There are specific extra-strict laws about banning parties since making that too easy would also make it easier for oppressive governments to ban democratic parties. Part of these laws is also the condition that the party has to be significant enough to have actual potential to cause damage.
How that works: The ONLY way to ban a party here in Germany, is to task our supreme court (Bundesverfassungsgericht) to review a parties incompatibility with our constitutional law and to declare a ban on them and all following organisation's trying to bring back that party. The review process often takes years and is extremely thorough since banning a party is a pretty extreme "ultima ratio" measure.

The idea behind banning a party is not to convince them or their voters that they were on the wrong path or to punish them in favor of other parties. The paragraphs for banning a party have been created explicitly to protect the state, its institutions and citizens from harm coming from said party and its structures, members, and supporters.
The voters will still be there. The members will still be extremist. But the party as an organized way of preparing harmful actions will be destroyed, and all following attempts to bring it back under a new name, with a new image, etc. will be outlawed as well.

Last but not least, our German supreme court, which specializes explicitly on dealing with issues related to the constitution and the constitutional rights of the people, will not start the process of banning a party on its own. It needs to be tasked with that by either the parliament (Bundestag), the federal council (Bundesrat), or the acting government (Bundesregierung).

After having been tasked with the review, the court will start a long investigation into the organisation, in our case the AfD, and will collect evidence that supports the claim that they are unconstitutional. For that, it also uses the 1100-page report of the "Verfassungsschutz", police reports, other court cases against the party, etc.

Should the court decide the AfD is actually unconstitutional, which in my opinion, and the opinion of many experts here, is highly likely, the party will be banned and ordered to seize all its operations, and to disband their headquarters and all other places they've worked in. They will also be forbidden to found a new party trying to replace it. If the party continues to organize action, not as a party any more but as a banned organisation, law enforcement will enforce the ban and disrupt any action taken by members of it, that contribute to the goals of the former party.

Now that the AfD has been officially found to be right-wing extremist, the pressure on our government, parliament and federal council is rising to ask the Supreme Court to review a ban of the AfD. There are no more excuses. If this actually happens, remains to be seen.

(As of right now, our conservatives and neo-liberals are still opposing a vote to start the judicial investigation against the AfD)

A little remark from @nicobruenjes, that really shows how strict the rules around banning parties are:

So far, the Supreme Court only banned two parties in the hostory of our Federal Republic: The SRP (Sozialistische Reichspartei) in 1952, which was an openly acting successor organisation of the Nazi Party NSDAP. And the KPD (Kommunistische Partei Deutschlands), the communist party of Germany in 1956.