> The idea that in a war there can be legal and illegal actions established by international treaties to protect civilians as much as possible can only work if two (or more) legitimate states are fighting each other

This is not true (the laws of war work and have been applied successfully in conflicts not involving two or more legitimate states) and it's an assumption that seems to have negatively informed the questions that followed.

> with leaders who can be held accountable for the orders they give.

Holding leaders accountable ("legitimate" political leaders, terrorist leaders, rebel leaders, we can do it) is good, but we also hold individuals accountable.

> But does it still make sense to talk about legality and international treaties when on one side there is a terrorist organization whose method of warfare consists of kidnapping or killing civilians?

Of course it does. The notion that one side is no longer accountable for harm done to civilians in violation of the law because the other side has harmed civilians in violation of the law is wrong.

> At this point, doesn't complying to international treaties only mean further endangering their own population?

Sometimes yes. It certainly does put troops in danger often enough. Everyone who is party to these treaties is well aware that a country could be safer in a conflict if they just quickly incinerated the other side, and they've chosen to be bound by these laws anyway.