Magical compulsions are bread and butter in fantasy and folklore- the noble hero swearing a binding oath, or bewitched with a slave collar, or given a geas- it happens all the time. It’s honestly a little unusual to see a work of fantasy without anything like it.

It’s also one of the most dangerous plot devices to use without damaging the plausibility and internal consistency of your worldbuilding. Why? Because power seems more power, almost without fail. If a tool of magical compulsion exists, a government, dark lord, powerful wizard, or someone would logically try to use it on as many people as possible, barring constraints. (Plus magical compulsion itself is kinda disgusting.)

Slave collars are a good example of this. There is no reason why a figure of power wouldn’t secure their entire governed population with them, if nothing prevents them. In the Wheel of Time, the a’dam, slave collars that can only be used on magic users, are controlled by the Seanchan Empire, which mandates that every single magic user must be bound with them (well, not all of them, but spoilers). In Jim Butcher’s Codex Alera, they can be used on everyone, and the only thing preventing the unscrupulous from doing so is a strong central government which would frown on that very strongly. (Though it still happens to an extent).
So right there we have two constraints- limited demographics and societal/authority disapproval. Both are versatile tools for allowing the use of magical compulsion without breaking the world around it.
An example of a worldbreaking magical compulsion takes place in Tamora Pierce’s Daughter of the Lioness duology. Blood oaths literally invoke the power of the gods to enforce them, and they don’t require any special invocations, spells, or materials (beyond the blood, of course). The economic cost to enact the blood oaths are almost nonexistent. There is literally nothing to prevent an unscrupulous ruler from forcing his subjects to sign blood oaths on pain of death and permanently enslaving his population, or a gang to try to take over a city the same way, etc. (It’s possible that the gods would look down on this and prevent it, but I don’t recall that being addressed in the books.) I’m a huge Tamora Pierce fan, but that really broke my suspension of disbelief. (Definitely check out her stuff if you haven’t, that’s the only major problem I have with it. I especially recommend The Immortals or Protector of the Small).
A third method of preventing magical compulsion from getting out of hand can be derived from that, as well- Cost of compulsion. The cheaper in time, money, and skill it is to lay a compulsion, the easier it is to abuse. Make compulsions more difficult, and eventually the economic cost will outweigh the economic benefits.
What about voluntary compulsions, though? They show up occasionally. Well, the restriction there is right in the name. A good example is from the webcomic Order of the Stick, where the protagonist’s father has sworn a blood oath (an expensive one requiring a magical tattoo and consent) which binds him to vengeance. Of course, that gets passed down to his son, so… Oops.
Then there’s straight up magical brainwashing. Compulsion, from The Wheel of Time is a good example. It’s kept under control both by societal factors (very, very illegal) and skill factors. (Doing a quick job on someone results in them becoming adoring zombies who can no longer function in society.)
It’s tough to slot in exactly how love potions fit in. They’re definitely abrogations of free will, but they’re such a diverse category it’s hard to pin them down. (They’re all aimed at only one or two things, sure, but they use a radically differing number of methods to do so).
Regardless, all methods of magical compulsion are generally pretty abominable, tricky to limit story wise, and should only be used by the villains if at all possible. Better yet, just don’t use them. On top of being kinda disgusting, they’re super overplayed tropes, and there are more than enough methods of nonmagical compulsion out there to obviate the need for magical ones.