Didn't you post a thread complaining about this 6 months ago or more? :) I seem to recall reading exactly this situation described.

A couple of comments ...
1) The module STARTS with "In previous adventures they have identified their enemy as the witch Esmae Amarantha, learned of her plan, and recovered a magical locket that can be used as a weapon against her."

If the characters don't know about the BBEG's plan, don't have or know about the locket and what it can do, or know the identity of the witch then the DM really didn't set the scene properly.

This is covered in part 1 where one of the things the party should learn is:

"The characters are in possession of a magical locket that was used by Esmae to draw the vestige of the Evening Glory from its prison in the Amber Temple. They should be able to use it to do the
same to Esmae."

NPCs needed in trying to locate Esmae include two options with knowledge of the locket and how to use it.

Spoiler: The locket
Show
"How Do I Use This Locket?
The characters have in their possession a magical locket recovered in DDAL04-10 The Artifact and attuned to Esmae in DDAL04-13 The Horseman such that they can use it to draw the vestige of the Evening Glory out of Esmae. Ixusaxa and Jeny both believe that it must be presented forcefully in some manner to do so; likely in a way similar to how it was used to pull for the possessing ghosts in the Amber Temple. They suggest being obvious and loud but are stumped on details as to how to do it and exactly what will happen. Note: The characters do not know this but to use the locket, the character must use an action to present it forcefully to Esmae and loudly demand the Evening Glory depart her body. Esmae then must attempt a DC 15 Charisma saving throw to prevent the vestige from being pulled out. The DC increases by 5 for attempt made to entrap the vestige."


Here is how Esmae is supposed to be played.

Spoiler: Esmae
Show
In order to ritual going she must use one action every round for ten rounds while possessed by the Evening Glory, either her own or a legendary action. If she cannot do so, the ritual is disrupted. In addition, she saves her legendary resistance ability for effects that prevent her from casting the ritual, banish her, or exorcise the Evening Glory.


Did you folks have the locket and at least a minimal idea of what it does and how to use it? Without that, the DM pretty much decided you weren't supposed to succeed at this encounter or at least left the only option to beat her down with all of her advantages and resistances. The DM also upped the difficulty a bit by adding a couple of witches.

2) The thug and witches are cannon fodder to slow the party down and stop them from focusing on Esmae. The armor is intended to delay the party.

However, the room at the top of the spiral stair is 10x20. Did the DM eliminate that forcing single file on a circular staircase with just the armor at the top?

If the room is the 10x20 from the module, the characters need to surround the armor. Rage/reckless ... shield bash from the paladin to knock it prone - possibly grapple to keep it there. Alternatively, leave it up for the ranged rogue but he can always step in and attack with his short sword then disengage. The cleric could throw up an upcast spirit guardians and later cast spiritual weapon. It is important to eliminate the armor before opening the door. The paladin should smite to take it down more quickly.

If the landing is 5'x5' with the stair ending there allowing only 1 character to attack in melee at a time then the DM decided to force you to lose since the delay factor goes way up. Still probably best with the rage/reckless barbarian in front backed up by the cleric with spirit guardians running. If the rogue can get a shot from third in line - do it since sharpshooter prevents issues with cover and he will still get sneak attack. However, it will probably take 3-4 rounds to take down, possibly longer with bad luck, if the DM makes it impossible to use all the characters to take out the armor.

Note: The module gives the armor instructions as "she directs to the armor to the top of the stairs to defend them against all comers" ... so running past isn't an option it might be if she actually told it to defend the door.

3) Once the way is clear ... party prepares to enter the room. The druid upcasts conjure animals from outside the door - the BBEG can't counterspell what they can't see. A 5tg level slot will conjure 16 wolves or 4 dire wolves/bears. These are either sufficient to harry the BBEG or eliminate the thug/witches so the players can focus on the BBEG.

The BBEG is supposed to use spells from range ... with bad luck, disintegrate will instantly kill one character. Finger of death could take out another. Fireball might be less of an issue since it affects everyone in the room but the BBEG might not care since they are immune to fire. The BBEG is not immune to prone so shield bash could help and magical weapons do full damage.

As for the next room, the witches don't have much in the way of spells, hit points or attacks ... an upcast spirit guardians actually has a decent chance of killing them.

The characters may deduce that since the thug and witches run to engage the characters that Esmae must be doing something towards the ritual and they will see her use her action or a legendary action to continue the ritual. So clearly, if they want to disrupt the ritual they have to prevent Esmae from using an action to continue the ritual. This could be either the DM forgetting to use the action :) or taking Esmae down to zero hit points or using the locket to recover the remnant of the dark god and make Esmae much easier to fight.

The biggest threat is probably the effect of the kiss. However, the kiss is a melee spell attack which means that it can be ended by dispel magic or potentially even counterspelled if someone in the party can cast it ... though that is probably too expensive to consider. This is one of those situations where a devotion paladin might be preferable to an ancients one since their 7th level aura blocks charm effects. :)

On the other hand, once it becomes obvious that the BBEG is using charm - which happens as soon as someone fails, this would be an ideal time for the cleric to cast calm emotions making the entire party immune to charm and the frightened condition. More useful is that your friends can choose to fail the save automatically .. which of course they would do since their friend the cleric is casting it even if they are also in love with the BBEG ... the spell isn't being cast on the BBEG so how can it harm them?

Once the party is immune to charm then although the BBEG still gets advantage to attack the ones affected by the mists ... it should go much quicker, especially if they can burn off the legendary saving throws. Keep in mind that the BBEG in this case has no choice in using the legendary saves. If they fail a save then they have to use one. "Legendary Resistance (3/day). If BBEG fails a saving throw, she succeeds instead." For this NPC, use of legendary saves is not optional. This means that they could potentially be burned off using sacred flame or another cantrip. On the other hand, the DM might have changed that to make the NPC more challenging.

Anyway, I think the key spell in this case is probably Calm Emotions from the cleric to suppress and prevent the charm effect during the fight.