Could the place be a disguised 'Mage's Magnificent Mansion' (to give the SRD name) where the caster invites them in, only to disappear for some reason? The spell description states only the caster can open the entrance, and the entrance is the only way to leave, so the party are stuck until they figure out......(add your plot).

Perhaps the Mansion's owner is magically trapped and needs the party's help to get out?