Using monster creation rules to make unconventional PCs

[Bryan]

I've got a few creative players who often want to make characters that don't necessarily fit with rules for any classes. Like one guy wanted to play a Dream Wolf which is an older fellow who sleeps and more or less astral projects a corporeal ghost wolf. The other wanted to play what was basically a selkie. And so it has gone for a couple years.
A ghost that is tied to an object carried by another PC.
A swarm of bug-size fairies that "crew" a human-size construct.
Etc.
Some can be kinda/sorta approximated using Race/Class/Magic Item combos but it has occured to me that it might be easier to just abandon the classes and use the monster creation rules to let them play whatever they want.
I'm wondering if anyone else has done this and if there were any tips, tricks or suggestions they could give me.

[Dualight]

I have been playing with how to do something that is not quite the reverse: I have been breaking down statblocks into something resembling classes (I read about how level adjustment worked in 3.5 and felt inspired).
As such, I have been staring at monster statblocks a lot.

As tyo actually answer your question: the big problem you'll encounter is progression. The monster creation rules (literally just guidelines, the rules amount to "whatever works") are about how to determine how strong a monster is once you have made it, and to give ideas on how to scale them up or down if the CR is not yet what you want. Unless it is for a one-shot or a short campaign where no one gains even a single level, the monster creation parts of the DMG will not help you.

That said, there is no reason you couldn't give players some monster-exclusive features by substituting (class/racial) features. For example, that Dream Wolf could be constructed using Wild Shape or the Bestial Spirit statblock as a base.

Speaking of characters that cannot be made without homebrewing, I have an anecdote that might help: First, for a Halloween one-shot, I played a cursed sword, mechanically it was mostly a Fighter, but i only had fixed mental stats, with the physical stats being based on whatever poor sap I could possess, with the DM assigning a save-based ability for me to try to possess those that touch my sword.

THere is one other approach: Sidekicks. The challenge their is that the TCE rules their are for CR 1/2 and lower, but that could work as a starting point.

[NontheistCleric]

Some ideas to represent these character ideas with normal classes:

Ghost wolf guy is an Astral Self monk. Just reflavor it as the whole wolf appearing when the abilities say only parts of it appear, and say the guy is levitating asleep in the middle of the astral form. Doesn't change anything mechanically, but he can have his full astral wolf at all levels.

Selkie guy is a Moon Druid. If you can't find a seal stat block, reflavor some existing aquatic thing to suit.

Ghost tied to an object is a Hexblade with a specter. The player just roleplays the specter as the main character instead of the Hexblade. The item it is bound to is the Hexblade's pact weapon, which must kill and absorb life energy periodically to release the specter.

Fairies are a Warforged Swarmkeeper Ranger. Roleplay it like the fairies control the warforged.

[Greywander]

I second the advice to see if you can refluff existing material to suit your needs. It will save you a lot of work. If that's just not working for you, though, you could try coming up with homebrew races or classes. I find it to be fun, though it can be a lot of work. It could be really fun to work with your players to design these things in collaboration.