Can we build the math from the ground up? (And does Vancian help or hurt that?)

[NigelWalmsley]

For example, what happens when someone encounters an illusion in combat? This is something that 5e has no suggestions for, despite having about 10 different powers that do so.

Having separate effects for in combat and out of combat causes a lot of problems. If Silent Image does something different to a guy you are fighting than a guy who is simply in the environment, you inevitably create incentives for stupid metagaming. Maybe the combat rules are more favorable, and the players start having the Fighter declare his action as drawing a sword (putting the party "in combat") every time the Wizard casts Silent Image. Maybe the non-combat rules are better, and the players invent increasingly contrived excuses for why the Wizard is totally not in combat right now.

It's a means to limit magic knowledge without sacrificing diversity. The various effects are physically printed under the heading of Fire Attack.

Sure, but such as system seems more akin to the Binder's Vestiges or the Cleric's Domains than it does Upcasting. At the point where you're talking about abilities whose only overlap is "does Fire damage", it seems bizarre to insist that these are one ability you power up, rather than merely a suite of abilities.

In my experience, people pick varied things not only to have different SFX and maybe 20% variation in effectiveness, but to bring something unique and necessary to the group.

In my experience, 90% of class choice comes down to "I like this class". That's why there's so much damn hue and cry over the Fighter sucking. People like the Fighter and want it to be good. If people just wanted to bring necessary abilities to the table, they would simply ignore the classes that sucked. People's class choices need to give them the opportunity to contribute in most situations, and to tailor their capabilities to the group. These are things the Wizard does.

Correction - the Wizard is better than a Beguiler when they try to do non-Beguiler things.

Sure, but that's exactly the paradigm you want the game to have, isn't it? You've got Beguiler stuff that the Beguiler is better off doing, and Wizard stuff the Wizard is better off doing. What does it matter that the Wizard's solution to the Beguiler stuff is a 6/10 instead of a 1/10? The Beguiler still does a better job of it. The only time that comes into play is if there isn't a Beguiler around, and at that point it's the difference between the adventure continuing and not. Where's the problem?

Beguiler already has a bit too much going on, but it's balanced by a simple fact - unless they charm/dominate someone, they have no innate way of dealing level-appropriate damage after levels 1-3.

So what? The Beguiler still has the toolkit to overcome pretty much any challenge, because challenges aren't just "do X damage". Having or not having a particular tool doesn't matter. What matters is being able to overcome challenges.

Fabricate - frankly, I don't think it should even be in the game. Maybe as a class feature for some specialized crafter-mage archetype. Not as a spell, certainly.

It depends what you mean by "spell". Fabricate certainly needs different mechanical constraints than Fireball does, but "magic up some non-magic goods" is absolutely the kind of effect that can and should exist in your system.

Ah, but Clerics would also lose most of their incredible spell access.

That's not the point. The point is that you don't need to take away the Wizard's toys to give other classes a niche. You can instead give them toys that are more efficient in their particular niche. If you want to rely on the Rogue for stealth, you don't need to nerf the Wizard, you need to make the Rogue as effective at stealth as the Beguiler.

And that's why not everyone should learn to Teleport

Of course not everyone should learn Teleport. Some people should learn Shadow Walk or Tree Stride or any of the other fast travel abilities that exist in D&D and the rest of the genre. What shouldn't happen is telling the rest of the players that this is the Wizard's turn (or the Fighter's turn or the Rogue's turn) to get to solve the problem, so they all get to sit down and shut up. That experience is the worst part of the game, and I categorically reject the notion that there is any problem that is solved by making it more common. Any problem that is interesting enough to spend table time on needs to be interesting enough for there to be tradeoffs between viable solutions.

There's also, say, Chaos Beast (about 44 HP) or Chimera (about 76 HP), against a few of which Fireballs are somewhat more valid.

Those creatures both have more than double the HP of our Bugbear, which means Fireball is falling behind against them too, just to a smaller degree.

Basically, as 3.5 is right now, you could very well be a Wizard with 2/3 (or even 1/2 if you're good with Wizards) casting and still contribute significantly (at least for the first 10-12 levels) to a party where a Beguiler or a Warmage or a Dread Necromancer exist, simply by covering for their weaknesses, even if it's not technically level-appropriate.

"Contribute significantly" seems like a phrase that is doing a lot of work there. What exactly does that mean? I could imagine a standard by which a 9th level Warblade "contributed significantly" in an 11th level party. That doesn't mean I'd pick a 9th level Warblade over an 11th level character given the choice.

I mean, I can imagine a spellcraft-on-the-fly system like choosing the effect/spell seed, shape of the effect (cone/cloud/burst/single target), and spell level to determine the resultant spell. Not sure if it'd be less trouble than just having four different spells, and certainly not as robust since you wouldn't have a lot of distinct effects in such a system. Doesn't mean that it'd be bad, though.

Such a system also makes it impossible to have Lightning Bolt, Fireball, and Cone of Cold without also having Line of Cold, Lightning Ball, and Cone of Fire. Like many things, there is a place for it within the rules, but it doesn't seem especially appealing as a core part of the system.

But the trick is always convincing the readers that what's written makes a lick of sense.

Again, I feel like the fact that virtually every major fantasy series pulled off this "trick" indicates that it's not really that hard. I think people are actually quite willing to accept that sufficiently hardcore badasses do magic. No one is out there demanding that Thor explain how he gets to have lightning powers despite being hammer guy. I think if you just wrote martial classes that got magic at high levels, people would simply accept that. Indeed, with respect to at least the Paladin and the Ranger, they already do.