Read. Before you sit down to write your own RPG, you should read (and, ideally, try) a number of other systems, just to see what's out there. D&D is only one way of doing things. Look at hings like Fate (available as a pay-what-you-will), GURPS, Mutants and Masterminds (
available as an SRD), Savage Worlds ($10 for a paperback; quickplay rules available), at least one White Wolf Storyteller-based game (Exalted, Vampire: The Masquerade, etc-- you might try to look at both New and Old World of Darkness books. I know there's at least one free nWoD quickstart adventure out there)... anything, really, especially if the rules look really different from those you know. The more sources you can draw on for inspiration, the better.
Also, when you're done with your reading, build the basic mechanics first. It took me four drafts of STaRS to get the core elements right; I imagine that you'll face a similar experience. Make sure the game flows properly before you worry about the exact numbers, make sure the numbers scale properly before you write core rules, make sure the core rules for combat and skills and such work before you start writing actual abilities and spells and such.
3d6 will have a strong bell curve. Almost fifty percent of rolls will be between 9 and 12. Extend that up to 8-13, and you're talking two thirds of all rolls. Modifiers will have a much larger impact than in a 1d20 system like D&D. On the plus side, you can make pretty reasonable assumptions about rolls:
- You have a 90% chance of rolling 7 or more
- You have a 75% chance of rolling 9 or more
- You have a 50% chance of rolling 11 or more
- You have a 25% chance of rolling 13 or more
- You have a 10% chance of rolling 15 or more
Does that help?