I generally agree with you here.
What I've found that works is something like this:
Acts that infringe on others (theft, imprisonment, harm, etc.) are evil. Acts that help others that do not gain you anything are good. Things that do neither are neutral.
Most people do some mix of all of the above, at various times.
Good people will do a lot of neutral things, and a lot of good things. They'll do some evil, but usually more "minor" evil, and usually only in great need and as a last resort, and they'll feel bad about it.
Neutral people mostly are the same as good people, but they do less good things (probably a lot!) and are slightly more likely to do evil. IOW, good and neutral people are mostly the same, except good people do more good. But both still do a ton of neutral.
Evil people do evil willingly and with little compunction. Maybe not *big* evil, but evil nonetheless. The harm it does to others doesn't really enter into it, but impacts on them do.
In concrete terms, a good person may steal bread to feed themselves or others in need. They'll do so if they have no money, and they'd likely try to get or work out a deal with the shopkeeper first. But if all else fails? Sure, they'll steal it. But they'll feel really bad about it, and probably try to make amends when they get the opportunity.
An evil person? They'll steal that bread just because they're a little hungry, it's there and they can get away with it.
A good person can still do evil, and an evil person can do good. And there's grey areas between them, and fuzzy bits.
I find this to be a good framework for gaming. It doesn't require you to think about what justification is required (as "it's good if you can justify it" just really asks you to be clever in your justification). It sets understandable lines. It's predictable - someone knows, in general, what is evil and what is good and what is neutral. Most of the things that would be evil/good line up really well, and the weird bits are usually in weird areas. It allows for interesting villains - characters that do evil things, but for good reasons, are still seen as evil, overall.
About the only weird one is Robin Hood, but it's worth noting that in a lot of the original legends, the money Robin stole was, effectively stolen in the first place. So there's that.