The one complication I had when thinking about this subject is what about neutral outsiders. Obviously a modron stays neutral if all it does is follow rules even if those rules hurt someone, and a salad stays neutral even if its randomness causes unintended harm to someone. So why were they different then the wizard and it came to me that the difference is they aren't people. Real people have a connection to the community and the people around them obligations implied and explicit and so on.

Thinking about this made me more hesitant about my opinions of the wizard in the tower, because the classic neutral wizard in the tower lacks any connection or obligation to the people around him. The classic high level hermit mage really has more in common with the modron than the peasant farmer. That leads me to a dilemma. My next thought is, would it change things if the wizard had built his tower with the pepoles labor on the peoples land under the promise that he would protect them? On the other hand isn't breaking promises more a matter of law and chaos then good and evil. For myself came to the eventual conclusion that breaking that obligation in such a way was both evil and chaotic but I am less confident in my prior opinion that obviously the wizard in the tower is evil.

Part of the complication is we often think of it as two sides of a coin evil or good with neutral just being in the middle but at least for D&D that's not quite right, neutral is its own thing not merely the absence of good or evil.

In the end I still belive that inaction can be evil but the WHY of the inaction is very important.