Ah, see, now this is a hippogriff of a very different color. The thread was originally framed in terms of theory and mathing things out, when you said "I think that this is mostly due to wrong mathematical implementation. Or at least wrong attempt of presentation of different cases." and started talking about progressions and functions. That's why you're getting discussion and feedback about the math side of things, and expectations and general system performance and such.
But it turns out that the actual problem you're trying to address is that you tend to run games for a lot of people who don't know the rules and who try to do fun/inventive/off-the-wall stuff all the time and get frustrated when things don't turn out as expected, and a lot of other people who powergame and don't care about the plot and just want to kill all the NPCs to show off how ub3r l33t they are, and these two types of people end up in the same group and have totally incompatible expectations and playstyles.
Basically, it's like if you had a group where every single arcane spellcaster only ever casts basic blasting spells with low save DCs and is getting frustrated that they can't kill anything, so you post a thread wondering how you can improve the sorcerer and wizard to be on par with the warmage, and everyone is talking about the tier system and good school specialization and battlefield control tactics and such, and you're getting frustrated that they aren't talking about wizards throwing fireballs and they're getting frustrated that you're claiming warmages are more powerful when they're factually weaker and both sides are talking past each other.
So, some observations:
1) Someone who tries to do one cool thing, fails, and then complains that their class is a sucky one-trick-pony doesn't understand the rules and their outcomes, and instituting houserules to help them out isn't going to make them understand the rules better; it's still likely that they'll try something else the rules don't support and get upset about that not working. Powergamers try to abuse the rules by definition (not to be confused with optimizers or min/maxers, but that's a whole 'nother topic), and instituting houserules to limit them isn't going to make them play nice with the rest of the party; no matter what kinds of houserules you implement, a powergamer is going to try to abuse those houserules.
You cannot fix playstyle incompatibilities with rules, unfortunately, so understand that changing up the skill system isn't going to magically fix the group dynamic problems, and this is something that would best be addressed out-of-game in conjunction with whatever houserules you decide to implement.
2) The issue of untrained/low-modifier characters seems to be less "I want them to still be somewhere on the d20 compared to an optimized build to avoid auto-success and auto-failure" and more "I want people to have fairly good chances of success for untrained tasks at all levels so PCs with 0 ranks can try cool stuff and succeed at it a reasonable portion of the time." That part can't be fixed with different progressions or long-term skill investment, because the issue crops up at low levels long before the high-level progression divergences.
Three good ways to address this come to mind, which can be used together or separately:
- Have skill "progressions" start with (or consist entirely of) big up-front bonuses that outweigh skill ranks for a few levels, something in the +3 to +5 range like PF and SWSE do, and then encourage players to scatter skill points around instead of maxing out a handful of skills. This won't solve the problem entirely, but it certainly helps when you can put a single rank in something and suddenly be moderately good at it.
- Implement a good "stunting" mechanic, that is, giving out hefty bonuses when players try something that (A) is fun and cool, (B) they describe well/flavorfully in-game, and (C) the character isn't already good at. That gives Mr. Frustrated an incentive and a reward for trying cool things and helps ensure that they succeed, without allowing Mr. Powergamer to stack that bonus on top of an already-high modifier.
- Let PCs save skill ranks and invest them as needed instead of investing everything at level-up, within whatever limits you feel are fair. If a PC can say "I wanna swim over to that underwater statue and do a thing! *invests 5 of his 30 saved skill points into Swim* Er, I totally took swimming lessons last summer, why do you ask?" then Mr. Powergamer can meticulously invest his ranks as desired and Mr. Frustrated can throw a bunch of skill points at a daunting task and they both come out happy.
3) The issue of optimized/high-modifier characters seems to be less "I want to institute diminishing returns and higher DCs because I think it's logical" and more "My players are powergaming one or two modifiers and using it on everything, which is boring and frustrating." This is a bit trickier to address, because the example you gave was optimizing combat stats, not skills, and the skills that can actually break games--Bluff, Diplomacy, Hide, and UMD, mostly; no one's breaking games with Decipher Script and Use Rope--break things even without automatic success because the things they do are powerful whether you can do them 100% of the time or just 20% of the time. To give specific advice here, I'd have to see some examples of the kinds of skill-related breakage you're seeing
But one thing to keep in mind in the meantime is that not all skills have to scale indefinitely. It's entirely within your rights to rule that you can only ever have 8 ranks in a skill because skills replicate real-world capabilities and "real-world" tops out at 5th, or declare that regardless of how you stack bonuses your modifier is hard-capped at +10, or something like that; it's like doing E6, but just for skills.
Does that all help address the actual underlying issues?