1. Have you talked to them? This should be the first rule of DM-player interaction, but it gets missed sometimes. Anyway, if you haven't, do so, if you need reasons we can provide many of them.

2. If the game works better with three players, play with three players. No amount of game balance or social inclusiveness is worth having a bad time during your recreational hours.

3. Relax. No one is policing your game to make sure it's run properly, do whatever you need to do personally to have fun- from your description everyone else in the group is taking that liberty.