This is because you make a party before entering the instance. Some instances are 3-player, some are 5, etc. When you make a party, the game doesn't know what instance you are planning on entering, so you can make a party larger than allowed. However, the quests for the dungeons do say how large a party may be.