Exp in a party is based on the highest lvl party member. If you are the highest lvl, and you gain no exp, neither does anyone else, and if you gain no class experience, neither does anyone else.
It's up to the specific party. Some parties want exp while running the dungeon, some only care about cp, and others are running for trophies/drops. Some may even be running only for a daily.
Best way to know what your party might want is to ask them yourself. Everyone wants something different.
Also: 5 player dungeons always give more experience than 3 player dungeons. Although, Malice (the next 5p after Vileshark), barely gives more exp than Branda, in lim at least, so Ulta is the next best dungeon for class lvling after VSHQ from my experience. Aside from Malice, all the other 5 players I've tried give close to 2x the exp as a 3player.
I'd recommend joining a guild. They're helpful in terms of teaching you about the game, and are generally more accepting if you don't know what you're doing as long as you're willing to learn.