For starters, factions serve as a limitation on what race you can play, and it's both a story based limitation and a more arbitrary one. To some degree, the previous games in the Warcraft setting, the RTS series, set up a certain animosity that can't easily be done away with and still retain any sense of continuity with previous games. In a world where orcs have destroyed a major human city and laid siege to another, a world where humans kept orcs penned in camps like cattle, it would be hard to justify an orc warrior in Stormwind. It wouldn't matter what the orc's intentions were - he'd be more likely to be driven out or worse than a death knight who has a faint hope of being mistaken for a living member of her race.
The arbitrary limit isn't unimportant - there are a lot of class/race combinations. By limiting all but one race to one of the factions, you don't have to have all the possible combinations to choose from. The initial decision of a faction has already limited some options for you.
Before Mists of Pandaria there was also something called the silhouette principle. In essence, it meant that in player vs. player combat, it was always easy to immediately know if you were facing a friendly or a hostile target, even before you could see any health bars or the like. The advent of the pandaren as playable characters for both factions weakened this argument. But while it may be weakened, it is not entirely dead. If an Alliance player sees a tauren or a horde player sees a worgen, they know immediately that's an enemy, without needing to look any further.
But finally, there's the questing experience. Frankly, it often lacks the variety one might expect - many quests the Horde and Alliance get are practically identical, especially in the content that comes with expansions. This is especially true when the quest experience isn't rooted in Horde or Alliance questgivers but rather neutral ones that give quests to all comers (such as is the case in many zones in Mists) but even so, there are also quests that are functionally identical even if they're given by a faction specific questgiver in a different location. Still, it can't be denied that (to give one example) the Horde and Alliance experiences in Ashenvale are fundamentally different, and there can be value in giving players a reason to play a character on the opposite faction.
The question then becomes - do factions serve enough of a purpose to keep them? Or to keep their restrictions as they are now? When RealID was first announced many of us were surprised, even outright shocked, when the restriction on cross-faction communication was lifted for our RealID/Battletag friends. This was a surprising change. And one that I would argue has fundamentally improved the game in many ways. But it also fundamentally weakened the faction barrier. Should it have? And should it have weakened it even more? With RealID allowing us to talk to our friends, it's interesting that it doesn't necessarily allow us to play the game with those same friends.
As for PvP, we already mentioned the weakened nature of the silhouette theory.
I should point out, I don't expect to ever see factions go away. Nor do I ever expect us to be allowed to play an Alliance orc, or a Horde gnome, not even through some paid service that let you switch factions without switching races. Why don't I expect that? Because I don't believe that Blizzard's character services are cynical cash grabs. If you're going to make it possible to pay money to faction switch without class switching, there's no reason to not allow players on one of the factions to just roll whatever race they want. There's a difference between a paid service for convenience, and one for something you just couldn't do that makes a real gameplay difference. Racial abilities would be even more complicated, progression raiders on both factions would all just pick the absolute best racial at will, it would be pointless to make it so you could only get these benefits by paying.
In the end, I'm fairly torn. I don't think that factions are as useful as they were once presented as - in a game world where blood elves can join with orcs despite the orcs trying to burn their city down and killing whole swaths of their people less than thirty years ago, it's not beyond the realm of possibility to imagine humans and orcs, or draenei and orcs, finally burying the hatchet somewhere besides each other's skulls. But, while I don't think factions are necessary, I also believe they're too entrenched to change in any significant way. They're certainly not going anywhere.