This is probably a good question to revisit since we're heading into a patch where lots and lots of us will be running pickup raids and groups. Souldreamer on WoW Ladies LJ asks: "just when do you give up on a raid?" It's a good question, and unfortunately, the answer probably depends on the raid itself. If, going in, you're not sure just how much DPS the raid can do, and your healer says he's actually specced prot, and you were planning on going to bed anyway, one wipe is probably enough to call it. On the other hand, if you've dropped a few bosses and have an issue with the tank losing aggro for a second on a boss, you'll probably go back for another few wipes just to see.
Do any of you have an actual policy? I tend to not get involved in PuGs at all if I think there's a chance they won't make it -- there are too many fish in the sea, and too many other things for me to work on rather than beating my head up against a boss. But maybe it would be good to set up a rule that most of us can agree on, something like, "three wipes and you're out." That might save a lot of time and frustration in the new Dungeon system.