Qualities of a good player
Dedication This is illustrated well by a player's gems and enchants. Does he have all his gear enchanted with the top-end enchants? Is his epic gear gemmed with epic gems? (You can put rare gems in blue gear, but not in 251+ gear.) Also informative is whether the person has a secondary spec.
Experience One of the very cool new features of armory is that you can see how many times someone has killed a boss. If he's killed a boss 10+ times, then he probably understands the fight basics very well. Also, you can see which bosses he's killed. If someone has killed the Lich King 10+ times, it shows a reasonable experience with unforgiving fight mechanics. This even counts if the player has made these kills on a main character but are wanting to bring an alt. Generally, some of the most experienced players are those with several max-level characters.
Class knowledge General, open-ended questions to ask someone are, "What's your spell rotation?" or "Why did you choose those talents?" The answers provide a good basis on which to judge a player's knowledge, even if you don't know the class yourself. The worst answers are, "I dunno, just whatever" and "I copied these from someone else." People who understand their class generally have reasons for spells they choose and what talents they pick. Even if they haven't gone to Elitist Jerks and researched every rotation and talent point, they should at least have a rationale for the choices they've made.
Ability Judging ability outside a raid fight is generally impossible. This is because ability is so much more than just pressing buttons in the right order. It's situational awareness, moving correctly, watching threat and boss positioning and much, much more. However, it's reasonable to get an approximation of someone's ability by asking, "What's your expected average DPS (TPS/HPS) on such-and-such boss?"
Attitude When you get a whisper from someone wanting to join a PUG, attitude and general demeanor can tell you a lot. If someone starts complaining before you've indicated that they can join your raid, it's not reasonable to expect that joining the raid will suddenly make him happy and less disruptive. Another risk is that someone may quit the raid early, which can cause a chain reaction of quits. You should ask a potential pugger general questions such as, "How much time do you have to raid?" and "Are you committed to staying in the raid 'til we down the last boss?" Sometimes open-ended questions like "What fight do you like most in ICC?" will provide some insight. I had a guy actually tell me one time, "Saurfang I guess. I've never done any past him. I'm just here for Deathbringer's Will, so I always leave after he's down."
Pre-planning for the win
I currently use PUG Plug to build pre-planned PUG raids. PUG Plug has very cool, built-in vetting tools that build off the concepts of dedication and experience that we discussed above. When someone signs up for a raid, you'll see his WoW Heroes score as well as a clickable link to his WoW-Heroes page. From the WoW-Heroes page, you can see gem and enchants selections. On the PUG Plug site, you will also get an aggregate experience number with a pop-up showing the player's boss kills across all his registered alts.
This site has made vetting pugs hugely easier. It allows you to rate raid members after the raid, so you can mark up the people who did well that you'd like to reinvite again and mark down those who demonstrated poor performance or bad attitudes. This helps a good bit for building future raids.
Have you ever wanted to write for WoW.com? Your chance may be right around the corner. Watch for our next call for submissions for articles via Seed, the Aol guest writer program that brings your words to WoW.com. The next byline you see here may be yours!