I'm writing to you as an officer in a guild and a raid leader for one of two groups in said guild.
Originally, our group struggled for numbers, mostly filled by alts from the other group. Then, we entered a golden age of progress, gaining several members who regularly attended. Then, we lost several to another guild. Then several more ceased regularly attending. Alts became less reliable. Several new recruits joined, and never show up for raids. Yet every new recruit, we check availability, everyone's availability for 2/3 raid nights ... I'm flabbergasted, and frustrated, with a hodgepodge group every night of non-ideal compositions born of necessity, less than ideal members not properly geared and less than committed to raiding, because we have no choice. We're not picking and choosing between members, we're taking whoever is available, which is usually just barely 10.
No penalty box
I'm aggressively recruiting, and doing my best to educate people on fights, but the fact of the matter is several of our members are underperforming drastically below their expected DPS for class, spec, and gear. Those same people are often failing mechanics, and failing to handle interrupts, dispels, etc.
I'm at my wit's end. We, my co-leader and myself, are at a point where something has to give. People need to start frankly not sucking, and taking some personal responsibility for their gear, enchants, and performance. Or, we honestly will have to either stop running this group, and PUG our raids, or leave. We want, above all else, to see progress. And our enjoyment of the game is declining rapidly. Is there anything we can realistically do?
The guild has no penal system. We have no DKP system. And there's no incentive to outperform anyone else when we barely have enough people to go any given night. I feel like we're in a situation where we're set up to fail.
I actually want to address your final paragraph first: Don't bother with a penalty system. Those never really work. Penalty systems only make the punished players feel resentful. The only meaningful penalty is "you don't get to go." Give players a chance to improve, and let them know if they don't, you'll be forced to replace them.
I hope your language is born out of frustration, in terms of people "not sucking" and taking personal responsibility. I'm not judging here; it's simply that if people aren't self-motivated to improve gear with things like gems and enchants, then there's only so much hope of those people really stepping up to the plate.
While I wouldn't angst about whether a player has the very best, bleeding-edge enchants, taking the basic steps is up to each individual. If you have to babysit every player at fundamental levels, you're kind of hosed from the start. You're forced back into the position where you have to recruit.
Set your own pace
That's easy for me to say, of course. You actually started off your letter with the dynamic that's causing all of your problems. It sounds as if you're a secondary raid group drawing members off the primary raid group. Since you're left struggling to find members, you start in a failing position. It even sounds like you don't feel the full support of your guild.
I hate to say it, but I see this kind of tough position all the time. When a guild has nearly enough members for two raids, and when you populate two raids with separate individuals, then those two raids actually become something more like two mini-guilds.
Your best bet is to accept that and run your separate raid much like that individual, separate guild. Draw your basic requirements and set expectations. And then keep recruiting until you've cycled enough reliable players to always be able to run.
For a 10-man raid, I like to have around 13 individuals on the overall roster. Real life happens. If any of your players are married couples, then real life happening to one will happen to two at the same time. While that means if all 13 individuals are online at the same time, three will have to sit out ... that's the price of making sure the raid always runs.
If you find yourself getting flack from your fellow guild leaders, your position is grounded in the fact that it's your job to run a successful raid. You can't do that if you're constantly taking hand-me-down alts and semi-motivated players.
Running two raids means you need all the tools necessary to execute a full raid. You can't simply run on "whoever is available." That's not a tenable position for 90% of the populace. Keep recruiting and pushing.
Coach your members where you can. Recruit to fill the holes where coaching is not possible. You have to stand up and run your raid. I wish I had a magic bullet for you, but there's not one. This kind of thing is the real challenge of raid leadership. Anyone who can master Super Mario Bros. can not-stand-in-fire. It takes a leader to deal with these rough situations.
Ready Check shares all the strategies and inside information you need to take your raiding to the next level. Be sure to look up our strategy guides to Cataclysm's 5-man instances, and for more healer-centric advice, visit Raid Rx.