Our guild isn't large by any means. In fact, we're a tiny guild, although that's not for want of effort. The Rogue Knights has been around on Elune since beta, and has had a constant stream of very good players throughout our time on the realm. However, a while back a group of players decided to leave the guild, and since then we haven't quite recovered. How can a few simple people make the difference? Well, these players have gone on to form the core players in some of the largest guilds on the server, so you can see how that might affect our guild.
Another reason we are still small is simple: we have standards. Not to say other guilds don't have standards, it's just that we are the sort of people that expect people to spell out their words in guild chat, to be kind and courteous to other players while in Azeroth, and expect them to help others whenever possible. These may sound like basic rules, but I have personally had to boot people from our guild for acts in instances or general chat that simply aren't nice. What this means is that the players that don't do these things, the gankers and the ninjas who happen to get invited to the guild, eventually these people leave the guild, either by themselves or by force. I see many of these players around the realm in my travels, and it is interesting how often they guild hop in their time in WoW.
I think it comes down to expectations of players. Some guilds will accept players based on their class, gear, etc. Our guild, being the ultimate of casual groups, tends to focus on how the player interacts with others, in the guild and outside the guild. What this all means, really, is that in the end, there is room for everyone in WoW, but not everyone in a guild. While some guilds focus on player attendance, others focus on player interaction, and still others focus on nothing at all. What a wonderful virtual world, in which everyone is accepted, and all find a home. No, we can't all be in raiding guilds, but not all of us want to be in one.