A guildie of mine discovered this last night, and reader Theronis sent us a note today about it -- after patch 3.0.2, defensive pets act a little differently. Before the patch, they would only attack an enemy automatically if they attacked you first, which made things a little tough with casters -- pets in defensive mode wouldn't actually move to attack until the first spell actually hit you. But after the patch, pets are a little bit quicker: now, they'll move to attack as soon as you attack something, so you won't have to wait around until you get hit before your pet springs into action.
However, this can be a drawback as well -- while Theronis says it's a good change and that his pets are smarter, my guildie had a problem: whenever he wanted to pull one mob out of a group, the pet would run in and pull them all. Anything you attack will get pounced on by your pet in defensive, and that might bring some unintended adds along for the ride.
Personally, I always keep my pets in passive -- it only takes a split second to hit CTRL-1 to send them attacking, and I can much better control, both soloing and in instances, what my pet is up to at any given moment. It can hurt my DPS a bit if I'm not careful (since I'm a BM Hunter, my pet is a big part of my DPS, so if I forget to send him, that's a bigger loss), but the tradeoff is that I can be careful about pulling adds and I have control of my pet all the time. If the "smarter" defensive works great for you, good, but if you find your pet springing in a little too early, you might try switching back to passive and just staying there most of the time.