Armor penetration is probably one of the most misunderstood stats in the game, for a number of reasons. First of all, it's only become popular lately -- while it's been in the game since 1.10, it's only started showing up regularly on items in Wrath. And even then it's really only a meta-meta stat: the core abilities like Strength and Agility are easy to understand, the next level of abilities are things like hit rating and crit rating, and then armor penetration, you could argue, goes another level after that: it's a stat that affects a stat affected by a stat. It's for that reason, then, that Xanthan argues we need a more elegant solution.
Armor penetration basically allows you to hit an opponent as if they're wearing less armor than they really are. That's not to hard to understand -- if you have a certain amount of armor penetration, then the opponent armor number in the equation that determines damage done is lower (edit: by a percentage, not a number) than it would usually be. But the confusion comes in when you see how armor penetration scales. It actually scales exponentially, not linearly -- if you have no ArP and you increase it by a little bit, you only get a little extra damage increase. But if you have a lot, and you add a little more onto that, then you'll get a bigger damage increase, due to the way the math works (I'm bad at math, but Xanthan has an excellent, clear description of the calculations in the forum thread, and we've posted some explanations before as well). Blizzard recently capped ArP at 100% (so you could never get into a place where you're reducing armor below the amount of armor that's there), but it's still possible to have the amount of armor reduced equal the amount of armor on a target, causing the equation of armor vs. armor penetration to divide by 0, and at that point, things get wacky, and terms like "infinite damage" come into play.