Armor penetration basically allows you to hit an opponent as if they're wearing less armor than they really are. That's not to hard to understand -- if you have a certain amount of armor penetration, then the opponent armor number in the equation that determines damage done is lower (edit: by a percentage, not a number) than it would usually be. But the confusion comes in when you see how armor penetration scales. It actually scales exponentially, not linearly -- if you have no ArP and you increase it by a little bit, you only get a little extra damage increase. But if you have a lot, and you add a little more onto that, then you'll get a bigger damage increase, due to the way the math works (I'm bad at math, but Xanthan has an excellent, clear description of the calculations in the forum thread, and we've posted some explanations before as well). Blizzard recently capped ArP at 100% (so you could never get into a place where you're reducing armor below the amount of armor that's there), but it's still possible to have the amount of armor reduced equal the amount of armor on a target, causing the equation of armor vs. armor penetration to divide by 0, and at that point, things get wacky, and terms like "infinite damage" come into play.
So there are a few things Blizzard could do here. I was talking about ArP with Matthew Rossi the other day (he was trying to explain it to me for the Phat Loot Phriday post), and he brought up the idea of Blizzard removing it completely, and just replacing it with a damage boost (which is essentially what it does anyway). Xanthan's solution takes it one better: instead of reducing an opponent's armor (thus creating the issue of what happens when you reduce it up to the extreme limit), he suggests that armor penetration increases an attacker's ability to do damage through armor -- in essence, he says, making it akin to how Haste just "increase your attacks per second rather than reducing your seconds per attack." Very well put.
Odds are that Armor Penetration is in for a change eventually -- at the lowest levels, where it's been in the game for a while, the math is harmless, if a little loopy. But as the stat scales, Blizzard will be dealing with some strange issues, so it might be better to change (or even remove) it now.