I've used Handbrake, Forty-TwoDVD-VXPlus, QuickTime, iPSP (PSP-formatted MPEG4 files play nicely on the iPod), DigiGami MovieToGo, Apple's Compressor, DiVa, DivX Doctor II, and ffmpegX to try to find that perfect iPod video format. More and more, however, I'm starting to think that the solution lies in higher resolution MPEG4 files, rather than in h.264, simply because the h.264 support on the iPod is limited to 320x240 and the MPEG4 support says it's up to 480, although people have reported success with files in the 740 range. This MacOSXHint would seem to agree, and in the comments to that hint, you'll find instructions for building your own custom version of Handbrake specifically for converting straight to iPod capable video. Even better, somebody over at Accelerate Your Macintosh! did all the legwork for you and is offering that specially compiled version of Handbrake for you via his .Mac space [link].
I just watched a copy of Final Fantasy VII: Advent Children with English Subtitles that I grabbed off of bittorrent. I plugged my iPod into the TV and watched the entire thing without noticing any artifacts. It was encoded in MPEG4, 128kbps bit rate, 44.1khz sample rate, and weighed in at 480.5MBs for an hour and forty minutes worth of video.
What's your magic encoding incantation of choice?