For a while now I've been using MediaLink ($20) from Nullsoft to stream video from my Mac to my PS3, but last evening I discovered that I had some MKV (Matroska container) files that I wanted to stream but which the PS3 doesn't natively support. Naturally I assumed I'd have to convert the files and Googling around did turn up a method for doing so using QuickTime Pro. Even beyond the need for QuickTime Pro, however, it is also a somewhat time-consuming process. That's when I ran across a brilliant piece of open-source software called PS3 Media Server.
PS3 Media Server is a cross-platform Java application (and, unfortunately, it looks like it), which not only streams video from your Mac to your PS3, but also utilizes the open source Mplayer/FFmpeg backend packages to transcode / remux media in real time. What this means is nothing short of remarkable: you no longer have to convert supported media (MKV/FLV/OGM/AVI) ahead of time for playback on the PS3. I was able to stream a 720p MKV file without a hiccup (though I do have both my Mac and PS3 connected to the router via ethernet cable). Being impressed at the performance I decided to try it with a 1080p movie trailer I had downloaded from Apple and after a brief blip about 10 seconds in it streamed beautifully as well. (Note that both of these files were already H.264, which the PS3 does natively support, so I believe the PS3 Media server was only remuxing and not actually transcoding.)
PS3 Media Server is a free download over at Google Code (donations requested). There's also a Mac-specific beta (pictured above) that offers more features (including iTunes and iPhoto library support). Overall I'm shocked by how well it has worked with what I've thrown its way (which is admittedly limited). Highly recommended.