The interaction between YouTube clips and iPod music on iOS is due to tight OS coupling. On iOS, a central music controller can handle the shift in priorities between the iPod app's audio and another application's audio playback. On OS X, that's not as much a given -- especially when you're an end-user, not a developer.
Auntie used a notification listening app to see if she could catch YouTube playback events on the Mac, to see if maybe there could be a way to sense that YouTube videos were starting and automate a solution for you. Unfortunately, there were none so that turned out to be a dead end.
Auntie consulted with TUAW technogenius Brett Terpstra who told her that, in theory, a custom Safari listener extension might be able to wake up on YouTube page loads, watch for clicks and monitor for playback requests. The extension could then reach out using proprietary URL schemes to a helper app that could control iTunes via AppleScript. This is, unfortunately, a highly theoretical exercise in what it would take to produce the effect you're looking for. There is no such solution at this time.
So what Auntie's going to recommend is this: Consider downloading any of the millions of floating iTunes controller apps that live in your Finder and offer direct access to iTunes playback without having to open the application itself. That way you can pause and resume playback whenever you're going to watch a video, but do so with the least work overhead.