One way of solving this dilemma is by using iTunes's powerful smart playlists to filter your library into a size that's a little more iPhone-friendly. The following smart playlists are examples from my own setup, but the smart playlist system is flexible enough that you can do almost anything with it to filter your songs. Naturally, these tips will work just as well for any of Apple's smaller capacity players, like iPod nanos or lower-capacity iPod touches and iPads; I've focused on the iPhone only because that's what I own.
I've rated almost every song in my library, and those with 4 or 5 stars end up in a playlist called "Top Rated," or in the language of smart playlists, "Rating is greater than three stars." This represents music that I want to listen to more often, and it ensures that songs I don't like don't end up playing when I shuffle songs in iTunes. It also guarantees that only music I actually want to listen to makes it onto my iPhone.
The only problem is that my "Top Rated" playlist is still too big -- almost 23 GB. It takes a little more paring down to get music to fit on my iPhone. Therefore, I created another playlist called "Favorites" which, as you might guess from the name, represents the best of the best in my music library. I set the following conditions for the smart playlist: "Rating is five stars," "Play Count is greater than 4," and I filtered out a bunch of genres that I don't necessarily want to listen to when I'm on the go, by setting "Genre is not" to those genres.
The "Favorites" smart playlist forms the foundation of my iPhone's music library, with a few standard playlists thrown in to round it out to about 1400 songs. However, I ran into another problem with the smart playlist that I use most on my Mac, one called "Not Recently Played." In order to keep from shuffling through the same songs all the time, I built a "Not Recently Played" playlist based on my "Top Rated" playlist. "Not Recently Played" has a simple setup: "Playlist is Top Rated" and "Last Played is not in the last 1 month." The issue here is that since this playlist is built off of my "Top Rated" playlist, it's far too large to fit on my iPhone. So, in order to get that same "Not Recently Played" functionality in my iPhone's music library, I had to create yet another smart playlist.
Here's how I filtered my "Not Recently Played" playlist into a more manageable size: I created another smart playlist called "iPhone NRP" with the condition "Playlist is Not Recently Played." Nested within that playlist condition by using the (...) button on the far right, I went through and set "Playlist is" to all of the various playlists I synced to my iPhone.
This accomplishes two things: it creates a "Not Recently Played" list for my iPhone with music that I haven't listened to in a month or more, and it pares it down to only songs that are already synced to my iPhone. This playlist doesn't add any music to my iPhone that wasn't already there, but it adds a new condition (not played in the last month) that draws more functionality out of my iPhone's music library.
These are just a few examples of how you can use smart playlists to your advantage when you have a large music library that you need to fit into a smaller device. With all of the metadata iTunes puts onto songs in your library, there are nearly limitless possibilities for setting up smart playlists.