Last week, I posted about creating your own iPhone ringtones, then iTunes 7.4.1 toddled along, so here are updated instructions.
iTunes uses the m4r file extension for ringtones. If you remove an AAC file from your library and rename it from .m4a to .m4r and then add it back to iTunes, the program reads it back in as a ringtone rather than a normal library track. You must then go back and rename that file back to m4a so you sync it to your iPhone. (That's what that unrecognized format error is all about.)
The secret lies in making sure the name change happens properly. This isn't a big problem on Windows. On the Mac, though, the Finder tries to keep you from renaming the file extension -- this isn't to keep you from making ringtones, it's to keep you from hurting your files. That file you just named "MyTone.m4r" may actually be named "MyTone.m4r.m4a" (unless you have the Finder advanced preference "Show all file extensions" turned on, usually a good idea if you plan to be tweaking your file types).
If you don't have the show all extensions preference on, you can do the rename in Terminal, or select the file you want to rename and Get Info (Cmd-I). Scroll down to the Name & Extension field and change the extension to m4r there. If the Finder is set to show all file extensions, you can pretty much just go ahead and change the extension; you may get a warning dialog but you may plow ahead safely.
At this time, the m4r items do not appear in your Library. You can only see them when you select an iPhone in the sources list and then click on the Ringtones tab. Also, there is no way to remove ringtone items from your library yet.
Thank you to Alex
Update: I've only tested this update on the Mac.