Kotaku obtained IRS documents showing that the contract's end and an expected drop in E3 revenue, due to the event's smaller size, caused the association to dramatically increase membership fees.
Kotaku speculates that the increase, as well as the decision to hold E3 in Santa Monica, may have been the reasons LucasArts and Activision/Vivendi jumped ship from the ESA. The ESA says membership dues were lower in the past because of income generated by E3, but revenues dropped significantly when the ESA board (made up of executives from publishers) moved the venue last year.
Not only was Santa Monica's E3 bad due to its invite-only policy (meaning publishers got to decide who came and who didn't), it was also horrible for journalists to cover -- not good for a "media and business summit." Thankfully, this year E3 will be back in one spot at the LA Convention Center. We're still waiting to hear what Activision/Vivendi has planned.