The EFF pointed out that the "unprecedented number of photographs" isn't the only problem. GAO's report also said that the FBI didn't test its system thoroughly for accuracy. It "has done little to make sure that its search results... do not include photos of innocent people," the nonprofit org wrote in its post. Since facial recognition technologies aren't perfect and still has issues recognizing people of color, FACE could return results with law-abiding people in the mix when law-enforcement agencies use the system to search for suspects.
It could cause even more issues if the government grants the feds' request for its databases to be exempted from several key provisions of the Privacy Act. For instance, the FBI doesn't want to tell people who ask if they're in the database and wants to be legally allowed to withhold that information. Thankfully, FACE isn't available to the public, so you at least don't have to worry that some random civilian is using it to look you up right now.
GAO's lengthy report has more details about the facial recognition system, lists the FBI's and the DOJ's shortcomings and discusses what it could have done better. As for the bureau's side, a spokesperson told the The Guardian when the publication asked for a comment: "The FBI believes GAO staff does not fully appreciate the nature of its face recognition service as being utilized for investigative leads only and not positive identifications."