webcrawler
Latest
Apple is crawling the web to help your Siri and Spotlight searches
Apple doesn't have to rely solely on outside web providers like Google or Microsoft to fuel your iOS and Mac searches. The company has confirmed the rumored existence of Applebot, a web crawler that collects site information for the sake of Siri and Spotlight queries. It behaves much like Google's crawler, looking for the familiar "robots.txt" file that tells it what results to exclude on a given site; it'll follow typical Google instructions if there isn't any Apple-specific rule set. It's not clear how long Cupertino has been running its bot, or whether there's anything more in the works. However, it's evident that Apple wants its online searches to work no matter what its partnerships look like in the future.
Edward Snowden used automated web search tools to collect NSA data
It's tempting to imagine that Edward Snowden obtained NSA data through a daring Mission Impossible-style raid, but it now appears that he didn't have to put in much effort. Intelligence officials speaking to the New York Times say that Snowden used a standard web crawler, a tool that typically indexes websites for search engines, to automatically collect the info he wanted. He only needed the right logins to bypass what internal defenses were in place. Since the NSA wasn't walling off content to prevent theft by insiders, the crawler could collect seemingly anything -- and Snowden's Hawaii bureau didn't have activity monitors that would have caught his bot in the act. Whether or not you believe the NSA's intelligence gathering policies justified a leak, it's clear that the agency was partly to blame for its own misfortune.
