crawler
Latest
Google pushes for an official web crawler standard
One of the cornerstones of Google's business (and really, the web at large) is the robots.txt file that sites use to exclude some of their content from the search engine's web crawler, Googlebot. It minimizes pointless indexing and sometimes keeps sensitive info under wraps. Google thinks its crawler tech can improve, though, and so it's shedding some of its secrecy. The company is open-sourcing the parser used to decode robots.txt in a bid to foster a true standard for web crawling. Ideally, this takes much of the mystery out of how to decipher robots.txt files and will create more of a common format.
Google's mobile-first search indexing is live after lengthy testing
After well over a year of testing, Google is ready to shift some of its search engine foundations to focus on mobile. The company has started the rollout for "mobile-first" search indexing, which uses the phone-optimized versions of pages for indexing and ranking whenever possible. Searchers are "primarily mobile," Google noted, and this should give them a better chance of finding what they wanted.
Apple is crawling the web to help your Siri and Spotlight searches
Apple doesn't have to rely solely on outside web providers like Google or Microsoft to fuel your iOS and Mac searches. The company has confirmed the rumored existence of Applebot, a web crawler that collects site information for the sake of Siri and Spotlight queries. It behaves much like Google's crawler, looking for the familiar "robots.txt" file that tells it what results to exclude on a given site; it'll follow typical Google instructions if there isn't any Apple-specific rule set. It's not clear how long Cupertino has been running its bot, or whether there's anything more in the works. However, it's evident that Apple wants its online searches to work no matter what its partnerships look like in the future.
Self-destructing site shows how long it takes Google to find you
Wondering how long it would take for Google's search engine to find your website if you didn't promote it? About 22 days, according to Matthew Rothenberg. He recently launched Unindexed, a purposefully short-lived web community that was set to self-destruct as soon as Google's indexing technology made it searchable. While Rothenberg didn't go out of his way to maintain the secret, participants knew that every view, post and shared link would bring the site closer to disaster -- it only took 346 views and 31 contributions before everything came crashing down. Thankfully, you can recreate this experiment yourself. Rothenberg has posted the source code, so his commentary on the sad state of web anonymity should last for much longer than a few weeks. [Image credit: Shutterstock]
Rubin: Vigil's new IP meant 'even more risk' for potential buyers
Despite many of its major brands and studios being snatched up during this week's auction, several of THQ's intellectual properties remain unclaimed, their respective studios dissolving as a result. Among those unclaimed properties is the Darksiders franchise, which failed to garner any bids during the auction proceedings."Having just finished a product, Vigil was farthest from release of their next game," THQ president Jason Rubin told Game Informer. "We were not able to garner any interest from buyers, despite a herculean effort. Additionally, they were working on a new IP, which meant even more risk for a buyer."Vigil's new IP, codenamed Crawler, was a show-stopper by all internal accounts: "When the teams got together recently to show each other their titles, Crawler dropped the most jaws," Rubin said. "It is a fantastic idea, and truly unique. The fact that nobody bid for the team and title is a travesty. It makes no sense to me."Darksiders will be sold (along with THQ's other remaining properties) in the coming weeks, though what that process entails remains unknown.
Daily iPhone App: 10000000 is a great, complex game with a strange name
Some games pare down a core idea to something really simple and elegant, cutting everything else away until one main bit of gameplay shines through. Canabalt is probably the perfect example of that recently -- it's just one core mechanic, done very, very well. But other games go the other direction: they add on system after system after system, and the art isn't in cutting things away, but it's in joining things together, juggling all kinds of balls and knives and torches, and yet still keeping the gameplay accessible and interesting. The recently released (and strangely named) 10000000 is of the second kind: It's a game with a ton of different things going on, but its charm is that even with so much happening, you can still "get it". Essentially, the game is a match-3 title: You can slide various tiles around, trying to match up three or more of them together. But it's also got a very in-depth RPG layer on top of it -- your character runs across the top of the screen, fighting monsters, unlocking chests, and trying to repair your castle (earning up to 10,000,000 points, which is where the game's name comes from). The gameplay's balanced between what's happening with your character at the top of the screen, and the effects of what you're matching on the tiles below. There's also loot, and skills, and a meta-mechanic that has you repairing doors to open up stages, and even bosses to fight as you race through timed dungeons. It's complicated, and the biggest problem with 10000000 is that it never backs down -- you need to keep a lot of systems moving at the same time, and it's not always clear where your attention should go. But there is a nice tutorial, and the stages do smartly ramp you up in difficulty, so the RPG elements keep you feeling rewarded, even when you lose track of what you're doing. The excellent old-school graphics and music deserve a mention as well -- they look really great and retro, and the aesthetic adds a lot to the old-school arcade feel. 10000000 is a really interesting title; it could probably have been pared down just a little bit, but the game's designers do deserve praise for including what seems like every system they could think of, and juggling them as adeptly as possible. The game is available right now as a universal build for $1.99.
Google bots learning to read webpages like humans, one step closer to knowing everything
Google just launched its Knowledge Graph, a tool intended to deliver more accurate information by analyzing the way users search. Of course, with a desire to provide better search results comes a need for improved site-reading capabilities. JavaScript and AJAX have traditionally put a wrench in Google bots' journey through a webpage, but it looks like the search engine has developed some smarter specimens. While digging through Apache logs, a developer spotted evidence that bots now execute the JavaScript they encounter -- and rather than just mining for URLS, the crawlers seem to be mimicking how users click on objects to activate them. That means bots can dig deeper into the web, accessing databases and other content that wasn't previously indexable. Looks like Google is one step closer to success on its quest to know everything.
Two new Halo 4 enemies give away their position on McFarlane toy box
Two new enemies for Halo 4 have been outed on the back of a Series 1 McFarlane toy package, dubbed the "Crawler" and "Watcher," a photo from All Games Beta shows. The box includes images of each new enemy: The Crawler looks like a hellhound stuck its head in a vat of boiling yellow glow-stick liquid, while the Watcher appears to be a moth-mounted, flying beetle creature.343 Industries better throw a leash on these Halo 4 image leaks before All Games Beta renames itself to "Official Halo 4 Pictures For Everyone Always, Enjoy."
Gold Capped: The Undermine Journal may have to close down
Every week, WoW Insider brings you Gold Capped, in which Basil "Euripides" Berntsen aims to show you how to make money on the auction house, and Insider Trader, which is all about professions. For Gold Capped's inside line on crafting for disenchanting, transmutation, cross-faction arbitrage and more, check in here every Thursday, and email Basil with your comments, questions or hate mail! The Undermine Journal is an invaluable site that a lot of auctioneers have been making great use of. I wrote about it when it was in alpha, as well as how to use its market alert to watch for cheap deals. Unfortunately, the days of having a convenient graph showing you the price history for your realm are numbered. There's a very real possibility that this service will be killed by Blizzard's new auction house interface, and we'd be back to each keeping our own spreadsheets.
Yarrr! There be treasure hidden in them thar data servers!
Have a need to embed Pirates of the Burning Sea information in your website, blog, or toolbar? Then you'll want to read Flying Lab's PotBS developer Brendan Weitzman's latest developer journal. There he talks about the special Crawler server. You can't play on this server -- Crawler's job is to go through all the data on the live servers, figure out what has changed, and copy the changes to itself so that the developers -- and the wider world -- can use this information. What sorts of information Crawler can give you, how much of it you can use and how often -- all these are answered within.Will this info change your life? Well, probably not. But isn't it a nice feeling to know if you did need this information, it would be there for you? See if your server is up without logging in -- or set up email alerts when your port gets thrown into contention. You can make those sorts of apps with this data. And maybe in the future, as they prepare ever more of their data for players to use, you could build something like WoW's Armory. We're just beginning to realize how games will be played in the future. They will be everywhere -- in your browser, on your cell phone, on your iPod.... This is how it starts, with a developer explaining how players can get at their data when they are not in the game.
Mazes of Fate getting ported from GBA
We're pretty inured to this kind of thing on the Wii, although we still enjoy getting outraged about it now and then. But now we're starting to see last-gen ports on the DS! Sure, we have the Phoenix Wright games already, but they totally get a pass since the ports are new to the US and Europe. RPGLand reports that Mazes of Fate, the first-person dungeon crawler developed by Argentina's Sabarasa Entertainment, is being prepared for a DS release by publisher Signature Devices and their in-house developer Graffiti Entertainment. Unlike Phoenix Wright, the GBA version of Mazes of Fate did come out in the US-- in December of last year. The DS is a good system for dungeon games, with its map-displaying second screen, and Graffiti is adding new dungeons, characters, and some touch-screen stuff. If you don't already have the game, this is the one to get. Or if you have a DS Lite and you get really embarrassed about the GBA protuberance.
TGS 06: Lost Regnum impresses
One game that we haven't heard a lot about is Goshow's Lost Regnum, but the game, which is a dungeon-crawling hack 'n slasher, found its way to the Tokyo Game Show.While currently it doesn't seem likely the game will make its way across the pond, IGN says the gameplay is straightforward enough that it should make for a great import.In Lost Regnum, you control one of four characters and can team up with a pal to play two-player co-op throughout the game. Graphically, the game seems to be pretty impressive with nice textures and detailed characters that move very well. And to make it even better, loading was nearly non-existent. However, all good things come at a price, right? It seems Goshow had to cut some corners by limiting the draw distance by the ever-dreaded fog.The biggest hurdle Goshow has, like any developer has in making a game of this ilk, is keeping the combat fresh and unrepetitive. If they can deliver, they may have a great game on their hands and one worthy of an import.(Via IGN)
