bigdata
Latest
Epic Games challenges VR devs to make sense of big data
We've seen a handful of neat ways to visualize big data and make it useful, and the folks behind Unreal Engine, Epic Games, think virtual reality is the next step for that. Currently a half-dozen international teams are taking part in the Big Data VR Challenge, and hope their expertise with games and VR will help 'em "find new ways to manipulate and interrogate" the massive amounts of info generated by science studies. As of now, projects include putting together a digital edition of one of medical history's largest collections of patient consultations (some 80,000 participants) from the 16th and 17th century and a cohort study of kids born between 1991 and 1992.
Los Angeles partners with Waze to head off traffic gridlocks
Los Angeles' traffic problems are legendary, which is probably why the City of Angels is following Boston's lead and partnering with Waze for real-time traffic alerts. The LA mayor's office Periscoped the announcement (as you do, I suppose), and tweeted that the partnership could lead to better understanding of how traffic moves in TInseltown and hopefully improve commutes. How's that? Well, the data inherently goes both ways. Specifically, Google's traffic app should tip off drivers to natural road hazards, car crashes and amber alerts while simultaneously informing the county about where backups and other issues are occurring. No word if it'll extend to pointing out where paparazzi are hiding, though.
White House appoints first ever chief data scientist
The White House has a new employee: Chief Data Scientist and Chief Technology Officer for Data Policy, DJ Patil. His high-level job description, according to the White House blog, is that he'll help "shape the practices and policies to help the United States remain a leader in technology and innovation." On a more granular note, newly appointed US chief technology officer Megan Smith says that he'll be working on the Precision Medicine Initiative that uses data to help physicians provide better care while protecting their privacy.
Poor privacy and data security policies are hurting innovation, says Intel
Look at the technology space objectively, and you might realize it's kind of in a weird place: Smartphones, fitness trackers and social networks have ushered us into an era of data -- the location of our friends and family in real time, the ability to monitor our homes remotely, health tracking and everything that goes with the quantifiable self. But consumer concerns about data security and privacy are at an all-time high. Intel's Malcolm Harkins, chief privacy and security officer, describes the landscape as a sort-of "third industrial revolution:" the evolution of the internet and the potential big data has to completely change the way our world works. Like any new technology, however, people are being cautious. "There's an inherent distrust brewing," he says of data collection. "If we don't rethink our practices, it will hinder the opportunities that are in front of us." We need to rethink data privacy, Intel says.
I'm a neurotic. IBM told me so.
I'm a late Wednesday afternoon tweeter. It's not a characteristic I'd necessarily include on any of my dating app profiles, but it accurately sums up my online behavior nonetheless. I'm also a tremendous neurotic (which should come as no surprise to anyone who knows me well) who embraces self-expression, challenges and change. I'm that personality pie chart you see up above. I'm an open book, or at least my Twitter profile is to IBM.
Researchers use cell-phone data, not precogs, to predict crime in London
Just this year we've seen open data give rise to recreations of Denmark in Minecraft, the ability to compare cities at the same scale and also collections of geo-mapped tweets and traffic lights. But what about a practical application for all of that info, one that has a more tangible benefit to society, like, say, crime prediction? That's what the University of Trento in Italy had in mind with its "Once Upon a Crime" study. The researchers coupled freely available (and anonymous, aggregated) demographic and mobile phone data with real crime data to forecast where in London an infraction might occur. Just how accurate was it? The Italian scientists say that their predictive algorithm was on-point, accurately anticipating whether an area would have either high or low levels of vice, 70 percent of the time. No, it's not quite enough to let Chief Anderton and co. start running wild just yet, but it could be a way to help cities struggling with budget woes decide what areas need more (or fewer) police patrols. [Image credit: Getty Images]
'Watch Dogs' web app turns real data into a virtual surveillance state
It seems like there's even more truth to developer Ubisoft's ode-to-hackers, Watch Dogs, than we first thought. Like we've previously reported, the game's depiction of a smart city that connects drawbridges, traffic lights and its population's smartphones (among other things) all to a single operating system is closer to fact than science-fiction, but the game studio has taken the notion one step further. With the We Are Data web app, you can wade through maps of publicly available geo-location information like tweets, Foursquare check-ins and even traffic light and CCTV camera placement -- all stuff you can find in the game's Chicago. As of now, you can only live out your Aiden Pearce fantasies with info from neighborhoods in London, Berlin and Paris, but there's quite a bit to click on should you be so inclined. The available datasets aren't nearly as extensive as, say, something like Urban Observatory's, but it's pretty neat nonetheless. If searching for public restroom-locations from your desk isn't quite your cup of tea, you could always leave the browser tab open in the background -- its ambient city sounds are oddly calming.
Microsoft puts on an 'infinite' light show to make data pretty
If you're Microsoft, how do you convince people that data is more than just a dry set of charts and graphs? You put on one heck of a light show, that's what. The company recently marked the launch of a database server app by opening the Infinity Room, a temporary art installation that dazzled the public with all the data behind the quarters in their pockets. The project, built by Universal Everything, relied on an array of choreographed LED lights and pixel spheres to produce animations in a mirrored room, giving guests the impression that they were standing in an endless sea of information. As you'll see in the video below, the effect is very Matrix-like -- and certainly more memorable than a run-of-the-mill slideshow. You sadly can't visit the Infinity Room in person anymore, but Microsoft has a 360-degree version at the source link if you want to get a sense of what it was like.
TED founder thinks big data needs a big makeover
Richard Saul Wurman isn't a fan of President Barack Obama's push for data.gov, an online repository for big data. "That's just politicians talking," the 79-year-old TED (technology, entertainment, design) conference founder told me. "I think there will be a pushback saying, 'We don't understand all this fucking data.'" The way Wurman sees it, that bulk collection of raw information has no value without a creative means of diagramming, mapping and comparing it all in a way that gives it meaning. "[You] have to have it in a form that you can understand. They're leaving that step out," he said. It's that approach to the organization of data that has directly informed the creation of Wurman's high-tech information-mapping project, Urban Observatory.
New York's next big neighborhood is its smartest
"It was always going to be tomorrow's city today. A new heart of New York City; Midtown expanding west." -- Thad Sheely, SVP operations for Related Companies Tourists come to stop and stare, and sometimes throw pennies. This isn't a long-standing tradition. There are no wishes to make here. It's just a construction site they're filling with change; "the largest development in New York City since Rockefeller Center." Its 28 acres span west from 10th Avenue to 12th Avenue and the Hudson River, and north from 30th Street to 34th Street. The site is home to the final piece of the High Line park; an extension of the number 7 subway line; five office towers and nearly 5,000 residences; 14 acres of public space; a public school; and an active rail yard, from which it gets its name. This is Hudson Yards: New York City's first truly smart neighborhood. Or, it will be when New York University's Center for Urban Science and Progress (CUSP), a partner for the development, finishes outfitting it with sensors.
White House advisors call for an internet privacy bill of rights
Nintey days ago, President Obama ordered a review about internet privacy, data use and other concerns sparked by the NSA spying revelations. Following a public consultation, the report has now been released by White House counselor John Podesta. Though flawed -- there's no mention of NSA spying activities, for instance -- it's already being lauded by consumer advocacy groups. The 85-page document notes that while the mountains of data collected by government and corporations like Google and Apple are incredibly useful, much more needs to be done to protect civil rights.
The evolution of mortality in America, presented as beautiful graphs
While everyone (and everything) dies in time, just how and when they die has changed a lot in the past few decades; it's not necessarily clear how you're likely to kick the bucket. Thankfully, Bloomberg has used visual data to make sense of death trends in the US. Some trends aren't surprising -- medicine, science and societal factors have helped Americans both live longer and avoid unnatural ends, like murders. As of 2010, roughly a third of all deaths were of people 85 and older. That's a big jump from 1968, when just 13 percent of people would live to become octogenarians.
Tell the White House who you trust (or don't trust) with your data
President Clinton's former Chief of Staff, John Podesta, is back in the White House, working with President Obama to assess the state of big data and privacy. A thorough review is currently underway, and you can contribute by filling out a survey over at the White House website. First, you'll be asked about how much you trust certain institutions with your data, with responses ranging from "not at all" to very much" (ha!), before the survey prompts you to rank your concern when it comes to certain types of content (like video and audio, or location info) and data practices in general. It's not clear what change, if any, may result from the agency's review, but if you feel strongly about sharing your opinions (and we know you do!), head on over to let the prez know what's on your mind.
Google and Microsoft are using the cloud to track climate change
Data.gov is getting a whole lot greener thanks to its new section dedicated to climate information. The new channel is the product of President Obama's Climate Data Initiative (PDF), and pulls information that can help predict the effects of climate change and prevent any damage that may result. The raw data comes from the likes of the Department of Defense, NASA and the US Geological Society, but probably isn't easy to grok for the average person. To help with that, Google and Microsoft have stepped in. Mountain View is donating 50 million hours of its Earth Engine's computing power -- the Global Forest Watch's backbone -- and is partnering with academics in the western US to produce a near real-time drought map and monitoring system. Redmond, on the other hand, has developed a tool (dubbed FetchClimate) that can both recall historical climate data and forecast future weather trends based on the stockpiles of information stored in Microsoft's Azure back-end. For example, the software giant says that this could allow state planners to predict extreme rainfall, preventing flood damage to infrastructure and transit lines as a result. These are still early days for the Initiative, but, as times goes on, more applications using its wealth of info will surely surface. For now, though, it's nice to see tech companies exploit government data instead of the other way around.
Monsanto pushes Big Data-driven planting but farmers are skeptical
Some farmers are worried that with the latest push from seed manufacturers, their planting techniques could be used against them. Monsanto and DuPont (two of the largest seed providers in the world) are urging farmers to implement data-driven "prescriptive planting" tech that suggests how densely rows of seeds should be planted and at what depth. It also gives detailed information about a farm's soil, according to a Wall Street Journal report. Some modern farming equipment already collects the above information for the farmer's personal use, but this new tech would upload it to seed-providers who will analyze the aggregated data and feed optimized planting info directly to the iPads or other tablet inside a tractor's cab. As Monsanto tells it, this could increase corn crop yields by as much as five to ten bushels per acre -- and with mass adoption, that number would rise. Critics, however, aren't nearly as optimistic. The American Farm Bureau Federation (a farming trade-group) has pointed out that seed companies have an implicit interest in higher crop returns and planting denser fields: Monsanto and its ilk stand to profit from the cost of their services as well as increased seed sales. Farmers fret that the shared data could lead to increased competition and higher seed prices, too. What's more, they're worried about a drop in the profits made from futures contracts and a possible fight related to who owns their crop data. Given Monsanto's history regarding ownership, though, the farmers' hesitation could be warranted. For the full story, be sure to hit the source link. [Image credit: Vampire Bear/Flickr]
Google is mapping the history of modern music
It's no surprise that Google has been tracking music uploads, but what's unexpected is exactly what the search giant is doing with all of that info. Interactive maps of music's ongoing journey are charted through Play Music's users' libraries, found over on Google's Research Blog. You could, for example, trace the ebb and flow of a genre era by era (rock remains one of the biggies while electronica's presence is relatively new), or even identify which release from a band is the most prominent. Looking at the Deftones, their biggest album is 2000's White Pony, and they're near the top of the alt-metal heap overall. Music nerds could lose a few days poring over the various ins and outs of the soundtrack to their lives, so be careful who you share this with. Perhaps best of all, Google says this likely won't be the last collaboration we see between the research and music teams.
eBay's new Utah data center goes green so you never have to stop bidding
Big data is big money, so when the power goes out and data centers go offline, companies like eBay stand to hemorrhage revenue. Which is why the mega auction e-tailer's been hard at work setting up a "greener" data center in South Jordan, Utah to avoid costly and unpredictable blackouts. The now operational site incorporates thirty fuel cells developed by Bloom Energy, a company with roots in NASA's Mars program, that turn natural gas into electricity via an electro-chemical process. What's more, eBay, using recovered energy generation technology provided by Ormat, is also attempting to offset its carbon footprint by harvesting the "heat waste" generated from natural gas pipelines and turning that into energy for its Utah site. Barring any unforeseen power failures, the company's green data center won't ever have to rely on the local grid. And that independence should ensure eBay users keep bidding and buying and filling the company's coffers.
Microsoft's Lab of Things now in open beta, lets researchers study all the things
Microsoft has released the first beta version of Lab of Things (LoT), and it's much more than just a fun name. LoT enables researchers to access data from connected devices around the world, collaborating and sharing it as they see fit. If, say, someone in the lab at University of Michigan wanted to determine how much TV people watch in London, all they'd need to do is install sensors as they please. Well, that and install a computer running HomeOS onsite to collect and transmit said data to an Azure Storage account. In theory, this gives academics a robust set of deployable tools without requiring coding knowledge to use them. The Windows company is taking a hands-off approach to the info, too -- researchers retain all rights to whatever they collect. The catch is, the LoT license doesn't allow for commercial use. The source link has a cute video explaining everything, but you'll need Silverlight to play it.
Japan's Terra Motors to introduce electric scooter with iPhone connection
The Apple Maps fiasco excepted, we have it pretty good where location-based services are concerned. But in areas like South East Asia, that's not entirely the case. Which is why Japan-based Terra Motors is prepping to launch the A4000i, an electric scooter that also features an iPhone connector (compatible with the 3GS and up) for big data collection. The company's positioning the A4000i as a means of gathering location data -- useful for mapping -- as well as mileage and battery performance (a lithium cell rated for 50,000km) for that region of the world. There's no word on exactly when Terra plans to launch this scooter overseas, but when it eventually does, expect to see the A4000i retail for around ¥450,000 (about $4,500 USD).
The Human Face of Big Data: an unlikely subject for a great book
Big data is, like many trendy IT buzzwords, an increasingly nebulous term. The Wikipedia definition, for example, is rather jargonistic and impenetrable. If you read big data conference information you'll typically see a lot of naked commercial stuff that might be terribly important to bigwigs but perhaps looks a little... dry... for the layman. Indeed, an awful lot of the hype around big data is very commercial in focus. At its heart, big data is concerned with how modern technology allows us to generate, store and process information on a massive scale. For example, Eric Schmidt, executive chairman of Google, said in 2010 "there were five exabytes of information created between the dawn of civilization through 2003, but that much information is now created every two days, and the pace is increasing." (one exabyte is a staggering 1,073,741,824 gigabytes.) As is so often the case in human endeavour, a lot of this ends up being about selling people things: think of Google's ad sensing network or Amazon's "people who bought this also liked..." engine, for example. You might be forgiven for thinking that's not the most logical subject for a high-production-value coffee table book, but that's exactly what creators Rick Smolan and Jennifer Erwitt have produced in The Human Face of Big Data; the book is also available as a US$2.99 iPad app, and all the profits from the tablet edition will be donated to charity. Through their crowd-sourcing firm Against All Odds and a team of more than 200 researchers, photographers, writers and illustrators, this is a project that aims to illuminate and explain the parts of big data that matter to people who aren't the CTO of a Fortune 500 company. So we get writeups of earthquake detection systems in Japan; of Shwetak Patel's sensor devices that can accurately calculate how much power each individual device in your house costs (and help inform you about which devices to replace with energy efficient ones); and of Nick Felton's obsessive gathering of personal data from how many miles he walks to how many hours he sleeps each year. We learn about Intel-GE Care Innovation's "Magic Carpet" prototype, which is a passive sensor net woven into the flooring of an elderly person's home that can learn the person's habits and routine and alert a relative or caregiver if it suddenly changes -- say, the person can no longer walk as fast, or starts spending long periods in bed. We hear from researchers John Guttag and Collin Stultz, who processed discarded EKG data of heart attack patients and identified subtle new early warning patterns to improve doctor's risk screening. It's full of interesting things, then, and it makes a good case that big data could be the first step towards the Internet developing a "nervous system" of sorts; a detailed sensor network generating reams of data, plus the ability to meaningfully process and act on that data in real time. You may now jump to the comment box and make a Skynet joke. It's worth pointing out that this is a most certainly a coffee table book, rather than an in-depth treatise, and as such it's more about the imagery than it is about the text. Most subjects get only a brief overview of a few hundred words, punctuated by some short essays of 1,500 words or so. This isn't the place to go for a lot of detail on each individual project, although of course most of them are covered in detail elsewhere on the web. The book is going to be delivered free to 10,000 "key influencers" around the globe, as part of Smolan and Erwitt's mission to "start a global conversation about Big Data, and who owns the data all of us generate it." Indeed, one of their concerns is that most of the conversation around big data is being driven by commercial interests, but it inherently affects all of us -- it is, in a very real way, made of us -- and this book attempts to redress that. It's a noble goal, for sure. The Human Face of Big Data is available in book form from Barnes & Noble internationally and from Amazon and IndieBound in the US. It costs around $35 and (in my opinion) would make a nifty gift for any CTOs you just happen to have in your social circle. The photography is attractive and enticing, the infographics are informative and in general it's the sort of book you flick through then end up reading half of as one thing after another catches your eye. The iPad app should be available now for $2.99, with all profits going to charity: water. It has content rather like most iPad magazine apps -- swipe to page through the book, scroll up and down to read each article, tap on various zones in some pictures to drill down into the detail -- that sort of thing. It's a nice app that uses the iPad Retina display to show off the great imagery from the print book, although inevitably some of the impact is lost in the transition to a much smaller canvas (the book measures 14 x 11 inches). Notably, the book also seems to have quite a bit more content -- partly, I think, because some of the more detailed illustrations like the stunning BibleViz (my personal favorite) won't scale down to the iPad's relatively small screen.
