The Brooklyn neighborhood remade as a VR game

Sponsored Links

Go east in Brooklyn, New York, past the blockchain software startups, gentrified co-working spaces and Edison-bulbed cocktail bars, and you reach Brownsville. In a former elderly home near the Langston Hughes Houses public housing complex is the Brownsville Community Justice Center. Within that center, more than 40 young people have been toiling for two years on an ambitious project: creating a faithful, block-by-block replica of their neighborhood and many of its residents in virtual reality for their own open-world video game.

They've set up a green screen with volumetric scanning equipment, experimented with game mechanics inside an Oculus Rift and learned to develop using the platform Unity. It's an uncommon blend of cutting-edge gaming technology, driven and developed by a hyper-local team in an area where 37 percent of adults have a high school diploma and the median household income is around $25,000.

Ray Graham has been working on the game from the start, in early 2016. He's 20 years old, loves Skyrim and hopes to attend college in the fall.

"If you're from Brownsville, you'd recognize it," he said. "And the people of Brownsville that we got in, you'd recognize [them]."

In Fireflies: A Brownsville Story, you'll play as two young characters -- one male, one female -- who have been separated by historical neighborhood conflicts and are both aspiring documentarians. You explore the majority-black neighborhood interviewing residents and taking photos, a mechanism Graham said was influenced by Rockstar Games' Bully.

While there are minigames and dreamscape sequences too, the final game may be similar to a "walking simulator" like Gone Home or Firewatch. But in a neighborhood divided into micro-territories according to which public housing development one comes from, being able to walk anywhere and speak to anyone is a significant experience.

"This video game is about space," said Nicholas Pilarski, a documentary filmmaker who oversaw production of Fireflies and teaches at Carthage College in Wisconsin. "It made total sense for us to work in virtual reality because we could build systems that allow young people to experience and walk to different parts of their community that the territorial divides might have not allowed to happen otherwise."

Pilarski describes himself as a community documentary maker, exploring how to make media not as a sole creator, but in joint authorship with the subjects. With Fireflies, the aim is to mollify historical conflicts and, in development, the makers brought together participants from several of the area's ubiquitous public housing developments. (Brownsville has the highest concentration of these buildings in New York City.) The idea is that if you can chat with digital people from a rival complex in the game you'll be more likely to do so in real life.

Players will encounter over 100 real-life residents in the game from chefs to store owners to Graham's mom's friends. They might wax lyrical on their first kiss, the best pizza in the neighborhood, the police or surviving the AIDS epidemic, creating a kind of oral history.

The NPC's stories were recorded by young Brownsville residents able to draw out more authentic dialogue than what an anthropologist or journalist might grab if they parachuted onto the block for a few days of field research. The project is by Brownsville, for Brownsville.

"I fell in love with the script because it feels real," said Graham, who comes from the Seth Low Houses just down the street from the community center. "It feels like it would be in reality -- people would say these things."

Image credit: Fireflies: A Brownsville Story

Brownsville is just over a square mile in size, and more than 100 people, most aged 15 to 21, have been involved with the game's mammoth development in some form. It started with a small team building computers from scratch two years ago, including Graham and Pilarski. The two then learned how to use the Unity game development platform with the help of arts collective Peoples Culture -- where Pilarski is artistic director -- as well as YouTube tutorials.

With Mantle, a company that generates 3D environments, the team aggregated GIS data from the city to map out infrastructure, like streets, foliage and building heights. They cross-referenced this with zoning and historical data to figure out, for instance, what bricks would be in the buildings or the age of trees.

Then, they systematically photographed the streets to capture details like storefronts -- like their own version of Google Maps, said Pilarski. The team also photographed the insides of specific local businesses and apartments, which players will enter in the game.

New York-based volumetric filmmaking company Scatter trained the team in DepthKit pro bono. Used by the New York Times as well as rapper Eminem, the technology combines a 4K camera and a Kinect depth sensor to create 3D models of the people for the game.

"I would say that it is a low-level indie budget game. But if you add in all of the donations of tech and supplies, we're going to end up inevitably having triple-A quality," said Pilarski.

"We never want it to seem like it came from a community center," said Julie Taylor, deputy director of the Brownsville Community Justice Center. The center has also produced an augmented reality app highlighting local businesses and is planning a projection mapping display. Its goals are to reduce crime and incarceration locally, and Taylor says one aim of Fireflies is to provide a vocational tech skills program for their paid interns and fellows who may not have traditional qualifications.

Graham, meanwhile, talks excitedly when it comes to VR game design. Scrolling through the center's game library, he explains how he was inspired by a technique from shooter Robo Recall that teleports items into your hand instead of the player having to physically pick them up. The Assembly taught him how to use documentary elements in a game. Keep Talking and Nobody Explodes is giving him ideas on how to integrate real-time teamwork dynamics.

The team aims to release an early version of the game in the neighborhood this summer. Later this year, they intend to offer it for free on Steam, iOS and Android, as well as in kiosks around Brownsville. "I'll walk down the street, and I'll just be yelled at by somebody, 'Hey, when is the game coming out?'" said Pilarski. "You go to a barbershop and ask a barber, 'Hey, were you in the video game?' 'Yeah, I was. When is it coming out? How do I play it? I can't wait to see myself. I can't wait to see my cousin.'"

Images: Brownsville Community Justice Center and Peoples Culture

In this article: art, av, entertainment, gaming, longread

All products recommended by Engadget are selected by our editorial team, independent of our parent company. Some of our stories include affiliate links. If you buy something through one of these links, we may earn an affiliate commission.