An inside look at a Verizon FiOS Super Headend and Video Hub
By Ben Drawbaugh 
posted December 17th 2009 5:00PM
posted December 17th 2009 5:00PM
What goes on at the Super Headend
Hidden away in a nondescript office park in Temple Terrace FL, is one of two Verizon Super Headends (the other is in Bloomington IL). From the outside one would never suspect that millions of customers would be watching TV from the equipment that is inside.
But this is where all of the programming is aggregated before it is sent via fiber throughout the country. The content creators obviously don't create special feeds just for a single MVPD, so just like everyone else, Verizon grabs the encrypted signal with a farm of satellite dishes. But what is different is that Verizon has only two farms for FiOS throughout the entire country. This is to provide redundancy, and that theme continues on to just about every piece of equipment as well (so yeah, there are like four of everything). It takes just 13 satellite dishes to pick up the almost 600 channels distributed from here, but there are also 7 backup dishes as well as three more ready to go in case the first backup fails. These can each be aimed at another satellite, but for the most part the dishes are never moved.
The Super Headend control center
The Video Hubs do most of the work
In the case of the Tampa Bay area, Verizon built the regional Video Hub in the same facility as the SHE, but this is the exception. Each VHO is connected to the same nationwide fiber network, but only grabs the channels that are available in that area. So in Tampa that is about 450 of the 600 channels. The VHO also aggregates local programming from your NBC affiliates (for example) or a regional sports network that isn't distributed via satellite. Since the VHOs don't have an satellite dish farm, they get these local channels via a fiber connection connected to the broadcasters, but also have antennas on the roof for redundancy -- just like the SHE, everything is redundant here too.
The Video Hub control center
But the VHO has more to do than just inserting local programming and sending it out to your house. For starters, local advertisers can have their commercial broadcast on ESPN HD, and the VHO is where that happens. The advertisers send them a DVD, VHS or whatever they've got, and the team at the VHO gets it on to their DVR and schedules it to run at the right time and on the right channel. This is also where the loud commercials can come in, and we were told of how difficult it is to get this right, and we feel for these guys and hope that an automated solution can be found, but it was nice for us to hear that it isn't a simple case of an over zealous advertisers trying to grab our attention. The other interesting thing is that regional blackouts are enforced at the VHO. So whenever ESPN HD blacks out a game and shows ESPNNews instead, the equipment that makes that happen is here. What struck us as odd though is that it isn't controlled by Verizon. Now obviously the call of what to blackout is all based on contracts and a complete mess, but while the equipment that controls the blackout is at the VHO, it is actually remote controlled by the network. So in other words, if the game you want to watch isn't available in your area, calling Verizon isn't going to do you any good.
Channel status monitors
The flow diagram of a single channel
The other thing that happens at the VHO is video on demand. The VOD servers are also in the VHO datacenter, so they are also monitored and it is even possible for Verizon to deliver different VOD offerings to different regions. One thing we are still waiting to hear on is where the CCI flags get set. The VHO manager said he'd have to look into it, but it makes sense that he wouldn't know since no FiOS regions seem to be setting the CCI flags.
So you see, inside and out FiOS really is pretty awesome
We don't say this very often, but Verizon FiOS really is different, in that their customers love their service. We often write posts about other providers that lead to lots of hateful comments, but whenever we write about FiOS, the most negative thing is that it isn't available everywhere. We also think that by even offering tours like this it goes to show how hard Verizon is trying at this video service thing. Don't get us wrong, FiOS doesn't do everything perfectly, but we'd be willing to bet their new customer retention stats are in a league of their own. So thanks to Verizon and all those who made this tour happen.
Very nice tour.
Now if only I could get Verizon FIOS here..
Yep! I live in comcastville, USA and I really wish that a FIOS truck would come drop a line to my house. =/
@BuffaloX Try looking for AT&T Uverse. Still over the fiber lines, in fact Verizon uses a lot of AT&T's lines to ship their service.
@Triscuit How about the little ol' town of DALLAS, TEXAS. I cannot get either FiOS or U-Verse. The best I can do at my house is 6Mb DSL or Time Warner's 8Mb cable that only works about 65% of the time.
There is some stupid agreement (contract?) that AT&T is the only high-speed provider in Dallas, preventing Verizon of even bringing FiOS here, even though surrounding communities have it already.
Actually Comcast does have a national media center in Denver Colorado that handles both Comcast cable and HITS. There also regional head-ends in large metro areas that handle regional networks. Then the local headends insert the PEG channels.
@(Unverified) Well there...while Comcast does have the CMC (Comcast Media Center) in Colorado, I'm pretty sure that it still does not deliver its video content to all (or any) of its headends throughout the country via fiber. CMC is simply a content transmission earth station which makes content available via satellite. Doesn't bode well for redundancy or extreme weather issues now does it! You like apples?
@danobegood From the CMC's own web site.
CMC offers a full range of terrestrial and satellite-based transmission capabilities providing global connectivity via several carrier class IP video transport providers and uplink video to satellites serving North and Latin America as well as European and Asian continents. Operating more than 30 antennas, which provide uplink services to any available domestic spacecraft, we transmit more than 500 video and audio services full time to over 20 satellites.
Terrestrial AND Satellite based transmission capabilities. I understand not every local or regional headend is connected to the CMC but if they can over terrestrial connectivity to third parties, I think it would be obvious that they would use it for their own communications as well.
@(Unverified)
Well neither terrestrial nor satellite is as reliable and robust as redundant sonet delivery. I think it is pretty clear that delivery over a fiber backbone is superior to terrestrial or satellite delivery. But this is just my two cents and a Buffalo nickel.
@danobegood What do you think terrestrial is? I don't know if Comcast uses SONET or not (I would consider it a good bet they do on the long huals) but that terrestrial network is fiber.
@(Unverified) Terrestrial in this industry means through or over the air. Possibly you are referring to the air between the fibers, or maybe the air between someplace else?
http://en.wikipedia.org/wiki/Terrestrial
more specifically: http://en.wikipedia.org/wiki/Terrestrial_television
But the real advantage that FIOS has is that every system is an 860Mhz system (on the TV side) so they can offer channels nationally almost as quickly as DirecTV/Dish as they have a national RF plan. Comcast still has a big mix of 550, 650, 750, 860, and a few 1000Mhz systems and dual cable systems so national RF plan isn't practical.
@(Unverified)
Essentially, FiOS has two cable pipes to the home...that is, it has one full wavelength (1550nm) dedicated to delivering the linear broadcast content which ends up in the 860MHz spectrum. The other wavelength (1490nm) is for delivering interactive services like VOD, Internet and Phone. On the Comcast side they have to stuff all the services into just the 1550nm delivery pipe. Think of it like a meter stick...broadcast TV (ABC, MTV, USA, FOX, HBO), they get delivered at centimeters 1 through 75, VOD is delivered on centimeters 76 through 85, Internet is delivered on centimeters 86 through 95 and Phone is delivered on centimeters 96 through 100 (although with docsis technology they would use the 5 centimeters for phone as internet and then your phone uses the internet transmission space instead of a dedicated space).
FiOS has a clear delivery advantage as it can use the full meter stick (1550nm, centimeters 1 through 100) to deliver just broadcast content and then the other pipe (1490nm) handles the VOD, Internet and Phone service delivery. This type of service delivery allows for transmission of more broadcast content, which in-turn means that you don't have to compress as much.
I'd like to see what Comcast's looks like, maybe they can show me where the "compressor" dial is so I can turn it all the way back. Some channels, SyFy SD, are so compressed that I see big squares on the screen. That's just crazy.
@Roberini, As a Cablevision to FiOS convert. On the traditional cable networks, I have to wonder who decides what channels to compress the hell out of, Cablevision basically chose them at random! Gotta love pixelation. :(
@Roberini I agree. Food TV used to be the best benchmark of what untouched HD looks like; in fact, it was the de-facto benchmark on AVS forum when the scandal broke.
Birtrates went from 17 megabits FiOS to under 7 on Comcast. I absolutely can't watch anymore. Live action shots on SciFi (I refuse to call it Sy**) are so compressed that the whole scene tiles over.
Fuck Comcast.
A significant shortcoming of FiOS video service is the quality of the source feeds that they receive from the content originators (e.g. HBO HD, SHO HD). These feeds exhibit significant tiling (macro-blocking) during high-motion scenes.
At least one pay movie channel, HDnet Movies, seems to encode and feed at a high enough bit rate. I have yet to see significant macro-blocking on HDnet Movies, even on very high motion / complexity scenes.
It would be nice if Verizon used its fiber network to acquire LOW COMPRESSION (high bit rate) feeds directly from the content creators.
(ie. abandon the low fidelity feeds sent over the C-band sats and acquire independent high video fidelity feeds from the movie networks, which are then back-hauled over VZ's terrestrial fiber infrastructure to the various head-ends)
FiOS could send us 30Mbps channels yielding near Blu-ray caliber video fidelity!! (Or at least they could send us as high a bit-rate as the video decode chip in the dumb Moto STB can handle :-) )
@enger
Seems like way too much work to watch movies that are cropped and with normalized audio.
I mean if you really want movies with great picture quality, you're only option is Blu-ray, or maybe Vudu, both of which look 100x better than HBO HD etc.
@BenD
From the Vudu requirements list on their web site:
HDX (1080p) requires 4.5 Mbps
Even sophisticated codecs, like AVC, would be hard pressed to deliver a pleasing (good image quality) rendition of high-motion programming when constrained to a bit rate that low. There is no free lunch. Delivering high a high bit rate is essential.
@enger
You can watch the codec meter instead of the movie if you want. What I'm telling you is that the picture quality of VUDU HDX BLOWS AWAY the picture and sound quality of anything I've ever seen on HBO HD or any other cable movie channel. And even if the PQ and AQ wasn't better, it'd still be more enjoyable because the movies aren't cropped and the audio isn't normalized. But then again, VUDU costs too much per movie so it isn't the best comparison anyways.
@enger HDNet and HDNet Movies mandate a minimum bitrate for their channels. I think it's something like 8Mbps.
@UnnDunn HDNet is much higher than that...basically HDnet maintains that the provider carry the channel without recompressing the signal.
What you are seeing on HBO, Cinemax, etc. could be one of two
things.
Overcompression on the content providers part either to save money on transponder space for distribution or based on "input" from the Cable and Satellite companies.
Overuse of overcompression at the Cable / Satellite company. I know a lot of cable companies are starting to go to 3:1 on the HD QAMs. Not necessarily bad but you can overuse it or use it on the wrong content. What I have seen is that if it non-sports 720p content, you can easily do 3:1. Non-sports 1080i content is a little tougher if you are intelligent on what shows goes onto which QAM. For example, CNN, CNBC, and other talking head channels can be added in with more challenging content like the Discovery networks, movie channels, etc to get a good 3:1 MUX. It's all just good or bad systems engineering.
@(Unverified)
He's speaking about FiOS which has been independently confirmed that the original feed is passed to the customer without further compression. There is some evidence that content creators have actually reduced the bit rate of their source feed so that Comcast and others wouldn't have to recompress it that much, that is what he is complaining about.
@BenD For many channels I'm sure that's true. But HBO for example switched to h.264 on their satellite feeds a while back, so VZ has to transcode those to MPEG-2 before shipping them out over the virtual QAM to its STBs. Which means first that HBO has compressed the signal more than you might like before it ever reaches VZ's hands, and second that you get another generational loss when VZ transcodes it.
@Enger 30Mbps feeds? Sure they're not sharing the QAM bandwidth with VOIP so their 860MHz plant has more room than you might think, but at 30Mbps they would only be able to fit one HD channel into each 6MHz band, which would mean they could only have 143 total channels on their network. Even Verizon can't do that.
@Fanfoot A solution cited by the (terrestrial) industry is to migrate to head-end switched video (all, or almost all channels will be sent via unicast to individual TVs). This addresses the issue of having more content bandwidth than distribution capacity. It allocates distribution capacity to the TV set (so to speak) rather than to the content.
It is, of course, unclear how many last-mile providers will adopt such technology. (Nodes must be relatively small, in order to allocate a QAM to each TV; this is certainly no issue for FiOS, but may be difficult for cable-TV operators with older plant.) Variations utilizing high bit rate base band video unicast distribution are also feasible.
The key phrase is high bit rate video. The DBS satellite guys are probably unable to follow-suit; if the terrestrial guys implement this high-bandwidth approach, the DBS guys will be hurting. To remain viable, they really need to convince consumers that HD video quality is unimportant (or at least a secondary issue).
@enger
Unicast technology like SDV only helps on channels that aren't watched much.
DBS companies are launching new birds like crazy and don't look like they are going to have any bandwidth problems anytime soon. The big concern for DBS is when VOD becomes preferred to linear TV. This is something they'll have a hard time competing with.
@BenD I concur with your comment on VoD versus linear. Pre-placing content on STB can be done, but opens pandora's box on piracy (and disk size).
On the main topic, I find it hard to believe that the DBS guys can distribute genuine bandwidth for real BluRay video for many hundreds (or more) channels. I'll be delighted to be proved wrong. Maybe they'll come up with multiple Gigabit transponders in the far infrared? :-)
@enger
Vudu has proven you can send Blu-ray comparable quality audio and video at ~4.5Mbps. DirecTV 12 is said to increase the overall throughput of DirecTV by 50% so the new birds are far more efficient then the older ones. The problem is that Vudu can do this with so little throughput because they aren't doing it in real time. But even if DirecTV put 1TB HDDs in their DVR filled with content at 4.5Mbps, they'd still have a hard time competing in the VOD space with two way services.
@enger
Frank Lee you don't have a clue. The satellite feeds (probably 95%) which provide content to FiOS are the same satellite feeds which provide content to most all other US based MSO's. Compression is a result of the way the MSO handles the content delivery, not how the originator handles it for a given MSO...that would be discrimination of service provider and would result in BIG LAWSUITS.
@danobegood
The assertion is that the content originators (e.g. HBO HD) are tailoring their feed to the constraints of the majority of the distribution operators (TW, Comcast, Cox and so forth). Most distributors are capacity limited and utilize statistical multiplexers to re-compress the incoming feeds, so that they can pack more of them onto a single QAM. The content supplier likely thinks "why provide a feed far in excess of what the distributor will pass through; my guys will do a better job of the compression than the distributor's hack statmux".
Meanwhile, the rare high capacity distributors (e.g. FTTH distributors, like FiOS) who have committed to NOT re-compressing the sources they are given, are stuck: few suppliers give them genuinely high fidelity source feeds. HDnet Movies seems to be a stand-out in the pay-movie lineup. I've never seen their channel macro-block. HDnet Movies acts as an existence proof of what FiOS is capable of. Meanwhile HBO-HD, SHO-HD and the rest fail to bolster FiOS' reputation of video fidelity: they macro-block readily on complex motion scenes.
The idea floated was to encourage Verizon to use its own terrestrial fiber network to obtain higher fidelity (minimally compressed) feeds directly from the content originators (abandon the lowest-common-denominator C-band feed). Any distributor could request a similar feed from a supplier; there wouldn't be any favoritism. However, I doubt most distribution operators have the requisite nationwide fiber backbone to economically accomplish the acquisition.
Thus, Verizon might obtain a sustainable competitive advantage by acquiring high fidelity feeds and making quality a high visibility selling point of their service.
I suppose Verizon could even offer a service to its competition. It could become a terrestrial primary feed distributor (replacement for the C/Ku-band satellites). Verizon has already deployed a 100Gbps per lambda fiber optic system. A single lambda on a single strand could backhaul three thousand Bluray-like (33Mbps) HD feeds. It would take a lot of C/Ku-band transponders to match that. :-)
They're showing QFHD/cinema4k at CES. FTTH and terrestrial fiber backhaul infrastructure can support delivery of that quality to the consumer. I hope Mr. Seidenberg shares a similar optimistic vision for the future.
Great tour Ben writeup, Ben. That was some really interesting coverage. I would never have known what the central distribution center of a major content delivery network like FiOS looked like. I would have just gone on unawares, enjoying it on my TV :)
One thing that came to mind, is the choice of location in FL. I'm familiar with FL locales, but aren't hurricanes a big threat to facility like this?
I realize it's redundant, but still a failover to another facility of this size and complexity can't be totally transparent, not to mention the incredible cost and time to rebuild the whole place. Additionally, would the frequent heavy storms cause bad reception on their dishes pulling down all the content?
I'm sure my thoughts are just uninformed, but I'm curious to hear about that.
@gadgetfanboy
oops, above should read "i'm NOT familiar with Florida locales"
@gadgetfanboy
People ask this all the time, and the answer makes perfect sense. As far as cities in Florida, Tampa has the least number of Hurricanes as any coastal city in the south east US. In fact only one major hurricane has hit the area in the past 100 years. But the real reason things like this are here is because power is relatively cheap compared to other parts of the country. And power is the primary expense in operating a facility like this.
That is pretty awesome. I've been debating a move to Bloomington, IL. Now I have a reason........
@TGrant
As weird as this sounds, Verizon does not have ANY fios in Illinois.
I have no idea why they have a hub here, but Verizon does not offer fios internet or TV in IL.
They plan to have it eventually, but I would not hold your breath.
@feffrey and TGrant ,
Correct, there is no Verizon FiOS anywhere in Illinois. Nor, will there be! Verizon is selling all their Illinois, Indiana, Michigan, Ohio, and Wisconsin territories to Frontier Communications, with goal of closing the sale in the Spring of 2010.
Bloomington Illinois was once a major regional headquarters for General Telephone. Verizon bought out General Telephone around 1998/2000. Major reason was to get a backup site, in a different part of the US.
If you can, switch to FIOS, it's amazing :)
Hey! This is right by my school!
I started reading this like 'Man I wonder where this Super-HQ is at..' and you guys said Temple Terrace, that's right by USF! Then I thought, hey.. isn't there a street down the road called Telecom Pkwy that goes back to a bunch of commercial offices?
Sure enough, I found that building with the satelite dishes on top of it on Google Maps in about 30 seconds..
http://www.google.com/maps?ie=UTF8&ll=28.059247,-82.371851&spn=0.001759,0.002411&t=h&z=19
I too have to say I love everything about FiOS. The service as a whole is great, and some of the other commenter's have opened my eyes to a reasonable explanation as to why i see that macro-blocking. When I had TWC I never encountered it, but now with FiOS, all the HBO channels are just plagued with it. I no longer even order the movie packages and simply stream everything uncompressed from my pc to my ps3 via my own local network. It really is a shame if the macro-blocking is due to the movie networks rather then FiOS itself. Get your act together movies! How dare you!
Dagnabbit BEN! Why didn't you call me before you went over! Sure you don't know me from a hole in the wall, but what's that got to do with it? Now I gotta sit here (*ahem*, in Riverview, neighbor!) and look at pictures of some building I could have sat outside of instead (while you got the tour). I'd have even sprung for beers. Maybe.
seriously fios is the best and the customer service is just as good. at least from my experience
i work in the Freehold NJ VHO and it is an amazing sight
The person who mans the desk closest to us in the first photo is a germaphobe. I imagine he puts new post-it notes on his laptop palm rest every morning. It's most likely where he stored his notes, but OCD will make you do strange things.
...tapping 6 times before I hit submit.
Good tour and cool.
the four monitors at the top look awesome! what kind of monitors have such a thin bezel? anything like those monitors available for PC?
hand moisturizer, paper cloths, hmmm
Very impressive yes, but FIOS is not television. It is streaming video more like what you do on your computer. Television is transmitted on a certain frequency either over the air or through a wire. IFIOS will improve over time and generations will not remember tuning a television in a few years. Folks, its all about control of the information you are exposed to. They want you to buy what is advertised and they reluctantly serve PEG community access agreements. There isn't very much attractive to thinkers or creative individuals on cable. Turn it off when you can.
FIOS video is not like streaming to a computer, it's pure and simple QAM modulated RF just the same as a cable company plant. I worked for a year on the edge modulation devices that took Gigabit IP in, and produced RF QAM on EIA channels out at VZ offices around the country.
@UDPvideo
Correct. This post originator clearly cannot see the parallel between 2-13, 14-69 VHF/UHF and 1-135 CATV QAM. Should read up on new TV transmission technology post 1992.
You guys just passed up the perfect opportunity to send out a blanket message to all Fi0S TV's:
"All your TV are belong to us!"
Once in a lifetime opportunity wasted yet again...
When did this tour happen? I work at that building. I wish I knew this was going to happen so i could photo bomb some of those pics! ;o)