Grid-Status
Latest
Strange goings on in the Second Life grid
Throughout Thursday, Second Life users report that the whole grid was acting a little oddly - and the statistics we have bear that out. Starting just after midnight, however, something else odd started to happen. From about 12:30 PM SLT (US Pacific) group and profile information began to load inconsistently along with other "random but inconsistent phenomena". It's nothing people can put their finger on, specifically, but given the day's concurrency curves, it's obviously having a significant impact on Second Life users.
Second Life login failure
At 3:50AM SLT (US Pacific) with 31,509 users online something serious went wrong with Second Life's backbone or web-services. Statistical data feeds ceased functioning and the login servers stopped responding to requests entirely. At present the cause is unknown and there is no word from Linden Lab about the problems. [Update: 4:05AM SLT - Statistical feeds have started reporting again. Apparently approximately 20,000 users were disconnected.] [Update: 4:15AM SLT - Linden Lab has acknowledged the problem] [Update: 4:40AM SLT - Logins are now functioning again] [Update: 4:50AM SLT - That was very short-lived. Apparently they've gone off again] [Update: 5:25AM SLT - Service has been restored. Linden Lab has identified the problem as a network failure]
Postcards from the future
A second round of network issues today in Second Life is announced with a blog post that is (once again) timestamped in the future. Forget the network problems between the Dallas and San Francisco colocation facilities. Forget that it's taken some simulators offline. Forget that about 10,000 people have been knocked offline. It's timestamped in the future! Is it possible for them to post Lotto numbers or something? [UPDATE: 6:50PM SLT - All in all, approximately 28,000 users were knocked off (or disconnected from) the Second Life grid during the network difficulties. Linden Lab now reports the issue as being resolved, and the numbers we are seeing from the live feeds reflect that.]
Widespread problems reported on the Second Life grid
We're getting widespread reports of systems failure in Second Life as of a short minutes ago [approximately 12:20AM SLT (US Pacific)]. Since then, distressed messages are becoming increasingly widespread. Teleports not working, inventory and asset problems, high-levels of packet-loss, presence issues (affecting group IMs and the friends-list); all among the usual suspects for a central services or network failure. Reports began to trickle in sporadically, but there has been a flood of them since 12:50AM SLT - at which time Linden Lab statistical data feeds suddenly went dark. Related to the rolling restart late yesterday? Quite possibly. The last Friday afternoon update Second Life had led to a weekend of disasters. There has been no word from Linden Lab at this time. [Update: 1:05AM - Linden Lab's statistical feeds came back to life after 15 minutes of silence. This may be a sign that the problems are easing]
Second Life transaction history failure
In the wake of the transaction reporting problems that started just before midnight on Sunday night, transaction reporting for Second Life accounts has stopped ... well, reporting transactions. As per usual, the data is most likely still down in there somewhere and unharmed, but the reporting interface itself or connectors to that data are not functioning. Linden Lab tells us that the web devs are looking into the problem. Hopefully it won't be too long before the interface is back in service, but it is traditional that it tends to lag behind a bit after being repaired. (Footnote: The Second Life/Linden blog is apparently still reporting times in PDT - apparently not having switched timezones with everyone else back to PST - but that might be okay, considering the post is timestamped an hour in the future) [UPDATE: This is now reported as fixed as of 10:30 am]
More network problems on the Second Life grid
At 12:55AM SLT (US Pacific) this morning, with 29,353 Second Life users online, Linden Lab suffered a network failure between the San Francisco and Dallas colocation facilities that house the servers that make up the Second Life grid. Roughly 1,400 simulators became inaccessible, and approximately 3,000 users were disconnected. The problem was resolved at 1:35AM SLT, and the grid has returned to smooth operation.
Voice system glitches on Second Life grid
Linden Lab reports that "many" Second Life users are unable to join or create voice chat sessions. We're not sure what "many" means, exactly. The voice system doesn't actually operate as a part of the Second Life grid, per se, and the servers and software that make it go are all the domain of their voice partner Vivox. Naturally, many people at both ends are running around in a dignified state of panic as they attempt to diagnose the flaw. Voice is not widely used by Second Life users - however, those that do use it, rely on it heavily. [UPDATE: The official blog reports this as fixed at 9:19 am Second Life Time]
Second Life transaction notification problems
We have some reports (and I've also experienced it myself in the last couple of hours) of L$ payments not showing up immediately for Second Life users this-evening. The money is there, but account balances in the top corner of the screen are not updating until you've paid someone at least one L$. More confusingly, L$ transactions of L$1 and higher dated 4 November SLT (US Pacific) tonight are not necessarily showing in the transaction history unless you include at least November 2 (or earlier) in the search range. Some users say that the transactions show up after an hour or so, some have to increase the search range several days in order to see L$ transactions that took place this-evening. [Update: 11:30PM SLT (US Pacific) - We are starting to receive reports of problems with inventory transfers as well. No word from Linden Lab at present]
More network problems on the Second Life grid today
At 3:50PM SLT (US Pacific time) today with 52,370 users online in Second Life, the Second Life grid suffered a second major networking failure for the day. Approximately 7,000 users were disconnected immediately, some regions became inaccessible and the login servers were affected (as were teleportation and sim-border crossings). The fault has been traced back to one of Linden Lab's ISPs. We're not sure if that ISP is Level 3 (who are a primary data carrier between the San Francisco and Texas colocation facilities), or some other provider whom we are unware of. Grid functions were restored at 5:15PM SLT, and logins seem to be proceeding normally again.
Second Life grid experiences transient network failure
At 1:25AM SLT (US Pacific time) with 26,285 users online in Second Life, we have reports that a network problem took out 1,100 simulators in the virtual world. We have second hand reports of information from Linden Lab's Concierge-level support service, but little direct verification other than a sudden observed drop in the number of users online, and a sudden increase in packet-loss to many users. The glitch appears to have been relatively shortlived with services apparently being restored 15 minutes later at 1:40AM SLT. As yet there is no public word from Linden Lab about the outage. [Update: 1:57AM SLT - Linden Lab has acknowledged the outage and is investigating. The cause is given as an infrastructure failure at the San Francisco colocation facility]
