The grid woes seem to be continuing. At present most of Second Life's systems appear to be malfunctioning. Teleports seem to be broken in most regions, sim border crossings have become problematic and asset systems are showing very low responsiveness (if any).

Linden Lab additionally reports that login systems appear to be erratic as well. Login concurrency is falling either due to an inability for users to log in, or due to inworld problems once they are there. Perhaps both. This is a repeat of similar problems over the last 8-10 hours. Perhaps catching up with a good book might be in order.

We are assured that the problem is being investigated. Again.

Second Life Saturday grid failure originally appeared on Massively on Sat, 12 Apr 2008 10:30:00 EST. Please see our terms for use of feeds.

Remember when Linden Lab disabled assorted services on Sunday with the intention of stabilizing the Second Life grid through its weekly peak period? Well the big switch (actually, that is something of a misnomer, as the shutdown and startup processes for the subsystems seem to be quite complex) was thrown again for Friday.

This time, an extra subsystem was thrown into the mix. Logins. That's right, as the grid came up to the busiest time of the day, Linden Lab made the call to prevent any more users from logging in.

Continue reading Who let users on the grid?

Who let users on the grid? originally appeared on Massively on Sat, 12 Apr 2008 01:30:00 EST. Please see our terms for use of feeds.

From 1PM SLT (US Pacific) until 4:30PM Linden Lab is going to try to stabilize the subsystem performance during the Sunday peak load on Second Life systems by selectively disabling services for the three and a half hour window.

Avatar profiles won't be available, group member lists and group information will be suppressed, no group proposals (and we presume notices) will be able to be created, and parcel traffic will read as zero. When 4:30PM comes around and these are re-enabled, you will need to either relog or clear your viewer's group cache to properly restore your use of the services.

This is supposed to alleviate load-related problems until additional, more permanent fixes can be rolled out.

Linden Lab attempts to maintain Sunday stability, disables services originally appeared on Massively on Sun, 06 Apr 2008 14:45:00 EST. Please see our terms for use of feeds.

Second Life will be shutting down in a few minutes for what's been described as a capacity upgrade for the asset cluster. We do not know if any of this is intended to address problems that the service has been experiencing since the last update, or if the work is entirely unrelated to the current difficulties.

Additional work include rolling out new search changes, and maintenance to the voice system. The maintenance window is six hours from 6AM SLT (US Pacific time) to Noon SLT, though these maintenance windows are known to run overtime. Linden Lab usually provides status updates on the blog during the course of maintenance.

Continue reading Second Life shutting down for capacity upgrade

Second Life shutting down for capacity upgrade originally appeared on Massively on Wed, 09 Jan 2008 08:45:00 EST. Please see our terms for use of feeds.

Linden Lab's Second Life grid platform is off to a particularly rocky start in 2008. Problems with databases, internal networking, server failures, system configurations and more have combined to make day-to-day operations of the platform something of a chore.

Since Wednesday the second of January, there have been daily failures affecting assets and inventory, communications, transportation, logins, disbursement of group funds, and even the statistical data feeds published by Linden Lab have ceased updating.

Continue reading Peering inside - Second Life grid problems a daily occurrence for 2008

Peering inside - Second Life grid problems a daily occurrence for 2008 originally appeared on Massively on Tue, 08 Jan 2008 11:30:00 EST. Please see our terms for use of feeds.

As predicted, today saw record concurrencies in popular virtual world Second Life, today being the second day of record concurrencies, despite ongoing database problems in the underlying Second Life grid technology.

A widespread outage this-morning resulting in a temporary closure, and another systems failure just minutes ago functioned as mere punctuation for a steady stream of minor problems throughout the day - few of which appear to be directly load-related.

Continue reading Users persevere despite Second Life problems

Users persevere despite Second Life problems originally appeared on Massively on Sun, 06 Jan 2008 22:30:00 EST. Please see our terms for use of feeds.

Intermittent asset problems have plagued the Second Life grid since approximately 1AM SLT (US Pacific) today. Residents report problems taking objects, rezzing objects, accessing inventory and creating/modifying objects.

At 5:29AM SLT, Linden Lab acknowledged the problem, and reported that they were investigating. However, Linden Lab closed grid logins, and disconnected users slightly over an hour later, at 6:53AM SLT. No ETA on repairs, or details as to the nature of the problem are available to us at the present time.

Continue reading Second Life grid closed due to asset problems

Second Life grid closed due to asset problems originally appeared on Massively on Sun, 06 Jan 2008 10:30:00 EST. Please see our terms for use of feeds.

Due to unexpected problems with the asset system, Second Life underwent an emergency shutdown at 1:00PM SLT (US Pacific). Technically, the grid itself was not shut down, but logins were closed and all users were forcibly logged out in order to address the asset problems.

Grid logins were restored at 1:45PM SLT and the grid status switched back to OPEN at that time - however, the Linden Lab blog entry with the notice and with status updates as to the progress of the work did not appear on the blog until 2:25PM SLT, at the time the work was completed, and cites 2:02PM SLT as the reopening time for logins.

Update: 3:30PM SLT - Residents are reporting the same asset problems that prompted the shutdown earlier this-afternoon.

Update: 4:00PM SLT - Linden Lab says: " Please do not use inventory related actions at this time" - that's got to be a bit of a barrier.

Update: 5:20PM SLT - 28,000 or so users logged out while things were being worked on (the messages we got from some users were very frustrated - others seemed more patient and resigned). Linden Lab reports that the issue is now resolved! Again.

Emergency shutdown for Second Life [UPDATED] originally appeared on Massively on Wed, 19 Dec 2007 17:45:00 EST. Please see our terms for use of feeds.

It looks like yesterday's sudden fix to deployed Second Life systems might have caused more problems than it solved. In the early hours of this-morning around 3AM SLT (US Pacific), large numbers of simulators started going off the air, and not coming back online.

While Linden Lab said they'd nearly gotten on top of the problem at 3:56AM SLT, things continued to be somewhat rocky, and at 6:50AM SLT things have started to get rather worse - with quite a number of users saying that they're opting to ride it out offline.

Continue reading Linden Lab struggling to keep Second Life simulators up today

Linden Lab struggling to keep Second Life simulators up today originally appeared on Massively on Thu, 13 Dec 2007 10:15:00 EST. Please see our terms for use of feeds.

Linden Lab Lab have just announced that a rolling restart of Second Life simulators to fix inventory issues (that resulted in some inventory service failures yesterday) will be rolled out at 11AM SLT (US Pacific).

That's a two hour rolling restart with 67 minutes notice. At two hours, it means it's one of the high-speed ones that restarts large numbers of regions at once in unpredictable patterns, and that usually spells large-scale disruption. Not a lot of warning - especially to people with meetings and events scheduled.

Very short notice rolling restart for the Second Life Grid originally appeared on Massively on Wed, 12 Dec 2007 13:10:00 EST. Please see our terms for use of feeds.

At 7:20AM SLT (US Pacific) today the Second Life grid began experiencing difficulties. The usual problems are in evidence: Problems teleporting, unusual difficulties with sim border crossings, account balances not showing, asset uploads failing, attachments failing, object rezzing failing, group IMs unreliable, problems accessing profiles and group information, and slow or failing logins.

At the time the problems commenced a mere 39,105 users were online on the Second Life grid. That number fell by over a thousand, fairly quickly and is still declining slowly.

Linden Lab has not yet issued any acknowledgement of the difficulties.

[Update: 8:20 AM SLT - Linden Lab has finally acknowledged the problems and is working to resolve them]

[Update: 8:52 AM SLT - Linden Lab considers the issue resolved. While concurrency seems to have taken an upturn since then, it's not rising very quickly yet, and there are still reports of problems, though not nearly so many]

Second Life grid services problems originally appeared on Massively on Mon, 10 Dec 2007 11:15:00 EST. Please see our terms for use of feeds.

At 5:10PM SLT (US Pacific) the Second Life grid suffered an unexpected subsystems failure, resulting in the disconnection of approximately 7,000 of the 45,000 users online at the time.

Users report problems with assets, inventory, simulator border crossings, teleportation and group IMs. As of 5:45PM SLT, problems seem to be easing, but no acknowledgement or notification from Linden Lab at this time.

Major Second Life grid failure originally appeared on Massively on Wed, 05 Dec 2007 20:45:00 EST. Please see our terms for use of feeds.

It's no surprise that Second Life has been having some trouble this week - mostly apparently related to bug fixes and upgrades, unfortunately. If you didn't much notice it, you probably didn't spend a whole lot of time online actively poking the asset and inventory servers (or trying to buy land) - particularly in the early hours of the SL morning.

Linden Lab have had just as rough a time with the platform this week apparently as well, with some of their own applications and batch jobs faring no better.

Continue reading Grid problems don't make Linden Lab happy either

Grid problems don't make Linden Lab happy either originally appeared on Massively on Fri, 30 Nov 2007 01:00:00 EST. Please see our terms for use of feeds.

Things settled down considerably after this-morning's breakage on the Second Life grid, but apparently it was too good to last.

Statistical data feeds went dark at 4:40PM SLT (US Pacific) today, and at approximately the same time DNS problems screwed up logins, teleports, region crossings, and actually just about everything crossing simulator boundaries. Linden Lab's ops team is trying to get everything fixed, but everything DNS-related has some settling lead-time.

Continue reading Linden Lab not having a good time - nor are the users

Linden Lab not having a good time - nor are the users originally appeared on Massively on Tue, 20 Nov 2007 20:45:00 EST. Please see our terms for use of feeds.

This morning's Second Life service outage, Linden Lab says, is the result of "essential maintenance". The maintenance wasn't announced ahead of schedule, but apparently only after it broke things.

It's hard to fault them for wanting to fix things, but a bit of warning is generally the done thing. In this case, notice of the maintenance work was only given after problems began at 12:10AM SLT (US Pacific), with the actual notice coming 34 minutes later at 12:44AM.

Continue reading Linden Lab sorry, busy breaking stuff

Linden Lab sorry, busy breaking stuff originally appeared on Massively on Tue, 20 Nov 2007 06:30:00 EST. Please see our terms for use of feeds.

Linden Lab advises this-morning (and we're certainly seeing it reflected in the figures) that while they thought the issue had been resolved on Saturday evening, that it was not, and that the issues that plagued the Second Life grid over the weekend are still ongoing.

Additionally, all voice services went dark at 2:06AM SLT (US Pacific).

Continue reading Second Life stability problems continue into the new week

Second Life stability problems continue into the new week originally appeared on Massively on Mon, 19 Nov 2007 11:00:00 EST. Please see our terms for use of feeds.

The new Second Life day is off to a poor start with what looks like another database crash inside Linden Lab's array of database clusters.

The crash (if that's exactly what caused it) took place at 1:25AM SLT (US Pacific time), one of the quietest parts of the day with only 32,055 users online. Obviously the problem is not solely load-related.

Continue reading Sunday not shaping up well for Second Life either

Sunday not shaping up well for Second Life either originally appeared on Massively on Sun, 18 Nov 2007 08:15:00 EST. Please see our terms for use of feeds.

It's not been a fantastic week for the Second Life service (actually more like ten days). Plagued with network problems, the grid has been experiencing everything from strange behavior to partial blackouts of large numbers of servers at once.

Today isn't any better, and actually is looking a whole lot worse.

Continue reading Linden Lab having a bad day

Linden Lab having a bad day originally appeared on Massively on Sat, 17 Nov 2007 21:30:00 EST. Please see our terms for use of feeds.

The Second Life website this-morning is directing people to the status page (which cheerfully announces that 'Second Life is open!').

This is causing people problems with Lindex currency transactions, accessing account data, and so on. The failure apparently began at 2:50AM SLT (US Pacific). No word from Linden Lab at this stage(though the Linden blog and application logins seem to be fine).

[Update: 6:15AM SLT - The website has been restored and related services are working properly again according to the reports we're getting. No word from Linden Lab about the glitch]

[Update: 7:00AM SLT - Linden Lab acknowledged the website problems and reports that they have been fixed. Also that there was yet another networking failure between the colocation facilities. Whatever is up with the networks, lately, it's not good]

Second Life website, some services down originally appeared on Massively on Sat, 17 Nov 2007 07:15:00 EST. Please see our terms for use of feeds.

Throughout Thursday, Second Life users report that the whole grid was acting a little oddly - and the statistics we have bear that out. Starting just after midnight, however, something else odd started to happen.

From about 12:30 PM SLT (US Pacific) group and profile information began to load inconsistently along with other "random but inconsistent phenomena". It's nothing people can put their finger on, specifically, but given the day's concurrency curves, it's obviously having a significant impact on Second Life users.

Continue reading Strange goings on in the Second Life grid

Strange goings on in the Second Life grid originally appeared on Massively on Fri, 16 Nov 2007 06:50:00 EST. Please see our terms for use of feeds.

At 3:50AM SLT (US Pacific) with 31,509 users online something serious went wrong with Second Life's backbone or web-services. Statistical data feeds ceased functioning and the login servers stopped responding to requests entirely.

At present the cause is unknown and there is no word from Linden Lab about the problems.

[Update: 4:05AM SLT - Statistical feeds have started reporting again. Apparently approximately 20,000 users were disconnected.]

[Update: 4:15AM SLT - Linden Lab has acknowledged the problem]

Second Life login failure originally appeared on Massively on Wed, 14 Nov 2007 07:15:00 EST. Please see our terms for use of feeds.

A second round of network issues today in Second Life is announced with a blog post that is (once again) timestamped in the future. Forget the network problems between the Dallas and San Francisco colocation facilities. Forget that it's taken some simulators offline. Forget that about 10,000 people have been knocked offline.

It's timestamped in the future! Is it possible for them to post Lotto numbers or something?

Postcards from the future originally appeared on Massively on Sat, 10 Nov 2007 20:45:00 EST. Please see our terms for use of feeds.

We're getting widespread reports of systems failure in Second Life as of a short minutes ago [approximately 12:20AM SLT (US Pacific)]. Since then, distressed messages are becoming increasingly widespread.

Teleports not working, inventory and asset problems, high-levels of packet-loss, presence issues (affecting group IMs and the friends-list); all among the usual suspects for a central services or network failure. Reports began to trickle in sporadically, but there has been a flood of them since 12:50AM SLT - at which time Linden Lab statistical data feeds suddenly went dark.

Related to the rolling restart late yesterday? Quite possibly. The last Friday afternoon update Second Life had led to a weekend of disasters. There has been no word from Linden Lab at this time.

[Update: 1:05AM - Linden Lab's statistical feeds came back to life after 15 minutes of silence. This may be a sign that the problems are easing]

Widespread problems reported on the Second Life grid originally appeared on Massively on Sat, 10 Nov 2007 04:00:00 EST. Please see our terms for use of feeds.

In the wake of the transaction reporting problems that started just before midnight on Sunday night, transaction reporting for Second Life accounts has stopped ... well, reporting transactions. As per usual, the data is most likely still down in there somewhere and unharmed, but the reporting interface itself or connectors to that data are not functioning.

Linden Lab tells us that the web devs are looking into the problem. Hopefully it won't be too long before the interface is back in service, but it is traditional that it tends to lag behind a bit after being repaired.

(Footnote: The Second Life/Linden blog is apparently still reporting times in PDT - apparently not having switched timezones with everyone else back to PST - but that might be okay, considering the post is timestamped an hour in the future)

[UPDATE: This is now reported as fixed as of 10:30 am]

Second Life transaction history failure originally appeared on Massively on Tue, 06 Nov 2007 11:35:00 EST. Please see our terms for use of feeds.

At 12:55AM SLT (US Pacific) this morning, with 29,353 Second Life users online, Linden Lab suffered a network failure between the San Francisco and Dallas colocation facilities that house the servers that make up the Second Life grid.

Roughly 1,400 simulators became inaccessible, and approximately 3,000 users were disconnected. The problem was resolved at 1:35AM SLT, and the grid has returned to smooth operation.

More network problems on the Second Life grid originally appeared on Massively on Tue, 06 Nov 2007 04:45:00 EST. Please see our terms for use of feeds.

Linden Lab reports that "many" Second Life users are unable to join or create voice chat sessions. We're not sure what "many" means, exactly. The voice system doesn't actually operate as a part of the Second Life grid, per se, and the servers and software that make it go are all the domain of their voice partner Vivox.

Naturally, many people at both ends are running around in a dignified state of panic as they attempt to diagnose the flaw. Voice is not widely used by Second Life users - however, those that do use it, rely on it heavily.

Voice system glitches on Second Life grid originally appeared on Massively on Mon, 05 Nov 2007 11:45:00 EST. Please see our terms for use of feeds.

We have some reports (and I've also experienced it myself in the last couple of hours) of L$ payments not showing up immediately for Second Life users this-evening. The money is there, but account balances in the top corner of the screen are not updating until you've paid someone at least one L$.

More confusingly, L$ transactions of L$1 and higher dated 4 November SLT (US Pacific) tonight are not necessarily showing in the transaction history unless you include at least November 2 (or earlier) in the search range. Some users say that the transactions show up after an hour or so, some have to increase the search range several days in order to see L$ transactions that took place this-evening.

[Update: 11:30PM SLT (US Pacific) - We are starting to receive reports of problems with inventory transfers as well. No word from Linden Lab at present]

Second Life transaction notification problems originally appeared on Massively on Mon, 05 Nov 2007 01:30:00 EST. Please see our terms for use of feeds.

At 3:50PM SLT (US Pacific time) today with 52,370 users online in Second Life, the Second Life grid suffered a second major networking failure for the day. Approximately 7,000 users were disconnected immediately, some regions became inaccessible and the login servers were affected (as were teleportation and sim-border crossings).

The fault has been traced back to one of Linden Lab's ISPs. We're not sure if that ISP is Level 3 (who are a primary data carrier between the San Francisco and Texas colocation facilities), or some other provider whom we are unware of. Grid functions were restored at 5:15PM SLT, and logins seem to be proceeding normally again.

More network problems on the Second Life grid today originally appeared on Massively on Fri, 02 Nov 2007 21:15:00 EST. Please see our terms for use of feeds.

At 1:25AM SLT (US Pacific time) with 26,285 users online in Second Life, we have reports that a network problem took out 1,100 simulators in the virtual world. We have second hand reports of information from Linden Lab's Concierge-level support service, but little direct verification other than a sudden observed drop in the number of users online, and a sudden increase in packet-loss to many users.

The glitch appears to have been relatively shortlived with services apparently being restored 15 minutes later at 1:40AM SLT. As yet there is no public word from Linden Lab about the outage.

[Update: 1:57AM SLT - Linden Lab has acknowledged the outage and is investigating. The cause is given as an infrastructure failure at the San Francisco colocation facility]

Second Life grid experiences transient network failure originally appeared on Massively on Fri, 02 Nov 2007 04:45:00 EST. Please see our terms for use of feeds.