Second Life Saturday grid failure

We believe that all the immediate problems are fixed right now. We had three unrelated problems nail us at once. There was a routing problem to about 300 regions (about 1.5% of Second Life), which has now been fixed; that was the source of the teleport problems. One of our inventory databases had trouble, and we swapped in a hot backup; that was the source of the login troubles, which have now been resolved. Finally, we had some other network infrastructure problems (related to nameservice) that has also been fixed. At the moment, everything appears to be working well. Please see the blog entries on the Second Life blog for more information about these issues: http://blog.secondlife.com/2008/04/11/failed-teleports-and-region-crossings/ , http://blog.secondlife.com/2008/04/11/more-database-unpleasantness/

Some of the network infrastructure work we've been doing recently is to mimize the impact of an upcoming power outage at one of the data centers that hosts our machines : http://blog.secondlife.com/2008/04/11/maintenance-events-scheduled-13-16-april/

Unfortunately, there have been several reports of ghost prims. This seems to have increased AFTER the blog indicated that the grid's problems had been resolved. Because this results in the loss of non-copy items, it's a significant problem.