2008年10月31日金曜日

EVE Online's server model - Massively

EVEと呼ばれる、世界で最大のオンラインゲームMMO(Massive MultiPlayer Online)サーバの記事。
EVEはすべてのプログラム、データをSingle Instanceで運用しており、全く分散化していない、との事、またそればさまざまな問題を起こしている要因担っている、という記事。
内容はゲーム専用用語が多く、ちょっとわかりにくいが、スケーラビリティを議論する上で参考となる記事。


Almost any time a discussion about EVE Online comes up, one way or another we end up talking about the server. EVE Online is unique among today's most popular MMOs for its single-server approach. While most MMOs deal with large number of users by starting up large numbers of separate servers with identical game universes, EVE maintains only a single copy of its game universe on a massive cluster of servers. CCP's decision to go with a server model that doesn't use any sharding or instancing whatsoever has had a major impact on in-game activities and how the game has developed.

Server woes:
Unfortunately for CCP, maintaining their vision of a single game universe has proven a lot more difficult and costly than anyone anticipated. Working with IBM, the EVE server cluster is maintained in London and is currently the largest supercomputer employed in the gaming industry. Even with this massive power behind the EVE universe, there are still problems as CCP tries to keep the server upgraded ahead of its ever-expanding playerbase.

In this article, I discuss the unique gameplay that is possible thanks to EVE's server model, the problems the server currently faces and what CCP is planning to do about it.

With a single game world, players are free to flock to whatever solar system they like even if the server that solar system is on can't handle the load. Popular trade hub Jita and several popular mission-running systems such as Dodixie suffer badly from lag at peak play times on the weekend. Massive fleet battles over important territory often suffer from the same crippling lag, turning what should be an amazing sci-fi space battle into a poor slideshow.

Server structure:
Each of EVE's 5000+ star systems is loaded as a separate process onto any one of hundreds of IBM blade servers, with some high-load systems being given a server all to themselves and many low-load systems being combined and run on servers together. These "SOL Servers" are tied into EVE's main database server where changes to the game take place (where the magic happens).

Since players need to move between solar systems, they are connected to proxy servers which keep track of which SOL server the player is on. It's an ambitious system but has worked well for over five years with constant upgrades going on in the background to keep up with the increasing number of players entering EVE daily.

Effect on PvP:
You could be forgiven for thinking that an MMO's server model doesn't affect its gameplay significantly but EVE Online has proven this wrong for five years running. Putting all players together in one server drastically increases the opportunity for PvP. Instead of the MMO norm of less than 5,000 potential players for you to interact with and barely 1000 online at peak hours, EVE's server houses over 300,000 players with a peak concurrent user record of over 40,000. Additionally, since there's only one server for all players, there's no option to sign up to a non-pvp version of the game. This puts all players in the same world with the same rules whether they like it or not. If all you want to do is trade, mine and run missions you're just as vulnerable to PvP as everyone else and that's a major factor in defining the harsh feel of the EVE universe.

If EVE did offer a non-PvP server option, roles such as the pirate or corporate spy wouldn't really be possible any more because most potential targets would be playing on the non-pvp server. The players on the non-pvp server would also suffer from having a duller, less challenging game experience. We'd have one server full of hunters with no prey and one server full of prey with no excitement to their game.

Ultima Online experienced this issue in the Renaissance expansion when they released Trammel, a server where non-consensual PvP was no longer possible. With all the cut-throat villains separated from the general population, the villains had nothing to do and the remaining players lost their opportunity to be heroes.

Territorial conflict:
The lack of instancing in EVE Online's game universe has had an even more profound impact on PvP than the lack of a non-PvP server. When a solar system is depleted of resources, is becoming overcrowded or is being camped by pirates, there is no second instance of the system to switch to. The ability to pursue attackers from system to system successfully or to lock down a star system and prevent your enemy from passing through allows for piracy and very real territorial control that just isn't possible with another server model. Conflict over resources and space arises as a natural consequence of gameplay and not from a developer-defined game mechanic. Real player alliances are forged and broken every week in EVE with complex politics behind them.

Economic impact:
EVE is often lauded for its realistic player-based economy and real working markets but neither of these would function well on a sharded server. Throwing all of the players together in one place forces the markets to act based heavily on the rules of supply and demand. Without enough players driving both sides of the supply and demand curve, a single player could interfere with the global market very easily for a very long time. This has been done before in games like World of Warcraft to manipulate prices for a profit. It worked because with so few players on each server, one particularly rich player's effect on the market can be proportionally massive.

In EVE's safer systems, even major price manipulations tend to be balanced out by other players in a matter of hours, making price manipulation in trading hubs a very expensive and risky venture. It's said that the number of players in EVE caused the markets to hit critical mass long ago, reaching a point where high demand is almost always met by players with adequate supply within reasonable time-frames. As a result, the game's market hubs are always stocked full of whatever you might need.

Upgrades:
CCP recently encountered a problem they hadn't seen since late 2005. Certain server nodes were running out of memory, filling up with legitimate user data and crashing. Their response has been a controversial change to introduce player limits to star systems under heavy load. Although this was changed to only affect trade hub Jita for now, it highlights hardware inadequacy that CCP are meeting head-on with another round of server upgrades. The current server hardware uses impressive processors and advanced solid-state RAMSAN disks with the fast access speeds and large storage capacity that EVE's servers require.The bottleneck at the moment is getting data from one processor or ram disk to another and this is where their latest project comes in.

CCP aims to link the processors and RAM drives of every SOL server together with high-speed low-latency "Infiniband" technology, allowing data transfer at rates of several gigabytes per second. This will allow any processes which can be threaded to be split off and run on a processor that's not being used heavily at runtime, which should massively increase the server's load-balancing ability. The infiniband project poses a huge task for EVE's programmers, who are in the unenviable position of having to rewrite large portions of the core server code. If all goes well with their internal infiniband tests, these major changes in server architecture could eventually spell the end of laggy fleet battles and node crashes.

Summary:
With the problems CCP are constantly facing with their server and the cost of its upkeep, other developers seem reluctant to take EVE Online's server model on board. However, this model affects a lot more than its running costs and complexity and may be practically required for any successful next-generation PvP based MMO. It makes avenues of gameplay such as meaningful politics, piracy and real territorial warfare not just a possibility but an unavoidable consequence of group play. Could the single server approach become commonplace in the next generation of PvP-based MMO? I, for one, hope it does.