Massively Multiplayer Middleware
大型多人游戲中間件
MICHI HENNING, ZeroC

Building scaleable middleware for ultra-massive online games teaches a lessonwe all can use: Big project, simple design.

為超大型在線游戲構(gòu)建可擴(kuò)展的中間件給我們所有人的教訓(xùn)：宏大的項(xiàng)目，簡單的設(shè)計(jì)。

Wish is a multiplayer, online, fantasy role-playing game being developed by MutableRealms.1 It differs from similar online games in that it allows tens of thousandsof players to participate in a single game world (instead of the few hundredplayers supported by other games). Allowing such a large number of players requiresdistributing the processing load over a number of machines and raises the problemof choosing an appropriate distribution technology.

Wish游戲是Mutable Realms[1]開發(fā)的幻想類多人在線角色扮演游戲。不同于類似的在線游戲，它允許數(shù)萬的玩家同時(shí)參與到一個(gè)游戲世界中（而其他游戲僅支持幾百人）。允許如此之多的玩家就要求在多臺(tái)機(jī)器上進(jìn)行分布式處理，由此產(chǎn)生了一個(gè)問題：選擇合適的分布式技術(shù)。

DISTRIBUTION REQUIREMENTS

分布式需求

Mutable Realms approached ZeroC for the distribution requirements of Wish. ZeroCdecided to develop a completely new middleware instead of using existing technology,such as CORBA (Common Object Request Broker Architecture).2 To understand themotivation for this choice, we need to examine a few of the requirements placedon middleware by games on the scale of Wish and other large-scale distributedapplications.

Mutable Realms帶著Wish游戲的分布式需求找到ZeroC。ZeroC決定開發(fā)一個(gè)全新的中間件，而不是使用現(xiàn)有的技術(shù)，比如CORBA（Common Object Request Broker Architecture，公共對(duì)象請(qǐng)求代理體系）[2]。為了理解這個(gè)選擇的動(dòng)機(jī)，我們需要考察像Wish這種規(guī)模的游戲和其他大規(guī)模分布式應(yīng)用程序?qū)χ虚g件的一些要求。

Multi-Platform Support. The dominant platform for the online games market isMicrosoft Windows, so the middleware has to support Windows. For the server side,Mutable Realms had early on decided to use Linux machines: The low cost of theplatform, together with its reliability and rich tool support, made this an obviouschoice. The middleware, therefore, had to support both Windows and Linux, withpossible later support for Mac OS X and other Unix variants.

多平臺(tái)支持。在線游戲市場(chǎng)的主要平臺(tái)是Microsoft Windows，所以中間件必須支持Windows。在服務(wù)器這邊，Mutable Realms很早就決定使用Linux：費(fèi)用低，并且可靠性高，以及工具軟件豐富，這是明擺著的選擇。因此，中間件必須要同時(shí)支持Windows和Linux，也許以后還要支持Mac OS X和其他Unix變種。

Multi-Language Support. Client and server software is written in Java, as wellas a combination of C++ and assembly language for performance-critical functions.At ZeroC we used Java because some of our development staff had little priorC++ experience. Java also offers advantages in terms of defect count and developmenttime; in particular, garbage collection eliminates the memory management errorsthat often plague C++ development. For administration of the game via the Web,we wanted to use the PHP hypertext processor. As a result, the game middlewarehad to support C++, Java, and PHP.

多語言支持。客戶端和服務(wù)器軟件是用Java編寫的，同時(shí)，對(duì)于性能關(guān)鍵的函數(shù)還混合使用了C++和匯編語言。在ZeroC，我們使用Java，因?yàn)槲覀兊牟糠珠_發(fā)人員先前沒什么C++經(jīng)驗(yàn)。Java同時(shí)在缺陷數(shù)和開發(fā)時(shí)間上有優(yōu)勢(shì)；特別是垃圾收集消除了內(nèi)存管理錯(cuò)誤，而這是C++開發(fā)的死疾。為了通過Web管理游戲，我們想用PHP。所以，這個(gè)游戲中間件必須支持C++、Java和PHP。

Transport and Protocol Support. As we developed the initial distribution architecturefor the game, it became clear that we were faced with certain requirements interms of the underlying transports and protocols:

傳輸和協(xié)議支持。當(dāng)我們?yōu)橛螒蜷_發(fā)最初的分布式構(gòu)架時(shí)，我們明顯面臨了對(duì)底層傳輸和協(xié)議的一些需求：

• Players connect to ISPs via telephone lines, as well as broadband links. Whilebroadband is becoming increasingly popular, we had decided early on that thegame had to be playable over an ordinary modem. This meant that communicationsbetween clients and server had to be possible via low-bandwidth and high-latencylinks.

• 玩家通過電話線或者寬帶連接ISP上網(wǎng)。雖然寬帶越來越普及，但是我們?cè)缫褯Q定游戲必須能通過普通modem玩。這意味著客戶端服務(wù)器之間通過低帶寬高延時(shí)鏈路進(jìn)行通訊必須是可能的。

• Much of the game is event driven. For example, as a player moves around, otherplayers in the same area need to be informed of the changes in the game worldaround them. These changes can be distributed as simple events such as, “PlayerA moves to new coordinates <x,y>.”

• 游戲大部分是事件驅(qū)動(dòng)的。比如，當(dāng)玩家四處溜達(dá)時(shí)，這些變化需要通知同一區(qū)域內(nèi)的其他玩家，使他們知曉其周圍的游戲世界中的變化。這些變化可以作為簡單的事件分發(fā)，比如“玩家A移動(dòng)到新坐標(biāo)<x,y>”。

Ideally, events are distributed via “datagrams.” If the occasionalstate update is lost, little harm is done: A lost event causes a particular observer’sview of the game world to lag behind momentarily, but that view becomes up-to-dateagain within a very short time, when another event is successfully delivered.

理想的，事件是通過數(shù)據(jù)報(bào)（datagram）分發(fā)的。即使偶爾有狀態(tài)更新丟失，也沒什么危害：一個(gè)事件丟失會(huì)造成某個(gè)觀察者對(duì)于游戲世界的視圖暫時(shí)性滯后，但很快，當(dāng)另一個(gè)事件成功傳送后，該視圖又會(huì)變成最新狀態(tài)。

• Events in the game often have more than one destination. For example, if a playermoves within the field of vision of five other players, the same positional updatemust be sent to all five observing players. We wanted to be able to use broadcastor multicast to support such scenarios.

• 游戲中的事件經(jīng)常有多個(gè)目標(biāo)。比如，玩家在其他5個(gè)玩家的視野內(nèi)移動(dòng)，位置更新必須發(fā)送給所有這5個(gè)看到他的玩家。我們想用廣播或多播來支持這樣的場(chǎng)景。

• Communications between clients and game servers must be secure. For an onlinesubscription-based game, this is necessary for revenue collection, as well asto prevent cheating. (For example, it must be impossible for a player to acquirea powerful artifact by manipulating the client-side software.)

• 客戶端和游戲服務(wù)器之間的通信必須是安全的。對(duì)基于訂購的在線游戲，這是必須的，既為了賺錢，也為了防做弊（例如，玩家無法通過修改客戶端軟件來得到超級(jí)裝備）。

• Clients connect to the game from LANs that are behind firewalls and use NAT (networkaddress translation). The communications protocol for the game has to be designedin a way that accommodates NAT without requiring knowledge of application-specificinformation in order to translate addresses.

• 客戶端能在局域網(wǎng)內(nèi)部通過防火墻和NAT（network address translation，網(wǎng)絡(luò)地址轉(zhuǎn)換）連入游戲。游戲的通信協(xié)議必須按以下方式設(shè)計(jì)：它能夠穿越NAT，但是地址轉(zhuǎn)換無需應(yīng)用程序特有的相關(guān)信息。

Versioning Support. We wanted to be able to update the game world while the gamewas being played—for example, to add new items or quests. These updateshave to be possible without requiring every deployed client to be upgraded immediately—thatis, client software at an older revision level has to continue to work with theupdated game servers (albeit without providing access to newly added features).This means that the type system has to be flexible enough to allow updates, suchas adding a field to a structure or changing the signature of a method, withoutbreaking deployed clients.

版本控制支持。我們需要能夠在游戲運(yùn)行中更新游戲，比如，增加新的物品或任務(wù)。這些更新不要求所有已部署的客戶端都馬上升級(jí)，即，舊版本的客戶端軟件必須能夠連接已升級(jí)的服務(wù)器繼續(xù)運(yùn)行（雖然不提供新增功能）。這意味著類型系統(tǒng)必須要足夠靈活以允許更新，比如給結(jié)構(gòu)添加新域，或者更改方法的簽名，而不會(huì)使已部署的客戶端無法使用。

Ease of Use. Although a few of the Wish game developers are distributed computingexperts, the majority have little or no experience. This means that the middlewarehas to be easy for nonexperts to use, with simple, threadsafe and exception-safeAPIs (application programming interfaces).

易用性。雖然有些Wish游戲的開發(fā)者是分布式計(jì)算的專家，但是大部分經(jīng)驗(yàn)不足甚至沒有經(jīng)驗(yàn)。這意味著中間件必須讓非專家也能容易使用，具有簡單的、線程安全的，和異常安全的API（application programming interface，應(yīng)用程序接口）。

Persistence. Much of the game requires state, such as the inventory for eachplayer, to be stored in a database. We wanted to provide developers with a wayto store and retrieve persistent state for application objects without havingto concern themselves with the actual database and without having to design databaseschemas. Particularly during development, as the game evolves, it is prohibitivelytime consuming to repeatedly redesign schemas to accommodate changes. In addition,as we improve the game while being deployed, we must add new features to a databaseand remove older features from it. We wanted an automatic way to migrate an existing,populated database to a new database schema without losing any of the informationin the old database that was still valid.

持久化。很多游戲都需要將狀態(tài)保存數(shù)據(jù)庫，如玩家的所有物品。我們需要給開發(fā)者提供一種方法，使之可以保存和獲取應(yīng)用對(duì)象的持久化狀態(tài)，但不必關(guān)心具體的數(shù)據(jù)庫，也不必設(shè)計(jì)數(shù)據(jù)庫模式（database schema）。特別是開發(fā)過程中，當(dāng)游戲不斷演化，為了適應(yīng)變化而反復(fù)地修改表結(jié)構(gòu)，所耗的時(shí)間會(huì)讓人望而卻步。而且，在游戲發(fā)布后，當(dāng)我們改進(jìn)游戲時(shí)，必須添加新的屬性到數(shù)據(jù)庫，以及刪除舊的屬性。我們需要一種自動(dòng)的方式，將現(xiàn)存的已填充的數(shù)據(jù)庫遷移到新的數(shù)據(jù)庫模式，而不丟失舊庫中任何仍然有效的信息。

Threading. Much of the server-side processing is I/O-bound: Database and networkaccess forces servers to wait for I/O completion. Other tasks, such as pathfinding,are compute-bound and can best be supported using parallel algorithms. This meansthat the middleware has to be inherently threaded and offer developers sufficientcontrol over threading strategies to implement parallel algorithms while preventingproblems such as thread starvation and deadlock. Given the idiosyncrasies ofthreading on different operating systems, we also wanted a platform-neutral threadingmodel with a portable API.

多線程。大多數(shù)服務(wù)端處理都是I/O密集型：數(shù)據(jù)庫和網(wǎng)絡(luò)訪問迫使服務(wù)器等待I/O完成。其他任務(wù)，比如尋路，是計(jì)算密集型，最好使用并行算法。這意味著中間件必須內(nèi)在地多線程化，并且為開發(fā)者提供足夠的對(duì)線程策略的控制，以實(shí)現(xiàn)并行算法，同時(shí)避免如線程饑餓和死鎖這樣的問題。考慮到不同操作系統(tǒng)上的線程特性，我們需要平臺(tái)無關(guān)的線程模型，并具有可移植的API。

Scalability. Clearly, the most serious challenges for the middleware are in thearea of scalability: For an online game, predicting realistic bounds is impossibleon things such as the total number of subscribers or the number of concurrentplayers. This means that we need an architecture that can be scaled by federatingservers (that is, adding more servers) as demands on the software increase.

可擴(kuò)展性。顯然，對(duì)中間件最重要的挑戰(zhàn)是在可擴(kuò)展性方面：對(duì)于在線游戲，不可能預(yù)測(cè)到如訂閱總數(shù)或同時(shí)在線玩家數(shù)這類數(shù)據(jù)的真實(shí)范圍。這意味著我們需要這樣的架構(gòu)，當(dāng)對(duì)軟件的需求增加時(shí)，它可以聯(lián)盟（federate）服務(wù)器來進(jìn)行擴(kuò)展（即添加更多服務(wù)器）。

We also need fault-tolerance: For example, upgrading a server to a newer versionof the game software has to be possible without kicking off every player currentlyusing that server. The middleware has to be capable of automatically using areplica server while the original server is being upgraded.

我們也需要容錯(cuò)性：比如，將服務(wù)器升級(jí)到游戲軟件的新版本必須是可能的，而不應(yīng)該踢掉當(dāng)前正在使用該服務(wù)器的每個(gè)玩家。中間件必須能夠在原服務(wù)器正在升級(jí)時(shí)，自動(dòng)使用一個(gè)副本服務(wù)器（replica server）。

Other scalability issues relate to resource management. For example, we did notwant to be subject to hardwired limits, such as a maximum number of open connectionsor instantiated objects. This means that, wherever possible, the middleware hasto provide automated resource management functions that are not subject to arbitrarylimits and are easy to use. Simultaneously, these functions have to provide enoughcontrol for developers to tune resource management to their needs. Wherever possible,we wanted to be able to change resource management strategies without requiringrecompilation.

其他可擴(kuò)展性問題涉及資源管理。例如，我們不想受到硬限制，如最大打開連接數(shù)，或最大實(shí)例化對(duì)象數(shù)。這意味著，只要有可能，中間件必須提供不受任何限制的自動(dòng)資源管理功能，并且使用方便。同時(shí)，這些功能必須給開發(fā)者提供足夠的控制，以按照他們的需要調(diào)整資源管理。只要有可能，我們希望能夠改變資源管理策略，而不要求重新編譯。

A common scalability problem for distributed multiplayer games relates to managingdistributed sets of objects. The game might allow players to form guilds, subjectto certain rules: For example, a player may not be a member of more than oneguild, or a guild may have at most one level-5 mage (magician). In computingterms, implementing such behavior boils down to performing membership tests onsets of distributed objects. Efficient implementation of such set operationsrequires an object model that does not incur the cost of a remote message foreach test. In other words, the object identities of objects must be visible atall times and must have a total order.

分布式多人游戲共同的一個(gè)可擴(kuò)展性問題是，管理分布式對(duì)象集合。游戲可能允許玩家們組成公會(huì)（guild），但要符合一定條件：比如，玩家不可以加入多個(gè)公會(huì)，或者公會(huì)最多有1個(gè)5級(jí)魔法師。用計(jì)算機(jī)術(shù)語說，實(shí)現(xiàn)這種行為歸結(jié)為對(duì)分布式對(duì)象集執(zhí)行成員資格測(cè)試。有效地實(shí)現(xiàn)這種集合運(yùn)算需要這樣的對(duì)象模型，它不用每次測(cè)試都發(fā)送遠(yuǎn)程消息。也就是說，對(duì)象的標(biāo)識(shí)對(duì)象要全程可見，并且具有全序關(guān)系。

In classical RPC (remote procedure call) systems, object implementations residein servers, and clients send remote messages to objects: All object behavioris on the server, with clients only invoking behavior, but not implementing it.Although this approach is attractive because it naturally extends the notionof a local procedure call to distributed scenarios, it causes significant problems:

在經(jīng)典RPC（remote procedure call，遠(yuǎn)程過程調(diào)用）系統(tǒng)中，對(duì)象的實(shí)現(xiàn)位于服務(wù)器，客戶端發(fā)送遠(yuǎn)程消息到對(duì)象：所有對(duì)象行為都在服務(wù)器，而客戶端只是調(diào)用行為，而不是實(shí)現(xiàn)它。盡管這個(gè)方法很有吸引力，因?yàn)樗鼘⒈镜剡^程調(diào)用的概念自然地?cái)U(kuò)展到了分布式環(huán)境中，但它造成了以下值得注意的問題：

• Sending a remote message is orders of magnitude slower than sending a local message.One obvious way to reduce network traffic is to create “fat” RPCs:as much data as possible is sent with each call to better amortize the cost ofgoing on the wire. The downside of fat RPCs is that performance considerationsinterfere with object modeling: While the problem domain may call for fine-grainedinterfaces with many operations that exchange only a small amount of state, goodperformance requires coarse-grained interfaces. It is difficult to reconcilethis design tension and find a suitable trade-off.

• 發(fā)送遠(yuǎn)程消息比發(fā)送本地消息慢好幾個(gè)數(shù)量級(jí)。減少網(wǎng)絡(luò)流量的一種明顯的方法是創(chuàng)建“胖（fat）”RPC：每次調(diào)用都發(fā)送盡可能多的數(shù)據(jù)，以便更好地分?jǐn)倐鬏數(shù)某杀尽Ｅ諶PC的缺點(diǎn)是，性能方面的考慮干擾了對(duì)象建模：雖然問題域可能需要細(xì)粒度的接口，以及許多僅僅交換少量狀態(tài)的操作，但為了良好的性能，卻要求粗粒度的接口。人們很難調(diào)和這種設(shè)計(jì)的緊張局勢(shì)并找到適合的折衷之法。

• Many objects have behavior and can be traded among players. Yet, to meet theprocessing requirements of the game, we have many servers (possibly in differentcontinents) that implement object behavior. If behavior stays put in the server,yet players can trade objects, before long, players end up with a potion whoseserver is in the United States and a scroll whose server is in Europe, with thepotion and scroll carried in a bag that resides in Australia. In other words,a pure client–server model does not permit client-side behavior and objectmigration, and, therefore, destroys locality of reference.

• 很多對(duì)象有行為，并且可以在玩家之間交易。但是，為了滿足游戲處理的要求，我們有很多服務(wù)器（可能位于不同的大陸），它們實(shí)現(xiàn)了對(duì)象的行為。如果行為停留在服務(wù)器不動(dòng)，而玩家可以交易對(duì)象，那么，不久以后，有個(gè)玩家會(huì)有一瓶藥在美國的服務(wù)器上，一個(gè)卷軸在歐洲的服務(wù)器上，而放這瓶藥和卷軸的袋子卻在澳大利亞的服務(wù)器上。換句話說，純粹的客戶端-服務(wù)器模型不允許客戶端的行為和對(duì)象遷移，并且因此破壞了訪問局部性（locality of reference）。

We wanted an object model that supports both client- and server-side behaviorso we could migrate objects and improve locality of reference.

我們需要一個(gè)同時(shí)支持服務(wù)器端和客戶端行為的對(duì)象模型，這樣我們就能遷移對(duì)象并改善訪問局部性。

DESIGNING A NEW MIDDLEWARE

設(shè)計(jì)新的中間件

Looking at our requirements, we quickly realized that existing middleware wouldbe unsuitable. The cross-platform and multi-language requirements suggested CORBA;however, a few of us had previously built a commercial object request brokerand knew from this experience that CORBA could not satisfy our functionalityand scalability requirements. Consequently, we decided to develop our own middleware,dubbed Ice (short for Internet Communications Engine).3

看到需求，我們馬上意識(shí)到現(xiàn)有的中間件沒有合適的。多平臺(tái)和多語言的需求使人想到CORBA；但是，我們幾個(gè)以前曾經(jīng)建立了一個(gè)商業(yè)性的對(duì)象請(qǐng)求代理，并且從這次經(jīng)歷知道，CORBA不能滿足我們的功能和可擴(kuò)展性需求。因此，我們決定開發(fā)自己的中間件，稱為Ice（互聯(lián)網(wǎng)通信引擎，Internet Communications Engine的簡稱）[3]。

The overriding focus in the design of Ice was on simplicity: We knew from bitterexperience that every feature is paid for in increased code and memory size,more complex APIs, steeper learning curve, and reduced performance. We made everyeffort to find the simplest possible abstractions (without passing the “complexitybuck” to the developer), and we admitted features only after we were certainthat we absolutely had to have them.

Ice中設(shè)計(jì)的首要重點(diǎn)是簡單性：我們從過去的痛苦經(jīng)驗(yàn)得知，每個(gè)功能的代價(jià)就是增加的代碼和內(nèi)存占用、更復(fù)雜的API、陡峭的學(xué)習(xí)曲線，以及降低的性能。我們盡了一切努力尋找最簡單的抽象（不是把復(fù)雜性推卸給開發(fā)者），并且我們只接受我們確信絕對(duì)必要的功能。

Object Model. Ice restricts its object model to a bare minimum: Built-in datatypes are limited to signed integers, floating-point numbers, Booleans, Unicodestrings, and 8-bit uninterpreted (binary) bytes. User-defined types include constants,enumerations, structures, sequences, dictionaries, and exceptions with inheritance.Remote objects are modeled as interfaces with multiple inheritance that containoperations with input and output parameters and a return value. Interfaces arepassed by reference—that is, passing an interface passes an invocationhandle via which an object can be invoked remotely.

對(duì)象模型。Ice將其對(duì)象模型限制到了最小化。內(nèi)置數(shù)據(jù)類型僅限于有符號(hào)整數(shù)、浮點(diǎn)數(shù)、布爾型、Unicode字符串和8位無解釋（二進(jìn)制）字節(jié)。用戶定義的類型包括常量、枚舉、結(jié)構(gòu)、序列、字典，以及帶繼承的異常。遠(yuǎn)程對(duì)象建模為帶多重繼承的接口，而接口包含具有輸入輸出參數(shù)和返回值的各種操作。接口通過引用傳遞，也就是說，傳遞接口會(huì)傳遞一個(gè)調(diào)用句柄，通過該句柄，就可以遠(yuǎn)程調(diào)用一個(gè)對(duì)象。

To support client-side behavior and object migration, we added classes: operationinvocations on a class execute in the client’s address space (instead ofthe server’s, as is the case for interfaces). In addition, classes canhave state (whereas interfaces, at the object-modeling level, are always stateless).Classes are passed by value—that is, passing a class instance passes thestate of the class instead of a handle to a remote object.

為了支持客戶端行為和對(duì)象遷移，我們添加了類：類上調(diào)用的操作執(zhí)行于客戶端的地址空間（而不是就接口而言的服務(wù)器端）。此外，類可以有狀態(tài)（而接口，在對(duì)象建模的層次，總是無狀態(tài)的）。類是按值傳遞的，也就是說，傳遞類實(shí)例會(huì)將該類的狀態(tài)傳遞到遠(yuǎn)程對(duì)象，而不是傳遞句柄。

We did not attempt to pass behavior: This would require a virtual execution environmentfor objects but would be in conflict with our performance and multi-languagerequirements. Instead, we implemented identical behavior for a class at all itspossible host locations (clients and servers): Rather than shipping code around,we provide the code wherever it is needed and ship only the state. To migratean object, a process passes a class instance to another process and then destroysits copy of the instance; semantically, the effect is the same as migrating bothstate and behavior.

我們沒有試圖傳遞行為：這將要求為對(duì)象建立一個(gè)虛擬執(zhí)行環(huán)境，這與我們的性能和多語言需求相沖突。相反，我們實(shí)現(xiàn)了類在其所有可能的主機(jī)位置（客戶端和服務(wù)器）上具有同一行為：我們不是到處分發(fā)代碼，而是在有需要的地方提供代碼，并且僅僅分發(fā)狀態(tài)。為了遷移對(duì)象，進(jìn)程傳遞類實(shí)例到另一進(jìn)程，然后消毀自己的實(shí)例拷貝；在語義上，其效果與同時(shí)遷移狀態(tài)和行為是相同的。

Architecturally, implementing object migration in this way is a two-edged swordbecause it requires all host locations to implement identical (as opposed tomerely similar) behavior. This has ramifications for versioning: If we changethe behavior of a class at one host location, we must change the behavior ofthat class at all other locations (or suffer inconsistent behavior). Multiplelanguages also require attention. For example, if a class instance passes froma C++ server to a Java client, we must provide C++ and Java implementations withidentical behavior. (Obviously, this requires more effort than implementing thebehavior just once in a single language and single server.)

在架構(gòu)上，以這種方式實(shí)現(xiàn)對(duì)象遷移是一把雙刃劍，因?yàn)樗笏械闹鳈C(jī)位置實(shí)現(xiàn)同一行為（而非僅僅類似）。這對(duì)版本控制有復(fù)雜的影響：如果我們?cè)谝粋€(gè)主機(jī)位置改變類的行為，我們必須改變那個(gè)類在所有其他位置的行為（否則就會(huì)有不一致的行為）。多語言也要注意。比如，如果類實(shí)例從C++服務(wù)器傳遞到Java客戶端，我們必須提供具有同一行為的C++和Java實(shí)現(xiàn)。（顯然，相比用單一語言，在單一服務(wù)器上，僅僅實(shí)現(xiàn)一次行為，這需要更多的努力。）

For environments such as Wish, where we control both client and server deployment,this is acceptable; for applications that provide only servers and rely on otherparties to provide clients, this can be problematic because ensuring identicalbehavior of third-party class implementations is difficult.

對(duì)于像Wish這樣的環(huán)境，我們控制了客戶端和服務(wù)器的部署，這是可以接受的；但對(duì)于只提供服務(wù)器，而依靠其他各方提供客戶端的情況，這就有問題，因?yàn)楹茈y確保第3方類的實(shí)現(xiàn)具有同一行為。

Protocol Design. To meet our performance goals, we broke with established wisdomfor RPC protocols in two ways:

協(xié)議設(shè)計(jì)。為了滿足性能目標(biāo)，我們摒棄了RPC協(xié)議中的2個(gè)既定觀念：

• Data is not tagged with its type on the wire and is encoded as compactly as possible:The encoding uses no padding (everything is byte-aligned) and applies a numberof simple techniques to save bandwidth. For example, positive integers less than255 require a single byte instead of four bytes, and strings are not NUL terminated.This encoding is more compact (sometimes by a factor of two or more, dependingon the type of data) than CORBA’s CDR (common data representation) encoding.

• 數(shù)據(jù)傳輸時(shí)沒有類型標(biāo)記，并且盡可能的以緊湊格式編碼：編碼沒有填充（結(jié)構(gòu)都以1字節(jié)對(duì)齊），并且使用了一些簡單的技術(shù)節(jié)省帶寬。例如，小于255的正整數(shù)需要1個(gè)字節(jié)而不是4字節(jié)，字符串不以NUL結(jié)尾。這樣的編碼比CORBA的CDR（common data representation，通用數(shù)據(jù)表示）編碼更緊湊（有時(shí)可達(dá)到2倍以上，視數(shù)據(jù)類型而定）。

• Data is always marshaled in little-endian byte order. We rejected a receiver-makes-it-rightapproach (as used by CORBA) because experiments showed no measurable performancegain.

• 數(shù)據(jù)總是以little-endian字節(jié)序組編（marshal）。我們不使用receiver-makes-it-right（接收者讓它正確）方法（它用于CORBA），因?yàn)閷?shí)驗(yàn)表明該方法沒有可觀的性能提升。

The protocol supports compression for better performance over low-speed links.(Interestingly, for high-speed links, compression is best disabled: It takesmore time to compress data than to send it uncompressed.)

協(xié)議支持壓縮，以在低速鏈路上有更好的性能。（有趣的是，對(duì)高速鏈路，最好禁用壓縮：壓縮數(shù)據(jù)太費(fèi)時(shí)，還不如不壓縮傳輸。）

The protocol encodes request data as a byte count followed by the payload asa blob. This allows the receiver of a message to forward it to a number of downstreamreceivers without the need to unmarshal and remarshal the message. Avoiding thiscost was important so we could build efficient message switches for event distribution.

協(xié)議將請(qǐng)求數(shù)據(jù)編碼為一個(gè)字節(jié)計(jì)數(shù)加一個(gè)blob凈荷。這允許消息接收者向下游接收者轉(zhuǎn)發(fā)消息，而無需解編（unmarshal）和重新組編（remarshal）。避免這種消耗很重要，這樣我們才能為事件分發(fā)建立高效的消息分派。

The protocol supports TCP/IP and UDP (user datagram protocol). For secure communications,we use SSL (secure sockets layer): It is freely available and has been extensivelyscrutinized for flaws by the security community.

協(xié)議支持TCP/IP和UDP（user datagram protocol，用戶數(shù)據(jù)報(bào)協(xié)議)。為了安全通信，我們使用SSL（secure sockets layer，安全套接字層）：它是免費(fèi)的，并且安全社區(qū)已經(jīng)對(duì)它進(jìn)行了廣泛的檢驗(yàn)。

The protocol is bidirectional, so a server can make a callback over a connectionthat was previously established by a client. This is important for communicationthrough firewalls, which usually permit outgoing connections, but not incomingones. The protocol also works across NAT boundaries.

協(xié)議是雙向的，所以服務(wù)器可以在客戶端先前建立的連接上進(jìn)行回調(diào)。這對(duì)穿越防火墻的通信很重要，防火墻通常允許向外的連接，而不能被連入。該協(xié)議同時(shí)還能穿越NAT邊界。

Classes make the protocol more complex because they are polymorphic: If a processsends a derived instance to a receiver that understands only a base type of thatinstance, the Ice runtime slices the instance to the most-derived base type thatis known to the receiver. Slicing requires the receiver to unmarshal data whosetype is unknown. Further, classes can be self-referential and form arbitrarygraphs of nodes: Given a starting node, the Ice runtime marshals all reachablenodes so graphs require the sender to perform cycle detection.

類使得協(xié)議更復(fù)雜，因?yàn)轭愂嵌鄳B(tài)的：如果進(jìn)程發(fā)送了一個(gè)派生類的實(shí)例給接收者，而接收者只知道該實(shí)例的基類的話，Ice運(yùn)行庫（runtime）會(huì)將該實(shí)例剪切（slice）為接收者所知道的最近基類。剪切要求接收者解編未知類型的數(shù)據(jù)。而且，類可以是自引用的（self-referential），并形成任意的節(jié)點(diǎn)圖：給定一個(gè)起始節(jié)點(diǎn)，Ice運(yùn)行庫會(huì)組編所有可達(dá)的節(jié)點(diǎn)，因此節(jié)點(diǎn)圖要求發(fā)送者執(zhí)行環(huán)檢測(cè)。

The implementation of slicing and class graphs is surprisingly complex. To supportunmarshaling, the protocol sends classes as individually encapsulated slices,each tagged with their type. On average (compared with structures), this requires10 to 15 percent extra bandwidth. To preserve the identity relationships of nodesand to detect cycles, the marshaling code creates additional data structures.On average, this incurs a performance penalty of 5 to 10 percent. Finally, forC++, we had to write a garbage collector to avoid memory leaks in the presenceof cyclic class graphs, which was nontrivial. Without slicing and class graphs,the protocol implementation would have been simpler and (for classes) slightlyfaster.

剪切和類圖的實(shí)現(xiàn)異常復(fù)雜。為了支持解編，協(xié)議按單獨(dú)封裝的剪切片發(fā)送類，每個(gè)都標(biāo)記其類型。平均而言（對(duì)比結(jié)構(gòu)），這需要10%-15%的額外帶寬。為了保存節(jié)點(diǎn)關(guān)系標(biāo)識(shí)，也為了檢測(cè)環(huán)，組編代碼會(huì)生成附加的數(shù)據(jù)結(jié)構(gòu)。平均而言，這會(huì)產(chǎn)生5%-10%的性能損耗。最后，對(duì)于C++，我們必須編寫垃圾收集器，以避免環(huán)形類圖表示中的內(nèi)存泄漏，這也不容易。如果沒有剪切和類圖，協(xié)議實(shí)現(xiàn)將更簡單，（對(duì)類來說）也略微更快些。

Versioning. The object model supports multiple interfaces: Instead of havinga single most-derived interface, an object can provide any number of interfaces.Given a handle to an object, clients can request a specific interface at runtimeusing a safe downcast. Multiple interfaces permit versioning of objects withoutbreaking on-the-wire compatibility: To create a newer version, we add new interfacesto existing objects. Already-deployed clients continue to work with the old interfaces,whereas new clients can use the new interfaces.

版本控制。對(duì)象模型支持多接口：對(duì)象可以提供任意多的接口，而不是單一的最終接口。給定一個(gè)對(duì)象的句柄，客戶端可以在運(yùn)行時(shí)利用安全的向下轉(zhuǎn)型，來請(qǐng)求一個(gè)特定接口。多接口允許對(duì)象的版本控制，而不會(huì)破壞已上線對(duì)象的兼容性：要?jiǎng)?chuàng)建一個(gè)新的版本，我們會(huì)在現(xiàn)有對(duì)象上添加新的接口。已經(jīng)部署的客戶端使用舊接口繼續(xù)運(yùn)行，而新客戶端可以使用新接口。

Used naively, multiple interfaces can lead to a versioning mess that forces clientsto continuously choose the correct version. To avoid these problems, we designedthe game such that clients access it via a small number of bootstrap objectsfor which they choose an interface version. Thereafter, clients acquire handlesto other objects via their chosen interfaces on bootstrap objects, so the desiredversion is known implicitly to the bootstrap object. The Ice protocol providesa mechanism for implicit propagation of contextual information such as versioning,so we need not pollute all our object interfaces by adding an extra version parameter.

如果盲目使用，多接口可以導(dǎo)致版本混亂，使得客戶端要不斷地選擇正確版本。為避免這些問題，我們這樣設(shè)計(jì)游戲：讓客戶端通過幾個(gè)引導(dǎo)對(duì)象訪問接口，客戶端只需為引導(dǎo)對(duì)象選擇一個(gè)接口版本。因此，客戶端通過其所選的引導(dǎo)對(duì)象接口，來獲得其他對(duì)象的句柄，引導(dǎo)對(duì)象完全知道需要的版本。對(duì)于比如版本這樣的上下文信息，Ice協(xié)議提供了一個(gè)隱式傳播機(jī)制，我們不必為所有對(duì)象的接口都添加額外的版本參數(shù)。

Multiple interfaces reduced development time of the game because, apart fromversioning, they allowed us to use loose coupling at the type level between clientsand servers. Instead of modifying the definition of an existing interface, wecould add new features by adding new interfaces. This reduced the number of dependenciesacross the system and shielded developers from each others’ changes andthe associated compilation avalanches that often ensue.

多接口縮短了游戲開發(fā)時(shí)間，因?yàn)椋税姹局猓嘟涌谶€允許我們?cè)诳蛻舳撕头?wù)器之間的類型級(jí)別中使用松耦合。不修改現(xiàn)有接口的定義，我們就可以通過添加新接口來增加新功能。這減少了整個(gè)系統(tǒng)的依賴數(shù)量，并且屏蔽了開發(fā)者相互間的變更，以及由此帶來的相關(guān)的編譯雪崩（compilation avalanche）。

On the downside, multiple interfaces incur a loss of static type safety becauseinterfaces are selected only at runtime, which makes the system more vulnerableto latent bugs that can escape testing. When used judiciously, however, multipleinterfaces are useful in combating the often excessively tight coupling of traditionalRPC approaches.

多接口的缺點(diǎn)是，它招致了靜態(tài)類型安全性的損失，因?yàn)榻涌谥辉谶\(yùn)行時(shí)被選擇，這使得系統(tǒng)更容易受到潛伏錯(cuò)誤的傷害，即那些逃過測(cè)試的錯(cuò)誤。但是，如果被明智地使用，相比傳統(tǒng)RPC方法中往往過緊的耦合，多接口更為有用。

Ease of Use. Ease of use is an overriding design goal. On the one hand, thismeans that we keep the runtime APIs as simple and small as possible. For example,29 lines of specification are sufficient to define the API to the Ice objectadapter. Despite this, the object adapter is fully functional and supports flexibleobject implementations, such as separate servant per object, one-to-many mappingsof servants to objects, default servants, servant locators, and evictors. Byspending a lot of time on the design, we not only kept the APIs small, but alsoreaped performance gains as a result of smaller code and working set sizes.

易用性。易用性是設(shè)計(jì)的首要目標(biāo)。一方面，這意味著我們要保持運(yùn)行庫API盡可能的簡單，盡可能的小。比如，29行的說明足以定義Ice對(duì)象適配器（object adapter）的API。盡管如此，對(duì)象適配器是功能完善的，并且支持靈活的對(duì)象實(shí)現(xiàn)，比如每個(gè)對(duì)象有單獨(dú)的servant（服務(wù)者）、servant與對(duì)象是一對(duì)多關(guān)系、默認(rèn)servant、servant locator（服務(wù)者定位器），和evictor（逐出器）。通過長時(shí)間的設(shè)計(jì)，我們不僅使API很小，而且由于更小的代碼和工作集而獲得了性能提升。

On the other hand, we want language mappings that are simple and intuitive. Limitingourselves to a small object model paid off here—fewer types mean less generatedcode and smaller APIs.

另一方面，我們想要簡單直觀的語言映射。我們把自己限制為小型對(duì)象模型，在這里得到了補(bǔ)償：更少的類型意味著更少的生成代碼和更小的API。

The C++ mapping is particularly important: From CORBA, we knew that a poorlydesigned mapping increases development time and defect count, and we wanted somethingsafer. We settled on a mapping that is small (documented in 40 pages) and providesa high level of convenience and safety. In particular, the mapping is integratedwith the C++ standard template library, is fully threadsafe, and requires nomemory management. Developers never need to deallocate anything, and exceptionscannot cause memory leaks.

C++映射尤其重要：從CORBA我們知道，設(shè)計(jì)不良的映射會(huì)增加開發(fā)時(shí)間和缺陷數(shù)，而我們想要更安全的東西。我們最終決定的映射很小（文檔只有40頁），并且提供了高水平的便利性和安全性。尤其是，映射集成了C++標(biāo)準(zhǔn)庫，是完全線程安全的，并且不需要內(nèi)存管理。開發(fā)者不用釋放任何東西，并且異常也不會(huì)引起內(nèi)存泄漏。

One issue we repeatedly encounter for language mappings is namespace collision.Each language has its own set of keywords, library namespaces, and so on. Ifthe (language-independent) object model uses a name that is reserved in a particulartarget language, we must map around the resulting collision. Such collisionscan be surprisingly subtle and confirmed, yet again, that API design (especiallygeneric API design, such as for a language mapping) is difficult and time consuming.The choice of the trade-off between ease of use and functionality also can becontentious (such as our choice to disallow underscores in object-model identifiersto create a collision-free namespace).

語言映射中，我們多次遇到的一個(gè)的問題是，名字空間沖突。每個(gè)語言都有它們自己的一套關(guān)鍵字、庫名字空間，等等，如果（語言無關(guān)的）對(duì)象模型使用了一個(gè)特定目標(biāo)語言中保留的名字，我們必須重新映射來繞過產(chǎn)生的沖突。這種沖突會(huì)出奇的微妙，并且反復(fù)地出現(xiàn)，以致于API設(shè)計(jì)費(fèi)時(shí)又費(fèi)力（特別是通用API設(shè)計(jì)，比如語言映射）。在易用性和功能性之間權(quán)衡的選擇也可能是有爭議的（比如我們選擇不允許對(duì)象模型的標(biāo)識(shí)符用下劃線來產(chǎn)生不沖突的名字空間）。

Persistence. To provide object persistence, we extended the object model to permitthe definition of persistence attributes for objects. To the developer, makingan object persistent consists of defining those attributes that should be storedin the database. A compiler processes these definitions and generates a runtimelibrary that implements associative containers for each type of object.

持久化。為了提供對(duì)象持久化，我們擴(kuò)展了對(duì)象模型，允許為對(duì)象定義持久化屬性。對(duì)開發(fā)者來說，讓對(duì)象持久化就是定義那些應(yīng)該存儲(chǔ)在數(shù)據(jù)庫中的屬性。編譯器處理這些定義，并生成運(yùn)行時(shí)庫，為每種類型的對(duì)象實(shí)現(xiàn)關(guān)聯(lián)容器。

Developers access persistent objects by looking them up in a container by theirkeys—if an object is not yet in memory, it is transparently loaded fromthe database. To update objects, developers simply assign to their state attributes.Objects are automatically written to the database by the Ice runtime. (Variouspolicies can be used to control under what circumstances a physical databaseupdate takes place.)

開發(fā)者訪問持久對(duì)象時(shí)，要在容器中用其鍵值查找它們，如果對(duì)象尚未在內(nèi)存中，它會(huì)透明地從數(shù)據(jù)庫加載。當(dāng)更新對(duì)象時(shí)，開發(fā)者只要對(duì)它們的狀態(tài)屬性賦值。Ice運(yùn)行庫會(huì)自動(dòng)地把對(duì)象寫入數(shù)據(jù)庫（可以使用各種策略來控制在何種情況下執(zhí)行物理數(shù)據(jù)庫更新）。

This model makes database access completely transparent. For circumstances inwhich greater control is required, a small API allows developers to establishtransaction boundaries and preserve database integrity.

這個(gè)模型使得數(shù)據(jù)庫訪問完全透明。對(duì)于需要更多控制的情況來說，小型的API允許開發(fā)者建立事務(wù)的邊界并維護(hù)數(shù)據(jù)庫的完整性。

To allow us to change the game without continuously having to migrate databasesto new schemas, we developed a database transformation tool. For simple featureadditions, we supply the tool with the old and new object definitions—thetool automatically generates a new database schema and migrates the contentsof the old database to conform to the new schema. For more complex changes, suchas changing the name of a structure field or changing the key type of a dictionary,the tool creates a default transformation script in XML that a developer canmodify to implement the desired migration action.

為了允許我們改變游戲，但不必反復(fù)地遷移數(shù)據(jù)庫到新的模式，我們開發(fā)了一個(gè)數(shù)據(jù)庫轉(zhuǎn)換工具。對(duì)于簡單的功能添加，我們只要向工具提供新舊對(duì)象的定義，工具會(huì)自動(dòng)地生成一個(gè)新的數(shù)據(jù)庫模式，并遷移舊庫的內(nèi)容以符合新的模式。對(duì)更復(fù)雜的變更，比如改變結(jié)構(gòu)的字段名，或者改變字典鍵值的類型，工具會(huì)創(chuàng)建一個(gè)默認(rèn)的XML轉(zhuǎn)換腳本，開發(fā)者可以修改它以實(shí)現(xiàn)所需的遷移動(dòng)作。

This tool has been useful, although we keep thinking of new features that couldbe incorporated. As always, the difficulty is in knowing when to stop: The temptationto build better tools can easily detract from the overall project goals. (“Insideevery big program is a little program struggling to get out.”)

這個(gè)工具很有用，盡管我們不斷地想到還可以納入的新功能。就像通常一樣，困難在于知道何時(shí)停止：建立更好的工具，這個(gè)誘惑很容易降低整個(gè)項(xiàng)目的目標(biāo)。（“Inside every big program is a little program struggling to get out.”，“在每個(gè)大程序中，都有一個(gè)小程序掙扎著要出來。”）

Threading. We built a portable threading API that provides developers with platform-independentthreading and locking primitives. For remote call dispatch, we decided to supportonly a leader/followers threading model.4 In some situations, in which a blockingor reactive model would be better suited, this decision cost us a little in performance,but it gained us a simpler runtime and APIs and reduced the potential for deadlockin nested RPCs.

多線程。我們建立了一個(gè)可移植的線程API，給開發(fā)者提供了平臺(tái)無關(guān)的線程和鎖原語。對(duì)于遠(yuǎn)程調(diào)用派發(fā)，我們決定僅支持leader/follower（領(lǐng)導(dǎo)者/跟隨者）線程模型[4]。在阻塞或者反應(yīng)模式更合適的情況下，這個(gè)決定使我們損失了一點(diǎn)性能，但是它讓我們得到了更簡單的運(yùn)行庫和API，并且減少了嵌套R(shí)PC調(diào)用中死鎖的可能性。

Scalability. Ice permits redundant implementations of objects in different servers.The runtime automatically binds to one of an object’s replicas and, ifa replica becomes unavailable, fails over to another replica. The binding informationfor replicas is kept in configuration and is dynamically acquired at runtime,so adding a redundant server requires only a configuration update, not changesin source code. This allows us to take down a game server for a software upgradewithout having to kick all players using that server out of the game. The samemechanism also provides fault tolerance in case of hardware failure.

擴(kuò)展性。Ice允許在不同的服務(wù)器上冗余的對(duì)象實(shí)現(xiàn)。運(yùn)行庫會(huì)自動(dòng)綁定到一個(gè)對(duì)象副本（replica），并且，如果副本不可用時(shí)，會(huì)自動(dòng)切換（fail over）到另一個(gè)副本。副本的綁定信息保存在配置中，并在運(yùn)行時(shí)動(dòng)態(tài)獲得，所以添加冗余服務(wù)器只需更新配置，不用修改源碼。這允許我們卸下游戲服務(wù)器做軟件升級(jí)，而不必把所有使用這個(gè)服務(wù)器的玩家都踢出游戲。同樣的機(jī)制還提供了硬件故障時(shí)的容錯(cuò)能力。

To support federating logical functions across a number of servers and to shareload, we built an implementation repository that delivers binding informationto clients at runtime. A randomizing algorithm distributes load across any numberof servers that form a logical service.

為了支持在多臺(tái)服務(wù)器上聯(lián)盟（federate)邏輯函數(shù)，以分擔(dān)負(fù)載，我們建立了一個(gè)實(shí)現(xiàn)庫（implementation repository），它會(huì)在運(yùn)行時(shí)傳送綁定信息到客戶端。一個(gè)隨機(jī)算法會(huì)分配負(fù)載到任意數(shù)量的服務(wù)器，讓它們組成一個(gè)邏輯服務(wù)。

We made a number of trade-offs for replication and load sharing. For example,not all game components can be upgraded without server shutdown, and a load feedbackmechanism would provide better load sharing than simple randomization. Givenour requirements, these limitations are acceptable, but, for applications withmore stringent requirements, this might not be the case. The skill is in decidingwhen not to build something as much as when to build it—infrastructuremakes no sense if the cost of developing it exceeds the savings during its use.

我們?yōu)閺?fù)制和負(fù)載共享作出了若干權(quán)衡。比如，不是所有的游戲組件都能在線升級(jí)，以及負(fù)載反饋機(jī)制將比簡單的隨機(jī)化提供更優(yōu)的負(fù)載共享。鑒于我們的需求，這些限制是可以接受的，但是，對(duì)于需求更嚴(yán)格的應(yīng)用，可能并非如此。技巧在于決定何時(shí)不要建立某個(gè)東西，其重要性等同于決定何時(shí)要建立它：如果基礎(chǔ)設(shè)施的開發(fā)成本超過使用它所節(jié)省的成本，這就毫無意義了。

SIMPLE IS BETTER

簡單就好

Our experiences with Ice during game development have been very positive. Despiterunning a distributed system that involves dozens of servers and thousands ofclients, the middleware has not been a performance bottleneck.

在游戲開發(fā)中，我們對(duì)Ice的感覺是非常肯定的。雖然運(yùn)行的是包含幾十臺(tái)服務(wù)器和數(shù)千客戶端的分布式系統(tǒng)，但中間件并不是性能瓶頸。

Our focus on simplicity during design paid off many times during development.When it comes to middleware, simpler is better: A well-chosen and small featureset contributes to timely development, as well as to meeting performance goals.

在設(shè)計(jì)中對(duì)簡單性的專注，讓我們?cè)陂_發(fā)中贏得了數(shù)倍的回報(bào)。對(duì)于中間件，簡單就好：一個(gè)精心挑選并且小型化的功能集，有利于按時(shí)開發(fā)，以及達(dá)到性能目標(biāo)。

Finally, designing and implementing middleware is difficult and costly, evenwith many years of experience. If you are looking for middleware, chances arethat you will be better off buying it than building it.

最后，設(shè)計(jì)和實(shí)現(xiàn)中間件是困難和昂貴的，即使對(duì)于有多年經(jīng)驗(yàn)的老手。如果你正在尋找中間件，很有可能你最好是購買一個(gè)而不是構(gòu)建一個(gè)。

REFERENCES

參考

1. Mutable Realms (Wish home page): see http://www.mutablerealms.com.

2. Henning, M., and S. Vinoski. Advanced CORBA Programming with C++. Addison-Wesley,Reading: MA, 1999.

3. ZeroC. Distributed Programming with Ice: see http://www.zeroc.com/Ice-Manual.pdf.

4. Schmidt, D. C., O’Ryan, C., Pyarali, I., Kircher, M., and Buschmann,F. Leader/ Followers: A design pattern for efficient multithreaded event demultiplexingand dispatching. Proceedings of the 7th Pattern Languages of Programs Conference(PLoP 2000); http://deuce.doc.wustl.edu/doc/pspdfs/lf.pdf.

MICHI HENNING (michi@zeroc.com) is chief scientist of ZeroC. From 1995 to 2002,he worked on CORBA as a member of the Object Management Group’s ArchitectureBoard and as an ORB implementer, consultant, and trainer. With Steve Vinoski,he wrote Advanced CORBA Programming with C++ (Addison-Wesley, 1999), the definitivetext in the field. Since joining ZeroC, he has worked on the design and implementationof Ice and in 2003 coauthored “Distributed Programming with Ice” forZeroC. He holds an honors degree in computer science from the University of Queensland,Australia.

    Originally published in Queue vol. 1, no. 10—    
    see this item in the ACM Digital Library