Massively Multiplayer Middleware
大型多人游戲中間件
MICHI HENNING, ZeroC

Building scaleable middleware for ultra-massive online games teaches a lessonwe all can use: Big project, simple design.

為超大型在線游戲構建可擴展的中間件給我們所有人的教訓：宏大的項目，簡單的設計。

Wish is a multiplayer, online, fantasy role-playing game being developed by MutableRealms.1 It differs from similar online games in that it allows tens of thousandsof players to participate in a single game world (instead of the few hundredplayers supported by other games). Allowing such a large number of players requiresdistributing the processing load over a number of machines and raises the problemof choosing an appropriate distribution technology.

Wish游戲是Mutable Realms[1]開發的幻想類多人在線角色扮演游戲。不同于類似的在線游戲，它允許數萬的玩家同時參與到一個游戲世界中（而其他游戲僅支持幾百人）。允許如此之多的玩家就要求在多臺機器上進行分布式處理，由此產生了一個問題：選擇合適的分布式技術。

DISTRIBUTION REQUIREMENTS

分布式需求

Mutable Realms approached ZeroC for the distribution requirements of Wish. ZeroCdecided to develop a completely new middleware instead of using existing technology,such as CORBA (Common Object Request Broker Architecture).2 To understand themotivation for this choice, we need to examine a few of the requirements placedon middleware by games on the scale of Wish and other large-scale distributedapplications.

Mutable Realms帶著Wish游戲的分布式需求找到ZeroC。ZeroC決定開發一個全新的中間件，而不是使用現有的技術，比如CORBA（Common Object Request Broker Architecture，公共對象請求代理體系）[2]。為了理解這個選擇的動機，我們需要考察像Wish這種規模的游戲和其他大規模分布式應用程序對中間件的一些要求。

Multi-Platform Support. The dominant platform for the online games market isMicrosoft Windows, so the middleware has to support Windows. For the server side,Mutable Realms had early on decided to use Linux machines: The low cost of theplatform, together with its reliability and rich tool support, made this an obviouschoice. The middleware, therefore, had to support both Windows and Linux, withpossible later support for Mac OS X and other Unix variants.

多平臺支持。在線游戲市場的主要平臺是Microsoft Windows，所以中間件必須支持Windows。在服務器這邊，Mutable Realms很早就決定使用Linux：費用低，并且可靠性高，以及工具軟件豐富，這是明擺著的選擇。因此，中間件必須要同時支持Windows和Linux，也許以后還要支持Mac OS X和其他Unix變種。

Multi-Language Support. Client and server software is written in Java, as wellas a combination of C++ and assembly language for performance-critical functions.At ZeroC we used Java because some of our development staff had little priorC++ experience. Java also offers advantages in terms of defect count and developmenttime; in particular, garbage collection eliminates the memory management errorsthat often plague C++ development. For administration of the game via the Web,we wanted to use the PHP hypertext processor. As a result, the game middlewarehad to support C++, Java, and PHP.

多語言支持。客戶端和服務器軟件是用Java編寫的，同時，對于性能關鍵的函數還混合使用了C++和匯編語言。在ZeroC，我們使用Java，因為我們的部分開發人員先前沒什么C++經驗。Java同時在缺陷數和開發時間上有優勢；特別是垃圾收集消除了內存管理錯誤，而這是C++開發的死疾。為了通過Web管理游戲，我們想用PHP。所以，這個游戲中間件必須支持C++、Java和PHP。

Transport and Protocol Support. As we developed the initial distribution architecturefor the game, it became clear that we were faced with certain requirements interms of the underlying transports and protocols:

傳輸和協議支持。當我們為游戲開發最初的分布式構架時，我們明顯面臨了對底層傳輸和協議的一些需求：

• Players connect to ISPs via telephone lines, as well as broadband links. Whilebroadband is becoming increasingly popular, we had decided early on that thegame had to be playable over an ordinary modem. This meant that communicationsbetween clients and server had to be possible via low-bandwidth and high-latencylinks.

• 玩家通過電話線或者寬帶連接ISP上網。雖然寬帶越來越普及，但是我們早已決定游戲必須能通過普通modem玩。這意味著客戶端服務器之間通過低帶寬高延時鏈路進行通訊必須是可能的。

• Much of the game is event driven. For example, as a player moves around, otherplayers in the same area need to be informed of the changes in the game worldaround them. These changes can be distributed as simple events such as, “PlayerA moves to new coordinates <x,y>.”

• 游戲大部分是事件驅動的。比如，當玩家四處溜達時，這些變化需要通知同一區域內的其他玩家，使他們知曉其周圍的游戲世界中的變化。這些變化可以作為簡單的事件分發，比如“玩家A移動到新坐標<x,y>”。

Ideally, events are distributed via “datagrams.” If the occasionalstate update is lost, little harm is done: A lost event causes a particular observer’sview of the game world to lag behind momentarily, but that view becomes up-to-dateagain within a very short time, when another event is successfully delivered.

理想的，事件是通過數據報（datagram）分發的。即使偶爾有狀態更新丟失，也沒什么危害：一個事件丟失會造成某個觀察者對于游戲世界的視圖暫時性滯后，但很快，當另一個事件成功傳送后，該視圖又會變成最新狀態。

• Events in the game often have more than one destination. For example, if a playermoves within the field of vision of five other players, the same positional updatemust be sent to all five observing players. We wanted to be able to use broadcastor multicast to support such scenarios.

• 游戲中的事件經常有多個目標。比如，玩家在其他5個玩家的視野內移動，位置更新必須發送給所有這5個看到他的玩家。我們想用廣播或多播來支持這樣的場景。

• Communications between clients and game servers must be secure. For an onlinesubscription-based game, this is necessary for revenue collection, as well asto prevent cheating. (For example, it must be impossible for a player to acquirea powerful artifact by manipulating the client-side software.)

• 客戶端和游戲服務器之間的通信必須是安全的。對基于訂購的在線游戲，這是必須的，既為了賺錢，也為了防做弊（例如，玩家無法通過修改客戶端軟件來得到超級裝備）。

• Clients connect to the game from LANs that are behind firewalls and use NAT (networkaddress translation). The communications protocol for the game has to be designedin a way that accommodates NAT without requiring knowledge of application-specificinformation in order to translate addresses.

• 客戶端能在局域網內部通過防火墻和NAT（network address translation，網絡地址轉換）連入游戲。游戲的通信協議必須按以下方式設計：它能夠穿越NAT，但是地址轉換無需應用程序特有的相關信息。

Versioning Support. We wanted to be able to update the game world while the gamewas being played—for example, to add new items or quests. These updateshave to be possible without requiring every deployed client to be upgraded immediately—thatis, client software at an older revision level has to continue to work with theupdated game servers (albeit without providing access to newly added features).This means that the type system has to be flexible enough to allow updates, suchas adding a field to a structure or changing the signature of a method, withoutbreaking deployed clients.

版本控制支持。我們需要能夠在游戲運行中更新游戲，比如，增加新的物品或任務。這些更新不要求所有已部署的客戶端都馬上升級，即，舊版本的客戶端軟件必須能夠連接已升級的服務器繼續運行（雖然不提供新增功能）。這意味著類型系統必須要足夠靈活以允許更新，比如給結構添加新域，或者更改方法的簽名，而不會使已部署的客戶端無法使用。

Ease of Use. Although a few of the Wish game developers are distributed computingexperts, the majority have little or no experience. This means that the middlewarehas to be easy for nonexperts to use, with simple, threadsafe and exception-safeAPIs (application programming interfaces).

易用性。雖然有些Wish游戲的開發者是分布式計算的專家，但是大部分經驗不足甚至沒有經驗。這意味著中間件必須讓非專家也能容易使用，具有簡單的、線程安全的，和異常安全的API（application programming interface，應用程序接口）。

Persistence. Much of the game requires state, such as the inventory for eachplayer, to be stored in a database. We wanted to provide developers with a wayto store and retrieve persistent state for application objects without havingto concern themselves with the actual database and without having to design databaseschemas. Particularly during development, as the game evolves, it is prohibitivelytime consuming to repeatedly redesign schemas to accommodate changes. In addition,as we improve the game while being deployed, we must add new features to a databaseand remove older features from it. We wanted an automatic way to migrate an existing,populated database to a new database schema without losing any of the informationin the old database that was still valid.

持久化。很多游戲都需要將狀態保存數據庫，如玩家的所有物品。我們需要給開發者提供一種方法，使之可以保存和獲取應用對象的持久化狀態，但不必關心具體的數據庫，也不必設計數據庫模式（database schema）。特別是開發過程中，當游戲不斷演化，為了適應變化而反復地修改表結構，所耗的時間會讓人望而卻步。而且，在游戲發布后，當我們改進游戲時，必須添加新的屬性到數據庫，以及刪除舊的屬性。我們需要一種自動的方式，將現存的已填充的數據庫遷移到新的數據庫模式，而不丟失舊庫中任何仍然有效的信息。

Threading. Much of the server-side processing is I/O-bound: Database and networkaccess forces servers to wait for I/O completion. Other tasks, such as pathfinding,are compute-bound and can best be supported using parallel algorithms. This meansthat the middleware has to be inherently threaded and offer developers sufficientcontrol over threading strategies to implement parallel algorithms while preventingproblems such as thread starvation and deadlock. Given the idiosyncrasies ofthreading on different operating systems, we also wanted a platform-neutral threadingmodel with a portable API.

多線程。大多數服務端處理都是I/O密集型：數據庫和網絡訪問迫使服務器等待I/O完成。其他任務，比如尋路，是計算密集型，最好使用并行算法。這意味著中間件必須內在地多線程化，并且為開發者提供足夠的對線程策略的控制，以實現并行算法，同時避免如線程饑餓和死鎖這樣的問題。考慮到不同操作系統上的線程特性，我們需要平臺無關的線程模型，并具有可移植的API。

Scalability. Clearly, the most serious challenges for the middleware are in thearea of scalability: For an online game, predicting realistic bounds is impossibleon things such as the total number of subscribers or the number of concurrentplayers. This means that we need an architecture that can be scaled by federatingservers (that is, adding more servers) as demands on the software increase.

可擴展性。顯然，對中間件最重要的挑戰是在可擴展性方面：對于在線游戲，不可能預測到如訂閱總數或同時在線玩家數這類數據的真實范圍。這意味著我們需要這樣的架構，當對軟件的需求增加時，它可以聯盟（federate）服務器來進行擴展（即添加更多服務器）。

We also need fault-tolerance: For example, upgrading a server to a newer versionof the game software has to be possible without kicking off every player currentlyusing that server. The middleware has to be capable of automatically using areplica server while the original server is being upgraded.

我們也需要容錯性：比如，將服務器升級到游戲軟件的新版本必須是可能的，而不應該踢掉當前正在使用該服務器的每個玩家。中間件必須能夠在原服務器正在升級時，自動使用一個副本服務器（replica server）。

Other scalability issues relate to resource management. For example, we did notwant to be subject to hardwired limits, such as a maximum number of open connectionsor instantiated objects. This means that, wherever possible, the middleware hasto provide automated resource management functions that are not subject to arbitrarylimits and are easy to use. Simultaneously, these functions have to provide enoughcontrol for developers to tune resource management to their needs. Wherever possible,we wanted to be able to change resource management strategies without requiringrecompilation.

其他可擴展性問題涉及資源管理。例如，我們不想受到硬限制，如最大打開連接數，或最大實例化對象數。這意味著，只要有可能，中間件必須提供不受任何限制的自動資源管理功能，并且使用方便。同時，這些功能必須給開發者提供足夠的控制，以按照他們的需要調整資源管理。只要有可能，我們希望能夠改變資源管理策略，而不要求重新編譯。

A common scalability problem for distributed multiplayer games relates to managingdistributed sets of objects. The game might allow players to form guilds, subjectto certain rules: For example, a player may not be a member of more than oneguild, or a guild may have at most one level-5 mage (magician). In computingterms, implementing such behavior boils down to performing membership tests onsets of distributed objects. Efficient implementation of such set operationsrequires an object model that does not incur the cost of a remote message foreach test. In other words, the object identities of objects must be visible atall times and must have a total order.

分布式多人游戲共同的一個可擴展性問題是，管理分布式對象集合。游戲可能允許玩家們組成公會（guild），但要符合一定條件：比如，玩家不可以加入多個公會，或者公會最多有1個5級魔法師。用計算機術語說，實現這種行為歸結為對分布式對象集執行成員資格測試。有效地實現這種集合運算需要這樣的對象模型，它不用每次測試都發送遠程消息。也就是說，對象的標識對象要全程可見，并且具有全序關系。

In classical RPC (remote procedure call) systems, object implementations residein servers, and clients send remote messages to objects: All object behavioris on the server, with clients only invoking behavior, but not implementing it.Although this approach is attractive because it naturally extends the notionof a local procedure call to distributed scenarios, it causes significant problems:

在經典RPC（remote procedure call，遠程過程調用）系統中，對象的實現位于服務器，客戶端發送遠程消息到對象：所有對象行為都在服務器，而客戶端只是調用行為，而不是實現它。盡管這個方法很有吸引力，因為它將本地過程調用的概念自然地擴展到了分布式環境中，但它造成了以下值得注意的問題：

• Sending a remote message is orders of magnitude slower than sending a local message.One obvious way to reduce network traffic is to create “fat” RPCs:as much data as possible is sent with each call to better amortize the cost ofgoing on the wire. The downside of fat RPCs is that performance considerationsinterfere with object modeling: While the problem domain may call for fine-grainedinterfaces with many operations that exchange only a small amount of state, goodperformance requires coarse-grained interfaces. It is difficult to reconcilethis design tension and find a suitable trade-off.

• 發送遠程消息比發送本地消息慢好幾個數量級。減少網絡流量的一種明顯的方法是創建“胖（fat）”RPC：每次調用都發送盡可能多的數據，以便更好地分攤傳輸的成本。胖RPC的缺點是，性能方面的考慮干擾了對象建模：雖然問題域可能需要細粒度的接口，以及許多僅僅交換少量狀態的操作，但為了良好的性能，卻要求粗粒度的接口。人們很難調和這種設計的緊張局勢并找到適合的折衷之法。

• Many objects have behavior and can be traded among players. Yet, to meet theprocessing requirements of the game, we have many servers (possibly in differentcontinents) that implement object behavior. If behavior stays put in the server,yet players can trade objects, before long, players end up with a potion whoseserver is in the United States and a scroll whose server is in Europe, with thepotion and scroll carried in a bag that resides in Australia. In other words,a pure client–server model does not permit client-side behavior and objectmigration, and, therefore, destroys locality of reference.

• 很多對象有行為，并且可以在玩家之間交易。但是，為了滿足游戲處理的要求，我們有很多服務器（可能位于不同的大陸），它們實現了對象的行為。如果行為停留在服務器不動，而玩家可以交易對象，那么，不久以后，有個玩家會有一瓶藥在美國的服務器上，一個卷軸在歐洲的服務器上，而放這瓶藥和卷軸的袋子卻在澳大利亞的服務器上。換句話說，純粹的客戶端-服務器模型不允許客戶端的行為和對象遷移，并且因此破壞了訪問局部性（locality of reference）。

We wanted an object model that supports both client- and server-side behaviorso we could migrate objects and improve locality of reference.

我們需要一個同時支持服務器端和客戶端行為的對象模型，這樣我們就能遷移對象并改善訪問局部性。

DESIGNING A NEW MIDDLEWARE

設計新的中間件

Looking at our requirements, we quickly realized that existing middleware wouldbe unsuitable. The cross-platform and multi-language requirements suggested CORBA;however, a few of us had previously built a commercial object request brokerand knew from this experience that CORBA could not satisfy our functionalityand scalability requirements. Consequently, we decided to develop our own middleware,dubbed Ice (short for Internet Communications Engine).3

看到需求，我們馬上意識到現有的中間件沒有合適的。多平臺和多語言的需求使人想到CORBA；但是，我們幾個以前曾經建立了一個商業性的對象請求代理，并且從這次經歷知道，CORBA不能滿足我們的功能和可擴展性需求。因此，我們決定開發自己的中間件，稱為Ice（互聯網通信引擎，Internet Communications Engine的簡稱）[3]。

The overriding focus in the design of Ice was on simplicity: We knew from bitterexperience that every feature is paid for in increased code and memory size,more complex APIs, steeper learning curve, and reduced performance. We made everyeffort to find the simplest possible abstractions (without passing the “complexitybuck” to the developer), and we admitted features only after we were certainthat we absolutely had to have them.

Ice中設計的首要重點是簡單性：我們從過去的痛苦經驗得知，每個功能的代價就是增加的代碼和內存占用、更復雜的API、陡峭的學習曲線，以及降低的性能。我們盡了一切努力尋找最簡單的抽象（不是把復雜性推卸給開發者），并且我們只接受我們確信絕對必要的功能。

Object Model. Ice restricts its object model to a bare minimum: Built-in datatypes are limited to signed integers, floating-point numbers, Booleans, Unicodestrings, and 8-bit uninterpreted (binary) bytes. User-defined types include constants,enumerations, structures, sequences, dictionaries, and exceptions with inheritance.Remote objects are modeled as interfaces with multiple inheritance that containoperations with input and output parameters and a return value. Interfaces arepassed by reference—that is, passing an interface passes an invocationhandle via which an object can be invoked remotely.

對象模型。Ice將其對象模型限制到了最小化。內置數據類型僅限于有符號整數、浮點數、布爾型、Unicode字符串和8位無解釋（二進制）字節。用戶定義的類型包括常量、枚舉、結構、序列、字典，以及帶繼承的異常。遠程對象建模為帶多重繼承的接口，而接口包含具有輸入輸出參數和返回值的各種操作。接口通過引用傳遞，也就是說，傳遞接口會傳遞一個調用句柄，通過該句柄，就可以遠程調用一個對象。

To support client-side behavior and object migration, we added classes: operationinvocations on a class execute in the client’s address space (instead ofthe server’s, as is the case for interfaces). In addition, classes canhave state (whereas interfaces, at the object-modeling level, are always stateless).Classes are passed by value—that is, passing a class instance passes thestate of the class instead of a handle to a remote object.

為了支持客戶端行為和對象遷移，我們添加了類：類上調用的操作執行于客戶端的地址空間（而不是就接口而言的服務器端）。此外，類可以有狀態（而接口，在對象建模的層次，總是無狀態的）。類是按值傳遞的，也就是說，傳遞類實例會將該類的狀態傳遞到遠程對象，而不是傳遞句柄。

We did not attempt to pass behavior: This would require a virtual execution environmentfor objects but would be in conflict with our performance and multi-languagerequirements. Instead, we implemented identical behavior for a class at all itspossible host locations (clients and servers): Rather than shipping code around,we provide the code wherever it is needed and ship only the state. To migratean object, a process passes a class instance to another process and then destroysits copy of the instance; semantically, the effect is the same as migrating bothstate and behavior.

我們沒有試圖傳遞行為：這將要求為對象建立一個虛擬執行環境，這與我們的性能和多語言需求相沖突。相反，我們實現了類在其所有可能的主機位置（客戶端和服務器）上具有同一行為：我們不是到處分發代碼，而是在有需要的地方提供代碼，并且僅僅分發狀態。為了遷移對象，進程傳遞類實例到另一進程，然后消毀自己的實例拷貝；在語義上，其效果與同時遷移狀態和行為是相同的。

Architecturally, implementing object migration in this way is a two-edged swordbecause it requires all host locations to implement identical (as opposed tomerely similar) behavior. This has ramifications for versioning: If we changethe behavior of a class at one host location, we must change the behavior ofthat class at all other locations (or suffer inconsistent behavior). Multiplelanguages also require attention. For example, if a class instance passes froma C++ server to a Java client, we must provide C++ and Java implementations withidentical behavior. (Obviously, this requires more effort than implementing thebehavior just once in a single language and single server.)

在架構上，以這種方式實現對象遷移是一把雙刃劍，因為它要求所有的主機位置實現同一行為（而非僅僅類似）。這對版本控制有復雜的影響：如果我們在一個主機位置改變類的行為，我們必須改變那個類在所有其他位置的行為（否則就會有不一致的行為）。多語言也要注意。比如，如果類實例從C++服務器傳遞到Java客戶端，我們必須提供具有同一行為的C++和Java實現。（顯然，相比用單一語言，在單一服務器上，僅僅實現一次行為，這需要更多的努力。）

For environments such as Wish, where we control both client and server deployment,this is acceptable; for applications that provide only servers and rely on otherparties to provide clients, this can be problematic because ensuring identicalbehavior of third-party class implementations is difficult.

對于像Wish這樣的環境，我們控制了客戶端和服務器的部署，這是可以接受的；但對于只提供服務器，而依靠其他各方提供客戶端的情況，這就有問題，因為很難確保第3方類的實現具有同一行為。

Protocol Design. To meet our performance goals, we broke with established wisdomfor RPC protocols in two ways:

協議設計。為了滿足性能目標，我們摒棄了RPC協議中的2個既定觀念：

• Data is not tagged with its type on the wire and is encoded as compactly as possible:The encoding uses no padding (everything is byte-aligned) and applies a numberof simple techniques to save bandwidth. For example, positive integers less than255 require a single byte instead of four bytes, and strings are not NUL terminated.This encoding is more compact (sometimes by a factor of two or more, dependingon the type of data) than CORBA’s CDR (common data representation) encoding.

• 數據傳輸時沒有類型標記，并且盡可能的以緊湊格式編碼：編碼沒有填充（結構都以1字節對齊），并且使用了一些簡單的技術節省帶寬。例如，小于255的正整數需要1個字節而不是4字節，字符串不以NUL結尾。這樣的編碼比CORBA的CDR（common data representation，通用數據表示）編碼更緊湊（有時可達到2倍以上，視數據類型而定）。

• Data is always marshaled in little-endian byte order. We rejected a receiver-makes-it-rightapproach (as used by CORBA) because experiments showed no measurable performancegain.

• 數據總是以little-endian字節序組編（marshal）。我們不使用receiver-makes-it-right（接收者讓它正確）方法（它用于CORBA），因為實驗表明該方法沒有可觀的性能提升。

The protocol supports compression for better performance over low-speed links.(Interestingly, for high-speed links, compression is best disabled: It takesmore time to compress data than to send it uncompressed.)

協議支持壓縮，以在低速鏈路上有更好的性能。（有趣的是，對高速鏈路，最好禁用壓縮：壓縮數據太費時，還不如不壓縮傳輸。）

The protocol encodes request data as a byte count followed by the payload asa blob. This allows the receiver of a message to forward it to a number of downstreamreceivers without the need to unmarshal and remarshal the message. Avoiding thiscost was important so we could build efficient message switches for event distribution.

協議將請求數據編碼為一個字節計數加一個blob凈荷。這允許消息接收者向下游接收者轉發消息，而無需解編（unmarshal）和重新組編（remarshal）。避免這種消耗很重要，這樣我們才能為事件分發建立高效的消息分派。

The protocol supports TCP/IP and UDP (user datagram protocol). For secure communications,we use SSL (secure sockets layer): It is freely available and has been extensivelyscrutinized for flaws by the security community.

協議支持TCP/IP和UDP（user datagram protocol，用戶數據報協議)。為了安全通信，我們使用SSL（secure sockets layer，安全套接字層）：它是免費的，并且安全社區已經對它進行了廣泛的檢驗。

The protocol is bidirectional, so a server can make a callback over a connectionthat was previously established by a client. This is important for communicationthrough firewalls, which usually permit outgoing connections, but not incomingones. The protocol also works across NAT boundaries.

協議是雙向的，所以服務器可以在客戶端先前建立的連接上進行回調。這對穿越防火墻的通信很重要，防火墻通常允許向外的連接，而不能被連入。該協議同時還能穿越NAT邊界。

Classes make the protocol more complex because they are polymorphic: If a processsends a derived instance to a receiver that understands only a base type of thatinstance, the Ice runtime slices the instance to the most-derived base type thatis known to the receiver. Slicing requires the receiver to unmarshal data whosetype is unknown. Further, classes can be self-referential and form arbitrarygraphs of nodes: Given a starting node, the Ice runtime marshals all reachablenodes so graphs require the sender to perform cycle detection.

類使得協議更復雜，因為類是多態的：如果進程發送了一個派生類的實例給接收者，而接收者只知道該實例的基類的話，Ice運行庫（runtime）會將該實例剪切（slice）為接收者所知道的最近基類。剪切要求接收者解編未知類型的數據。而且，類可以是自引用的（self-referential），并形成任意的節點圖：給定一個起始節點，Ice運行庫會組編所有可達的節點，因此節點圖要求發送者執行環檢測。

The implementation of slicing and class graphs is surprisingly complex. To supportunmarshaling, the protocol sends classes as individually encapsulated slices,each tagged with their type. On average (compared with structures), this requires10 to 15 percent extra bandwidth. To preserve the identity relationships of nodesand to detect cycles, the marshaling code creates additional data structures.On average, this incurs a performance penalty of 5 to 10 percent. Finally, forC++, we had to write a garbage collector to avoid memory leaks in the presenceof cyclic class graphs, which was nontrivial. Without slicing and class graphs,the protocol implementation would have been simpler and (for classes) slightlyfaster.

剪切和類圖的實現異常復雜。為了支持解編，協議按單獨封裝的剪切片發送類，每個都標記其類型。平均而言（對比結構），這需要10%-15%的額外帶寬。為了保存節點關系標識，也為了檢測環，組編代碼會生成附加的數據結構。平均而言，這會產生5%-10%的性能損耗。最后，對于C++，我們必須編寫垃圾收集器，以避免環形類圖表示中的內存泄漏，這也不容易。如果沒有剪切和類圖，協議實現將更簡單，（對類來說）也略微更快些。

Versioning. The object model supports multiple interfaces: Instead of havinga single most-derived interface, an object can provide any number of interfaces.Given a handle to an object, clients can request a specific interface at runtimeusing a safe downcast. Multiple interfaces permit versioning of objects withoutbreaking on-the-wire compatibility: To create a newer version, we add new interfacesto existing objects. Already-deployed clients continue to work with the old interfaces,whereas new clients can use the new interfaces.

版本控制。對象模型支持多接口：對象可以提供任意多的接口，而不是單一的最終接口。給定一個對象的句柄，客戶端可以在運行時利用安全的向下轉型，來請求一個特定接口。多接口允許對象的版本控制，而不會破壞已上線對象的兼容性：要創建一個新的版本，我們會在現有對象上添加新的接口。已經部署的客戶端使用舊接口繼續運行，而新客戶端可以使用新接口。

Used naively, multiple interfaces can lead to a versioning mess that forces clientsto continuously choose the correct version. To avoid these problems, we designedthe game such that clients access it via a small number of bootstrap objectsfor which they choose an interface version. Thereafter, clients acquire handlesto other objects via their chosen interfaces on bootstrap objects, so the desiredversion is known implicitly to the bootstrap object. The Ice protocol providesa mechanism for implicit propagation of contextual information such as versioning,so we need not pollute all our object interfaces by adding an extra version parameter.

如果盲目使用，多接口可以導致版本混亂，使得客戶端要不斷地選擇正確版本。為避免這些問題，我們這樣設計游戲：讓客戶端通過幾個引導對象訪問接口，客戶端只需為引導對象選擇一個接口版本。因此，客戶端通過其所選的引導對象接口，來獲得其他對象的句柄，引導對象完全知道需要的版本。對于比如版本這樣的上下文信息，Ice協議提供了一個隱式傳播機制，我們不必為所有對象的接口都添加額外的版本參數。

Multiple interfaces reduced development time of the game because, apart fromversioning, they allowed us to use loose coupling at the type level between clientsand servers. Instead of modifying the definition of an existing interface, wecould add new features by adding new interfaces. This reduced the number of dependenciesacross the system and shielded developers from each others’ changes andthe associated compilation avalanches that often ensue.

多接口縮短了游戲開發時間，因為，除了版本之外，多接口還允許我們在客戶端和服務器之間的類型級別中使用松耦合。不修改現有接口的定義，我們就可以通過添加新接口來增加新功能。這減少了整個系統的依賴數量，并且屏蔽了開發者相互間的變更，以及由此帶來的相關的編譯雪崩（compilation avalanche）。

On the downside, multiple interfaces incur a loss of static type safety becauseinterfaces are selected only at runtime, which makes the system more vulnerableto latent bugs that can escape testing. When used judiciously, however, multipleinterfaces are useful in combating the often excessively tight coupling of traditionalRPC approaches.

多接口的缺點是，它招致了靜態類型安全性的損失，因為接口只在運行時被選擇，這使得系統更容易受到潛伏錯誤的傷害，即那些逃過測試的錯誤。但是，如果被明智地使用，相比傳統RPC方法中往往過緊的耦合，多接口更為有用。

Ease of Use. Ease of use is an overriding design goal. On the one hand, thismeans that we keep the runtime APIs as simple and small as possible. For example,29 lines of specification are sufficient to define the API to the Ice objectadapter. Despite this, the object adapter is fully functional and supports flexibleobject implementations, such as separate servant per object, one-to-many mappingsof servants to objects, default servants, servant locators, and evictors. Byspending a lot of time on the design, we not only kept the APIs small, but alsoreaped performance gains as a result of smaller code and working set sizes.

易用性。易用性是設計的首要目標。一方面，這意味著我們要保持運行庫API盡可能的簡單，盡可能的小。比如，29行的說明足以定義Ice對象適配器（object adapter）的API。盡管如此，對象適配器是功能完善的，并且支持靈活的對象實現，比如每個對象有單獨的servant（服務者）、servant與對象是一對多關系、默認servant、servant locator（服務者定位器），和evictor（逐出器）。通過長時間的設計，我們不僅使API很小，而且由于更小的代碼和工作集而獲得了性能提升。

On the other hand, we want language mappings that are simple and intuitive. Limitingourselves to a small object model paid off here—fewer types mean less generatedcode and smaller APIs.

另一方面，我們想要簡單直觀的語言映射。我們把自己限制為小型對象模型，在這里得到了補償：更少的類型意味著更少的生成代碼和更小的API。

The C++ mapping is particularly important: From CORBA, we knew that a poorlydesigned mapping increases development time and defect count, and we wanted somethingsafer. We settled on a mapping that is small (documented in 40 pages) and providesa high level of convenience and safety. In particular, the mapping is integratedwith the C++ standard template library, is fully threadsafe, and requires nomemory management. Developers never need to deallocate anything, and exceptionscannot cause memory leaks.

C++映射尤其重要：從CORBA我們知道，設計不良的映射會增加開發時間和缺陷數，而我們想要更安全的東西。我們最終決定的映射很小（文檔只有40頁），并且提供了高水平的便利性和安全性。尤其是，映射集成了C++標準庫，是完全線程安全的，并且不需要內存管理。開發者不用釋放任何東西，并且異常也不會引起內存泄漏。

One issue we repeatedly encounter for language mappings is namespace collision.Each language has its own set of keywords, library namespaces, and so on. Ifthe (language-independent) object model uses a name that is reserved in a particulartarget language, we must map around the resulting collision. Such collisionscan be surprisingly subtle and confirmed, yet again, that API design (especiallygeneric API design, such as for a language mapping) is difficult and time consuming.The choice of the trade-off between ease of use and functionality also can becontentious (such as our choice to disallow underscores in object-model identifiersto create a collision-free namespace).

語言映射中，我們多次遇到的一個的問題是，名字空間沖突。每個語言都有它們自己的一套關鍵字、庫名字空間，等等，如果（語言無關的）對象模型使用了一個特定目標語言中保留的名字，我們必須重新映射來繞過產生的沖突。這種沖突會出奇的微妙，并且反復地出現，以致于API設計費時又費力（特別是通用API設計，比如語言映射）。在易用性和功能性之間權衡的選擇也可能是有爭議的（比如我們選擇不允許對象模型的標識符用下劃線來產生不沖突的名字空間）。

Persistence. To provide object persistence, we extended the object model to permitthe definition of persistence attributes for objects. To the developer, makingan object persistent consists of defining those attributes that should be storedin the database. A compiler processes these definitions and generates a runtimelibrary that implements associative containers for each type of object.

持久化。為了提供對象持久化，我們擴展了對象模型，允許為對象定義持久化屬性。對開發者來說，讓對象持久化就是定義那些應該存儲在數據庫中的屬性。編譯器處理這些定義，并生成運行時庫，為每種類型的對象實現關聯容器。

Developers access persistent objects by looking them up in a container by theirkeys—if an object is not yet in memory, it is transparently loaded fromthe database. To update objects, developers simply assign to their state attributes.Objects are automatically written to the database by the Ice runtime. (Variouspolicies can be used to control under what circumstances a physical databaseupdate takes place.)

開發者訪問持久對象時，要在容器中用其鍵值查找它們，如果對象尚未在內存中，它會透明地從數據庫加載。當更新對象時，開發者只要對它們的狀態屬性賦值。Ice運行庫會自動地把對象寫入數據庫（可以使用各種策略來控制在何種情況下執行物理數據庫更新）。

This model makes database access completely transparent. For circumstances inwhich greater control is required, a small API allows developers to establishtransaction boundaries and preserve database integrity.

這個模型使得數據庫訪問完全透明。對于需要更多控制的情況來說，小型的API允許開發者建立事務的邊界并維護數據庫的完整性。

To allow us to change the game without continuously having to migrate databasesto new schemas, we developed a database transformation tool. For simple featureadditions, we supply the tool with the old and new object definitions—thetool automatically generates a new database schema and migrates the contentsof the old database to conform to the new schema. For more complex changes, suchas changing the name of a structure field or changing the key type of a dictionary,the tool creates a default transformation script in XML that a developer canmodify to implement the desired migration action.

為了允許我們改變游戲，但不必反復地遷移數據庫到新的模式，我們開發了一個數據庫轉換工具。對于簡單的功能添加，我們只要向工具提供新舊對象的定義，工具會自動地生成一個新的數據庫模式，并遷移舊庫的內容以符合新的模式。對更復雜的變更，比如改變結構的字段名，或者改變字典鍵值的類型，工具會創建一個默認的XML轉換腳本，開發者可以修改它以實現所需的遷移動作。

This tool has been useful, although we keep thinking of new features that couldbe incorporated. As always, the difficulty is in knowing when to stop: The temptationto build better tools can easily detract from the overall project goals. (“Insideevery big program is a little program struggling to get out.”)

這個工具很有用，盡管我們不斷地想到還可以納入的新功能。就像通常一樣，困難在于知道何時停止：建立更好的工具，這個誘惑很容易降低整個項目的目標。（“Inside every big program is a little program struggling to get out.”，“在每個大程序中，都有一個小程序掙扎著要出來。”）

Threading. We built a portable threading API that provides developers with platform-independentthreading and locking primitives. For remote call dispatch, we decided to supportonly a leader/followers threading model.4 In some situations, in which a blockingor reactive model would be better suited, this decision cost us a little in performance,but it gained us a simpler runtime and APIs and reduced the potential for deadlockin nested RPCs.

多線程。我們建立了一個可移植的線程API，給開發者提供了平臺無關的線程和鎖原語。對于遠程調用派發，我們決定僅支持leader/follower（領導者/跟隨者）線程模型[4]。在阻塞或者反應模式更合適的情況下，這個決定使我們損失了一點性能，但是它讓我們得到了更簡單的運行庫和API，并且減少了嵌套RPC調用中死鎖的可能性。

Scalability. Ice permits redundant implementations of objects in different servers.The runtime automatically binds to one of an object’s replicas and, ifa replica becomes unavailable, fails over to another replica. The binding informationfor replicas is kept in configuration and is dynamically acquired at runtime,so adding a redundant server requires only a configuration update, not changesin source code. This allows us to take down a game server for a software upgradewithout having to kick all players using that server out of the game. The samemechanism also provides fault tolerance in case of hardware failure.

擴展性。Ice允許在不同的服務器上冗余的對象實現。運行庫會自動綁定到一個對象副本（replica），并且，如果副本不可用時，會自動切換（fail over）到另一個副本。副本的綁定信息保存在配置中，并在運行時動態獲得，所以添加冗余服務器只需更新配置，不用修改源碼。這允許我們卸下游戲服務器做軟件升級，而不必把所有使用這個服務器的玩家都踢出游戲。同樣的機制還提供了硬件故障時的容錯能力。

To support federating logical functions across a number of servers and to shareload, we built an implementation repository that delivers binding informationto clients at runtime. A randomizing algorithm distributes load across any numberof servers that form a logical service.

為了支持在多臺服務器上聯盟（federate)邏輯函數，以分擔負載，我們建立了一個實現庫（implementation repository），它會在運行時傳送綁定信息到客戶端。一個隨機算法會分配負載到任意數量的服務器，讓它們組成一個邏輯服務。

We made a number of trade-offs for replication and load sharing. For example,not all game components can be upgraded without server shutdown, and a load feedbackmechanism would provide better load sharing than simple randomization. Givenour requirements, these limitations are acceptable, but, for applications withmore stringent requirements, this might not be the case. The skill is in decidingwhen not to build something as much as when to build it—infrastructuremakes no sense if the cost of developing it exceeds the savings during its use.

我們為復制和負載共享作出了若干權衡。比如，不是所有的游戲組件都能在線升級，以及負載反饋機制將比簡單的隨機化提供更優的負載共享。鑒于我們的需求，這些限制是可以接受的，但是，對于需求更嚴格的應用，可能并非如此。技巧在于決定何時不要建立某個東西，其重要性等同于決定何時要建立它：如果基礎設施的開發成本超過使用它所節省的成本，這就毫無意義了。

SIMPLE IS BETTER

簡單就好

Our experiences with Ice during game development have been very positive. Despiterunning a distributed system that involves dozens of servers and thousands ofclients, the middleware has not been a performance bottleneck.

在游戲開發中，我們對Ice的感覺是非常肯定的。雖然運行的是包含幾十臺服務器和數千客戶端的分布式系統，但中間件并不是性能瓶頸。

Our focus on simplicity during design paid off many times during development.When it comes to middleware, simpler is better: A well-chosen and small featureset contributes to timely development, as well as to meeting performance goals.

在設計中對簡單性的專注，讓我們在開發中贏得了數倍的回報。對于中間件，簡單就好：一個精心挑選并且小型化的功能集，有利于按時開發，以及達到性能目標。

Finally, designing and implementing middleware is difficult and costly, evenwith many years of experience. If you are looking for middleware, chances arethat you will be better off buying it than building it.

最后，設計和實現中間件是困難和昂貴的，即使對于有多年經驗的老手。如果你正在尋找中間件，很有可能你最好是購買一個而不是構建一個。

REFERENCES

參考

1. Mutable Realms (Wish home page): see http://www.mutablerealms.com.

2. Henning, M., and S. Vinoski. Advanced CORBA Programming with C++. Addison-Wesley,Reading: MA, 1999.

3. ZeroC. Distributed Programming with Ice: see http://www.zeroc.com/Ice-Manual.pdf.

4. Schmidt, D. C., O’Ryan, C., Pyarali, I., Kircher, M., and Buschmann,F. Leader/ Followers: A design pattern for efficient multithreaded event demultiplexingand dispatching. Proceedings of the 7th Pattern Languages of Programs Conference(PLoP 2000); http://deuce.doc.wustl.edu/doc/pspdfs/lf.pdf.

MICHI HENNING (michi@zeroc.com) is chief scientist of ZeroC. From 1995 to 2002,he worked on CORBA as a member of the Object Management Group’s ArchitectureBoard and as an ORB implementer, consultant, and trainer. With Steve Vinoski,he wrote Advanced CORBA Programming with C++ (Addison-Wesley, 1999), the definitivetext in the field. Since joining ZeroC, he has worked on the design and implementationof Ice and in 2003 coauthored “Distributed Programming with Ice” forZeroC. He holds an honors degree in computer science from the University of Queensland,Australia.

    Originally published in Queue vol. 1, no. 10—    
    see this item in the ACM Digital Library