Architecting Distributed Object Systems
March 4, 2000
Copyright (c) Jay Walters 2000.
|Jay Walters has over fourteen years of experience developing OLTP systems using a variety of RDBMS platforms, and has spent the last six years using distributed object technologies. He spent three years as the architect responsible for overall system performance on the first wagering system built using object technology and an RDBMS. He was most recently a consultant with the Oracle Consulting Object Technology Center of Excellence working with the Java and distributed computing features of Oracle Application Server. He is currently working for an internet auction portal where he has responsibility for the design of the search engine and all aspects of the underlying RDBMS.|
This paper describes some techniques and patterns which can be used to build high performance distributed applications using object oriented technology. This paper makes no attempt to provide a complete catalog of solutions, however it does attempt to cover some of the more significant issues in the development of distributed object systems. The techniques shown in this paper are implementation language independent and can be implemented in C++, Java or Smalltalk. Some of these techniques are available in commercially available products such as Transaction Monitors, others are for use in more custom environments. This paper defers discussing database design issues as they will be covered in a separate paper.
This paper addresses the design of distributed object systems, or n-tier systems. The expected implementation environment for these applications is browser or dedicated application on the client, Java or C++ application servers in one or more middle tiers and an RDBMS for persistance. Throughout the rest of this document I will use the term Application Server to mean a stand alone program, or a component running in the context of an Application Server product. A CORBA compliant ORB can be used for communications between middle tier components and JDBC for Java or a proprietary API such as Oracle's OCI for C++ can be user for middle tier to RDBMS communications. Another solution in the Java technology space is to use Enterprise Java Beans (EJB). The ideas in this paper are for the most part independent of the infrastructure and tools just discussed, and will serve anyone designing a distributed object system.
We will start with some basic development ground rules, move into a discussion of interfaces between client and application server and finish up with some patterns which can be used inside a Application Server. Some references and URLs which provide more information will be provided at the end. In a separate appendix the full text for some of the patterns will be provided. Credit for the patterns has been given where applicable.
“A designer knows he’s achieved perfection not when there is nothing to add, but when there
is nothing left to take away.”
Antoine de Saint-Exupery
“The best strategy for efficiency is to produce a clean and simple design.
Only such a design can remain relatively stable over the lifetime of the project and serve as a basis for
Some level of complexity is inherent in distributed object systems. This complexity impacts both the development of the system, and the management of the system after delivery. It is critical to aggressively manage the complexity of the project and the product, from the user requirements through the use of the latest language or development tool feature. This is not often a consideration in software architecture, however it should be.
There are three very good reasons to minimize complexity. First, it is estimated that 50% or more of all software development time consists of “system discovery” (trying to understand how the system works)[Mowbray 1995]. Clearly the simpler the application is, the easier this process of discovery will be, and the more efficient the software development process will be. Second, a simpler system is more likely to be correct, and it is easier to verify it’s correctness through testing. Third, a simple architecture is more likely to have good scalability and performance characteristics. That is not to say that all simple architectures are scalable, or all simple algorithms are efficient. You just have a better chance of getting it right if it’s simple.
Another name for this concept is elegance. A simple example of an elegant solution is the binary search algorithm, because it is fast and simple, and because it also suggests a wide range of similar algorithms for solving related problems.
Scalability and performance must be architected into the solution from the beginning, they can’t be retrofitted at the end. This section will discuss three techniques which can be used to insure that your application will exhibit good scalability and performance.
The first technique is to reduce network traffic by keeping processing as close to the data as possible. This locates business logic as close to the database as possible and presentation logic as close to the client as possible. At the client this ensures timely feedback to the user when they make an input mistake; no sense making a network round trip to figure out they tried to type a name into the birthdate field. On the database side this minimizes network overhead moving objects between the database and the Application Server or client. By removing business logic from the Application Server this technique leads to ‘thin’ Application Servers.
The second technique is to cache frequently used data in the Application Server. As you analyze the business logic you may find certain transactions that will exhibit better performance if they are executed in the Application Server, perhaps using cached data. Frequently these are computationally intensive transactions, or transactions which read a lot of stable data for validation. If the transactions are executed frequently enough that the performance benefit is worth the effort to maintain the cache, then this technique is applicable. By building up the Application Server with business and cache management logic this technique leads to ‘thick’ Application Servers.
In building a large application you will probably find some transactions that will work well with thin servers and some that will work well with thick servers. You should take this into account when you partition your application into different servers.
The last technique is to design the system to become more efficient under load if possible. This means that as the system utilization increases, the system’s efficiency increases.
As an example of these points, I worked on a system which handled lottery and racing betting. We were under strict response time requirements and needed to support high peak transaction volumes. There were somewhat complicated validation rules for the bets and for storage efficiency we stored the selections as bitmaps in raw columns. Furthermore, the data required to validate a bet changed very rarely, so we cached all of the validation data in the Application Server (thick server) and performed all of our bet validation there, before inserting the bet into the database. As the system load increased we increased the batch size of our database transactions from 1 to 8 bets. For some other transactions such as defining drawings and entering the winning numbers we handled the entire transaction within the database (thin server) because it was not time critical and was infrequent.
In a distributed object system scalability is often highly dependent on the interfaces between the Client, the Application Server and the Database Server. Good interface design will provide the same benefits as good object design, namely loose coupling between tiers and encapsulation of algorithms within a given tier. This means that you can generally tune individual classes or tiers until late in the development cycle because the modifications will be transparent to the other parts of the application.
A good rule for the design of interfaces within distributed applications is to expose services, not fine grain objects in the Application Server. One presentation of this rule is the Process-Entity design pattern provided by BEA as part of their WebLogic Enterprise Developer Center. This service oriented interface is related to the Gang of Four Facade pattern. These services should be designed around the business objects and the ways in which they need to be manipulated as part of business transactions. An example of this is to have an interface (object) with an operation for creating new orders (service) instead of having an order object and an order line object, each with a number of operations to set the various attributes. The goal of this rule is two-fold. First it reduces the number of network round trips per client function. Second it simplifies the system by reducing the complexity of the interface. This rule illustrates the challenge of designing a distributed object system, trading off purity of object oriented design versus a design with acceptable performance.
Some specific rules of good design for these interfaces are:
· Each interface should contain a single set of closely related operations (services).
· Keep each interface as simple and uncluttered as possible.
· Consider how the client will use the interface to make sure you actually minimize network round trips.
· Use exceptions to signal errors because they can return the information required by the caller to process the error.
· Be very careful when using unbounded sequences. Ensure that the maximum number of items in the sequence is guaranteed to be small, or that you always want every item.
This section will discuss some common design patterns and how they can be applied to building distributed object applications. Some of these patterns are incorporated in commercially available or soon to be available software such as Transaction Monitors or Enterprise Java Bean Containers.
· Share expensive resources across method invocations. A classic example of an expensive resource is a database connection. This is often referred to as funneling when referring to the practice of funneling multiple clients into a smaller number of database connections. A simple version of this pattern is implemented by Transaction Monitors. Another version of this pattern, the Resource Pool, is described in the appendix.
· Attempt to keep the Application Server stateless. This allows clients to move between servers easily for load balancing and to tolerate failures of an Application Server. Generally stateless servers are also easier to build, less prone to failure and quicker to recover if they do fail. The exporting of service level interfaces from the Application Server to the client helps in this regard as these services are usually complete transactions so no state needs to be kept across distributed method invocations.
If you need to keep state across method invocations, it can be encapsulated in a cookie, the Memento pattern [OO Design Patterns] and sent to the client, or stored in the database. If you decide to cache state in the Application Server the client software may need to be aware of that state when recovering from a server failure.
· Use the Template Method pattern [OO Design Patterns] to build a standard template for client and database transactions. Use the Command pattern [OO Design Patterns] to encapsulate transactions as objects to simplify your transaction processing framework. You may also use the Composite pattern [OO Design Patterns] to create more complex transactions or commands out a set of simple transactions or commands.
· Lazy evaluation is a good technique to use to place the cost of collecting data for an operation on the operation which requires the data, as opposed to charging each transaction a portion of the cost. The reverse of lazy evaluation is known as amortization. An example of this tradeoff is, do you update the year to date sales total for a customer every time you load a new order for them, or do you sum up the orders when a request is made for year to date sales for the customer.
· A specialization of “lazy evaluation” is “copy on write”. This is a good technique to use for reducing memory use in environments which are primarily read, occasional write. It is applicable when the modified objects don’t need to be shared, only the original needs to be shared. This technique can also be used to manage read consistency within cache data structures.
This technique is often used within C++ class libraries to efficiently implement a string class. When a programmer copies a string object into another string object all that is copied is the pointer to the data, so both copies are sharing the same data. If one string object is changed then that object will make it’s own copy of the data and modify the private copy. Other objects can continue to share the initial data.
This technique can be used to manage cache consistency by copying an object in the cache when it is updated, but not deleting the old object until all transactions which reference it are done with it. This is referred to as read consistency. If this technique can be used then the amount of locking required on the cache can be dramatically reduced, since only the structure needs to be locked not the individual elements in the cache. This approach depends on the cache consistency requirements and the types of transactions. It is most applicable for transactions which need a consistent view of the cached data, but don’t necessarily need the most up to date data.
The following books contain relevant information:
CORBA Design Patterns, Mowbray and Malveau, 1995.
(High level enterprise type patterns).
Design Patterns, Elements of Reusable Object-Oriented Software, Gamma, et al. 1995.
(The classic on software design patterns.)
The Essential CORBA, Systems Integration Using Distributed Objects, Mowbray and Zahavi,
(Covers architecture and design issues.)
The following URLs are also relevant:
- Object Management Group (OMG) owner of CORBA specifications.
Share expensive Application Server resources across multiple clients. Allow management of the resource outside the scope of a client transaction.
Two-tier systems typically require each client to have their own dedicated resources. This approach leads to scalability problems as the number of clients increases. Multi-tier systems alleviate this problem by multiplexing a large number of clients onto a small number of resources. This can be done because most of the clients are idle most of the time.
Simple approaches to this problem, such as a connection pool, multiplex the resources but process the resource commands in the thread of execution of the client transaction. This approach does not support cross-transaction operations such as priority handling of commands, batching of commands and recovery operations. These operations are supported by a resource pool because it assigns a separate thread of execution to each resource.
Use the Resource Pool pattern
· to multiplex many clients onto a limited number of resources.
· to support a resource recovery policy outside the context of a client transaction.
· to support priority or other cross-transaction resource usage policies.
· AbstractCommand (Template Method Pattern [OO Design Patterns])
declares abstract primitive operations that concrete subclasses will define to perform operations on a resource.
implements a template method defining the skeleton of the algorithm for operating on the resource. This skeleton calls both abstract and concrete methods in the AbstractCommand to implement the algorithm.
instantiates one or more ResourceCommands and passes them to ResourcePool for processing.
- processes ConcreteCommands.
- may implement policies for selecting next ConcreteCommand to process.
- maintains resource in a consistent and ready state.
- executes in it’s own thread.
· ConcreteCommand (Template Method [OO Design Patterns])
- implements the primitive operations required to carry out the command specific steps.
- may be more than one queue per ResourcePool.
- holds ConcreteCommands waiting to be executed by command processors.
- implements some ordering policy for ConcreteCommands.
- manages one or more queues of ResourceCommands waiting to be processed.
- manages a pool of CommandProcessors
- declares an interface for executing a ConcreteCommand in a separate thread.
· The Client Transaction creates a Concrete Command and passes it to the Resource Pool for processing. This may be a synchronous or asynchronous process, meaning the Client Transaction may wait for the results or it can poll for results.
· The Resource Pool places the Concrete Command on the appropriate Command Queue.
· Command Processors select one or more Concrete Commands from the Command Queue and process them according to their selection policy. If using the synchronous process the results will be returned to the Client Transaction at completion, if asynchronous then the results will be made available for return during the next polling operation.
There are two major types of resource pools, passive and active. A passive resource pool does not use separate threads for the CommandProcessor flow of control, but instead an active implementsWhat pitfalls, hints, or techniques should you be aware of when implementing the pattern? Are there language-specific issues?
An implementation Bullet. Description of Bullet
Sample Code and Usage
Code fragments that illustrate how you might implement the pattern in C++ or Smalltalk.
The Resource Pool is one of the core patterns within a transaction monitor.Examples of the pattern found in real systems. We include at least two examples from different domains.
It may be advantageous to hide much of this complexity behind a Facade [OO Design Patterns].
Depending on the requirements for the selectNext() selection algorithms it may be appropriate to implement this using the Strategy [OO Design Patterns] pattern.
If a cross-transaction batching strategy is used then the AbstractCommand/ConcreteCommand should use a pattern similar to the Composite [OO Design Patterns] to handle composite transactions.
The AbstractCommand/ConcreteCommand implements a pattern very similar to the Command [OO Design Patterns] pattern.
Process-Entity (See BEA Process-Entity design pattern)
Reduce the network traffic and system overhead of client interactions with remote database records by providing a single server object known as a "process" object that handles all client interactions with data records, known as "entities". By having a single CORBA object represent all the fine-grained data in the database, minimal server-side resources are used to service each client. Also, the process object can selectively pass data fields to the client, transferring only the necessary data rather than a full database record.
Two-tier systems treat the database layer as a set of shared objects. It might seem natural then to represent database records as shared CORBA objects. However, there are problems with scalability if this approach is used. Performance is vastly improved if each client has its own CORBA object that can interact with the database on its behalf. Performance is also improved if data is selectively sent to the client when needed rather than sending all fields in a record. The net result is reduced message traffic and less overhead for managing CORBA objects on the server.
This pattern is almost universally applicable in mission-critical applications. It is indicated in situations where a client needs to interact with database records stored on a server computer.
The client program gets a reference to a process object from a Factory. The process object implements all interaction with the database. Database records (entities) are retrieved when needed to service client invocations on the process object. Process object methods will return specific data fields to the client if needed for client-side processing.
Design the Process Object to pass the minimal amount of information required for a particular client operation on each method invocation. Process Object methods should be designed to do as "dense" processing as possible. Clients should not have to invoke more than one process method to accomplish a task. If more than one method needs to be invoked, then the process object should do the call to the additional methods within the processing of the method invoked by the client. When serial calls to a process object are required for a transaction, then a stateful object should be used (See the Iceberg Transction Processing Framework document).
Avoid the use of attributes in your CORBA IDL. Attributes are expensive to retrieve over the network. Instead, call a method to return a struct with all the values your client is likely to need for an operation.
SmallTalk MVC (Model-View-Controller) design pattern [SmallTalk]. Flyweight design pattern. [OO Design Patterns]
Transaction Monitor (Enterprise Structural)
To provide a basis for enabling distributed method invocations to be bound into atomic units called transactions.
Service Request Broker
Consider the case of building a high volume system which will handle fund transfers between accounts. In the simple case, a transfer from account A to account B, the state of both being maintained within a homogenous data store. This is known as a local transaction, and in this case the resource manager for the data store can handle the atomicity of the transaction and ensure that either both accounts are updated or neither is. In the more complex case where the state is stored within heterogenous data stores then we’ll need some mechanism to manage the atomicity of the transaction. This type of transaction is known as a global transaction. The Transaction Monitor is this third party which manages the state of a global transaction.
We also have a very large number of clients, more than the number of connections which any of the resources involved in the transactions can support. The naïve approach might be for the client to connect to each resource as the transaction requires, however for most resource managers the process of connecting is very expensive. Thus we need some virtual connection mechanism where each transaction views itself as having dedicated connections to all of the required resources, while the resources see only a small number of real connections. The Transaction Monitor provides this facility through a mechanism known as connection pooling.
This pattern is almost universally applicable in mission-critical applications. Use the Transaction Monitor pattern when:
· Transactions modify multiple resources requiring the use of two-phase commit protocols.
· Large numbers of clients need to be funneled onto a smaller number of Application Servers and scarce or expensive resources.
· The load on each application server needs to be dynamically balanced.
· High availability is required.
Processes the transaction and updates the appropriate resources via their ResourceManagers.
- Can control the commit or rollback of the transaction if ClientTransaction does not.
Sends a transaction message to the TransactionMonitor.
- Can control the commit or rollback of the transaction if ApplicationService does not.
Updates the underlying resource as requested by the ApplicationService.
- Performs commit or rollback of the transaction as requested by the TransactionMonitor.
- Accepts transaction delineation commands from the Transaction Monitor.
- Performs work under the auspices of the transaction. Provides durability, isolation
- Accepts commands to perform the phases of the commit algorithm from the TM.
- Awaits the transaction outcome of the second phase, as directed from the TM.
Starts the appropriate ApplicationService and passes the ClientTransaction’s message to it.
- Notifies ResourceManagers to commit or rollback the transaction.
- Accepts a Client’s requests to start and end transactions.
- Propagates the transaction when communication with other services takes place.
- Interfaces to RMs to inform them on the status of global transactions during termination
(commit or rollback) and recovery.
- Handles the atomicity of global transactions.
There are several commercially available Transaction Managers, CICS, Encina, Top End, Tuxedo. Most RDBMS products provide XA compliant resource managers allowing their easy integration with commercial Transaction Managers.