Performance Tuning Web Applications
Dr. Paul Dorsey, Dulcian, Inc.
Michael Rosenblum, Dulcian, Inc.
The main performance tuning techniques applied to client/server applications consisted of rewriting poorly written SQL code and tuning the database itself. These two techniques covered all but the most pathologically badly written applications. However, in contrast, web applications are frequently unaffected by these .performance improvement approaches. The causes of slowly running web applications are often different from those of client/server applications. This paper describes some of the reasons for performance tuning problems, as well as how to pinpoint and solve them.
Typical Web Application Process Flow
Poorly written SQL and a badly designed database will make any application run slower; but to improve the performance of a slow web application requires examination of the entire system, not just the database. A typical 3-tier web application structure is shown in Figure 1. (Note the numbering of the areas in the diagram since these will be used throughout this paper).
Figure 1: Web Application Process Flow
As shown in Figure 1, there are numerous possible places for web applications to experience bottlenecks or performance killers as described in the following nine step process:
Step 1: Code and operations executed on the client machine - When a user clicks a “Submit” button, data is collected and bundled into a request that is sent to the application server.
Step 2: Transmission of client request to the application server
Step 3: Code in the application server executed as a formulation of the client request to retrieve information from the database.
Step 4: Transmission of request from application server to the database
Step 5: Database reception, processing and preparation of return information to the application server.
Step 6: Transmission over internal network of information from database to the application server
Step 7: Application server processing of database response and preparation of response transmission to the client machine.
Step 8: Transmission of data from the application server to the client machine
Step 9: Client machine processing of returned request and rendering of application page in browser.
Possible Web Application Performance Problem Areas
Traditional tuning techniques only help with Step 5 and ignore all of the other eight places where performance can degrade. This section describes how problems can occur at each step of the process.
Step 1. Client Machine Performance Problems
The formulation of a request in the client is usually the least likely source of system performance problems. However, it should not be dismissed entirely. Using many modern AJAX architectures, it is possible to place so much code in the client that a significant amount of time is required before the request is transmitted to the application server. This is particularly true for underpowered client machines with inadequate memory and slow processors.
Step 2. Client to Application Server Transmission Problems
Like the client machine itself, the transmission time between the client machine and the application server is a less common cause of slowly performing web applications. However, if attempting to transmit a large amount of information, the time required to do so over the Internet may be affected. For example, uploading large files or transmitting a large block of data may slow down performance.
Step 3. Application Server Performance Problems
The application server itself rarely causes significant performance degradation. For computationally intensive applications such as large matrix inversions for linear programming problems, some performance slowdowns can occur, but this is less likely to be a significant factor in poorly performing applications.
Step 4. Application Server to Database Transmission Problems
Transmission of data from the application server to the database with 1 GB or better transmission speeds might lead you to ignore this step in the process. It is not the time needed to move data from the application server to the database that causes performance degradation, but it is the high number of transmission requests. The trend in current web development is to make applications database-independent. This sometimes results in a single request from a client requiring many requests from the application server to the database in order to fulfill. What needs to be examined and measured is the number of roundtrips made from the application server to the database.
Inexpert developers may create routines that execute so many roundtrips that there is little tuning that a DBA can do to yield reasonable performance results. It is not unusual for a single request from the client to generate hundreds (if not thousands) of round trips from the application server to the database before the transmission is complete. A particularly bad example of this problem encountered by the authors required 50,000-100,000 round trips. Why would this large number be needed? Java developers who think of the database as nothing more than a place to store persistent copies of their classes use Getters and Setters to retrieve and/or update individual attributes of objects. This type of development can generate a round trip for every attribute of every object in the database, which means that inserting a row into a table with 100 columns results in a single Insert followed by 99 Update statements. Retrieving this record from the database then requires 100 independent queries.
In the application server, identifying performance problems involves counting the number of transmissions made. The accumulation of time spent making round trips is one of the most common places where web application performance can suffer.
Step 5. Database Performance Problems
In the database itself, it is important to look for the same things that cause client/server applications to run slowly. However, additional web application features can cause other performance problems in the database.
Most web applications are stateless, meaning that each client request is independent. Developers do not have the ability to use package variables that persist over time. Consequently, when a user logs into an application, he/she will be making multiple requests within the context of the sign-on operation.
The information pertaining to that session must be retrieved at the beginning of every request and persistently stored at the end of every request. Depending upon how this persistence is handled in the database, a single table may generate massive I/O demands resulting in redo logs full of information which may cause contention on tables where session information is stored.
Step 6. Database to Application Server Transmission Problems
Transferring information from the database back to the application server (similar to Step 4) is usually not problematic from a performance standpoint. However, performance can suffer when a Java program requests a single record from a table. If the entire table contents are brought into the middle tier and then filtered to find the appropriate record, performance will be slow. The application can perform well as long as data volumes are small. As data accumulates, the amount of information transferred to the application server becomes too large, thus affecting performance.
Step 7. Application Server Processing Performance Problems
Processing the data from the database can be resource-intensive. Many database-independent Java programmers minimize work done in the database and execute much of the application logic in the middle tier. In general, complex data manipulation can be handled much more efficiently with database code. Java programmers should minimize information returned to the application server and, where convenient, use the database to handle computations.
Step 8. Application Server to Client Machine Transmission Problems
This area is one of the most important for addressing
performance problems and often receives the least attention. Industry standards
often assume that everyone has access to high performance client machines so
that the amount of data transmitted from the application server to the client
is irrelevant. As the industry moves to Web 2.0 and AJAX, very rich UI
applications create more and more bloated screens of 1MB or more. Some of the
Since most web pages only need to logically transmit an amount of information requiring 5K or less, the logical round trips on an open page should be measured in tens or hundreds of characters rather than megabytes.
Transmission between the application server and the client machine can be the most significant cause of poor web application performance. If a web page takes 30 seconds to load, even if it is prepared in 5 rather than 10 seconds, users will not experience much of a benefit. The amount of information being sent must be decreased.
Step 9. Client Performance Problems
How much work does the client need to do to render a web application page? This area is usually not a performance killer, but it can contribute to poor performance. Very processing-intensive page rendering can result in poor application performance.
Locating the Cause of Slowly Performing Web Applications
In order to identify performance bottlenecks, timers must be embedded into a system to help ascertain which of the nine possible places is causing the application performance to degrade. Most users will say “I clicked this button and it takes X seconds until I get a response.” This provides no information about which area or combination of areas is causing the slow performance.
Strategically placed timers will indicate how much time is spent at any one of the nine steps in the total process.
Using Timers to Gather Data About Slow Performance
This section describes the strategy for collecting information to help pinpoint web application bottlenecks.
Steps 1 & 9
Placing timers in the client machine is a simple task. A timer can be placed at the beginning and end of the client-side code.
Steps 2 & 8
Transmissions to and from the application server are difficult to directly measure. If you can ensure that the clocks on the client and application server are exactly synchronized, it is possible to put time stamps on transmissions, but the precise synchronization is often very difficult. A better solution is to determine the sum of time required for these two transmissions by measuring the total time from transmission to reception at the client and subtract the amount of time spent from when the application server received the request and sent it back.
This information will not reveal whether or not the problem is occurring during Step 2 or Step 8, but it will detect whether or not the problem is Internet related. If the problem is related to slow Internet transmission, the cause is likely to be large data volume. This can be tested by measuring the round trip time required to send and retrieve varying amounts of information.
Steps 3-7
The time spent going from the application server to the database and back is easy to measure by calculating the difference between the timestamps at the beginning and end of a routine. Depending upon how the system is architected, breaking down the time spent between Steps 1-3 can be very challenging.
The processing time to/from the database can be very difficult to measure. Most Java applications directly interface with the database in multiple ways and places in the code. There is no isolated servlet through which all database interaction passes. In the database itself, if the application server sends many requests from different sessions, the database cannot determine which information is being requested by which logical session, making it very difficult to get accurate time measures.
If the system architecture includes Java code that makes random JDBC calls, there is no way to identify where a performance bottleneck is occurring between Steps 3-7. Time stamps would be needed around each database call to provide accurate information about performance during this part of the process.
A more disciplined approach for calling the database is needed. This can be handled in either the application server or the database. For the application server, a single servlet can be created through which all database access would pass to provide a single location for placing time stamps. This servlet would gather information about the time spent in the application server alone (Steps 3 & 7) as well as the sum of Steps 4, 5, and 6 (to/from and in the database). Since the time in the database (4 & 6) will be negligible, this is an adequate solution.
To measure the time spent in the database, create a single function through which all database access is routed. The session ID would be passed as a parameter to the function to measure the time spent in the database as well as the number of independent calls.
Solving Web Application Performance Problems
Solving the performance problems in each of the nine web application process steps requires different approaches, depending upon the location of the problem.
Solving Client Machine Performance Problems (Steps 1 & 9)
Performance degradations in the client machine are usually due to AJAX-related page bloat burdening the client with rich UI components that could be eliminated. Determine whether or not all functionality is needed in the client. Can some processing be moved to the application server, database, or eliminated entirely?
Resolving Performance Issues between the Client and Application
Server (Step 2)
If the performance slowdown occurs during the transmission of information from the client to the application server, you need to decide whether or not any unnecessary information is being sent. To improve performance, decrease the amount of information being transmitted or divide that information into two or more smaller requests. This will reduce the perceived performance degradation. Making web pages smaller or creating a larger number of smaller web pages is also a possibility.
Solving Performance Problems in the Application Server (Step 3 &
7)
If the application server is identified as a bottleneck, examine the code carefully and/or move some logic to the database. If too many roundtrips are being made between the application server and the database, are Getters/Setters being overused? Is one record being retrieved with a single query when a set can be retrieved? If performance cannot be improved because the program logic requires bundles (or thousands of round trips), rewrite Java code in PL/SQL and move more code into the database.
Solving Performance Problems in the Client (Step 9)
If too much information is being moved to the client, the
only solution is to reduce the amount of that information. Changing
architectures, making web pages smaller, removing
Measuring Performance
Simply understanding a 9-step tuning process is not enough to be able to make a system work efficiently. There should be a formal, quantitative way to measure performance.
Necessary vocabulary:
· Command is an atomic part of the process (any command on any tier).
· Step is a complete processing cycle in one direction (always one-way) that can either be a communication step between one tier and another, or a set of steps within the same tier. A step consists of a number of commands.
· Request is an action consisting of a number of steps. A request is passed between different processing tiers.
· Round-trip means a complete cycle from the moment the request leaves the tier to the point when it returns with some response information.
Under the best of circumstances, the concept of a round-trip is redundant, but in real life getting precise measurements for all nine steps is extremely complicated:
· Steps 1,3,5,7,9 – both the start and finish of the step are within the same tier and the same programming environment
· Steps 2,4,6,8 – start and end are in different tiers.
Having entry points in different tiers means that there must be time synchronization between tiers, otherwise time measurements are completely useless. This problem can be partially solved in closed networks (like MilNet); but for the majority of Internet-based applications, it is a roadblock since there is no way to get reliable numbers.
The concept of a “round-trip” enables us to get around this issue. The 9-step model could be also represented as shown in Figure 2.
Figure 2: Round-TripTiming of 9-step Process
At the client level:
· From the moment that a request was initiated to the end of processing (user clicked the button/response is displayed)
· From the moment that a request was sent to the application server to the moment a response was received (start of servlet call/ end of servlet call)
At the application server level:
· From the moment that a request was accepted to the moment a response was sent back (start of processing in the servlet/end of processing in the servlet)
· From the moment that a request was sent to the database (start of JDBC call / end of JDBC call)
At the database level:
· From the moment that a request was accepted to the moment a response was sent back (start of the block/end of the block)
· Now there is a “nested” set of numbers that are 100% valid because they are measured on the same level. This allows us to calculate the following:
o Total time spent between the client and application server both ways (step 2 + step 8) = round trip 2 minus round trip 3
o Total time spent between the application server and the database both ways (step 4 + step 6) = round trip 4 minus round trip 5
Although there is no way to reduce this to a single step, it is significantly better than no data at all because two-way timing provides a fairly reliable understanding of what percentage of the total request time is lost during these network operations. These measurements provide enough information to make appropriate decisions about where to spend the most tuning resources, which is the most critical decision in the whole tuning process.
Conclusions
There is much more to tuning a web application than simply identifying slow database queries. Changing database and operating system parameters will only go so far. The most common causes of slow performance are as follows:
Excessive round trips from the application server to the database - Ideally, each UI operation should require exactly one round trip to the database. Sometimes, the framework (such as ADF) will require additional round trips to retrieve and make session data persistent. Any UI operation requiring more than a few round trips should be carefully investigated.
Large pages sent to the client - Developers often assume that all of the system users have high-speed Internet connections. Everyone has encountered slow loading web pages taking multiple seconds to load. Once in a while, these delays are not significant. However, this type of performance degradation (waiting 3 seconds for each page refresh) in an application (such as a data entry intensive payroll application) is unacceptable. Applications should be architected to take into account the slowest possible network to support when testing the system architecture for suitability in slower environments.
Performing operations in the application server that should be done in the database - For large, complex systems with sufficient data volumes, complete database independence is very difficult to achieve. The more complex and data-intensive a routine is, the greater the likelihood that it will perform much better in the database. For example, the authors encountered a middle tier Java routine that required 20 minutes to run. This same routine ran in 2/10 of a second when refactored in PL/SQL and moved to the database.
Poorly written SQL and PL/SQL routines - In some organizations, this may be the #1 cause of slowly performing web applications. This situation often occurs when Java programmers are also expected to write a lot of SQL code. In most cases, the performance degradation is not caused by a single slow running routine but a tendency to fire off more queries than are needed.
Keeping all nine of the potential areas for encountering performance problems in mind and investigating each one carefully can help to identify the cause of a slowly performing web application and point to ways in which that performance can be improved.
About the Authors
Dr.
Michael Rosenblum
is a Development DBA at Dulcian, Inc. He is responsible for system tuning and
application architecture. He supports Dulcian developers by writing complex
PL/SQL routines and researching new features. Mr. Rosenblum is the co-author of
PL/SQL for Dummies (Wiley Press,
2006). Michael is a frequent presenter at various regional and national Oracle
user group conferences. In his native