Business Requirements, Solution Architecture, Infrastructure Architecture, Application Architecture, Detailed Designs and followed by code.  A common pattern and sequence of artefacts which are delivered in the Software Development Life Cyle which most of us are familiar with in Enterprise System Development.  By the time a single line of code is etched into our system, we have already made a number of design decisions, selected key frameworks and began building and implementing the solution for the realisation of the business system.  Till this point, most well thought out and managed projects will have already made an attempt to specify the performance requirements of the system through Non-Functional Requirements, where response times and volumetrics are explicitly stated for the purposes of sizing the capacity of the system.

Typically, the volumetrics can be used to model the “work” the system must perform to process the business activities using metrics like throughput and transactions per second.  Response times are then used to constrain the system wait times and ensure the system can respond back to the end-user or client system in a timely manner.  These performance dimensions are both efficient and accurate methods to ensure architects can design the system to meet the business capacity requirements.

From a functional perspective, system testers verify the business rules are correctly implemented and the system is fault free and tolerant in all business scenarios.  Then, from a non-functional perspective, the performance & volume team ensure that the system is fault tolerant and performs to specification throughout the expected usage volumes.  As performance & volume testing is typically performed very late in the SDLC, performance problems, especially in custom build scenarios, can cause significant rework to the system and potentially jeopardise production deployment timelines and cost the program in schedule overruns and budget blowouts.  We can raise program risks and issues throughout the program to track and put mitigation plans in action to avoid performance problems, however, its very difficult to ensure a system will perform as designed without having a specific focus on performance throughout the design and development life cycle.

Enter The Performance Architect.

Having a focus on performance during all phases of the program (starting form requirements) is critical in truly managing all performance risks inherent in all programs.  The Performance Architects first job is to ensure the non-functional requirements are focused on setting the correct business objectives so the performance requirements are accurate, valid and achievable by the system.  Having set accurate and achievable performance targets, the following solution and technical architectures will remain “in-check” and will ensure the designs are performance considerate.  How is this achieved?  Architecture reviews, design reviews, instrstructure design reviews and finally code reviews.  Furthermore, code refactoring provides a very powerful and potent method of ensuring the system attains our performance objectives through the deployment of Code Profiling and Component Performance Testing patterns.  An implementation of this pattern is what l call the Performance Unit Test which l discuss in detail in my blog entry.

Typically we generate a plethora of Architecture documents as part of the SDLC, and l’m definitely not an advocate of adding more documents to the pile, however, the humble Performance Architecture or Performance View should be used to augment a solution design and provide clarity and guidance for major system components, such as Web Server and Application Server configurations.  A good example here is the need for a Web Server cache.  A performance architecture will, for example, define the need and configuration for a Web Server Cahce (like CacheRight), ensure bandwidth requirements are modelled, provide view for metrics collections and model the response times of each sub-system.

At Odecee, we have created an architecture practice which involves specialisation in performance through formal offerings in Peformace Architecure.  The uptake in our performance services in-and-around our performance architecture competency have added direct value to their SDLC and its pleasing to see the outcomes achievable by deploying augmenting performance-centric practioners into client architecure teams.

So,  Ill finish by asking my readers a simple question; can we afford to neglect this vital design elements of an enterprise business system?

Recently, some of our clients had asked us to determine an early performance view of an application that they are either considering to purchase or have recently purchased. Considering all things that make up a J2EE application, this can be difficult to represent. So, how can we present a view of the systems performance to the customer and satisfy all their concerns around product performance?

The answer is relatively trivial. A performance view can be derived using two methods; bottom-up or top-down.

Lets consider the bottom-up approach. This will typically involve an application architecture review, where all major architectural components of the system are identified and then analysed. Analysis of the components typically involve inspecting the configuration of the component in a view to derive the performance sociability of the component. A good example here would be the Hibernate 2nd Level Cache. As developers of Hibernate will testify, the configuration of the 2nd level cache can be very tricky and unforgiving. The cache configuration will typically involve the caching of both static and/or dynamic database data. Static data can be cached without any thought as the data is read-only and will not change during the runtime of the JVM. However, the same cannot be said about dynamic data. Dynamic data requires very careful thought as the developer is playing against transaction management and isolation levels. A simple oversight in design can cause data integrity issues that are hard to find when performance testing.

With view of all the architectural components and the way they configured to interact with each other, a performance architect can derive a scalability view of the system. Depending on the required performance detail, a model can then be formulated which can attempt to stochastically derive system configuration values to select optimal configuration parameters for the system.

Now, lets look at the top-down view. Applying operational analysis, we can define the expected end user usage characteristics and create a load model; also known as an Application Simulation Model. This load model is considered the system input for which the system will be measured for performance. The operational model is the commonly used performance and volume testing method. Applying the load model and measuring system metrics like CPU, memory, network utilisation etc. the performance engineer can view the system scalability on a given platform.

Having defined the two approaches, lets consider the analysis of the data and look at the results that we can obtain from both approaches. To the skillful performance engineer, the data provided from the architectural analysis using the bottom-up approach will give clues and allow conclusions to be derived of the systems performance capabilities. Continuing with the Hibernate 2nd level cache example, a performance engineer could conclude that the requirement to distribute updates to other caches in a cluster of JVM’s will incur an N+1 update overhead. This synchonise operation is, at best, an interprocess call or, worst case, a network call to write the cache state changes to the other JVM’s. Not very good for performance if the system requires a large number of JVM’s. This heuristic view of a component behavior is usually enough to flag a performance issue. The point here is that although the performance engineer is certain of a scalability issue, the results are theoretical and will require proof using operational model methods. Generally speaking, the performance engineer will not be able to derive a concrete scalability result from the bottom-up approach.

Operational analysis, or top-down approach, will provide clearer, more tangible result for the scalability of the system. The results of the test will generally be represented as something like: 100 logged in users, CPU at 80% capacity, memory at 80% capacity and network at 80% capacity. Normalising this view using a well defined, divisible entities is what l’ve found to provide the best view of system scalability.

Let me further clarify what is meant by well defined, divisible entities. To achieve the most potent results, the software and hardware configuration needs to be as simple as possible and representative of the minimum components of the system; for example, a single node for the JVM, a single node for the webserver and a single node for the database. This way, this forms the fundamental unit of configuration. From this point, the application can be horizontally or vertically scaled. Here is a view of this configuration:

The diagram illustrates a simple system configuration and describes the method which virtual users access the system.

By ensuring that the system remains simple and each software and hardware components are configured to the minimum required level to access all possible system functions, we can run our Application Simulation Model load into the system and measure the key system metrics. These measured metrics grouped together and presented as a whole is what l call the Unit of Scalability. Once this fundamental view is acquired, other scalability attributes are very easily derivable. For example, the Unit of Scalability can be used to derive whether the system scales vertically or horizontally in a linear manner. To determine the horizontal scalability attributes, simply doubling the hardware and software components and executing the same simulation model will reveal the systems linear scalability capabilities.

Having calculated the Unit of Scalability and further derived the systems vertical and/or horizontal scalability attributes, we can accurately quote system capacity requirements, and more importantly, present our customers with a simple view of a systems performance and scalability.

As a developer, I feel performance is too often neglected by the development team. I’ve been on a few projects that have been really compromised by the performance aspects of a JEE system, specifically because of the lack of performance testing during the development cycles of the SDLC. Often, as performance issues are identified during performance testing (albeit transactions not meeting Service Level Agreements, bloated memory profiles, transaction timeouts etc.) the development team will be summoned to identify the performance related issues, mostly at the end of the testing cycles and very late in the SDLC.

So, what do the development team do to identify performance issues in a system? They hypothisise, they reason, they eliminate suspects from potential problem candidates, then finally settle on bringing out a profiler of choice. Having suspected areas of concern, they target the profiler into the functional area an begin capturing runtime behavior.

Each captured profile is analysed by the developer and performance relating defects are identified and corresponding defects raised. These defect are then fixed and released through the testing cycles. This is the classic and well known process of finding performance related defects during performance testing, raise the testing defect, fix the defect in the development cycle, then finally release back to testing. Can we see parallels to our novel approaches of the past? Very waterfall indeed.

In the past, we had looked for innovative ways to put a stop to this type of waterfall development by attempting to identify and rectify as many defects during development as possible. We have developed sophisticated unit testing, integration testing, application automation scripting systems in a view to sufficiently unit test our code and pick up defects early in development; however, we have neglected performance aspects of our code.

And it still remains neglected.

A fellow odecee colleague and I decided to have a think about his problem and devise a new standard for our development teams. And on that one day, the Performance Unit Test was born.

We begin to describe the Performance Unit Test as a developer deliverable which forms a part of the overall Unit Testing process. It’s function is to prove the applications performance concerns are met. Each Performance Unit Test must be repeatable and (if possible) automated for simple execution. Results must be analyzed by the unit tester, fixing any identified performance problems. Results of the Performance Unit Test form part of the Code Reviews and are verified by the reviewer. The developer must step through the Performance Unit Test results with the reviewer before the components are formally signed off.

Employing the Performance Unit Test in the development process can seem expensive at first, given that it extends the development times of a particular component, however, as with an effective Unit Testing implementation, has the capacity to greatly reduce risk of performance related issues late in the SDLC and will ensure your application code delivered into performance testing is highly performant and optimised.