When writing Java applications that interact with a relational database, it is typical to use the JDBC PreparedStatement class to execute queries and updates.  There are several key reasons for using this class instead of the simpler Statement class:

• Simplified use of variables.  The use of bind variables avoids having to escape the variables being added to the query.  This avoids dealing with formatting variables date formats, escaping string terminators, and eliminates SQL injection attacks.
• Query Plan Re-Use.  The removal of dynamic variables from the query string allows the database to cache the resultant query plan.  The database engine will see two queries using the same query string and can re-use the previously calculated query plan.
• Re-Use of Prepared Statements.  A prepared statement can be re-used for many operations against the database.  This may be used multiple times within a single transaction, or re-used across many transactions using client-side Prepared Statement caching.

There are exceptions to every rule however, and there are cases where Prepared Statements do not provide the optimal solution.  This article discusses a recent scenario in which using Prepared Statement functionality had a serious detrimental impact on performance, and discusses the options available for resolving the issue.

Problem

The Open Street Map project is an Internet community that aims to become the “Wikipedia of Maps”.  The project is building a vector data set containing features such as roads, geographical boundaries, coastlines, buildings, points of interest, and in fact any feature which exists in the physical world.  It has many parallels to Google Maps, but the data is freely available in vector form.  This geo-spatial data set is a technology enabler allowing all kinds of new and innovative tools and applications to be built.

The central Open Street Map database is approximately 1 terabyte in size, and contains many hundreds of millions of rows.  Statistics showing the growth and activity over time can be found at the following URL:

http://wiki.openstreetmap.org/wiki/Stats

A complete snapshot of the dataset without history can be downloaded in compressed XML form and is currently approximately 6.6 gigabytes expanding to approximately 125 gigabytes when uncompressed.  This snapshot known as the “planet” file provides the basis for consumers of the data to build their own databases and applications to use the data.  Due to the size of the planet file, it is not feasible to download too often and therefore consumers cannot get access to new data in real-time.  A new planet file is made available weekly at the following URL:

http://planet.openstreetmap.org

To solve the real-time problem of access to data, a tool called Osmosis is used to extract regular changesets from the database and post them on a public web server for download.  These changesets are available in day, hour, and minute intervals.  Osmosis is written in Java, and uses JDBC to access the production PostgreSQL relational database.  Osmosis queries changes from 8 major tables, and several supporting tables.  It extracts these changes based on timestamp.  It queries for all data created within a particular time interval, and writes the changes to a new changeset file.

Using these change files, it is possible for consumers to keep their own databases synchronised with the production database.  Using the minute changesets, it is possible to keep a replica database within approximately 6 minutes of the primary database.  This near real-time feed eliminates the need for most applications to directly access the production database drastically improving performance for applications and allowing the production database to scale to a much larger user base.

Recent enhancements to Osmosis resulted in many of the queries to be re-written.  Unfortunately, the performance of these queries was unacceptable.  The queries need to execute in such a way that their execution time is proportional to the amount of data being extracted.  The new queries were taking several hours to run regardless of the time interval being queried.  This is bad when extracting day changesets, and completely unworkable when extracting minute changesets.

Diagnosis

The database contains the following primary entity types:

• Nodes – Represent a single point on a map.  Defined in terms of a latitude and longitude (eg. a point of interest).
• Ways – Represent a linear path between several nodes (eg. a road).
• Relations – A logical grouping of several nodes, ways and/or relations.

Each of those tables has a corresponding tag table which allows name/value pairs to be attached to describe the feature.  For example a road may have the tags highway=primary and name=<street name> attached to it.

There are several other key tables in the database, the key ones being:

• Changesets – All edits are grouped into changesets.  Each node/way/relation is related to a changeset.
• Users – All edits are performed by a registered user.  Each changeset is related to a user.

The database contains many other tables, but not important to this discussion.  For example, ways have a join table to the node table and relations have a generic join table to nodes, ways and relations.

The problematic query is shown below:

SELECT e.id, e.version, e.timestamp, e.visible, u.data_public,
u.id AS user_id, u.display_name, e.changeset_id, e.latitude, e.longitude
FROM nodes e
INNER JOIN (
SELECT id, version FROM
nodes WHERE timestamp > ? AND timestamp <= ?
) t ON e.id = t.id AND e.version = t.version
INNER JOIN changesets c ON e.changeset_id = c.id
INNER JOIN users u ON c.user_id = u.id


The query could be simplified, but the sub-select is included to improve code re-use.  The sub-select can be replaced with a different query that has different selection criteria without affecting the remainder of the query.

A test date range of 1 second in duration was used for analysis.  If a 1 second query can be made to run quickly, larger duration queries typically follow suit with their duration approximately proportional to data size.  The sample date range was from “2009-06-08 00:00:00” to “2009-06-08 00:00:01”.  This resulted in the following query:

SELECT e.id, e.version, e.timestamp, e.visible, u.data_public,
u.id AS user_id, u.display_name, e.changeset_id, e.latitude, e.longitude
FROM nodes e
INNER JOIN (
SELECT id, version FROM
nodes WHERE timestamp > '2009-06-08 00:00:00' AND timestamp <= '2009-06-08 00:00:01'
) t ON e.id = t.id AND e.version = t.version
INNER JOIN changesets c ON e.changeset_id = c.id
INNER JOIN users u ON c.user_id = u.id


This query was evaluated using the EXPLAIN command which produced the following query plan.

Query plan with 1 second interval

This query plan is reasonable and doesn’t explain the poor performance. The large node and medium changeset tables are joined via nested loops, and the small user table is joined via a hash join.  This query took less than 1 second to execute. It was apparent that the Java implementation was using a different query plan but not immediately obvious why.

The Java implementation was run again and was still executing several minutes later without result. The database process was examined and was causing heavy disk IO so it wasn’t blocked on a database lock.

Full JDBC tracing was enabled to determine if the JDBC driver was doing anything unusual such as messing with isolation levels. This trace can be enabled in the PostgreSQL driver by adding the following parameter to the connection URL:

loglevel=2


Examination of the trace showed the code performing a standard connection to the database and beginning a transaction as expected, then performing a prepare/bind/execute of the query, reading the query results, then committing the transaction and closing the connection. Nothing unusual was found in the trace.

The only difference that could be found between the Java initiated query and a console query was the use of bind variables. The Java code was modified to avoid the use of bind variables and directly pass the dates within the query. This was tested and found to produce efficient queries.

Some further investigation was done into PostgreSQL query planning and it was found that prepared statements can produce non-optimal query plans due to the fact that a query plan is created prior to the bind variables being available. In our case the time interval is typically small and selects a small subset of the data in the node table. The query planner doesn’t know this and assumes that a much larger dataset will be selected. As a result it resorts to full table scans which perform horribly when a small time interval is being selected. This is mentioned at the following URL:

http://jdbc.postgresql.org/documentation/83/server-prepare.html

This behaviour was verified at the console by preparing a statement and retrieving the query plan as follows:

PREPARE mystatement (timestamp, timestamp) AS
SELECT e.id, e.version, e.timestamp, e.visible, u.data_public,
u.id AS user_id, u.display_name, e.changeset_id, e.latitude, e.longitude
FROM nodes e
INNER JOIN (
SELECT id, version FROM
nodes WHERE timestamp > $1 AND timestamp <= $2
) t ON e.id = t.id AND e.version = t.version
INNER JOIN changesets c ON e.changeset_id = c.id
INNER JOIN users u ON c.user_id = u.id;
EXPLAIN EXECUTE mystatement('2009-06-08 00:00:00', '2009-06-08 00:00:01');


This produced the following plan:

Query plan with bind variables

This plan is using a full table scan on the node table which is completely inappropriate for returning 27 rows from a table containing in excess of 500 million rows.  The database server has 32GB of RAM, so smaller tables such as the users and even changeset tables can be scanned without major problems, but the main node/way/relation tables and their child tables must use existing indexes.

Resolution

At this point we knew that prepared statements were causing non-optimal query plans but had to find a way to resolve it. We could modify the queries to avoid bind variables, but this opens up a can of worms with having to format dates correctly and deal with potential timezone issues, bind variables produce much more reliable code.

After some investigation, two options were pursued.

Option 1

PostgreSQL provides a number of parameters that can be set on a per connection basis to affect how the query plan is built. These parameters are boolean parameters that are true by default. They can be selectively set to false. There are many of these parameters, but several key ones are listed below:

• enable_seqscan – If false, the query planner will avoid sequential scans where possible.
• enable_mergejoin – If false, the query planner will not use merge joins.
• enable_hashjoin – If false, the query planner will not use hash joins.

These parameters were experimented with and it was found that all three had to be set to false in order to make the Java initiated queries perform efficiently. The PostgreSQL documentation specifies that these parameters should be avoided if possible, however they do provide a quick simple way of brute forcing the query planner into giving the desired behaviour.

Option 2

The problem was occurring due to the use of server side prepared statements. This prevents the query planner from getting access to the actual variables that will be used. In this case it is incorrectly assuming that a much larger dataset is going to be returned than is actually the case. The Osmosis application has been designed to retrieve all data in a small fixed number of queries meaning that there are no gains to be had from re-using prepared statement query plans, instead it is much more important to get the best query plan possible.

The PostgreSQL JDBC drivers provide the option of using client side prepared statements. This is discussed at the following URL:

http://jdbc.postgresql.org/documentation/83/server-prepare.html

At the time of writing, this approach is still being investigated but if it can be made to work will provide a cleaner solution than Option 1.

Conclusion

It is often assumed that JDBC Prepared Statements are a better approach than dynamic SQL from both a performance and functional point of view. While they are undoubtedly the right approach to create clean bug free code, this investigation showed that they can be severely detrimental in scenarios where an optimal query plan is critical to meeting performance requirements. In an application where the number of queries is low and the data sizes are large, the overheads of creating query plans from scratch are minimal and the impact of a sub-optimal query plan is expensive.

Prepared Statements should still be the preferred method used for building SQL queries where possible. They are likely to provide the best performance in a high volume online transactional application with large numbers of small queries due to their ability to be cached both at the client and server side. However when performance issues arise, careful attention should be paid to the impact of late-binding of dynamic variables. If bind variables are causing sub-optimal query plans to be generated, measures must be taken to provide manual hints to the query planner or find a way of disabling server side prepared statements.

As with most optimisations, normal Prepared Statement functionality should not be deviated from unless performance issues can be measured and attributed to them. This is another non-obvious issue that a performance engineer needs to be mindful of and identify when it occurs.

What are the hallmarks of a quality software development team?  What attributes stand out from software which gives it the feeling of quality?  These questions and many more, lead to the seemingly elusive quality answer.  Certainly, answers to these quality questions have found themselves entrenched in today’s modern software development life cycle process, however, lm continually presented with quality challenges in all projects which l have been fortunate to be a part of.

Throughout my career, l have seen many quality processes and patterns, all with great intentions, both succeed and fail.  Some processes have been too strict and focussed on impractical code coverage targets, whilst others were focused on minimal unit testing but strong focus on integration testing.  Some programs embraced automation, others rejected it.  Coupled with the heterogeneous and diverse business domains, it can be very difficult to select and accept quality patterns as each of them present their own unique challenges to the quality problem.

One thing that l have proven irrefutable is that repeatability is a key enabler of quality.  Repeatability is, after all, the critical component of the Scientific Method which has roots from the past times of Galileo.  Repeatability reinforces the strength of observation, and observation allows identification of patterns, facilitating classification, behavioural analysis and many other exploitable scientific methods of analysis. Repeatability in software development is also a critical dimension in quality assurance. Any process that is automated is highly repeatable and precise, leaving no room for error. This is why computers were created in the first place! However small a task, automation can add significant value to the overall quality dimensions of the system.

On a macro scale, teams within a program will have a number of concerns which they provide services for. For example, the environments or infrastructure teams will build and maintain all environments during the project. They will ensure deployments are executed to the target environments and availability targets of the environments under-management are maintained. The environments team could benefit from an automated deployment system, which is developed to ensure consistency in configuration and deployments. Again, repeatability in deployments increases the quality of the deployments and manifests into higher application availability times and lowers defects relating to environment misconfigurations.

Until now, l’ve described and illustrated that repeatability can increase the quality of deployments if facilitated through an automation framework. This approach is not only relevant for the environments team, and a more holistic approach to automation is required to ensure all facets of the SDLC benefit from the quality enablers of repeatability. This is the task of the Build Pipeline.

The build pipeline has cross-cutting concerns which define the processes for all technology teams in the SDLC. Primarily focussing on development, build engineering and environments teams, the build pipeline formalises and provides clarity into the software development and release processes of the program. Each of the development, build engineering and environments teams will assume custodianship of a portion of the build pipeline to enable the end-to-end artefact creation and assurance process.

Figure 1 - The Build Pipeline

The build pipeline begins with the developers local build. Here, the developer will execute unit testing for the purposes of check-in. Once verified, the process is repeated on the Continuous Integration server, where an additional deployment and integration test cycle is executed. Once verified, the build engineering team will deploy the application into a deployment test environment and reverify. Finally, the environment team will take the, now highly verified software artifacts, and fan-out the deployments into all downstream environments, albeit, System Integration Testing, User Acceptance Testing, even, Production. One key point to mention here, the verification processes in the build pipeline will serve as a quality contract between the teams. For example, the build engineering teams will assert the applications units and integration’s are tested (through unit and integration testing performed in both development and continuous integration) and fail the pipeline operation if this contract is broken.

Realising the build pipeline requires diligence, commitment, collaboration and attention to detail. Once employed, the build pipeline has a proven quality yield that reduces cost through shorter testing cycles and reduced defect numbers. Furthermore, consistent and repeatable deployments through automation will promote the consistency and fault-free environment configurations. Sure, there are upfront costs associated in establishing the build pipeline, however, ignoring the need for one and assuming quality is achievable through our novel methods of the past, have time and time again, proven to be a false economy.

Business Requirements, Solution Architecture, Infrastructure Architecture, Application Architecture, Detailed Designs and followed by code.  A common pattern and sequence of artefacts which are delivered in the Software Development Life Cyle which most of us are familiar with in Enterprise System Development.  By the time a single line of code is etched into our system, we have already made a number of design decisions, selected key frameworks and began building and implementing the solution for the realisation of the business system.  Till this point, most well thought out and managed projects will have already made an attempt to specify the performance requirements of the system through Non-Functional Requirements, where response times and volumetrics are explicitly stated for the purposes of sizing the capacity of the system.

Typically, the volumetrics can be used to model the “work” the system must perform to process the business activities using metrics like throughput and transactions per second.  Response times are then used to constrain the system wait times and ensure the system can respond back to the end-user or client system in a timely manner.  These performance dimensions are both efficient and accurate methods to ensure architects can design the system to meet the business capacity requirements.

From a functional perspective, system testers verify the business rules are correctly implemented and the system is fault free and tolerant in all business scenarios.  Then, from a non-functional perspective, the performance & volume team ensure that the system is fault tolerant and performs to specification throughout the expected usage volumes.  As performance & volume testing is typically performed very late in the SDLC, performance problems, especially in custom build scenarios, can cause significant rework to the system and potentially jeopardise production deployment timelines and cost the program in schedule overruns and budget blowouts.  We can raise program risks and issues throughout the program to track and put mitigation plans in action to avoid performance problems, however, its very difficult to ensure a system will perform as designed without having a specific focus on performance throughout the design and development life cycle.

Enter The Performance Architect.

Having a focus on performance during all phases of the program (starting form requirements) is critical in truly managing all performance risks inherent in all programs.  The Performance Architects first job is to ensure the non-functional requirements are focused on setting the correct business objectives so the performance requirements are accurate, valid and achievable by the system.  Having set accurate and achievable performance targets, the following solution and technical architectures will remain “in-check” and will ensure the designs are performance considerate.  How is this achieved?  Architecture reviews, design reviews, instrstructure design reviews and finally code reviews.  Furthermore, code refactoring provides a very powerful and potent method of ensuring the system attains our performance objectives through the deployment of Code Profiling and Component Performance Testing patterns.  An implementation of this pattern is what l call the Performance Unit Test which l discuss in detail in my blog entry.

Typically we generate a plethora of Architecture documents as part of the SDLC, and l’m definitely not an advocate of adding more documents to the pile, however, the humble Performance Architecture or Performance View should be used to augment a solution design and provide clarity and guidance for major system components, such as Web Server and Application Server configurations.  A good example here is the need for a Web Server cache.  A performance architecture will, for example, define the need and configuration for a Web Server Cahce (like CacheRight), ensure bandwidth requirements are modelled, provide view for metrics collections and model the response times of each sub-system.

At Odecee, we have created an architecture practice which involves specialisation in performance through formal offerings in Peformace Architecure.  The uptake in our performance services in-and-around our performance architecture competency have added direct value to their SDLC and its pleasing to see the outcomes achievable by deploying augmenting performance-centric practioners into client architecure teams.

So,  Ill finish by asking my readers a simple question; can we afford to neglect this vital design elements of an enterprise business system?

Open for extension, closed for modification. The open-closed principle is on the of the great object oriented design principles of our time. Its a design principle that is very fragile where the smallest of changes can easily undo the closed-ness of a system. So, l’ll to summarise the principle for the purposes of our discussion, and to allow me to make some assertions on the validity of the principle for Enterprise Business Systems.

The open/closed principle states “software entities (classes, modules, functions, etc.) should be open for extension, but closed for modification” [Meyer, Bertrand (1988). Object-Oriented Software Construction. Prentice Hall. ISBN 0136290493]. In principle, each software entity can allow it’s behaviour to be changed without the alteration to it’s source code.

So, l’ll start by giving an example of a successful outcome for the Open-Closed principle. My last project required the extension of Spring to enable some extra behavior during bean factory construction. So, we hunted and found the Spring class that implemented the XML file parsing and decided to extend the functionality. It was really simple, NewSpringClass extends SpringClass, then add the new behavior. Done! I did not need to download, modify then build the Spring code to add the new behavior. The Spring classes were implemented in a way that allowed for simple extension, otherwise we could have found ourselves doing things that we should be banished for doing.

Now, lets look at another modification scenario; changing business requirements. As new business requirements become known, we need to add new or modify old behavior of the system. How often is this done by extending the functionality of a system classes instead of modifying them? Well, thinking back, extension is rarely done, it’s mostly modification.

For example, when the business want the behavior of the system to debit accounts in a scenario instead of crediting the account, this will most definitely result in a modification. Does this mean that l’ve not implemented by Enterprise System correctly? Fortunately enough, this is not the case. I’ve used most of the good design practices like keeping my concerns separated, ensuring that my layers are well defined, designed my service interfaces at the correct granularity, avoided anti-patterns, etcetera. So, why is it that my changes to the Enterprise System is mostly modification?

The answer is the rate of change of the system during a standard development process. Behavior and functionality of a framework like Spring can be defined priori, where the changes of system requirements is significantly smaller than the change expected from Enterprise Systems. So, the Open-Closed principle breaks down when the requirements of a system are unpredictable. That’s the definitive point.

So, given l have demonstrated that the rate-of-change of Enterprise Business Systems has the propensity to force the Open-Closed principle out the door, what other options do we have in this architectural domain?  I believe that there are many; although the answers may not be in other patterns or principles.  All solutions in this domain need to be carefully considered where options of each solution carefully weighed-up, prioritised and recorded.  Applying a pragmatic view will always present a viable, well thought out solution.

Tree Walking, the art of navigating an object graph or relational model through the use of the Visitor (GoF) or any other arbitrary algorithm. The concept is relatively trivial, in that you start at a node in the interconnected system and then make navigation decisions based on some logic or rules. Most times, the structure of the tree is known a-priori and therefore the navigation is analogous to navigating a map.

Object To Relational mapping, or better known as ORM is one of the application that we can see Tree Walking in practice. For example, given a domain model which defines associations between entities can be instantiated to reveal a complex tree. Here is a typical example:

In the above example, we can see the domain model where the associations and cardinality of each entity is illustrated. Although the above UML represents classes, this is not an instance view. The instance view is significantly different and more complex to represent, given that there are some unbounded association properties of the model. Therefore, the representation of the instance view can be complex for the given model and is best left to the imagination.

Continuing with our ORM discussion, we can map the above model onto the database tables using metadata and achieve the ORM solution. Up till now, a majority of the implementation details have been inherited from the ORM tool selection, e.g. using annotations or XML to implement the metadata. What’s left now, is the answers to the “hard” questions, typically relating to resource management, performance views, dependency management, transaction management and a whole list of other Application Architecture concerns.

One particular Application Architecture concern that l have seen poorly defined and designed time and time again, is the handling of object graphs that are central to ORM. To further elaborate, object graphs are used all throughout the client and business tiers of an application. They can be constructed from the view and passed to the business tier for persistence, or can be generated from the business tier for consumption (i.e. rendering) by the web tier. The business tier, is where my arch nemesis, the Tree Walker often inhabits. For example, ORM implied constraints like lazy loading, can be a large enough force for some Application Architects to mandate a form of Tree Walking to mitigate N+1 selects problems, specifically to improve application performance. The choice seems obtuse to me, as the complexity associated to Tree Walkers often leads to architectural debt, which is more than not, remediated later in project maintenance cycles; mostly by means of the removal of the Tree Walker.

A recent sighting of a Tree Walker got me gasping as the unnecessary complexity it introduced into the persistence layer. After inspecting the anatomy of the Tree Walker, as with all my other Tree Walker sightings, l was able to dismiss it’s requirement in the solution very easily. This particular application architecture was simple, JSF, Hibernate and Spring all contained in a single WAR deployed in a Servlet container (i.e. no EJB). I was able to dismiss its requirement citing:

• Hibernate session is in scope of the View and does not need to a shortened connection lifecycle.
• Lazy collections can be hydrated from the database when the view is walking the object graph in the rendering phase of the request.
• Why not use already proven and mature Hibernate HQL to fetch the required object graph shape?

The main purpose of the Tree Walker in this instance, was to (a) optimise the database access by ensuring all associations in the object graph are lazy when first loaded into memory from the database and (b) reduce the time connections to the database is kept open; hence avoiding the Hibernate LazyIniitializationException.

Even if the database session was not in the scope of the View (i.e. in the business tier and behind an EJB), point 3 from the above list would still hold strong. This made me think about my previous blog entry on a similar topic; N+1 has leaked into my service interfaces. Does the Tree Walker stop the ORM constraints from leaking in to my service interfaces? Well, no. A developer still needs to define the directions for the Tree Walker which it will use to navigate the object graph. Although the service interfaces may appear to be more simple, the fact that you need to define a parameter to accept the directions for the Tree Walker means you have not avoided service interface pollution.

Time and time again, I often think what drives Application Architects, Designer and Developers to develop a solution that is, simply, not required. Is the art of simplicity something that can be learned? How can one reject thoughts that lead us to develop complex architectures when they are clearly not required? This brings me to a belief that I have held for a very long time; intelligence alone cannot buck the forces of over-engineering; it is wisdom that guides a truly skilled architect to a solution that’s both elegant and simple.

Recently, some of our clients had asked us to determine an early performance view of an application that they are either considering to purchase or have recently purchased. Considering all things that make up a J2EE application, this can be difficult to represent. So, how can we present a view of the systems performance to the customer and satisfy all their concerns around product performance?

The answer is relatively trivial. A performance view can be derived using two methods; bottom-up or top-down.

Lets consider the bottom-up approach. This will typically involve an application architecture review, where all major architectural components of the system are identified and then analysed. Analysis of the components typically involve inspecting the configuration of the component in a view to derive the performance sociability of the component. A good example here would be the Hibernate 2nd Level Cache. As developers of Hibernate will testify, the configuration of the 2nd level cache can be very tricky and unforgiving. The cache configuration will typically involve the caching of both static and/or dynamic database data. Static data can be cached without any thought as the data is read-only and will not change during the runtime of the JVM. However, the same cannot be said about dynamic data. Dynamic data requires very careful thought as the developer is playing against transaction management and isolation levels. A simple oversight in design can cause data integrity issues that are hard to find when performance testing.

With view of all the architectural components and the way they configured to interact with each other, a performance architect can derive a scalability view of the system. Depending on the required performance detail, a model can then be formulated which can attempt to stochastically derive system configuration values to select optimal configuration parameters for the system.

Now, lets look at the top-down view. Applying operational analysis, we can define the expected end user usage characteristics and create a load model; also known as an Application Simulation Model. This load model is considered the system input for which the system will be measured for performance. The operational model is the commonly used performance and volume testing method. Applying the load model and measuring system metrics like CPU, memory, network utilisation etc. the performance engineer can view the system scalability on a given platform.

Having defined the two approaches, lets consider the analysis of the data and look at the results that we can obtain from both approaches. To the skillful performance engineer, the data provided from the architectural analysis using the bottom-up approach will give clues and allow conclusions to be derived of the systems performance capabilities. Continuing with the Hibernate 2nd level cache example, a performance engineer could conclude that the requirement to distribute updates to other caches in a cluster of JVM’s will incur an N+1 update overhead. This synchonise operation is, at best, an interprocess call or, worst case, a network call to write the cache state changes to the other JVM’s. Not very good for performance if the system requires a large number of JVM’s. This heuristic view of a component behavior is usually enough to flag a performance issue. The point here is that although the performance engineer is certain of a scalability issue, the results are theoretical and will require proof using operational model methods. Generally speaking, the performance engineer will not be able to derive a concrete scalability result from the bottom-up approach.

Operational analysis, or top-down approach, will provide clearer, more tangible result for the scalability of the system. The results of the test will generally be represented as something like: 100 logged in users, CPU at 80% capacity, memory at 80% capacity and network at 80% capacity. Normalising this view using a well defined, divisible entities is what l’ve found to provide the best view of system scalability.

Let me further clarify what is meant by well defined, divisible entities. To achieve the most potent results, the software and hardware configuration needs to be as simple as possible and representative of the minimum components of the system; for example, a single node for the JVM, a single node for the webserver and a single node for the database. This way, this forms the fundamental unit of configuration. From this point, the application can be horizontally or vertically scaled. Here is a view of this configuration:

The diagram illustrates a simple system configuration and describes the method which virtual users access the system.

By ensuring that the system remains simple and each software and hardware components are configured to the minimum required level to access all possible system functions, we can run our Application Simulation Model load into the system and measure the key system metrics. These measured metrics grouped together and presented as a whole is what l call the Unit of Scalability. Once this fundamental view is acquired, other scalability attributes are very easily derivable. For example, the Unit of Scalability can be used to derive whether the system scales vertically or horizontally in a linear manner. To determine the horizontal scalability attributes, simply doubling the hardware and software components and executing the same simulation model will reveal the systems linear scalability capabilities.

Having calculated the Unit of Scalability and further derived the systems vertical and/or horizontal scalability attributes, we can accurately quote system capacity requirements, and more importantly, present our customers with a simple view of a systems performance and scalability.

As a developer, I feel performance is too often neglected by the development team. I’ve been on a few projects that have been really compromised by the performance aspects of a JEE system, specifically because of the lack of performance testing during the development cycles of the SDLC. Often, as performance issues are identified during performance testing (albeit transactions not meeting Service Level Agreements, bloated memory profiles, transaction timeouts etc.) the development team will be summoned to identify the performance related issues, mostly at the end of the testing cycles and very late in the SDLC.

So, what do the development team do to identify performance issues in a system? They hypothisise, they reason, they eliminate suspects from potential problem candidates, then finally settle on bringing out a profiler of choice. Having suspected areas of concern, they target the profiler into the functional area an begin capturing runtime behavior.

Each captured profile is analysed by the developer and performance relating defects are identified and corresponding defects raised. These defect are then fixed and released through the testing cycles. This is the classic and well known process of finding performance related defects during performance testing, raise the testing defect, fix the defect in the development cycle, then finally release back to testing. Can we see parallels to our novel approaches of the past? Very waterfall indeed.

In the past, we had looked for innovative ways to put a stop to this type of waterfall development by attempting to identify and rectify as many defects during development as possible. We have developed sophisticated unit testing, integration testing, application automation scripting systems in a view to sufficiently unit test our code and pick up defects early in development; however, we have neglected performance aspects of our code.

And it still remains neglected.

A fellow odecee colleague and I decided to have a think about his problem and devise a new standard for our development teams. And on that one day, the Performance Unit Test was born.

We begin to describe the Performance Unit Test as a developer deliverable which forms a part of the overall Unit Testing process. It’s function is to prove the applications performance concerns are met. Each Performance Unit Test must be repeatable and (if possible) automated for simple execution. Results must be analyzed by the unit tester, fixing any identified performance problems. Results of the Performance Unit Test form part of the Code Reviews and are verified by the reviewer. The developer must step through the Performance Unit Test results with the reviewer before the components are formally signed off.

Employing the Performance Unit Test in the development process can seem expensive at first, given that it extends the development times of a particular component, however, as with an effective Unit Testing implementation, has the capacity to greatly reduce risk of performance related issues late in the SDLC and will ensure your application code delivered into performance testing is highly performant and optimised.

Over the course of my career, I have been eagerly reading, interviewing colleagues and experimenting with my own unit testing methodologies. It was only recently, that a fellow odecee colleague and I sat in a room and attempted to map out what we thought was a practical and repeatable unit testing methodology that we could take to other projects and cement a position on a unit testing pattern for odecee.

After brainstorming our ideas and categorising all the types of development testing, we were able to define a taxonomy that represented all types of tests that a development team should be creating. After some further refinement, I described the process in an odecee white paper with view to lay the foundations for odecee’s unit testing methodology.

Central to our discussions was the topic of this blog; when should a developer avoid testing in the container? To better illustrate the question, it’s best if I define what container avoidance is. First and foremost, all JEE applications will have a target deployment environment which typically includes an installation of some JEE container (WebSphere Application Server, Tomcat etc.) where the application that is being developed will be deployed to. Now, all containers provide foundation services to which the application can call upon to access some resource; albeit a database, LDAP directory, Connector, JMS provider, even the humble HTTPRequest object. The application under development will typically have a dependency on one or more resources managed by the container and will therefore need the services during the execution of the application. If you want to run your unit tests on an application that has a target container which is managing resources, then testing your application outside your container will require a strategy to provide replacements for your container managed services. The process in which a solution is derived that substitutes a container service for a fake one, albeit a stub or mock, is what I call container avoidance.

One implementation of a container avoidance strategy is to use an off-the-shelf framework like Spring, where each class in your application is configured as an “bean” and defined to have dependencies to other “beans”. Loosely coupling your java classes in this manner puts you in an ideal situation to abstract all your (potentially) container service providers. Now, swapping out your providers should be a simple configuration task; a task that can be abstracted by your unit testing framework, like the AbstractTransactionalDataSourceSpringContextTests. Seemingly simple really, but does require some planning and commitment to the testing pattern.

Once all your providers are abstracted, we have achieved container avoidance and can now run our primary test objects outside the container and weave our test code.

So, now that we have a method to avoid most container services, we’ve managed to reach further into our classes using our unit test code than ever before. Has this process increased the likelihood of a “works-first-time” deploy? No, it certainly has not. In the view to implement container avoidance, we’ve missed testing other aspects of the system under test, like Deployment Descriptors, System Configurations, View Concerns and many more things that make up the working application.

Although we have definitely achieved a engineering feat, l would characterise container avoidance as an academic task; one which can easily miss the pragmatic viewpoint of unit testing: proving the business function.

Well, where too from here? How far should we go with abstracting our providers? Only datasources? Only EJB’s? What about Web Services? It’s a tough question. And the answer depends on your philosophy of unit testing. Complex systems have many moving parts, orchastrated in a perfect harmony that will reliably deliver business value time and time again. Therefore, we should try our hardest to test the business functions and not the finest code granularity in a view to attain high levels of code coverage. This means using the container services by deploying your application and testing all facets of the system, including container configuration, in the test cycle. Lets not forget, if it was not for the services of the container, we would be forced to write multi-threaded applications on every customer engagement. Now that would surely be another engineering feat.

The rise in popularity of Dependency Injection over the last three years has been an interesting observation. “Lightweight containers” like Spring and Pico have really intrigued and excited the masses of Java Developers, so much so, that Spring (for example) is automatically adopted and accepted to be mandatory for almost all J2EE applications.

Looking back to when I was first reading about Spring, I clearly recall how the DI concepts and simple semantics of the XML configurations really got my creative juices flowing. I remember thinking about all the possibilities the framework could offer; specifically the view of configuring all my components so they had no knowledge of their dependents. Just after I created my “Hello World!” application, I quickly realized that the concepts of the Abstract Factory had just been reinvented, this time under the “Lightweight Container” marketing. Sure, the framework provides externalised definitions of object dependencies and promotes coding to interfaces, but its no different than my trusted old Abstract Factory pattern. Ignoring my cautious voice, the shiny marketing got the better of me and I convinced myself Spring could present a better alternative to my novel approaches; I’d decided to use it on my next project.

After about three months of using Spring in angst, I recall moments where I was enjoying the flexibility I’d attributed to Spring. Spring promoted the definition of loosely coupled Java classes which really gave me an advantage during unit testing. Heck, with a bit of thought, I could wire every one of my java classes using Spring bean definitions and define all my dependents as a reference to another Spring bean. Now, I can pass in Mocks, Stubs, basically anything that implemented the contract. What power! It gave me incredible leverage to test my primary test object by mocking or stubbing my fixtures or dependents; now, obtaining higher levels of code coverage was really easy. I just had to ensure that all the java classes that I needed to test were defined as a Spring bean, making sure that all the primary test objects dependents were also defined as bean references. This way, by overriding specific bean definitions with Mock or Stub ones, I could replace them at runtime to exercise every concern of my application code.

One observation that I made during this time was the reduced number of the new operator usage. Looking though my source, everything was injected into my classes and the new operator had practically vanished. I recall the infamous “Dont call me, we’ll call you” cliche had whispered in my mind.

Looking back over my first Spring project, I’d say we achieved a test code coverage of about 80%. This was really exciting and I’d give all the credit to our decision to use Spring. A few years later, a few more gray hairs and a few more Spring enabled projects, I’ve realised that Spring is only an enabler to achieve high unit test code coverage. Sure, Spring privides some really nice JDBC, JNDI, JMS etc helpers, but the XML configurations aspect that it gives to an application can get really messy; so much so that it can become a real development impedance. How is this possible? Read my Aspects of Spring on a Monolithic Codebase blog; it sums up my growing pains.

So, the one resounding question that I have in my mind is “What real benefit does Spring offer to a system?”. After many discussions and hours of mental deliberation, I categorise the benefits of Spring by percentage as 90% unit testing flexibility and 10% utility.

Where to from here? Well, that depends on the importance you place on code coverage. Period. If your aiming for high levels of code coverage, then Spring’s your answer; just expect to deal with a pervasive XML configuration aspect of your codebase.

Proposing alternatives to Spring could be futile given the wild acceptance of the framework. In my humble view, Abstract Factories are formidable alternatives and can also be configuration driven and, again, cause a proliferation of dependency descriptor externalisation, however, they force you to think about your providers and give you a break from the Spring alternative to think about when you need to abstract your providers. This way, your decisions to define your configurable components can be made with more pragmatism and care.

In conclusion, using Spring has many advantages, most of which are, sadly, concerned with the testability requirements of your codebase. Separating your application tiers by demarcating with a layer of Spring configuration will be my next approach. This way, I can achieve my provider view using the attractive semantics of Spring bean definitions.

Over the years, l’ve read lots of commentary, white papers, best practice papers, books on the topic of TDD. I’ve heard the rant of many TDD evangelists who preach about how total code coverage brings you closer to code quality perfection and how you’ve failed when you’ve not been able to achieve these goals. Sure, this is an extreme example of evangelical preaching, where in actual fact, most of these individuals commonly drum down their hard line views of testing by using words like “pragmatism” and statements like “do what works best”. But, why do l feel as if I’ve failed if l have not got 100% code coverage? It’s because l, to some degree, shared some of the religious views about testing.

I’m now at a point where I’m beginning to rethink some of my beliefs about testing after many years in the trenches. So, l’m at the crossroads settling on a methodology that, l feel, works the best. What level of unit testing is really required to meet the business needs?

I’ll start by analysing two “special interest” projects to see what the outcomes they delivered based on the business expectations.

Firstly, let me talk about a project that was one extreme; no mandated position on unit testing. The project was highly successful, where the business expectations were met and exceeded. Donning my evangelist hat, I’d say the project outcomes were a fluke and it was a miracle that we were able to make any changes to the application without having a negative impact on functionality. Looking back, the project was definitely not a fluke; we made lots of changes to the application without any regressive impact. We knew our issues and had the right processes in place to gate-check the application functionality pre-release. For example, a week before each release, every developer had an area of expertise in the application which they would spend approximately a week testing the functional area and making any spot fixes as need be. We were not very clever about our testing methodology, but we delivered on time, on budget and exceeded customer expectations.

Now, let me tell a story of another, very different project, one that’s in stark contrast to the first one. This application had literally 98% code coverage. Unit tests, integration tests, front end screen tests, water tight code reviews, continuous integration, nightly deploys, every agile practice and quality assurance process under the sun. Did the code meet the business expectations? Well, yes; but it was expensive. It took twice as long to develop an application feature, and we mandated near perfect code coverage. Was this approach more successful than my first example? Not really. Sure, we had more confidence in making changes to the code base and having an “immediate view” of regression impact of the change. But the business paid a price for all of that. A very heavy price. One would think, given the money it cost for development to test the application, that the number of defects would be significantly reduced; but they weren’t. We had lots of functional and non-functional defects detected by testing which was effectively misinterpretation of  business requirements or some gap in the business logic.

Which, from a business perspective, was more successful? Both. The corollary is that a heavily tested application cost lots of money and takes longer to build. This l have seen first hand. So, time to answer the titan question from my own experiences.

As a developer, you need to test the components that you write; theres no arguing that. Otherwise, how else can you prove the functionality of your components? Bu, just how far should we push the envelope?

My view is simple. We all need to be pragmatic about how we approach our unit testing. We should always stop and ask ourselves “are we going to far with our unit testing?”. As a developer, we are faced with this question constantly. We should always do the most to prove our components are functionally correct, but also write the least amount of unit test code to ensure our testing solution remains simple yet effective.  After all, a good developer is a lazy one.