Performance Model Calibration – What are your production users doing?

September 13th, 2011 by Oscar Huseyin Leave a comment

Capacity Planning, Modelling and Management can be seen as a daunting process as each systems yield their own unique challenges in measuring and reacting to customers usage patterns.  Most of my customer are concerned and interested in one thing; maintaining and adhering to the Service Level Agreements as defined in their service contracts. My customers interest in the art form that is capacity planning and modelling really end here. The rest of the assurance process that make up the complex and process driven method of capacity measurement, analysis, forecasting, capacity procurement, service augmentation and enablement lies with the infrastructure teams and driven by the Performance Architect.

So, how do you measure and plan for adequate capacity? Well, the answer is not that trivial. There are a number of really good books in this space, my favourite one being The Art of Capacity Planning, which focuses on production systems capacity measurement and planning, however, does not focus on new systems, e.g. Predictive capacity modeling. Given the differences between the two starting points, l thought l would write this article to demystify the black art that is capacity management for new systems.

Let’s take the simple case, the system is already in production and requires a capacity plan. The approach is trivial, identify what your users are doing, what time of day they are doing it and how many times they are doing it. Measure the system resource consumption and viola, a system utilisation view is presented.  Take this view, multiply it by your projection requirements and there you have it, a view of how much resources you will need in the future.

Let’s take the more complex view; the system is currently under development and user usage patterns are assumed, e.g. not realised yet as the system does not exist. This is where your models are very important (and your assumptions critical) in setting the boundaries for your capacity models. Enduring your performance requirements are accurate and agreed by your business stakeholders is a critical-success-factor in creating and maintaining a valid performance simulation model to base your performance testing strategy and identify your capacity requirements.  Be warned though, this is an assumptions based model and will need to be ratified as soon as your system is deployed and being used by your customers in production.

This is where calibration of performance simulation models is critical in ensuring your performance testing strategy delivers quality data into capacity planning processes.  Without calibrating your performance models with data from production usage, your assumption based performance models will deteriorate over time and will eventually become irrelevant; providing incorrect views of the capacity requirements and performance of your production system.  A good rule of thumb here is to recalibrate your performance simulation models at the end of every month after a new release of features or software into production.  This way, you have enough time to measure what your users are going in production and feed that back into the development and test cycles.

Unfortunately, l see huge number of projects neglecting calibration oriented due-diligence  and it often leads to production incidents.  If you find yourself asking “why have my performance tests passed and given me a stable view in our pre-production environment, yet the application performance requirements have failed when we deployed into production?”, well, the answer is in the question “how relevant is your performance simulation model?”.

Ensuring your performance testing approach remains relevant and accurate requires calibrating your performance models on a frequent basis.  The act in thinking about, designing and implementing a calibration approach helps you solve a number of other concerns relating to production monitoring and data gathering.  A presupposition to calibrating your performance model is understanding the monitors that are important in production! Wow! Talk about killing two birds with one stone.  As if that was not enough benefits to jump right into calibration, approaching calibration activity often yields massive insightful benefits into how to correlate your metrics and provide alarm scenarios when capacity shortages or problem scenarios are on the event horizon.  I will rest my case here.

So, to keep your performance testing approach in a healthy and relevant state, model calibration is a critical requirement to performance due-diligence.  The benefits extend into production support teams and furthermore, save the business in over-capacity spend and reduced production outages.

Till next time.