7 steps to performance testing bliss :)

This post goes out to those that approached me recently with the question "I would like to learn performance testing. How do I get started?"

It's a step by step approach to getting started on performance testing.
A couple of points may cause deja-vu from previous posts, but repeating those was necessary evil towards creating this summary guide

Step 1 : Identify tests !
Identify the performance test needs of the application under test (AUT). The AUT could be a web application, a guest OS, host OS, API or just a client server application.

Hints to identify crucial components:

1. Log user behavior in production to measure the most concurrently and frequently used features

2. Identify brittle system components that may not scale well.
Dive deep into the system architecture diagrams to figure out these nuances. An example to explain this would be files or database tables that hold ever increasing information

Remember, throughput is just a unit of transactions/time, and is not restricted to requests/sec. Your client server application may have a file upload feature. Your throughput could be the size of the file that's uploaded single or multiple.

Step 2: Match environments !

If your QA env is one that's rebuilt frequently, it would be wise to automate this comparison

Hints to compare key differentials:

1.System resources on the server in production like CPU, Memory, Hard disk, Swap.. you get my drift.
2.System specifics like log level correlation, any other processes running on the server other than the application under test
3.Network and system architecture comparison, including load balancer settings, if implemented for any server
4.If production system has redundancy in servers, QA env should mimic atleast to scale, if not the same count of redundancy

Step 3: Plan !

Now that you've identified what to test, write a detailed plan with what features you intend to load/stress test, what features you intend to measure performance of, the min throughput you will apply etc. An earlier post  explains this in detail.

Step 4: Automate !

Identify your automation needs based on the performance test plan.

If it's a web application where user behavior can be completely tracked via HTTP requests, use a tool like JMeter (refer to my earlier post on performance test tools for details)
To get started on JMeter, read this section of their user manual.

If all the user actions on the web application cannot be tracked by HTTP requests (client side AJAX UI behavior), you may want to use UI automation tools that let you customize the code to run it multi-threaded. (hint: Selenium IDE and RC/Server)

If you are trying to increase concurrent calls to specific system methods/features, scripting multi threaded behavior using the application's API calls would be a good way of also implicitely load testing API calls.

Step 5: Execute !
Key points to remember while executing performance tests:

1.Isolate the QA env where performance tests are run.  Functional tests or any other concurrent use of this system will skew your results

2.Restore system state after stress tests. The automated stress tests should include a step to restore system to default state. Ensure that this step is run regardless of whether your tests completed or crashed. Ofcourse, it would be prudent to check the state before starting the tests.

Step 6: Analyze !
I am writing up detailed posts on this topic, meanwhile an earlier post covers some of the basics as well .
Result analysis involves tracking the different values being measured and monitored and checking if they satisfy the verification/pass criteria
In case of inconsistencies, isolate the test/behavior that causes failures.
Identify the cause of the bottlenecks with profilers, stepping through code etc.
Resolve bottlenecks by tuning your settings, adding more memory, load balancing servers, indexing database tables, improving slow running queries etc.

Step 7: Report !

Performance test results are best reported in two formats. One is a basic summary that highlights what the stakeholders care to know or should know. And a detailed report for archival and comparison with regression runs.

Stakeholder summary could contain max concurrency/load/throughput and average response time for best and worst concurrency cases
Detailed report could contain the resource monitoring graphs, output from the file handle monitoring scripts and logs created by the automation tool.

Nifty tools for performance analysis

Happy New Year ! I am back from holiday hibernation to spread testing joy :)

Picture this: You've run your web app performance tests and bench-marked response times for your application and you are trying to make sense out of the numbers and the only thing that occurs to you is "Why? why me?" :)

Now let's drop these nifty little tools in your lab (pun intended) and watch the magic:

YSlow (from Yahoo) or Page Speed (from Google)
These are lightweight tools/ plug-ins that don't actually simulate load on your application. Instead they predict performance issues based on whether your application conforms to basic rules/ best practices. A simple rating system that lists all the ways in which your application is built to perform or break.
Why should I ramble on when google can say it better: http://code.google.com/speed/page-speed/docs/rules_intro.html

BrowserMob
I am planning to evaluate this tool, so don't consider this a review yet, more an fyi. Most performance test tools send http requests and receive response without simulating an actual browser. This tool lets you figure out performance implications via real browser simulation. Check out http://browsermob.com/load-testing

Stay tuned for the my performance test series to be continued, with some serious coaching on JMeter and such.

Cheers,
Dharma
(The more you teach, the more you learn)

QA Challenge in the holiday spirit !

I've started a QA challenge via a LinkedIn discusion forum and wanted to present the same question here. It's in the spirit of Christmas and holiday cheer.

If you were an Elf and were asked to test Santa's sleigh so he can have a nice safe journey around the earth and everybody can wake up to their presents, what would be your test plan? (If it helps, consider it a software model of a sleigh or a real sleigh.. but be clear about it in the test plan)

It's probably the eggnog, but I think it's fun :)

Reply privately, if you do, because I am also using this challenge to find good QA engineer prospects for my team. Figured it would be a fun way to find good talent after going through like a million resumes.

If any of you do respond, I'll ask you to re-post here as a comment after the holidays.

Happy holidays!
Dharma

Six degrees of separation from a skill

My pet peeve as a hiring manager is what I like to call the "six degrees of separation from a skill" syndrome :)

Beware when you see everything but the kitchen sink listed as skills, especially in short career spans. More often than not, my phone screens of such candidates have taken bizarre and disappointing turns such as these:

1.This one is the killer.. inspiration for the title of this blog:

A candidate's resume listed JAVA, which was in the JD. When asked what was his level of experience in JAVA, his response was.. brace yourself for this..

"I was working with a developer in the same building who was using JAVA for his programming"

2.Other lame but fairly common ones:

When asked how recent is your Perl experience/ how do you rate yourself 1-5?
A brief silence on the other end usually followed by

"I use a Perl script written by someone else and I can understand the code and even modify it"

I call the next one "look over someone's shoulder and quickly add it to your resume" syndrome

When asked to explain experience with say performance testing when a ton of tools are listed on the resume, candidate says

"I was involved in a project where my team/colleagues created the script and monitoring and I pressed the start button (or something to that effect)"

I am yet to hear amusing ones such as "My mother read aloud Perl scripts to me when I was in her womb hence it's on my resume":)


ps: For anyone wondering why I interrupted my sequence of performance testing related posts with this inane rant, I am phone screening a lot of candidates currently for QA engineer, so it's been on my mind. Now that it's out of my system and I can move on back to technical topics.

Cheers,
Dharma

Best practices in performance testing

Essential Do's and Dont's in performance testing

Please read my previous detailed posts on Web application performance testing methodology before moving on to this post.

1.Lab resources
It is important to match the architecture of the system between QA lab and production. It could be a scaled down version, in which case the results are analyzed and predicted to scale

How-to match production

• Hardware specifications of the servers should match
• Load balanced architecture of the servers should be replicated in lab
• Database clustering/replication should match, including type of replication set up in production
• Software versions should match
• Log level settings and log archival settings should match (otherwise production with a more verbose log level or without sufficient archival will die pretty soon and you can never recreate it in the lab)
• Innovate ways to simulate hardware or other resources that cannot be matched apples for apples in the lab for budget constraints. Understand the blueprint of the system to determine these vulnerabilities

2.Result archival
The results from each test should be recorded and easily accessible for comparison with future/past results. There is no way to determine improvement or degradation in performance without comparison data

3.Isolated testing
Exercise different actions or transactions in the application separately as atomic actions to isolate bottlenecks easier. ie., don't execute all actions on one or more page(s) sequentially in the same script.

Different concurrent threads may end up executing different actions together and it will not be possible to isolate which action caused the breakdown or bottleneck

There is merits to running tests that exercise a combination of actions together. But this should be one of the scripts/plans of execution to simulate real world scenarios, not to analyze specific component performance.

3.Analyze bottlenecks

Other than the action/transaction separation discussed above, it's also important to exercise different system components separately.
Examples of such isolation methods:

• Exercising just the HTTP requests for required transaction rates without involving the GUI components. Command line tools like HTTPerf are very useful for these "headless performance tests" as they are called. Ofcourse JMeter can also be used for this, but the time involved in recording, removing image links etc that are not needed and then playing back the bare script is not worth the time in this context.
• Exercising selected actions (as discussed above)
• Only invoke requests/ transactions relevant to execute a specific action.. less is more !
• Exercise database queries separate from application layer. Determine the queries used by the application for each action (where applicable) and run them separately as JDBC transactions from the performance test tool
• Exercise authentication/LDAP separately by using LDAP transactions. The time it takes to login may have more to do with LDAP than with login code in the application

Related Posts to look forward to (coming soon to a browser near you):
Stress testing
Scalability testing
Performance test tool cheat sheets/Tool instructions

Your comments/suggestions will be much appreciated !

Tweet much?

All Things QA can now be followed on Twitter @ http://twitter.com/allthingsqa for quick and easy access to updates.

Happy tweeting :)

Web Application Performance Testing tools

Highlights and lowlights of a few popular performance testing tools:

JMeter

• Open source, active project
• Supports HTTPs traffic via HTTPS spoofing (works quite well)
• Supports various protocols other than HTTP, like JDBC to analyze SQL bottlenecks directly
• Supports cookie and session management
• Can be executed in distributed mode from remote host to prevent the host machine from throttling the test
• Captures screen (if specified) to be used as functional test (capturing screen is not recommended during load/performance tests)

OpenSTA

• OpenSource, not actively developed or maintained
• Support for HTTPS is patchy/unreliable at best
• Good scripting interface, better than JMeter for customization and parametrization
• Falls apart easily under load even in distributed mode
• Easy access for DOM objects to be accessed from scripts
• Monitoring of servers possible

HTTPerf

• Free, open source
• Command line tool (not GUI based)
• Granular support via command line options for inputs, delays, target throughput etc
• SSL supported

Grinder

• Very similar to Jmeter; Open source JAVA based record and playback tool
• Supports HTTP and HTTPS
• HTTP(S), JDBC, LDAP and more protocols supported

Mercury Load Runner

• Not inexpensive
• Method level Diagnostics on application
• Monitoring of servers during test built in
• Record and playback via GUI

IXIA and Avalanche

• Not inexpensive :)
• Large scale/higher concurrency use, loads that JMeter cannot handle well
• More granular control over inputs and target throughput and output monitoring

Persistence testing web applications

Pre-requisites:
My previous post on performance testing is the prequel to this post

Persistence / stability / soak testing is the next step after figuring out how well your application performs under load.

It's time to determine if it continues to perform well under reasonable load over a period of time.

Reasonable load in this context:
Either 60% of stress point or average throughput that application is subject to on production

Steps:
Let the load test with above load run uninterrupted over a period of time (Industry standard for minimum time seems to be 72 hours but we tend to cut corners to stay within tight project deadlines, maximum time depends on the project deadline)

During this time, continue all the monitoring an measurement described in previous post, but not as frequent. Example, if you were monitoring/measuring every minute in a shorter performance test, in persistence test, you would need to monitor every 15 mins or so.

It would be essential to cause small infrequent spikes in the load during persistence tests, to simulate real world behavior.

These tests analyze if the application is stable, even what (if any) is the performance degradation over time.

Performance Testing Web Applications

This section explains how the terms performance, load and stress testing seamlessly integrate towards end to end performance testing

Performance testing:

What to measure:

• Response time of an action (Performance testing tool)
• Actual throughput vs target throughput (Performance testing tool)
• Responsiveness of the action/app (Example successful HTTP requests -200 response code) (Performance testing tool)

Baselining:
Exercise each action in the application with no load/concurrency
In the absence of a specific requirement of supported throughput, measure the actual throughput without increasing/manipulating target throughput under no load

If a requirement is clear as to minimum throughput to be supported, then execute the action specifically for target throughput set to the min required value and after execution, verify the actual generated throughput

Throughput is transaction rate (business transactions in a given interval of time)
For instance: bytes transferred/second, requests/second etc


Load testing:

What is load:
Number of concurrent users/connections
Number of concurrent threads accessing each action in the application
Throughput for each action

What to measure:
1.Breaking/stress point
Steadily increase the load (as defined above) for each action and execute test
The point where the test returns 500 response code (Server error due to high load) is the breaking/stress point for your application (Will be continued as part of the stress testing section in future)

2.Stability under acceptable load
Acceptable load is the load that you can expect your application to be subject to in production
Three ways to arrive at this acceptable load:

• Management specifies in a requirement doc
• Monitor production usage of the application by tailing logs or such and determine the load or frequency of use of each action
• 60% of the breaking point/stress point, if accepted by stakeholders

Measuring stability for increased load means checking responsiveness and that latency introduced by load is within acceptable level (latency in this context is is response time with load - response time without load)

What's acceptable level defines verification criteria
In the absence of requirement of response time, observed latency is provided to stakeholders. If accepted, this is the baseline/yardstick
If the latency is not accepted by stakeholders, back to the drawing board for developers to tune the performance of the system (that's a topic for another day)

What to monitor:

1.System resources on the application server (System monitoring tool - Nagios, VMStat, PerfMon)
Monitor the CPU and memory utilization, Disk and Network I/O, Swap size during different load levels and between load tests to track/determine memory leaks
Nagios UI interface displays all the utilization info from the server as graphs
PerfMon results can also be analyzed as graphs
VMstat results must be collected and exported to Excel and graphs must be generated from this info
The reason I stress on graphs, is it's easier to find spikes in utilization when observed as graphs between time and utilization

One quick test for memory leaks is to run a high load test, stop the test and re-run it. Expect to see a dip in utilization before it rises again. If the spike continues with no dip, there you have your memory leak bug ..woo hoo !

2.File handles - Small scripts can be written to monitor count of file handles

3.Garbage collection- Profiling tools (JProfiler) -- note: Profiling tools slow down the system, so do not measure speed during profiling

4.Sessions in app server - Monitor the application server manager to ensure sessions are cleaned up appropriately after user logs out/test ends

Verification criteria:
If tests result in acceptable latency and 200 result codes, no memory leaks, test passes -- that's your baseline !
If tests result in out of proportion latency and 404/500 response codes, memory leaks, file a bug

Performance regression testing
All the measurements of baselines/yardsticks should be noted and compared against all future builds.

If performance is better in future build, new measurement is the new baseline
If performance is worse, file a bug and don't change the baseline. If the bug cannot be resolved, hold on to the old baseline and new results to track this long term

Yet to come in future posts:

Persistence testing
Stress testing
Scalability testing
Best practices in performance testing
Tools for performance testing will be covered in detail (JMeter, HTTPerf, OpenSTA etc)
Performance testing Linux/windows systems (IOMeter etc)

Stay tuned .. !

Test Management tool selection

Most QA teams (especially start ups where you have to start testing before you can say the word "test management") start by writing test cases in Excel files, word documents and post-its ;)

As a lead/manager, your first task should be to end the madness and start using a test management tool or evaluate the current system and replace if it's ineffecient.

Why?
To render tests repeatable and to track, archive and present test results to the stakeholders (ps: these are the magic words to convince management to let you set up the tool)

What?
I am not here to sell a specific product, just to suggest how to arrive at one.

Allright, then ..how?

Considerations/criteria for selecting a test management system:

1.BUDGET:
Determine the budget ceiling from stakeholders/management. There is no point wasting time evaluating tools that serve you pancakes in bed if they are out of your reach.

If the budget is too low or non existent, research freewares. Downside is set-up and configuration time and mostly lack of features. But most are better than post-its :)

2.LICENSE:
Budgets are not written in stone.. they change.. frequently! So, evaluate your selection based on the licensing model that's preditable

License/cost models:
Per registered user license, recurring cost
Per seat/concurrent user count license, persistent (one time cost)
Recurrring cost for max count (not per user)
Persistent / one time cost for max count or independant of cost
Hosted offsite vs Installed onsite

Persistent/One time costs are higher because it's Capital Expense. It will pay for itself in reasonable time frame and become free afterwards

To arrive at the best license model, you need to:

- Predict reasonably the growth of team
- Ascertain the possible concurrent connections to the app (for instance offshore/onshore models have lesser concurrency then team count)
- Length of time of usage of the tool
- Offsite hosting takes away the rigour of managing hardware and software and upgrades
- Availability of hardware and personnel to manage onsite installation vs offsite hosting


3.FEATURES:

- Intuitive and configurable interface
- Configurable fields across all components (example: able to modify/add test case fields, result fields etc)
- Import test cases from other documents/excel/applications
- Export test cases into files
- Search/ filter interface
- Modify test cases / Bulk update test cases
- Assign test cases to individual QA
- Execute/assign results to tests via sessions
- Track and archive test session results
- Integration with other SDLC apps or tools used by the team
- Associate failed cases with bugs
- Good result reporting interface
- Web applications have the merit that you can provide tests and results as URLs anywhere for quick and direct access
- Scalability - Evaluate the app/tool loading it with considerable tests and sessions and check performance (essential for web apps, most essential for hosted apps)

Bug priority and severity guidelines

Guidelines for assigning severity and priority to a bug

These guidelines are based on industry standards and my experience on what worked well in various projects/companies. These definitions and criteria should be tailored to best suit a specific project team

Severity vs Priority

Training the team on using two different fields for severity and priority in a bug tracking system could seem like an overhead, but ideally QA's definition of the impact of a bug (severity) should be kept separate from PM's decision on whether they want to make an exception depending on time constraints and defer it (priority/when to fix)

In the absence of such segregation, the priority field would be modified when a bug is triaged out of a release

*The term 'critical functionality' refers to new features deemed as acceptance tests by QA/Product Mgmt or existing features that impact customers the most

Severity/Priority assignment:

Severity: Critical/Show-Stopper
Priority: Must fix before Feature Complete (in agile context) /Code Freeze milestone/ Could be deferred to an immediate patch

1. Failure of critical functionality with no workaround
2. Data loss or corruption
3. Billing data or invoice display inconsistencies
4. Issues blocking testing
5. Critical functionality discrepancy between requirements and implementation
6. Critical functionality error handling failures with high occurence likelihood and resulting in high impact to customers
7. Security concerns with customer visibility


Severity: High
Priority: Should fix before code freeze (agile and waterfall context)

1. Partially impaired critical functionality with less than desirable workaround
2. Failure of non-critical functionality with no workaround
3. Non critical functionality discrepancy between requirements and implementation
4. High customer impact UI discrepencies
5. Security concern with no real customer impact


Severity: Medium
Priority: Attempt to fix before code freeze or deferred to next release (ie, release candidate should not be tainted for a medium priority fix)

1. Impaired non critical functionality with satisfactory workaround
2. Impaired critical functionality during low occurence edge cases
3. High impact spelling errors
4. Non critical functionality error handling failure


Severity:Low
Priority:If it may be fixed before code freeze else deferred to next release

1. Rare occurrence
2. Low customer impact feature failure
3. UI layout or low impact spelling errors

Test Planning 101

A Test Plan ..

* is a contract of the tests identified and agreed upon as a verification release criteria for a feaure.

* is signed off by all stakeholders (test cases don't need that level of granular review from teams outside of QA, since they are by products of the test plan)

* is written during spec/requirement review. Test Plan includes but is not limited to all use cases in a spec.

* highlights acceptance tests (applicable especially in agile model)

Essential components of a test plan:
(Sections that are not self explanatory have been explained alongside)

PURPOSE & SCOPE (Scope could be iteration specific in an agile mode or release version specific in a waterfall SDLC model)

COMPONENTS/FEATURES TO BE TESTED
1.Verification (Positive tests)
2.Validation (Negative tests)

COMPONENTS/FEATURES NOT TO BE TESTED

SECURITY TESTS

LOAD & PERFORMANCE TESTS

STRESS TESTS (Includes failover testing etc)

BROWSER COMPATIBILITY TEST MATRIX

PRODUCTION DEPLOYMENT SMOKE TESTS

RISKS AND ASSUMPTIONS

DELIVERABLES (Example: Test Cases via this tool/URL, Bugs via this tool etc)

HARDWARE & SOFTWARE (eNV) REQUIREMENTS (Especially necessary when more resources than currently available in QA Lab are required to be prepared before the code is dropped to QA)

SCHEDULE AND STAFFING (This may be handled independant of the test plan, but essentially mentions the QA resoUrces selected for this release)

RESOURCES (Spec and requirement documents, Engineering/Architecture docs)