Thursday, December 24, 2009

QA Challenge in the holiday spirit !

I've started a QA challenge via a LinkedIn discusion forum and wanted to present the same question here. It's in the spirit of Christmas and holiday cheer.

If you were an Elf and were asked to test Santa's sleigh so he can have a nice safe journey around the earth and everybody can wake up to their presents, what would be your test plan? (If it helps, consider it a software model of a sleigh or a real sleigh.. but be clear about it in the test plan)

It's probably the eggnog, but I think it's fun :)

Reply privately, if you do, because I am also using this challenge to find good QA engineer prospects for my team. Figured it would be a fun way to find good talent after going through like a million resumes.

If any of you do respond, I'll ask you to re-post here as a comment after the holidays.

Happy holidays!
Dharma

Saturday, December 19, 2009

Six degrees of separation from a skill

My pet peeve as a hiring manager is what I like to call the "six degrees of separation from a skill" syndrome :)

Beware when you see everything but the kitchen sink listed as skills, especially in short career spans. More often than not, my phone screens of such candidates have taken bizarre and disappointing turns such as these:

1.This one is the killer.. inspiration for the title of this blog:

A candidate's resume listed JAVA, which was in the JD. When asked what was his level of experience in JAVA, his response was.. brace yourself for this..
"I was working with a developer in the same building who was using JAVA for his programming"
2.Other lame but fairly common ones:

When asked how recent is your Perl experience/ how do you rate yourself 1-5?
A brief silence on the other end usually followed by
"I use a Perl script written by someone else and I can understand the code and even modify it"

I call the next one "look over someone's shoulder and quickly add it to your resume" syndrome

When asked to explain experience with say performance testing when a ton of tools are listed on the resume, candidate says
"I was involved in a project where my team/colleagues created the script and monitoring and I pressed the start button (or something to that effect)"


I am yet to hear amusing ones such as "My mother read aloud Perl scripts to me when I was in her womb hence it's on my resume":)


ps: For anyone wondering why I interrupted my sequence of performance testing related posts with this inane rant, I am phone screening a lot of candidates currently for QA engineer, so it's been on my mind. Now that it's out of my system and I can move on back to technical topics.

Cheers,
Dharma

Monday, December 14, 2009

Best practices in performance testing

Essential Do's and Dont's in performance testing

Please read my previous detailed posts on Web application performance testing methodology before moving on to this post.

1.Lab resources
It is important to match the architecture of the system between QA lab and production. It could be a scaled down version, in which case the results are analyzed and predicted to scale

How-to match production
  • Hardware specifications of the servers should match
  • Load balanced architecture of the servers should be replicated in lab
  • Database clustering/replication should match, including type of replication set up in production
  • Software versions should match
  • Log level settings and log archival settings should match (otherwise production with a more verbose log level or without sufficient archival will die pretty soon and you can never recreate it in the lab)
  • Innovate ways to simulate hardware or other resources that cannot be matched apples for apples in the lab for budget constraints. Understand the blueprint of the system to determine these vulnerabilities

2.Result archival
The results from each test should be recorded and easily accessible for comparison with future/past results. There is no way to determine improvement or degradation in performance without comparison data

3.Isolated testing
Exercise different actions or transactions in the application separately as atomic actions to isolate bottlenecks easier. ie., don't execute all actions on one or more page(s) sequentially in the same script.

Different concurrent threads may end up executing different actions together and it will not be possible to isolate which action caused the breakdown or bottleneck

There is merits to running tests that exercise a combination of actions together. But this should be one of the scripts/plans of execution to simulate real world scenarios, not to analyze specific component performance.

3.Analyze bottlenecks

Other than the action/transaction separation discussed above, it's also important to exercise different system components separately.
Examples of such isolation methods:

  • Exercising just the HTTP requests for required transaction rates without involving the GUI components. Command line tools like HTTPerf are very useful for these "headless performance tests" as they are called. Ofcourse JMeter can also be used for this, but the time involved in recording, removing image links etc that are not needed and then playing back the bare script is not worth the time in this context.
  • Exercising selected actions (as discussed above)
  • Only invoke requests/ transactions relevant to execute a specific action.. less is more !
  • Exercise database queries separate from application layer. Determine the queries used by the application for each action (where applicable) and run them separately as JDBC transactions from the performance test tool
  • Exercise authentication/LDAP separately by using LDAP transactions. The time it takes to login may have more to do with LDAP than with login code in the application

Related Posts to look forward to (coming soon to a browser near you):
Stress testing
Scalability testing
Performance test tool cheat sheets/Tool instructions

Your comments/suggestions will be much appreciated !

Tweet much?

All Things QA can now be followed on Twitter @ http://twitter.com/allthingsqa for quick and easy access to updates.

Happy tweeting :)

Sunday, December 13, 2009

Web Application Performance Testing tools

Highlights and lowlights of a few popular performance testing tools:

JMeter
  • Open source, active project
  • Supports HTTPs traffic via HTTPS spoofing (works quite well)
  • Supports various protocols other than HTTP, like JDBC to analyze SQL bottlenecks directly
  • Supports cookie and session management
  • Can be executed in distributed mode from remote host to prevent the host machine from throttling the test
  • Captures screen (if specified) to be used as functional test (capturing screen is not recommended during load/performance tests)

OpenSTA
  • OpenSource, not actively developed or maintained
  • Support for HTTPS is patchy/unreliable at best
  • Good scripting interface, better than JMeter for customization and parametrization
  • Falls apart easily under load even in distributed mode
  • Easy access for DOM objects to be accessed from scripts
  • Monitoring of servers possible

HTTPerf
  • Free, open source
  • Command line tool (not GUI based)
  • Granular support via command line options for inputs, delays, target throughput etc
  • SSL supported

Grinder
  • Very similar to Jmeter; Open source JAVA based record and playback tool
  • Supports HTTP and HTTPS
  • HTTP(S), JDBC, LDAP and more protocols supported

Mercury Load Runner
  • Not inexpensive
  • Method level Diagnostics on application
  • Monitoring of servers during test built in
  • Record and playback via GUI
IXIA and Avalanche
  • Not inexpensive :)
  • Large scale/higher concurrency use, loads that JMeter cannot handle well
  • More granular control over inputs and target throughput and output monitoring

Saturday, December 12, 2009

Persistence testing web applications

Pre-requisites:
My previous post on performance testing is the prequel to this post

Persistence / stability / soak testing is the next step after figuring out how well your application performs under load.

It's time to determine if it continues to perform well under reasonable load over a period of time.

Reasonable load in this context:
Either 60% of stress point or average throughput that application is subject to on production

Steps:
Let the load test with above load run uninterrupted over a period of time (Industry standard for minimum time seems to be 72 hours but we tend to cut corners to stay within tight project deadlines, maximum time depends on the project deadline)

During this time, continue all the monitoring an measurement described in previous post, but not as frequent. Example, if you were monitoring/measuring every minute in a shorter performance test, in persistence test, you would need to monitor every 15 mins or so.

It would be essential to cause small infrequent spikes in the load during persistence tests, to simulate real world behavior.

These tests analyze if the application is stable, even what (if any) is the performance degradation over time.

Sunday, December 6, 2009

Performance Testing Web Applications

This section explains how the terms performance, load and stress testing seamlessly integrate towards end to end performance testing

Performance testing:

What to measure:
  • Response time of an action (Performance testing tool)
  • Actual throughput vs target throughput (Performance testing tool)
  • Responsiveness of the action/app (Example successful HTTP requests -200 response code) (Performance testing tool)

Baselining:
Exercise each action in the application with no load/concurrency
In the absence of a specific requirement of supported throughput, measure the actual throughput without increasing/manipulating target throughput under no load

If a requirement is clear as to minimum throughput to be supported, then execute the action specifically for target throughput set to the min required value and after execution, verify the actual generated throughput

Throughput is transaction rate (business transactions in a given interval of time)
For instance: bytes transferred/second, requests/second etc


Load testing:

What is load:
Number of concurrent users/connections
Number of concurrent threads accessing each action in the application
Throughput for each action

What to measure:
1.Breaking/stress point
Steadily increase the load (as defined above) for each action and execute test
The point where the test returns 500 response code (Server error due to high load) is the breaking/stress point for your application (Will be continued as part of the stress testing section in future)

2.Stability under acceptable load
Acceptable load is the load that you can expect your application to be subject to in production
Three ways to arrive at this acceptable load:
  • Management specifies in a requirement doc
  • Monitor production usage of the application by tailing logs or such and determine the load or frequency of use of each action
  • 60% of the breaking point/stress point, if accepted by stakeholders

Measuring stability for increased load means checking responsiveness and that latency introduced by load is within acceptable level (latency in this context is is response time with load - response time without load)

What's acceptable level defines verification criteria
In the absence of requirement of response time, observed latency is provided to stakeholders. If accepted, this is the baseline/yardstick
If the latency is not accepted by stakeholders, back to the drawing board for developers to tune the performance of the system (that's a topic for another day)

What to monitor:

1.System resources on the application server (System monitoring tool - Nagios, VMStat, PerfMon)
Monitor the CPU and memory utilization, Disk and Network I/O, Swap size during different load levels and between load tests to track/determine memory leaks
Nagios UI interface displays all the utilization info from the server as graphs
PerfMon results can also be analyzed as graphs
VMstat results must be collected and exported to Excel and graphs must be generated from this info
The reason I stress on graphs, is it's easier to find spikes in utilization when observed as graphs between time and utilization

One quick test for memory leaks is to run a high load test, stop the test and re-run it. Expect to see a dip in utilization before it rises again. If the spike continues with no dip, there you have your memory leak bug ..woo hoo !

2.File handles - Small scripts can be written to monitor count of file handles

3.Garbage collection- Profiling tools (JProfiler) -- note: Profiling tools slow down the system, so do not measure speed during profiling

4.Sessions in app server - Monitor the application server manager to ensure sessions are cleaned up appropriately after user logs out/test ends

Verification criteria:
If tests result in acceptable latency and 200 result codes, no memory leaks, test passes -- that's your baseline !
If tests result in out of proportion latency and 404/500 response codes, memory leaks, file a bug

Performance regression testing
All the measurements of baselines/yardsticks should be noted and compared against all future builds.

If performance is better in future build, new measurement is the new baseline
If performance is worse, file a bug and don't change the baseline. If the bug cannot be resolved, hold on to the old baseline and new results to track this long term

Yet to come in future posts:

Persistence testing
Stress testing
Scalability testing
Best practices in performance testing
Tools for performance testing will be covered in detail (JMeter, HTTPerf, OpenSTA etc)
Performance testing Linux/windows systems (IOMeter etc)

Stay tuned .. !

Saturday, December 5, 2009

Test Management tool selection

Most QA teams (especially start ups where you have to start testing before you can say the word "test management") start by writing test cases in Excel files, word documents and post-its ;)

As a lead/manager, your first task should be to end the madness and start using a test management tool or evaluate the current system and replace if it's ineffecient.

Why?
To render tests repeatable and to track, archive and present test results to the stakeholders (ps: these are the magic words to convince management to let you set up the tool)

What?
I am not here to sell a specific product, just to suggest how to arrive at one.

Allright, then ..how?

Considerations/criteria for selecting a test management system:

1.BUDGET:
Determine the budget ceiling from stakeholders/management. There is no point wasting time evaluating tools that serve you pancakes in bed if they are out of your reach.

If the budget is too low or non existent, research freewares. Downside is set-up and configuration time and mostly lack of features. But most are better than post-its :)

2.LICENSE:
Budgets are not written in stone.. they change.. frequently! So, evaluate your selection based on the licensing model that's preditable

License/cost models:
Per registered user license, recurring cost
Per seat/concurrent user count license, persistent (one time cost)
Recurrring cost for max count (not per user)
Persistent / one time cost for max count or independant of cost
Hosted offsite vs Installed onsite

Persistent/One time costs are higher because it's Capital Expense. It will pay for itself in reasonable time frame and become free afterwards

To arrive at the best license model, you need to:

- Predict reasonably the growth of team
- Ascertain the possible concurrent connections to the app (for instance offshore/onshore models have lesser concurrency then team count)
- Length of time of usage of the tool
- Offsite hosting takes away the rigour of managing hardware and software and upgrades
- Availability of hardware and personnel to manage onsite installation vs offsite hosting


3.FEATURES:

- Intuitive and configurable interface
- Configurable fields across all components (example: able to modify/add test case fields, result fields etc)
- Import test cases from other documents/excel/applications
- Export test cases into files
- Search/ filter interface
- Modify test cases / Bulk update test cases
- Assign test cases to individual QA
- Execute/assign results to tests via sessions
- Track and archive test session results
- Integration with other SDLC apps or tools used by the team
- Associate failed cases with bugs
- Good result reporting interface
- Web applications have the merit that you can provide tests and results as URLs anywhere for quick and direct access
- Scalability - Evaluate the app/tool loading it with considerable tests and sessions and check performance (essential for web apps, most essential for hosted apps)

Friday, December 4, 2009

Bug priority and severity guidelines

Guidelines for assigning severity and priority to a bug

These guidelines are based on industry standards and my experience on what worked well in various projects/companies. These definitions and criteria should be tailored to best suit a specific project team

Severity vs Priority

Training the team on using two different fields for severity and priority in a bug tracking system could seem like an overhead, but ideally QA's definition of the impact of a bug (severity) should be kept separate from PM's decision on whether they want to make an exception depending on time constraints and defer it (priority/when to fix)

In the absence of such segregation, the priority field would be modified when a bug is triaged out of a release

*The term 'critical functionality' refers to new features deemed as acceptance tests by QA/Product Mgmt or existing features that impact customers the most

Severity/Priority assignment:

Severity: Critical/Show-Stopper
Priority: Must fix before Feature Complete (in agile context) /Code Freeze milestone/ Could be deferred to an immediate patch

1. Failure of critical functionality with no workaround
2. Data loss or corruption
3. Billing data or invoice display inconsistencies
4. Issues blocking testing
5. Critical functionality discrepancy between requirements and implementation
6. Critical functionality error handling failures with high occurence likelihood and resulting in high impact to customers
7. Security concerns with customer visibility


Severity: High
Priority: Should fix before code freeze (agile and waterfall context)

1. Partially impaired critical functionality with less than desirable workaround
2. Failure of non-critical functionality with no workaround
3. Non critical functionality discrepancy between requirements and implementation
4. High customer impact UI discrepencies
5. Security concern with no real customer impact


Severity: Medium
Priority: Attempt to fix before code freeze or deferred to next release (ie, release candidate should not be tainted for a medium priority fix)

1. Impaired non critical functionality with satisfactory workaround
2. Impaired critical functionality during low occurence edge cases
3. High impact spelling errors
4. Non critical functionality error handling failure


Severity:Low
Priority:If it may be fixed before code freeze else deferred to next release

1. Rare occurrence
2. Low customer impact feature failure
3. UI layout or low impact spelling errors

Thursday, December 3, 2009

Test Planning 101

A Test Plan ..

* is a contract of the tests identified and agreed upon as a verification release criteria for a feaure.

* is signed off by all stakeholders (test cases don't need that level of granular review from teams outside of QA, since they are by products of the test plan)

* is written during spec/requirement review. Test Plan includes but is not limited to all use cases in a spec.

* highlights acceptance tests (applicable especially in agile model)

Essential components of a test plan:
(Sections that are not self explanatory have been explained alongside)

PURPOSE & SCOPE (Scope could be iteration specific in an agile mode or release version specific in a waterfall SDLC model)

COMPONENTS/FEATURES TO BE TESTED
1.Verification (Positive tests)
2.Validation (Negative tests)

COMPONENTS/FEATURES NOT TO BE TESTED

SECURITY TESTS

LOAD & PERFORMANCE TESTS

STRESS TESTS (Includes failover testing etc)

BROWSER COMPATIBILITY TEST MATRIX

PRODUCTION DEPLOYMENT SMOKE TESTS

RISKS AND ASSUMPTIONS

DELIVERABLES (Example: Test Cases via this tool/URL, Bugs via this tool etc)

HARDWARE & SOFTWARE (eNV) REQUIREMENTS (Especially necessary when more resources than currently available in QA Lab are required to be prepared before the code is dropped to QA)

SCHEDULE AND STAFFING (This may be handled independant of the test plan, but essentially mentions the QA resoUrces selected for this release)

RESOURCES (Spec and requirement documents, Engineering/Architecture docs)

Wednesday, December 2, 2009

One stop for all things QA

Roughly 6 million things about QA (they dont tell you)

Topics that I hope to cover via this blog:
Test planning - Strategy, Tools
Automation - Strategy, Tool evaluation and instructions
Performance Testing - Strategy, Tool evaluation and instructions
Offshore model - Highlights and pitfalls
Career Track - What you should consider before becoming a QA
Interview Preparation - Go get 'em :)
Bugs - Filing, prioritizing and tools suggestions
Agile testing - Learn what the hype is all about
Helpful Resources - Books and links I've found useful

For more interactive discussions on these and other QA topics beyond the scope of the blog, please sign up to get access to chat/forums on allthingsqa.ning.com (I may ask for a LinkedIn public profile link or equivalent before accepting the signup, just to ensure credibility of opinions posted in the forums)

Please check back/follow the blog for updates.. this is just the beginning :)

Cheers!
Dharma