Software QA FYI - SQAFYI

Testing and Quality Assurance Techniques

By: Sandy Sorkin

Objectives
1. Automate the testing process
2. A structured review process
3. Structured development with reusable
code and reusable tests
4. Appropriate progress and performance
metrics
5. On-going quality initiatives
6. Identify testable conditions
7. Organize testing
8. Teamwork
9. Communications
10. Enjoy your job


Observation - 1
Nothing is obvious
Specifications must be written
Examples
Graphics
Quantify everything
Observation - 2
Do it twice
Test and retest
Design all tests to be repeatable
Test bed should be maintainable
Test initialization & re-initialization


Observation - 3
Everything has a limit
Identify the limits
Test the limits
Language imposed limitations
Platform imposed limitations
Specification imposed limitations
Observation - 4
Design systems with testing in mind
Insert diagnostic tools (instrumentation)
Control totals
Audit trails
Balancing routines
File comparisons
Observation - 5
Practice tact and diplomacy
Don’t be critical all of the time
Offer positive comments
Encourage the right behavior
It is better to find agreement than to win
Observation - 6
The specifications may be wrong
All specifications will change
Include test plans in the specs
Identify a formal change process
Don’t fight change
Get better estimates of change impact
Change Requests:
must be in writing and
require 3 estimates


Documentation estimate\
                       \ 
Programming estimate---Total change request estimate
                      /
Testing estimate     /



First Shopping List
• 1 box cereal
• 2 cans orange juice
• 3 1/2 gallons of milk
• 1 loaf of bread

Shopping List (Spec)
• olives
• laundry detergent
• oatmeal
• juice
• coffee
• dessert for tonight

Shopping List (Spec) Questions
• olives (large, jumbo, can, jar, pimentos)
• laundry detergent (brand, size, aroma)
• oatmeal (instant, flavored, can, box)
• juice (orange, tomato, apple, cranberry)
• coffee (instant, Jamaican, Hawaiian)
• dessert for tonight


Bug Propagation
1 bug in analysis
can result in
3-15 bugs in design
can result in
2-10 bugs in code
Worst case: 1 bug in analysis causes 15 bugs in
design and they in turn create 10 bugs in the code.
Total = 150 code defects. How many will you
find in testing?

Impediments
What are the major impediments to quality
initiatives in this organization?
1.
2.
3.
4.
Impediments, Opportunities, and Managing
• No time
• “We just don’t have time to change the way
we test or develop systems. Quality
improvement is a great idea, but we don’t
have anyone available right now.”

Impediments, Opportunities, and Managing
• Inertia
• “Things aren’t so bad, why should I want to
do anything differently. No one has yelled
at me in over two weeks.”
• Need a management buy-in
• “If management doesn’t tell us to improve
quality, gives us time, and a budget, nothing
is going to happen. Discussing quality with
us is simply preaching to the choir.”
• We’re not ready and we need training
• “We have to get the rest of the shop in
order, before we can consider quality
improvement suggestions. When will we
have time for training?”

Who is responsible for testing?
Customer
They know what they want better than I do.

Business Analysts
They should do all of the testing. They’re always
talking to the customers.

Programmers
If they do their job properly, no one else has to
test.

Quality Assurance
Its their job.

Quality Questions
1. What is quality.
2. What does it cost?
3. How is quality measured?
4. Where does quality come from?

Quality Improvement Suggestions: 1
Joint Application Design sessions (JAD/JAR)
Define business objectives
Insist on written specifications
Use prototyping tools
Write user manuals (before coding)
Estimate the testing effort
Allocate resources to automate testing

Quality Improvement Suggestions: 2
Instrumentation
Written unit test plans (before coding)
Enforce development methodologies
Enforce coding standards
Earlier involvement of QA personnel
Capture and report metrics
More documentation

Quality Improvement Suggestions: 3
Risk and Contingency analysis
Follow-up process for suggestions
Reusable tests
Maintainable test beds
Walkthroughs
Earlier Reviews
Written unit test plans

Concise Project Management Course
• Milestones
• 40 hour rule
• 80 hour rule
• Deliverables
• Measurable
• Reviewable
• Achievable
• Document everything
• Review everything promptly
Front End Coding Back End
Traditional 35% 15% 50%
Quality Process 42% 18% 28%
Front End Coding Back End
Traditional 35% 15% 50%
Quality Process 42% 18% 28%
12% productivity increase
50+% fewer defects
12-15% faster to market

It may cost less to leave the
defects out of the system, than to
pay to put them in, pay to find
them, and then pay to take them
out again.

It may cost less to leave the
defects out of the system, than to
pay to put them in, pay to find
them, and then pay to take them
out again.


Reliability Metrics
Mean Time Between Failures
• MTBF1 Crash, software inoperable
• MTBF2 Functional failure
• MTBF3 Quality failure
• MTBF4 Client Server failure

Mean Time To Repair
• MTTR1 Actual time to fix
• MTTR2 Total time in queue


Test Scripts (Generic Tests)
Unit Test Scripts
Component level tests
System and Acceptance Test Scripts
Functional tests
Test threads
End-to-end tests
Start-to-finish


No time to perform a regression test?
Alternatives
• Parallel testing
• Volume testing with live data
• Volume testing with test data
• Local, regional, expanded regional testing
• Statistical sampling
• Testing changes only
• Combine sample data with known problems




Traceability Matrix

Customer    |->| Functional   |->|System Test|->|Test  |
Requirements|--| Specification|--|Scripts    |--|Cases |
                \      
                 \       
                  Code Modules
                     \
                      \
                     Data  Structures
                     
                     
                     
                     


Black Box Testing

Specification:
C = (A*B)                    Dynamic Testing
A: Integer, 1 to 100         Validation
B: Integer, -5 to +5  

              Test Plan
        Inputs         Expected Result
        A    B          C
        1    1          1
        1    6          Error
        5    5          25
        
        
White Box Testing

Specification:
C = (A*B)                   Static Testing
A: Integer, 1 to 100        Verification
B: Integer, -5 to +5  



Code Example 1:              Code Example 2:
Move 0 to A, B               Move 0 to A, B
GetInput(A,1,100,B,-5,5)     GetInput(A,0,100,B,-5,5)
Compute C = A*B              Compute C = A * B + .000001
Compute Tot = Tot + C        Compute Tot = Tot + C


Quotes

• It compiled, it’s got to  • It worked yesterday.
• It usually works.         • It works on my machine.
• No reasonable             • I tested it for you.
customer would ever         • Of course it doesn’t pass 
do that.                      that test.
• Trust me, it’s okay.      • What could go wrong?
Print C      
        
        
        
Screen Edits
• Screen defaults
• Function keys
• Escape key
• Minimize screen
• Maximize screen
• Drag screen
• resize screen
• Initial screen size
• Initial screen location
• O/S Characteristics
• Resolution
• Color
• Fonts
• Menu bars
• Button bars
• Navigation bars
• Slide bars
• Screen title

Button Edits
• Single click
• Double click
• Look & feel
• Space bar
• Tab
• Enter
• Hot key
• Other events that can
be triggered
• Escape
• Default setting
• Color
• Relational edits
• Focus box


Character Entry
• Leading spaces
• Trailing spaces
• Embedded spaces
(multiples)
• Permitted spaces
• Special characters
(numeric, CTRL,
ALT, SHIFT, foreign)
• Specific valid or
invalid characters
• Font
• Color
• Case sensitivity
• Entry templates
• Minimum field length
• Maximum field length

Recommendation Guidelines
1. All recommendations must be written.
Verbal doesn’t count.
2. Provide a means to measure the results.
Results that can not be predicted, or quantified, make
the suggestion seem less credible.
3. Offer a brief assessment of the current situation.
4. Keep it short.
Try to get everything on one page.

Root Cause Analysis
• When was the error made?
• Who made the error?
• What was done incorrectly?
• How could the error have been prevented?
• Why wasn’t the error detected earlier?
• How could the error have been detected earlier?
• How was the error found now?
• Has this happened before?
• Is the problem in any other systems?

Question
Which best describes your job?
1. Defect detection
2. Defect prevention


Assessing Readiness to Automate
1. Are good testing policies and procedures in place today?
2. Is there time to learn how to use the automated testing tool?
3. Will there be ongoing training?
4. Are the development team and test team in agreement
about the use of automated test tools and procedures?
5. Are you ready to make a total commitment to the testing tool?
6. Are your development language and platform supported?
7. How many platforms will you test?
8. Are you testing embedded systems?
9. Is the budget adequate?
10. How much support will your organization require?


1. Design all tests to be repeatable
2. Design the test bed to be maintainable


Question
Which activity will enable you to make the
greatest contribution to your organization?
1. Running tests
2. Identifying testable conditions        

Other Resource

... to read more articles, visit http://sqa.fyicenter.com/art/

Testing and Quality Assurance Techniques