08 - Software Testing

ucla | CS 130 | 2024-11-25

Table of Contents

Software Testing

  • to detect diffs bw observed and expected (errors due to omission or commission)
  • test-driven dev suggests writing testable code via:
    • writing tests first before code
    • common for unit testing
    • requires “seams” to insert tests

      Levels of Testing

  • Unit testing (devs) tests indiv modules
  • Regression testing: (devs or testers) incremental differential tests of previously tested sw after change
  • Integration testing (testers) interface bw multiple sw modules
  • System testing (testers) end-to-end testing
  • Acceptance testing (users) tests user requirements on a release candidate of the system

Isolating Item-Under-Test (IUT)

  • to reduce uncertainty, isolate - e.g., mock/patch
  • Test stub: mock items called by IUT
  • Test driver: mock items that call IUT

JUnit - Test Automation

  • testing is hard bc failure causes may be sparse
    • e.g.,
  • so automate via frameworks - e.g., JUnit for Java testing

    Features

  • Annotations, assert statement, test fixtures for common shared code/data, test aggregation for suites
  • Annotation Examples
    • code example
        public class DummyTest {
  private static List<String> list;
	
  @BeforeClass
  public static void beforeClass() {
      list = new ArrayList<String>();
  }
	
  @Before
  public void before() {
      list.add("Alex");
  }
	
  @Test
  public void getElement() {
      String element = list.get(0);
      assertEquals(element, "Alex");
  }
	
  @After
  public void after() {
      list.remove("Alex");
  }
	
  @AfterClass
  public static void afterClass() {
      list = null;
  }
  }
  • Assertions
    • code example
  • other features

Control Flow Testing

  • statement coverage - percent of statements executed by tests
  • branch coverage - % of branches explored by tests
  • path coverage - % of control flow paths explored by tests
  • Control Flow Graph (CFG) Notation

    Analyzing Test Coverage

  • given
  • solution:
    1. Draw CFG
    2. Fill test coverage table

      Branch Coverage

  • measured via decision takes T or F branch
  • example

    Path Count Estimation

  • entails estimating number of loop iterations for bounded progs
  • for loop free with $k$ decisions => $2^k$ feasible paths
  • for loops -> perform loop unrolling s.t. number of paths = $2^\text{# of non-det. decisions}$
  • e.g., given we can follow the steps of CFG then loop unrolling:
    • draw CFG
    • unroll loop
    • draw unrolled CFG
    • calc coverage table
  • nested loop estimation via polynomial combination

Symbolic Execution

  • path counts evaluate whether branch flattens to true or false in nondeterministic paths
  • unconditional code is deterministic
  • but, infeasible paths are dead or unreachable code
  • symbolic execution - unroll loop -> use fresh vars in beginning and with state updates -> for each decision propagate both branches: for each path, the condition exercised is the path condition and the result is the path effect e.g.,
  • examples
    • given1
    • solution1
    • example 2

Generating Test Input

  • symbolic execution can generate test inputs - determine path conditions then find concrete input values that make the path condition true

Regression Testing

  • testing to ensure incremental changes don’t take a step “backward” and cause failures of previously succeeded test cases
  • goal is to select a set of minimum tests to ensure no regressive changes

    Regression Test Selection (RTS)

  • idea is given older version of prog $P$, create test suite for $P$ called $T$ and use it to generate coverage matrix $C_{PT}$-> create new version $P’$ and calc delta bw new and old version -> use to select $\subset T$ s.t. safe RTS covers new version
  • e.g., Harrold & Rothermel’s RTS
    1. Build CFG of original prog $P$
    2. run test suite $T$ on $P$
    3. build coverage matrix $C$
    4. build CFG of $P’$
    5. traverse both CFGs in parallel
    6. select tests for dangerous edges