The Office of Information and Technology (OIT) Mobile Application Program (MAP) has tasked Booz Allen Hamilton with supporting tasks under the Mobile Applications (Apps) Verification and Validation (V&V) contract. The V&V team is tasked with supporting the comprehensive testing of multiple Apps from two projects within the Mobile Applications Program: Mobile Development and Mobile Health External Development. They provide the management, business/functional, and technical services needed to assist in providing VA-directed programmatic planning, analysis, design and oversight to guide the V&V of these Apps.
The V&V team sees Apps through the Test Intake process, which includes a Functional Review Findings process to verify that the App is ready to test; then through Test Execution and Analysis, which includes Functional, 508, and Security testing. Once Test Execution and Analysis is complete, a Test Summary Debrief meeting is held to deliver all documentation of test analysis and results so that the App may proceed into compliance review, Pilot/Initial Operating Capability (IOC), and National Release. Throughout the process, the V&V team is tasked with engaging compliance review bodies to ensure that Apps have been reviewed for compliance during development, go through V&V compliance testing, and that compliance best practices are documented and utilized for Apps.
V&V performs testing, following Software Quality Assurance (SQA) development to ensure that Apps function as desired and meet security and compliance standards without defects. V&V is required in order for an App to enter Pilot, Initial Operating Capability (IOC) and National Release.
- Fully Development Application
- Prior to V&V, an App must be fully developed.
- Software Quality Assurance (SQA) testing must be completed.
- It is the expectation of the V&V team that once the App is submitted, testing may be completed successfully (e.g., No Critical or High Severity Level defects or anticipated compliance failures). Therefore, it is highly encouraged that Development teams and Project Managers (PMs) worked closely with compliance reviewers during development.
- Completed Test Intake Form
- A Test Intake Form requires the following documentation (preferably stored in JIRA or Wiki):
- Business Requirements Document
- Concept Paper
- Design Document
- Data Model
- Release Management Plan
- Requirements Traceability Matrix
- Requirements Specification Document (RSD)/ARD Mobile Apps Addendum
- SQA Checklist and/or Unit Test Results
- Sustainment Plan
- System Design Document Mobile Apps Addendum
- System Security Plan Addendum
- User Guide
- User Stories/Acceptance Criteria
- If an intake form is missing required documentation, the form will be returned until the documentation and/or information can be provided before an App is able to enter into the Test Planning and Execution stage.
- It is highly recommend that the App Project already exists in the Mobile Application Environment (MAE) JIRA and documentation is stored there.
When to Engage
- When an application has been approved the Business Owner and is entering development, alert the V&V team through your Project Manager. If possible, the V&V team may attend scrum calls for awareness.
- Once development and all associated documentation and testing have been completed, the Test Intake Form and Addendum should be submitted via email. See your Project Manager for more information.
V&V Review Process
The V&V Test Framework is divided into three main stages, all of which are covered and further defined in the Master Test Plan.
- Test Preparation
Receive Apps test request
Receive completed intake sheet and all relevant Apps materials from VA
Review and validate test request with stakeholder and prioritize application
Acquire Apps software
Deliverable: Test Intake Form
Receive requirements and conduct Functional Product Review
Document and submit Functional Review Findings Report
Deliverable: Functional Review Findings Report
Develop/Modify Master Test Plan
Create Test Plan & Test Procedures for Functional, 508 and Security Testing
Create Requirements Traceability Matrix (RTM)
Generate test data
Test Plan Review
Identify and select tools and test techniques.
Establish automated test framework, as needed
Manage configuration of test devices, test data, test procedures and associated test artifacts
Set-up test environment and perform dry-run
Deliverables: Requirements Traceability Matrix, Test Plan Package
- Test Execution and Evaluation
Execute Test Procedures
Record test results
Document incidents and findings in JIRA
Track incidents and findings in JIRA
Generate defect report
Perform additional testing, as necessary
Deliverables: Test Plan Package, Defect Fix Prioritization List
Test Analysis and Reporting
Collect data and analyze test results
Document and submit Test Summary Reports
Complete and submit SQA Checklist
Complete and submit Enterprise Testing Services (ETS) Intake Assessment
Deliverables: Test Summary Report, RTM Updates, SQA Checklist
Support User Acceptance Testing (UAT) and Compliance
Create UAT Test Scripts
Participate in UAT
Hold daily scrum meetings with Compliance Approvers
Conduct Compliance Debrief meeting
- Initial Operating Capability (IOC)
- Participate in IOC Entry Meeting
- Participate in IOC Evaluation
- Participate in IOC Exit Meeting
- Deliverable: IOC Entry and Exit Request Summary
Note that there will be Go/No Go decisions before proceeding to:
- Test Planning (decided during Functional Review Findings Meeting)
- Test Execution (decided during Test Readiness Review Meetings)
- UAT/Compliance Review (decided during Test Summary Review Meetings)
- IOC Entry (decided during Compliance De-Brief Meeting)
(No Go decisions may be made due to lack of documentation, critical defects, etc.)