The U.S. Department of Veterans Affairs' VA Mobile App Store houses three types of apps:

1. Apps built by the VA Office of Connected Care (OCC).
2. VA-certified apps built by other VA groups.
3. Third-party apps that VA specialists recommend.

The Developer Portal explains how Connected Care builds their apps, how they ensure those apps are safe, secure and useful. In the Developer Portal, prospective VA Mobile app developers can learn about the effort and the level of expertise it takes to deliver a VA-branded app. (See Development Stages and VA Mobile App Compliance Requirements for more information.)

Mobile App Development Overview

OCC, working closely with the VA Office of Information and Technology (OIT), has created the Mobile Applications Platform (MAP). Supporting MAP are OCC and OIT teams that own specific operations, from development operations, to reviewing the apps' compliance with all VA requirements, to brokering data to and from VA records, to support. Below are some key features of the MAP solution:

• Agile-based methodology
• Microservice Architecture
• Amazon Web Services (AWS) cloud
• Docker-Jenkins-Kubernetes containerized platform
• Highly automated Continuous Integration, Delivery, and Deployment (CI/CD) workflows
• Github code repository, Jira and Confluence project management

OCC App development teams must use only approved tech stack tools.

About CI/CD

VA Mobile app developers continually improve their apps, and MAP's Continuous Integration and Continuous Deployment (CI/CD) methodology approach allows them to efficiently release their improvements and still maintain required VA quality. Developers iteratively develop, deploy, test, and release app versions by automating key phases of the development lifecycle. Developers can continually improve an app in increments, so the development lifecycle is circular. The following diagram shows the CI/CD pipeline:

A VA-sponsored development team determines the requirements and obtains the needed approvals.

About the MAP Workflow

The MAP workflow is iterative and software may go through several pipeline cycles before the app is ready for Production. Below is a summary of the MAP steps:

1. The app team submits an app's design and requirements for approval to OCC and OCC creates the appropriate MAP account credentials. (See Getting Started)
2. The app team develops locally within a Docker and Kubernetes container environment. (See Local Mobile App Development.)
3. The app deploys to the VA External Cloud Mobile Application Program (VAEC-MAP) Sandbox environment for testing. The Sandbox looks and acts like a VA environment but is not a part of it, so the app can be modified and tested without jeopardizing VA systems. Passing Sandbox testing automatically triggers deployment to Staging. (See MAP Sandbox Environment.)
4. The app deploys to the Staging environment for testing. Staging provides a real integration environment that ensures the app functions in the Production environment. Developers can test code against the VA system, ensure compliance through Compliance Body reviews or Self-Certifications, and submit updated documentation to the MAP Verification and Validation (V&V) team for review. (See MAP Staging Environment.)
5. The app is ready for deployment to Production and Field Testing (FT) after V&V findings are approved by OCC management. (See MAP Production Environment.)
6. The app is ready for national release when the app's Business Owner provides final approval for release.

Development teams may continue to release new application versions with added functionality as illustrated in the above diagram.

Concepts and Principles

OCC adheres to development concepts related to automation, containerization and design principles. These concepts and more are described by the U.S. Digital Services team. Developers working on VA apps must be familiar with these principles. Below are some links to the guiding principles, behaviors, and processes:

• Open source technologies
• Agile/Scrum
• Building Twelve-Factor App
• DevOps
• The Agile Manifesto
• Project Management Institute (PMI) Project Management Body of Knowledge (PMBOK)
• The U.S. Digital Services Playbook
• The TechFAR Hub
• Lighthouse API Platform

Navigating the Developer Portal

The pages in the Developer Portal match the major development stages OCC apps follow to reach the VA App store. (See About the Map Workflow, above.)

To navigate to another section in the Developer Portal, select any menu item listed in the top navigation for Developers. You can also use the navigation links at the end of each section to move forward and backward between sections. Selecting the UP link takes you back to the Developer Portal Home page.

You can access printer-friendly versions of the Developer Portal pages by selecting the printer-friendly version link that is at the bottom of each page. For a printable version of the entire Development Portal, select the printer icon on the Developer Portal Home page below.

Process

This is an overview of the process the Office of Connected Care (OCC) follows to develop and maintain VA mobile apps. There are two paths that an application could follow depending on the purpose of the application and the systems the application will integrate with.

Please submit a request to have your application be part of the VA Mobile App Store by completing the VA Mobile App Intake Form (VA Network access required). The following instructions will provide you with additional details about requirements based on the purpose of your application.

Planning

A new VA app requires the following planning activities:

• Conduct a trademark clearance search using the US Patent and Trademark Office (USPTO) Trademark Electronic Search System (TESS). Determine if a trademark has already been registered (or applied for) that is similar or already in use.
• Present names and logos to the VA Office of General Counsel.
• Draft a Help Desk Support plan and funding.
• Draft a Sustainment and app retirement plan.
• Review VA Mobile App Compliance Requirements
  • Applications must receive authorization from Privacy, Security, 508 and adhere to VA branding standards.

If the application being developed will integrate with VA systems and be housed in the VA External Cloud (VAEC) then the app team must also be prepared to complete the following activities:

• If the app accesses VA data, the development team must draft a System Design Document (SDD) or Service Design Document (SRVDD)
• Present a System Design Document (SDD) or Service Design Document (SRVDD) to the Office of Connected Care VA Mobile Architecture team for approval. This must be approved to move into the MAP Staging Environment.
• Obtain current federal identification credentials and request a Mobile App Program (MAP) account. This lets the team access the MAP development tools within the VA firewall.
• Conduct Agile Sprint 0 activities, including the creation of epics, stories, and a backlog within the MAP space in the Atlassian suite.
• Review VA Mobile App Compliance Requirements
  • Applications must receive authorization from Privacy, Security, 508 and adhere to VA branding standards.

Project Management

VA app development teams use Atlassian Jira to track all project activity. Teams maintain project documentation in Confluence wiki spaces for each respective project. Only VA MAP development personnel with the appropriate credentials have access to these sites. Most of the development operation work requests are created from Jira tickets. Teams create the required project documentation with Confluence wiki templates. The Developer Help wiki space maintains a general guide to MAP operations, and the Mobile Application Cloud Migration (MACM) Chief Product Team wiki space maintains detailed MAP standard operating procedures.

Local Mobile App Development

Getting Started

VA app development can begin on local computers outside the VA network using a Mobile App Platform (MAP) Docker solution. Developers should install the following technical stack:

• Git
• Docker Desktop

How to Kickstart a New Project

Developers once onboarded will have access to starter projects maintained by our Shared Services team to help with service and application development.

Once familiar with the VA Mobile Framework build platform and the VA Mobile App Compliance Requirements, developers implement a MAP Docker Starter Kit to move their projects into the MAP Sandbox environment.

The MAP Sandbox Environment

Once an app development team completes creating an app locally from the Starter Kits, obtain accounts to the Sandbox Active Directory, they then deploy their containers to the VA Enterprise Cloud Mobile Application Program (VAEC-MAP) Sandbox. MAP Sandbox is an ephemeral environment where development namespaces can be deleted and recreated as needed. 

The Sandbox is similar to the staging and production environments except that, for safety, it is not part of the VA network. Developers test their apps to see how they would work if they were on the VA network. They also gather all the compliance artifacts they need to pass Verification and Validation (V&V). As a best practice for the required System/Service Design Document (SRVDD/SDD), teams should have their approved System/Service Design Document (SRVDD/SDD) before their application/service can deploy to staging.

Deploying and working in the Sandbox is similar to the local development Docker-Kubernetes environment configuration. The main difference is that the build and deploy scripts run on a Jenkins build automation server. Deploying to the MAP Sandbox has the following advantages:

• Reduces environment dependency: Reduce the dependency that development teams have on any one particular environment by providing the capability to launch and terminate them at will.
• Better resource utilization: By terminating environments that are no longer being used, capacity is freed up for other development teams.
• Knowledge transfer: When team members know that their environments will be terminated at specific times, automation becomes the only solution to the institutional knowledge of how the environment gets configured.

These are some of the functions developers can add to their applications once they reach the Sandbox:

• Create namespaces with the "create-namespace" Jenkins job
• Manage Consul and Vault values for testing
• Create Kubernetes pods
• Connect without VA VPN access
• Run builds
• Deploy apps
• Create databases (MongoDB, Oracle)
• View application logs in Kibana
• Integrate with mock VA services to control test scenarios and data (the Sandbox has no connection to real VA services)
• Jenkins build pipeline to build, test, and deploy applications.
• Static code quality and security scanning.

Sandbox Requirements

Deploying to the Sandbox environment confirms that the development team has tested for the following issues:

• Application automation works (including automated testing)
• Application environment is created properly
• Jenkins file is created properly
• Product configuration is accurate
• Dev branch should build & deploy in Sandbox
• Release branch should build & deploy in Sandbox

Deploying to the Staging Environment

To move to the Staging environment or SQA environment, the development team submits requests to have the build pipeline and any additional configurations set up. Once ready, the development team may proceed with the automated build and deployment of their application in the SQA environment. If any issues occur in Staging, OCC may require the app to return to the Sandbox to fix the issues.

The MAP Staging Environment

The Staging environment is a real VA integration environment where developers ensure apps are able to correctly function in the VA Pre-Production environment. Once the code is deployed to Staging, developers then test their code on a genuine VA system, ensure compliance through VA Mobile App Compliance Requirements, and undergo review by the Verification and Validation (V&V) team.

In Staging, developers test their code's functionality and that they comply with VA standards, and then present them to the V&V team.

Preparing to Deploy to Production

The Application/Service Teams submit their application/service request for V&V review in parallel to submitting their requests for permission to deploy to the Production environment. The Application/Service Teams performs two request activities, described in the following sections which involve drafting a Technical Review Board ticket (TRB) and submitting an NGD (Next Generation Deployment) ticket.

Application/Service Teams Issues an NGD Ticket

The Application/Service Teams V&V team issues an NGD ticket, which is a reference to OCC's current generation of software development operations, to the Technical Review Board (TRB) for review.

Technical Review Board (TRB) Completes Qualitative Review

The Technical Review Board (TRB), comprising Connected Care and Office of Information and Technology stakeholders, reviews all NGD production tickets and makes final approvals for apps to move to the Mobile Application Platform (MAP) Production Environment.

Release Candidate Types

The first release of an app requires working through the entire CI/CD pipeline. Subsequent releases may qualify to work through a reduced pipeline without sacrificing VA quality standards. The development team's project manager decides whether a release candidate is a Major, a Minor, or a Patch release. The requirements and documentation to release a version varies accordingly:

Major Releases (Version X)

A Major release is an application's first release to production, or any subsequent release that has added functionality or architecture changes. (Major Releases of existing products have breaking changes: There are removed or changed features and dependent modules have to modified to be compatible with the new version.) Major Releases must progress through the entire MAP pipeline and submit a complete set of compliance documents during review. OCC decides if applications that do not access VA networks ("informational" applications), are eligible to comply with a reduced set of compliance certifications.

Minor Releases (Version X.Y)

A Minor release is an incremental feature set enhancement. Minor releases may introduce new features but may not change behavior for existing APIs. Minor releases are always backward compatible with their parent Major release. If an application adds a feature, but the module is backwards compatible, it qualifies as a minor release. Minor release candidates resubmit updated credentials for most of the compliance requirements during review.

Patch Releases (Version X.Y.Z)

Only bug fixes, technical debt, flexline findings, OIS findings, and/or minor improvements that impact no other changes to the application qualify as patch releases.

VA Verification and Validation

The Verification and Validation (V&V) Team performs a comprehensive and thorough review of documentation and testing of the applications/services capabilities for the Department of Veteran Affairs/VA Office of Connected Care (OCC). The VA OCC Application/Service Teams are required to submit their application/service release requests through the V&V Automation by utilizing the Jenkins YAML process that creates a JIRA Mobile Application Software Quality Assurance (MASQA) ticket and Markdown Report and notifies the V&V Team that the application/service is ready for review. The V&V Team receives the JIRA MASQA ticket and reviews the “Release Types” (Major, Minor, or Patch) to determine what documentation is required to be reviewed and what testing is required. Verification & Validation (V&V) - Process (VA Staff Only).

The V&V Team reviews all the required documentation and creates a Review Findings Report (RFR) that provides information on the application/service and whether the supplied information passed or failed the V&V review. V&V creates a JIRA MASQA ticket for documentation that fails and assigns it to the Application/Service Team for resolution.

About IAM: Identity and Access Management shared services

VA applications may utilize two Identity and Access Management network services (IAM) to simplify accessing the applications. Secure Logon Access Instructions. The two service types are:

• Single Sign-on Authentication services, which simplify accessing the apps.
  • Single Sign-on External (SSOe) provides authentication services for external users (currently Veterans, third party vendor support coming soon).
  • Single Sign-on Internal (SSOi) provides authentication services for Staff.
• Access via the Master Veteran Index (MVI) to retrieve identifiers for Veterans known to the VA by other identity systems VA recognizes, like: Login.gov and ID.me.

If an application utilizes these services, the development team must submit requests before the application moves to Production.

The Mobile Application Platform (MAP) Production Environment

Preparing your Application for National Release

Once your application has been approved by MAP leadership, an application moves into production for end-user testing and national release activities. Once the application is in the production environment, the application is tested by the field to ensure everything is working properly by the end-users.

Once fully tested an application is then marketed across the Department of Veteran Affairs and can be made available on the VA Mobile App store for download and use.

Field Testing Processes

OCC apps typically undergo a phase called Field Testing, which includes processes and support activities that ensure OCC delivers the best possible product. OCC's Field Testing team states:

Field testing strives to continuously improve product quality through data collection and analysis. OCC's Field Testing team uses evidence-based methods to test and develop products for implementation across the VA enterprise. Field testing provides the data gathering and analysis required in order to successfully carry out the implementation of a product.

The field testing process consists of several steps that conclude with an assessment of lessons learned. Lessons learned are identified and reviewed to optimize processes going forward and share valuable insight from the field with other product development teams. Field testing is a bridge between app development and the implementation/rollout of the product. It is separate from usability testing or User Acceptance Testing (UAT). Field testing seeks to find out how well a product will fit in a real-world environment. Its objective is to determine how well a product fits (or does not fit) into existing workflows, assess potential changes that need to be made prior to a broader deployment, and anticipate the value/impact to VA. The data gathered from these activities serves to provide guidance for future app functionality, and the implementation and rollout strategy.

National Release Activities

Once an application completes field testing and resolves any potential bugs found, the application is ready for National Release and can be made readily available through any or all of the options below:

• VA Mobile Application Store
• VA Google Play Store
• VA Apple Store
• VA App Catalogue
• Patient Video Tablets
• Digital Prescription Pad

Once the application is available on the applicable platforms listed above, the application should be marketed by the application team as laid out in the implementation strategy and made available for the use of our VA Veterans or VA Providers.

Human Factors Engineering

VA mobile apps must meet safety and usability requirements set forth by the Human Factors Engineering (HFE) division within the Clinical Informatics and Data Management Office (CIDMO).

What is Human Factors Engineering?

As a discipline, human factors engineering studies how people interact with devices, products, and systems. Behavioral science, engineering, and other disciplines come together to help assure that devices and systems are usable by the people who are meant to use them. Human Factors Engineering (HFE) is a division within the Clinical Informatics and Data Management Office (CIDMO) in the Office of Health Informatics (OHI).

HFE’s mission is to maximize efficiency, effectiveness, satisfaction, and safety of VA systems by using analysis methods and design principles based on knowledge of human capabilities and limitations. This effort takes place throughout the engineering lifecycle. HFE focuses on collaborative efforts to optimize design, minimize risk of human error, and ensure systems and services are engineered for human use.

What does HFE do?

HFE offers analysis, design, and testing methods to improve the safety and usability of systems used by Veterans, Caregivers, and VHA employees. Its goal is to maximize system performance. HFE promotes the benefits of the Human Centered Design process, which aims to make systems usable and useful by focusing on the users, their needs, and end-user requirements.

In support of OHI’s mission, HFE provides guidance on human performance risks and issues in order to enable informed decision-making. It makes recommendations that result in streamlined policies, processes, procedures, and operations. It also advises on ways to reduce operating costs, increase safety, performance, efficiency, and effectiveness of health information technology across VA.

For VA Mobile, the HFE team works with developers to ensure that mobile applications meet usability standards and “acceptable use” criteria. These reviews are meant to ensure a minimum level of usability by identifying issues that would lead to dissatisfaction among end users.

What is the value of involving HFE?

Use HFE early to gain knowledge of user needs when there is high design flexibility, available resources, and low risk. Without early validation of customer or end user needs through HFE, risk is back loaded. The goal is to identify any potential risks early in the development process, and avoid the need for design changes later in the process due to unforeseen risks. If risks are not identified until later in the system development lifecycle, there may be much less design flexibility and a lack of resources to addresses issues that may arise.

How and when can you get involved?

The earlier HFE can engage with project teams, the higher the likelihood of positively impacting information architecture and optimizing the user interface design. Considering usability early in the design process will reduce the number of iterations needed to create an effective, efficient, and safe product. This ultimately saves time and money. When first setting development requirements, consider the opportunity to consult with HFE, so we can provide recommendations for the best way to incorporate HCD in your development process.

To learn more about HFE and its services or request support contact vha10p2hfq@va.gov. Developers should visit HFE’s Mobile App Review Process to learn more about how HFE can help ensure a positive, meaningful and valuable experience for your users.

HFE Mobile App Review Process

The VHA Office of Health Informatics Human Factors Engineering (HFE) works with developers to ensure that mobile applications (apps) meet usability standards and “acceptable use” criteria. The HFE compliance review ensures that an app meets an acceptable level of usability for the end user. This is done by identifying any issues that would lead to end user dissatisfaction with the app. When HFE receives a compliance review request from the app developer, HFE conducts a Heuristic Evaluation (HE), (VA Network Access Required: https://dvagov.sharepoint.com/sites/vhahumanfactors/boksitepages/methods/heuristic%20evaluation.aspx) where a human factors expert evaluates the app against a pre-defined set of widely accepted usability heuristics (considered standards of best practices in both business and academia), and then reports any issues found, categorized as “Serious”, “Moderate”, or “Minor” severity.

What is a Heuristic Evaluation?

HEs are not a pass/fail metric. The purpose of an HE is to identify issues that are expected to impact use in a negative way. The “Serious”, “Moderate”, and “Minor” ratings indicate how likely the issue is to impact a user’s experience with the app. HFE recommends addressing all “Serious” issues if possible, and, as resources allow, addressing “Moderate” and “Minor” issues. HFE will work with the app development team to track any changes made, along with justifications for why certain changes might not be pursued for each finding.

HEs offer a number of benefits for development teams compared to other HFE methods:

• HEs can be completed at various points in the development process.
• HEs typically do not require advance planning.
• HEs can typically be turned around approximately one week, depending on size.
• HEs can serve as a pretest for user testing if additional testing is needed.

What is Usability Testing?

In some cases HFE may recommend or require additional evaluations, such as usability testing. Usability testing is usually done when end-user input and perspective can provide additional insight into an app's usability. Like HEs, Usability Testing can be done at various points throughout the development process. There are two types of Usability Testing methods, and the final determination on which method HRE recommends is made in collaboration with the app development team.

• Formative Usability Testing (VA Network Access Required: https://dvagov.sharepoint.com/sites/vhahumanfactors/boksitepages/methods/formative%20usability%20test.aspx) occurs during the development process, with a focus on eliminating usability issues.
• Summative Usability Testing (VA Network Access Required: https://dvagov.sharepoint.com/sites/vhahumanfactors/BoKSitePages/Methods/Summative%20Usability%20Test.aspx) occurs upon completion of development, and focuses primarily on how the system meets usability benchmarks.

Benefits of Engaging HFE Early

App development teams are also encouraged to notify HFE when apps are registered. By working closely with HFE early in the app development process, the development team can ensure that its app will be intuitive, engaging, and well received by the end user.

The following steps describe how an app development team might engage HFE for iterative expert reviews prior to the final compliance review:

1. App developers request a review of a completed set of mock-ups (wireframes) or a partially completed app at any time before or during development. Developers provide HFE with the mock-ups or test environment and any associated documentation. A meeting is scheduled to discuss the app and address any clarifying questions.
2. HFE conducts an initial evaluation of the app. This evaluation typically results in annotated screenshots and a findings table. The document is returned to developers as early as one week from the initial meeting.
3. Business Owners and developers review the evaluation and document a response for each finding. Responses are discussed in a follow-up meeting with HFE.
4. This process repeats at intervals appropriate to the app’s development; ideally, HFE will review after each significant change.

In addition to expert reviews, HFE recommends development teams consider involving end users as often as possible.

Note: The HFE User Interface (UI) Design Confirmation Review is a separate compliance review requirement.

HFE is managed under the Clinical Informatics and Data Management Office (CIDMO)

For more information pertaining to Human Factors Engineering, please refer to our Human Factors Engineering SharePoint site (VA Network Access Required: https://dvagov.sharepoint.com/sites/vhahumanfactors/SitePages/HFE%20Division%20Home%20Page.aspx)

If you would like to request HFE services, please contact VHA10p2HFQ@va.gov

References

Jakob Nielsen: Ten Usability Heuristics

BJ Fogg: Persuasive Technology: Using Computers to Change What We Think and Do

Human Factors Engineering (HFE) Compliance Self-Certification

VHA Health Informatics Human Factors Engineering (HFE) staff developed the Self-Certification Program to allow project teams to bring “usability thinking” and planning to earlier stages of mobile applications (apps) development. It will free up HFE to work with app developers at the beginning of the development cycles (when the impact is greater), resulting in a better user experience for Veterans and clinicians.

A well-designed application allows the user to easily and intuitively complete tasks, without having to learn new conventions, or to think about where to find information or how to operate the app. The Self-Certification process allows project teams to certify that their apps meet usability criteria. An app that is efficient, effective and satisfying to the user will likely be used more often and not abandoned after the limited use.

What is assessed?

A compliance review of an app is composed of two primary parts: the Mobile User Interface (UI) Design Certification (UI Cert) and the Heuristic Evaluation (HE). The UI Cert is performed by comparing the app against a criteria list of key design standards and documenting any deviations, along with a severity rating. The HE is performed by reviewing an app, while being mindful of 10 industry standard heuristics, and assigning a severity rating to issues that violate those standards.

Supporting documentation for the Self-Certification process is posted on the link below, under "References."

What is the process?

• Project teams refer to the HFE UI Design Criteria and Heuristic Standards when the app is at the beginning stages of development, such as requirements analysis, and incorporate the best practices into the app design. A link to the criteria and standards appears below, under "References."
• Project teams can incorporate Usability Testing during the development of the app. Contact HFE to inquire about the testing services. A link to the HFE website appears below (under "References") and contains contact information.
• After the app has had a completed OI&T Verification and Validation (V&V) review and been approved to move forward for the final Mobile Health certifying body reviews, project teams download the HFE Self-Certification materials and commence with their UI Design Compliance Review.

References:

• HFE Self-Certification Initiative Training Materials (VA Network Access Required) - https://dvagov.sharepoint.com/:w:/s/vhahumanfactors/EQQ6XWchJlhGrGWDCATwDxkBPLLaGjwPU_1r4clMQCmp4A
• HFE Self-Certification Template (VA Network Access Required) - https://dvagov.sharepoint.com/:x:/s/vhahumanfactors/EQ4qpUzdMuJLqHDsAZW7seIBFllPZIvjA87pZMPnTrrARQ
• For more information please visit the HFE SharePoint site (VA Network Access Required) - https://dvagov.sharepoint.com/sites/vhahumanfactors/SitePages/HFE%20Division%20Home%20Page.aspx

Code Review

All apps must pass a Fortify code scan in the staging environment. The VA Office of Information Security (OIS) Software Assurance (SwA) Program Office ensures VA app developers adhere to VA requirements when they conduct the scan validation process. From OIS SwA:

OIS Authorization Requirements SOP section Application Security Testing effectively requires CI/CD pipelines to be certified by OIS Software Assurance to increase the confidence in the security of continuous deployments. Certification tests OIS-licensed Fortify tool integrations in pipelines and pipeline workflows by reviewing scans produced for completeness and correctness. Projects, more specifically CI/CD pipelines for applications (or libraries, microservices, or scannable blocks of VA code) registered with OIS Software Assurance that successfully complete the Fortify scan validation process are then considered certified for their authorization period, during which time automated releases may be made according to applicable policies.

Data Security

The Veterans Health Administration (VHA) Healthcare Security Requirements (HCSR) Office reviews your mobile application (App) to ensure that it meets VA software data security standards. We ensure that your App complies with the Health Insurance Portability and Accountability Act of 1996 (HIPAA) and Security Rule specifications for protecting electronic Protected Health Information (ePHI). Additionally, we ensure that you implement sufficient security control during development to protect the Veteran’s PHI and personally identifiable information (PII) data. The objective of the data security reviews are to protect the Veteran’s ePHI/PII data from compromise or breach, be it of an intentional or unintentional nature, by ensuring the proper security control measures are implemented to prevent the theft, interception or alteration of the data.

We ensure that if your App is designed or developed to transmit, store or process a Veteran’s ePHI/PII data has the appropriate security control measures to:

• Restrict access to the ePHI/PII data by password, personal identification number (PIN) or other appropriate access control mechanism that meets VA requirements
• Encrypt stored ePHI/PII data with Federal Information Processing Standard (FIPS) 140-2 (or its successor) validated encryption
• Encrypt ePHI/PII data transmitted to or from VA with FIPS 140-2 (or its successor) validated encryption
• Impose the removal mechanisms needed to sanitize all stored ePHI/PII, according to VA policy

We review your Business Requirement Documents (BRDs), application user stories and technical development documents, such as Requirements Specification Documents (RSDs) and System Design Documents (SDDs), if they are available.

References

The VHA HCSR Office conducts the data security compliance reviews in accordance with the provisions of the HIPAA Security Rule and VA Handbook 6500, Risk Management Framework for VA Information Systems - Tier 3: VA Information Security Program.

Patient Safety

About the Patient Safety Assessment

Patient safety reviews of mobile applications are conducted by The Veterans Health Administration (VHA) Informatics Patient Safety Office (IPS), whose purpose is to improve the safety of Health Information Technology (HIT) products used by the VA and Veterans.

How to request an IPS Compliance Review

Project Managers (PMs) designated to a mobile application by Connected Care may request an IPS review. In order to help IPS understand the context of use, identify safety critical tasks, walkthrough realistic scenarios, and manage the review process efficiently and effectively, PMs are asked to complete an IPS Mobile Application Review Request Form (also available to MAE members as an attachment IPS Mobile App Review Request form) in its entirety. The PM should then send the completed form and associated documentation via E-mail to VHA10P2AIPSMobileTeam@va.gov. IPS will respond via E-mail and include the contact information for the person assigned to lead the patient safety review. Note: Requests for a preliminary review before the compliance review stage (i.e. wireframe or preliminary executable) may be granted if sufficient IPS resources are available during the time the review is requested. As much of the review request form as possible should be completed when requesting a preliminary review.

IPS Review Results

When IPS has completed its review the PM will receive an E-mail with an attached document that provides a detailed description of the review conducted and any patient safety issues found. This is presented in a standardized review form, Informatics Patient Safety (IPS) Mobile App Inspection Template. IPS welcomes questions about the review results and also requests that after the PM has discussed the results with the application business owner the response to each issue is communicated back to the IPS person who led the application’s review.

IPS Recommendations for Enhancing Patient Safety

Patient Safety can be enhanced by:

• Consideration for Patient Safety at all stages in the development life-cycle for mobile applications
• Certain application characteristics and functionality

Development Process Enhancements

• Clearly define usage context: Users, Tasks, Technology, Environment of use
• Identify high-level tasks (clinical/medical tasks) early
• Connect user stories to high-level tasks
• Explicitly connect functionality to high-level tasks (this is sometimes called a Conceptual model)
• Pay particular attention to safety-critical tasks. A safety critical task is any action or decision whose failure could cause patient harm or a significant delay in care delivery
• Provide support for users needing immediate help during safety critical tasks
• Employ independent clinical review and usability testing of patient safety critical tasks
• Develop or utilize a user accessible post-deployment issue reporting system

Application Design Enhancements

When presenting information to the user:

• Make safety critical information salient
• Provide context qualifying information near data presented - e.g. units of measure
• Show deviation from the norm in results
• Inform the user when system errors/events occur they need to be aware of
• Provide meaningful error messages
• Show how to get immediate help
• Show how to report patient safety issues
• Show who can see data entered
• Show the app version/build number
• User language of the user (especially for safety critical tasks)

When supporting the entry of data by the user Help avoid incorrect action selection & data entry

• Have a confirmation step in some situations
• Have an undo capability

References

1. Bagian, J.P., Lee, C., Gosbee, J., DeRosier, J., Stalhandske, E., Eldridge, N., Williams, R., & Burkhardt, M. (2001). Developing and deploying a patient safety program in a large health care delivery system: You can’t fix what you don’t know about. Joint Commission Journal of Quality Improvement, 27, 522–532.
2. Chapman R. J., Taylor, L. M., & Wood, S. D. (2012). Cataloging Errors from Reported Informatics Patient Safety Adverse Events. Proceedings of Human Factors and Ergonomics Society Healthcare Symposium, Baltimore, MD. (Available to MAE members as an attachment here).
3. DeRosier, J., Stalhandske, E., Bagian, J.P., & Nudell, T. (2002). Using Health Care Failure Mode and Effect Analysis™: The VA National Center for Patient Safety’s prospective risk analysis system. Joint Commission Journal of Quality Improvement, 28, 248–267.
4. Draft Guidance for Industry and Food and Drug Administration Staff - Applying Human Factors and Usability Engineering to Optimize Medical Device Design. http://www.fda.gov/MedicalDevices/DeviceRegulationandGuidance/GuidanceDocuments/default.htm
5. Johnson, J., ‎ Henderson, A., (2012). Conceptual Models: Core to Good Design. Morgan & Claypool.
6. NIST EHR Usability Protocol (Lowry, et al. 2012)
7. (NISTIR 7804) Technical Evaluation, Testing and Validation of the Usability of Electronic Health Records.
   • http://www.nist.gov/manuscript-publication-search.cfm?pub_id=909701
   • http://www.nist.gov/manuscript-publication-search.cfm?pub_id=907849#
8. Wharton, C. Rieman, J., Lewis, C. and Polson, P. (1994). The Cognitive Walkthrough Method: A Practitioner's Guide. In J. Nielsen and R. Mack (eds.) Usability Inspection Methods (New York: Wiley) 105-140.
9. Wood, S. D., Chapman R. J., Taylor, L. M., Wright, P., & Scott J. (2014). Identifying Latent Design Issues in Mobile Products to Prevent Patient Harm. Proceedings of Human Factors and Ergonomics Society Healthcare Symposium, Chicago, IL. (Available to MAE members as an attachment here).

Privacy and Application Data Security

The VHA Privacy and Application Security offices ensure that mobile applications adhere to Privacy regulations and statutes as well as VA policy. We use the Privacy and Security checklist to determine if data is stored, transmitted or entered by the user, provider or employee. We also determine if there is sensitive information like Private Health Information (PHI) or Personally identifiable information (PII) stored on the App. If it is, the Data Security branch determines how the App protects the information. For an overview of the Healthcare Security process, please review this presentation.

Privacy Issues

Here are examples of the issues we examine to see how your App handles Privacy issues.

Apps for Veterans:

• If the App retrieves VA Data from a VA database and stores it on the mobile device, an end user license agreement (EULA) states that the Veteran owns the locally stored data.

• If the App retrieves VA data from VA database but it is not stored on the Veteran’s device, the App's EULA states that the Veteran is not being provided a copy but is only being given access to the data through the device.

• If the App does not transmit a Veteran's self-entered data to VA, it securely stores it on the device as determined by an Information Security Officer.

• If the App transmits a Veteran's self-entered data to VA it does so in a manner that is covered by a Privacy Act system of records. The App's EULA states that the VA will receive the data entered by the user on the device.

Apps for Providers:

• If the App transmits data from a Provider to VA, it does so in a manner that is covered by a Privacy Act system of records like the Mobile Application Environment (MAE) -VA 173VA005OP2.

• If the App displays any VA data retrieved from VA database and displayed to VA Providers in performance of their official duties, it does not store that data on the device.

• If the App retrieves VA data from VA database that a VA Provider modifies and then transmits to VA for inclusion, the record arrives in the appropriate Federal Record or in a Privacy Act System of Records. (For example, 24VA10P2)

Section 508 Compliance

The Office of Information andTechnology (OIT) Section 508 Office tests mobile content to verify that it complies with Section 508 standards, including 1194.21 (Software and Platforms), 1194.22 Web, 1194.24 (multimedia), and 1194.31 Functional Requirements. We work to facilitate access to mobile technologies for anyone with disabilities.

We manually test VA platform-specific Apps directly on the device using its built-in accessibility features and supported methods. For example, we test with VoiceOver on iOS and Talkback on Android.

We perform some automated testing on web-based content, but we also perform manual tests to ensure the (mobile application) App properly implements accessibility techniques on the device’s browsers.

Note: App platforms have varying accessibility support. It is important to develop using the accessibility support of each platform you develop.

Registration

The Section 508 office maintains its own project system. It is important to register your App as early in Planning or Development as possible. To register your App, visit:

http://www.insidetms.va.gov/VA_SubmissionForms/section508/default.asp

Testing

The following list describes some of the key areas that the Section 508 Office tests. It does not include all testing activities.

• Use of a color contrast checker to verify that all text and images of text meet color contrast requirements
• Manual tests to verify that color is not the sole means of communicating information
• Manual tests with the device's screen reader to verify that all text and controls are spoken and properly identified
• Manual tests with the device’s screen reader to verify that the user can operate all controls using accessible methods supported by the Operating System and device. This testing includes:
  • Text entry
  • Selecting tabs
  • Operating pickers, sliders and other controls
• Manual tests with screen readers to verify that all images that convey meaning or that are part of controls have appropriate alternative spoken text
• Manual tests with screen readers to verify that screen review methods read the content in the correct sequence

For further information on the Section 508 Office, you can refer to their web site here.

References

WCAG 2.0

Section 508 ICT Baseline

Access Board

Apple iOS Accessibility API

Android Accessibility Layer

VA Mobile Branding Requirements and Resources

App developers certify that their apps comply with VA's look and feel by certifying the VA Branding compliance review while the app is in the The MAP Staging Environment. The review is a checklist established by the Office of Public and Intergovernmental Affairs (OPIA). The review compares the app's look and feel to the VA Mobile Style Guidelines to ensure the app follows the appropriate Logo, Usage, Typography, Color Palette, Chiclet, Splash Page, Naming Conventions, and Splash Screen.

The Branding review addresses questions like:

• Does the app name provide sufficient information to describe the purpose of the app?
• If the app name has a shortened version, does the abbreviation retain the name's meaning?
• Has the app name successfully gone through a Trademark Clearance Search using the Trademark Electronic Search Engine (TESS; www.uspto.gov)?
• Does the app have a Department of Veteran's Affairs (VA) Header, clearly identifying it as property of the Department of Veterans Affairs?
• Does the app follow the VA approved logos found in the App Logo Library?
• Does the app use the correct font as approved for use within VHA?
• Does the app use the correct VHA color palette?
• Does the app have an approved application chiclet?
• Does the app have an approved app splash page?
• Is the app logo in the correct file format?
• Is the app chiclet in the correct file format?

The app team then makes the app available to an additional reviewing body for peer review. The peer review is a secondary assessment of the app to ensure the initial review was conducted properly. This ensures a different perspective of the app in addition to uncovering issues that may only surface on one operating system (OS) or device type.

The VA App Verification and Validation team then verifies the Branding review records and reports its findings to MAP leadership, OPIA, and OCC leadership.

Branding Resources

• VA Mobile Style Guidelines
• VA Mobile Branding Requirements and Resources
• OPIA Branding for VA Apps Presentation

Web App Security

Web Application Security Assessment (WASA )

Web Application Security Assessments (WASAs) are an in-depth penetration test for common vulnerabilities, such as SQL Injection, Authorization Bypass and Cross-Site Scripting (XSS) within web applications. Development teams submit a WASA questionnaire to OCC when the app reaches the staging environment. OCC periodically requests the VA Cybersecurity Operations Center (CSOC) Vulnerability Scanning Team (VST) perform vulnerability scans of VA apps.

Mobile Application Security Assessment (MASA) Penetration Testing

Specific to mobile apps, the Mobile Application Security Assessment (MASA), is a process for reducing risk and improving compliance with industry regulations by comprehensively analyzing an application’s security system. The NSOC uses a variety of tools and act as an "attacker" to identify application vulnerabilities on a running application (runtime analysis). One aspect of their test is to use tools such as Nessus to scan an app's infrastructure. NSOC security engineers also perform manual testing. This phase, sometimes also called Enterprise IT Security, is a formal Mobile Applications Program (MAP) requirement that must be completed before promotion to production. This testing is performed in a test environment, not in production. The app Project Manager. is responsible for contacting NSOC.

VA Mobile App Intake

Step One

Request approval and learn what documentation is required.

Before you can publish your app to any of the OCC App Publishing Options, you must obtain approval for your app project by completing the VA Mobile App Intake (VA Network access required).

There are three internal classifications of apps. You will need to read the definitions below to determine which classification your app falls under. There is general information you will need to be prepared to complete regardless of whether your app is classified as an OCC app, a VA-sponsored app, or a third-party-app.

General Information Required

• A high-level executive summary: Be prepared to describe the app as you would in the description field of an app store download page. The description should include the target audience(s) and the job the app helps the target audience(s) accomplish.
• App platform(s).
• App audience(s).
• The address(es) for the app: native/web URL, Apple's App Store URL, and/or Google Play URL, if applicable.
• The purpose of the app.
• What features or capabilities does the app have over similar apps that already exist and are listed on the VA App Store.
• Where a user gets technical support for the app (be prepared to provide detailed support instructions).
• Is the app required to be behind the VA firewall in order to use. If so, be prepared to identify sign in method(s) required for the app: LOGIN.GOV, ID.me, or DS Logon Level 2 (Premium),  MyHealtheVet Premium.
• Review the OCC app publishing options, so you can be prepared to request which OCC App Catalogs you want your app published on.

OCC App Information Required

• Project manager (PM) and contracting PMs Contact Info.

VA-Sponsored App and Third-Party App Information Required

• Whether VA funds were used to purchase the app or license the app.
• Contact information of the person and office sponsoring these apps contact info.
• Be prepared to agree to review the content published for the app biannually and the apps continued relevance for the intended target audience(s) if approved for the VA  App Store.
• Which company makes the app.
• The vendor address.

Once the Office of Connected Care approves the VA app, a workflow is triggered. If the app is an OCC app, you will receive an information packet and Mobile App Platform (MAP) account instructions. If it is not approved, you will receive an email from Connected Care explaining the reason. Once approved, be prepared to provide additional metadata to include VA branded icons.

Once your app has been published to one or multiple of the OCC app publishing options, you may need to make edits and/or updates. Please contact: OCCVAMobileIntakeSupport@va.gov.

Once your request has been reviewed by the committee, you will receive additional information on how to provide the documentation required to publish your app. Reviewing all the information on the OCC VA Mobile Developer Portal will be helpful in determining the required workload and/or level of effort to get your app through the various Development Stages.

Step Two

Discover app publishing avenues!

Once approved and documentation received, it's time to publish your app!

Connected care has many application publishing options, which include the following:

• VA App Store (external)
• Patient Video Tablets (PVTs) (internal)

The VA App Store is an external app store. Veterans and other staff can access it from their computers or mobile devices. There are other avenues to publishing apps outside of Connected Care. The VA App Catalog is an internal app store and accessible by only GFE devices which are enrolled in VA's Enterprise Mobility Management (EMM). This store is managed by the VA Enterprise Mobility team (Solution Delivery Mobility and Endpoint Engineering team). There are also commercial app stores, Apple App Store and Google Play. These publishing avenues outside of Connected Care have different points of contact and processes to follow, as noted in the section: Publishing Apps Outside of OCC.

OCC App Ownership

OCC Apps

OCC apps are developed by the VHA Office of Connected Care in close cooperation with the VA Office of Information and Technology (OIT). They have passed extensive review and testing to ensure that they comply with all Federal and VA requirements for safety, security, and usefulness, as outlined in the Developer Portal. Only tested and approved apps display VA’s Mobile App logo or appear as a VA app in app stores. They are tools for Veterans, their Caregivers, and VA staff. The app development teams manage their projects and deliverables on the Mobile App Platform (MAP).

OCC PMs can suggest an app for consideration through the VA Mobile App Intake (VA Network access required).

VA-Sponsored Apps

The VA Mobile Enterprise App Store may also include apps built or contracted by the VA that originate outside the Office of Connected Care (OCC). They have passed all necessary VA requirements and testing to ensure that they meet Federal and VA requirements for safety, security, and usefulness, as outlined in the VA Software Compliance Requirements of the Developing VA Apps overview.

If you are a VA organization and are planning or developing an app you want to display in the VA App Store as a VA-branded app, it must comply with VA standards. You must upload your code to VA’s VA Mobile Framework (VAMF) for testing and release.

VA organizations can suggest an app for consideration through the VA Mobile App Intake (VA Network access required).

Third Party Apps

The VA Mobile Enterprise App Store may also include links to apps produced outside the VA. These are apps a VA sponsor has recommended as useful. Listed Third Party Apps do not connect to the VA network and VA does not ensure that these apps are safe or secure.

VA sponsors can suggest an app for consideration through the VA Mobile App Intake (VA Network access required).

00

Pathways to Publishing

OCC App Publishing Options

Connected Care provides several places to publish an app depending on the audience or security restrictions. Connected Care manages the VA App Store, the Launchpad apps, companion apps to the VA App Store, and the Patient Video Tablets (PVTs). The request will be reviewed, if additional information is needed, you will be contacted.

VA App Store

Connected Care manages the VA App Store. Current app intake allows for access to 40-plus apps specifically created for Veterans and their providers. The VA App Store is a "one-stop shop" for Veterans and VA providers to find VA apps created jsut for them, canceling the need to go to commercial app stores. 

Each app on the VA  App Store has a content detail page that includes an app description, instructions for signing in (if needed), screenshots, training materials, and a way to provide feedback to VA.

Connected Care PMs can submit an app for consideration through the VA Mobile App Intake page (VA Network access required).

Patient Video Tablets (PVTs)

Connected Care manages the Patient Video Tablets (PVTs),a locked-down internal tablet for Veterans. Any app — including third-party or non-VA, publicly available apps —  requested to be installed on PVTs requires approval by the Tablet Workgroup and final sign-off by Dr. Heyworth. Once approved, these apps will be available for install on PVTs either by auto-install or on-demand. When intaking apps for PVTs, the scope of apps presented to Veterans' PVTs is determined, i.e. all PVTs (eg. VVC), specific PVT-identified use cases, or individually approved PVTs.

Publishing Apps Outside of OCC

If you wish to publish your app on commercial app stores, VA provides a VA App Catalog as well as a process that considers the audience and security restrictions of the app. The apps on the commercial app stores are considered informational or educational apps because they do not require a VA network connection nor require interfacing with any VA network systems or databases. Informational or educational apps do not provide personal information or private health information about the user.  

VA App Catalog

This app intake process is outside of OCC. This process is managed and supported by the Mobile & Endpoint Engineering Team. The VA approved apps that reside within the VA App Catalog, include both the use of interaction with VA Data as well as non-PII/PHI Interaction of VA Data. To request an application to be added within the VA App Catalog, a security assessment of the requested applications will be required. These applications are only available for Enterprise Mobile Management (EMM) Enrolled GFE mobile devices.

ESE Mobile App Intake (VA Network access required).

Google Play

The app intake process for Google Play is outside conducted of Connected Care. Read through and follow the directions at https://wiki.mobilehealth.va.gov/display/PDFMA/Google+Play+Store+SOP. Please note that all required items and all first-time items mentioned in each SOP are required to get the app into Google Play. If any of the required items are not provided, then Google Play will not make the app available for testing or production use.

Apple's App Store

The app intake process for the Apple App Store is outside of Connected Care. Read through and follow the directions at https://wiki.mobilehealth.va.gov/display/PDFMA/Apple+App+Store+SOP. Please note that all required items and all first-time items mentioned in each SOP are required in order to get the app initially into Apple's App Store. If any of the required items are not provided, then Apple's App Store will not make the app available for testing or production use.

About Federal Web Data Privacy Protection

All U.S. Federal web pages that you view, or interact with, in a browser must comply with a set of general rules to protect your private data. These include all pages in the .gov domain, including web sites like this one on mobile.va.gov, and the VA Web apps (apps that run on any device in a browser), and VA Mobile apps that run on Apple or Android devices, listed in the VA App Store. These rules include:

• Rules under the Privacy Act rights
• Information collected and stored automatically
• Use of cookies and tracking technologies
• Registration and log in
• Password protection
• Saving of passwords by browser
• Logging out
• Information sharing
• Digital analytics

You can read more about how the VA Privacy Services addresses these issues, here: www.va.gov/privacy-policy.

About Google Analytics

Some Federal agencies may participate in the Digital Analytics Program (DAP). As part of this program, Federal web sites may analyze their web traffic with Google Analytics 360 tools. DAP forbids passing any personally identifiable information -- including health information -- to Google Analytics. Information is "anonymized" (scrambled) before Google Analytics receives it. Users can receive recommendations related to personal privacy by accessing the Google Privacy Checkup. Google Analytics also has an opt-out tool to prevent sharing of information from web browsers.

About VA Apps Privacy Protection

In addition to the general federal rules listed above, The VA's Privacy Service ensures that all VA online applications listed on the VA App Store comply with rules specific to web or mobile apps. These rules include:

• Passing data to or from the VA, like health records.
• Terms protecting private data are included in the End User License Agreement (EULA)
• How a device may store private data

You can read an overview of these topics from an app developer's perspective, here: Privacy and Application Data Security.

Protecting Personal Health Information

The Veterans Health Administration (VHA) also ensures that the apps developed by the Office of Connected Care OCC further protect your personal health information. Apps with the padlock symbol on their app logos protect any online exchange by requiring they must occur over a secure connection within the VA network. Veterans and their designated Caregivers typically need a DS-Logon, ID.ME or My HealtheVet subscription to satisfy this requirement. VA Providers and Staff must be logged into the VA network via their Personal Identification Verification (PIV) card to review any user data.

About Third Party Apps Privacy Protection

The VA App Store includes apps from three sources: Veterans Health Administration (VHA) Office of Connected Care (OCC) apps, VA-Sponsored apps, and suggested Third Party Apps. (See OCC App Ownership for more information about these app types. Third Party apps are only recommendations: VA does not test them.) The VA Privacy Services office tests and certifies that OCC apps and VA-Sponsored apps have passed its rules to ensure that the apps cannot reveal private information to unapproved parties.