VPN providers often point to a penetration test or security assessment as evidence of quality. A summary that says “tested by an independent firm” is a claim about an event, and what that event covered is a separate question. The UK’s National Cyber Security Centre (NCSC) publishes guidance on penetration testing that explains what these tests are designed to do and where their limits lie. The guidance is written for organisations commissioning tests, not for consumers comparing VPNs, so this guide applies its points by analogy and labels them as such. It is general information and assesses no provider.
What the NCSC says a penetration test is
The NCSC guidance, first published on 8 August 2017 and reviewed on 10 January 2022, opens by calling penetration testing a core tool for analysing the security of IT systems that is “not a magic bullet”. It defines a penetration test as a method for gaining assurance in the security of an IT system by attempting to breach some or all of the system’s security, using the same tools and techniques as an adversary might.
It adds that a test should be viewed as a way of gaining assurance in an organisation’s own vulnerability assessment and management processes, not as the primary way of identifying vulnerabilities, and compares it to an external financial audit that checks whether the internal team’s processes are sufficient.
What a test can tell you
According to the NCSC, a well-scoped test can give confidence that the products and security controls tested have been configured in accordance with good practice and that there are no common or publicly known vulnerabilities in the tested components, at the time of the test. A test report is expected to include the issues found, a risk assessment for each, and a way to resolve them. Findings are normally given a severity rating, and any departure from the standard rating should be documented and justified.
What a test cannot tell you
The same guidance sets out several limits that matter when reading a published summary:
- A snapshot in time. A test can validate only that systems are not vulnerable to known issues on the day of the test. The NCSC notes that a year or more may pass between tests, so problems can exist for long periods without being detected.
- Dependence on scope. The scoping stage fixes the technical boundaries, the types of test, and the time and effort allowed. If testers find components outside the scope that affect security, the exclusion may be recorded as a limitation on testing.
- Dependence on the testers. Tests cannot be entirely procedural, so the quality of the work is closely linked to the abilities of the people doing it.
- Different testing bases. Testers may be given full information about the target (open box) or none (closed box). Closed-box work models an outside attacker, but the NCSC notes that limited information and time can leave vulnerabilities undiscovered.
- Not suited to every target. The NCSC describes penetration testing as appropriate for a specific operational system made up of products and services, and says it is not an appropriate technique for product-specific testing.
The last point is relevant to VPN apps, which are products. A published VPN assessment may be a different kind of review, such as a code or configuration review, so the report’s own description of its method matters more than the label “penetration test”.
Reading a provider’s published summary
Using the NCSC’s framework as a checklist, these are the questions a summary should answer:
- Who and when? Which firm performed the work, who commissioned and paid for it, and on what dates?
- What was in scope? Which apps, versions, servers and back-end systems were included, and which were excluded?
- What method? Open-box, closed-box, code review, or something else?
- What was found? A report that lists issues with severity ratings tells a reader more than a statement that none were significant.
- What happened next? Were the findings fixed, and was that confirmed by retesting? The NCSC says the organisation, not the test team, owns risk decisions and fixes.
- How current is it? Does the tested version match what is being sold today?
The guides on logging policies and independent audits and on the 2017 OpenVPN audits show how age and scope affect what a published audit can support. A related distinction, between stated policy and verified practice, runs through the guide to warrant canaries and transparency reports.
Common questions
Does a clean report mean the service is secure?
No. Under the NCSC’s description, a test shows the state of the tested components against known issues at a point in time, within an agreed scope.
Is an independent test the same as an audit of a no-logs policy?
Not necessarily. A security test looks at vulnerabilities, while a logging audit looks at data practices. Each has its own scope, and one does not substitute for the other.
The bottom line
The NCSC describes penetration testing as a useful way to gain assurance about a defined system at a point in time, not a guarantee of security and not a primary method for finding vulnerabilities. When a VPN provider cites a test, the claim becomes meaningful only to the extent that the scope, method, dates, findings and follow-up are published. A short statement that a test took place is the provider’s claim; the full report is the evidence, and even that evidence is a snapshot.
Sources
- Penetration testing: how to get the most from penetration testing (ncsc.gov.uk, published 8 August 2017, reviewed 10 January 2022)