What is an SD Speed test tool?
It is a tool designed to measure and compare VPN brand connection Speed with each other specially after connecting to OTTs such as Hulu, HBOMax, Netflix, Paramount Eg. Source/Destination and protocol are kept constant in each test. Test iterations are performed 4 times a day, results are also logged for trend analysis.
SD Speed Test Methodology
- For better projection of user experience, we are using the same method in SD speed test too that app uses to get the best server for connection. This allows us to see what Speeds users are getting with different VPNs.
- Service providers from different Sources and Destinations at different times.
- We dial all our connections via OpenVPN (UDP), Strongswan (IKEv2) and WireGuard at the moment.
- The number of Destinations and Sources that we use in our VPN speed test too are limited but kept constant for all VPN service providers alike. We are using Top 10 destinations only and each destination is being dialled via Top 3 sources only.
- The number of VPN services that we use for comparison are limited too. We’ve got IPVanish, ExpressVPN, NordVPN, Surf Shark and PureVPN in the list at the moment.
Speed Metrics used in Tool:
- Latency (ms): This metric is used to track and compare latency of different vendors from specific source(s) to specific destination(s). Lower latency means the transmission is faster, hence better speed.
- TCS(s): TCS is the Time To Connect To Server in seconds. This represents the Time b/w VPN Client connection initiation to Connection procedure completion as provided in logs of Respective Service/Daemon. We use this Metric to track and compare time taken to connect to the server. Lower the time, the better it is.
- UTC %: Unable to Connect %. Formula : (Connection Failed/Total Connection) X 100. We use this Metric to track and compare Failure% while attempting to connect to the server.
- Download1 (MBps): It represents the Download Speed of Public CDN that is in the same region as VPN Server. Higher the Download Speed, the better it is.
- Download2 (MBps): It represents the Download Speed of Public CDN that is not in the same region as VPN server. Again, higher the Download Speed, the better it is.
- Iperf Receiver (MBps): Iperf Reception (downlink speed) offered to client machines tested from Iperf Server that is located in the same region as VPN server.
- Iperf Sender (MBps): Iperf transmission (uplink speed) offered to client machines tested from Iperf Server that is located in the same region as VPN server.
- Degradation%: We use this metric to track Speed Degradation on Server after Connecting to VPN. The formula to calculate Speed Degradation is as follows: (SpeedBefore – SpeedAfter)/SpeedBefore.
Properties:
Download1, Download2 , Iperf Receiver & Iperf Sender carry similar attributes, details are given below:
- SpeedBefore: Data transmission Rate in MegaBits Per Second before connecting to VPN.
- SpeedAfter: Data transmission Rate in MegaBits Per Second after connecting to VPN.
Testing Procedure:
For a selected destination:
For VPN vendors:
- Create VPN dialling Config (grids for all) and connect VPN tunnel
- Take average of 3 X Download1 download speeds
- Take average of 3 X Download2 download speeds
- Perform Iperf Test (if fails retry 3 X after 30 Seconds)
- Connect VPN (if fails, try 3 Times (3 servers received via Speedtest API)
If VPN connection fails, Ignore Current Destination Tests for All remaining competitors. If Connection Succeeds, perform following tests without VPN tunnel:
- Take average of 3 X Download1 download speeds
- Take average of 3 X Download2 download speeds
- Perform Iperf Test (if fails try 3 X after 30 Seconds)
- Perform comparison of Speed for all competitors
- Post Results to database
Repeat procedure for other destinations
Speed Test Tool Clients:
CTE clients are VPS machines from different cloud providers in the following locations :
- United States – New york
- United States – Los Angeles
- United Kingdom – London
- Australia – Sydney
- Australia – Melbourne
- France – Paris
- Romania – Bucharest
- Germany – Frankfurt
- Canada – Vancouver
- Canada – Toronto
- Canada – Quebec
- Netherlands – Amsterdam
- Singapore – Singapore
- Russia – Moscow
- Sweden – stockholm
- Italy – Rome
VPN Destinations:
Following VPN server destinations are tested in CTE:
- Netherlands
- United Kingdom
- India
- Australia
- United States
- Germany
- Russia
- Brunei Darussalam
- France
- Singapore
- Canada
VPN Speed Test Tool Execution Architecture:
All tests are executed via python script executed in a docker container.
VPN Speed Test Enhancements:
- Enable CTE to start testing Speed via App.
Why?
Better projection of user experience.
What?
4 hours iteration/day
4-8 Competitors
Testing can be enabled via 3 sources without comprising the existing CTE system.
Simple image for reference:
