The software and data that are used to put together the Compass are open-source and in the process of becoming Digital Public Goods. The script and datasets can all be found on UNDP’s GitHub , developed in partnership with GitHub’s volunteer programme.

1. Indicator selection

UNDP conducted a review of the 200+ indicators that formed the first version of the Digital Development Compass. This indicator evaluation was carried out in two phases:

• Phase 1: Existing indicators were reviewed according to newly established indicator criteria. A number of composite indexes that were included earlier in the Compass were also exploded to remove instances of double-counting and improve relevancy. The term “exploded” is used, in the sense of ‘to break up into pieces’, to refer to the process of surfacing all of the individual indicators contained within a composite index.
• Phase 2: New indicators were added after the research and review process.

Currently, there are a 145 indicators included in the Compass. Indicators were identified by conducting online desk research into the public data available related to the sub-pillars and pillars of the Digital Development Framework mentioned above.

During this review process, an Expert Committee was appointed, comprising a globally representative group of experts with specialized experience in digital development. The Committee’s vision is to support the Digital Development Compass as it develops year on year. At its core, the Committee will remain flexible, evolving organically and in parallel with the Compass.

Based on a series of advisory consultations with the Expert Committee, questions for data evaluation have been created to assess data quality of all existing and newly selected indicators. In case the examined indicator does not meet the evaluation criteria, the indicator will be excluded from the Digital Development Compass.

Table 2. Data evaluation questions

2. Data collection

The methodology used for data collection depends on the format the data are published in. Largely, data are retrieved from the data source automatically. Links to data sources are collected in a spreadsheet. Then, automation is used to scrape spreadsheets, PDFs and documents and convert them into a machine-readable format.

However, while automation can be applied to most websites from which data are collected, certain platforms require manual intervention. In these cases, files containing data in the correct format are uploaded manually.

99 percent of all indicators are automatically updated when organisations release new reports and datasets. The below diagram visualises how updates in the source data triggers a change in the Compass webpage.

Figure 4. Source

When any of the links to source data expire, an automated email notification is sent to administrators, and new links are updated manually.

3. Score calculation methodology

The Digital Development Compass aims to put countries into one of the five maturity levels for each of the sub-pillars. This section includes the approach to normalization, weighting indicators and imputation of missing scores.

Normalisation

The indicators used within the compass have been normalized using Min-Max rescaling.

Min-Max rescaling

The following formula creates a score between 0 and 1. It does so by creating a scale with a maximum possible score of 1 that corresponds to the highest scoring country’s score; and a minimum possible score of 0 that corresponds to the lowest scoring country’s score.

Scaling and normalization of categorical data

The categories are identified in the dataset and numerical values are assigned to each category. Each category has a unique numerical representation. The maximum possible value that a category can attain within the dataset is determined to establish the scaling range. The maximum value provides a reference point for rescaling all other values. Once the maximum possible value is determined, the numerical values of the categories are rescaled to a predefined range, typically spanning from 0 to a chosen maximum value. This rescaling process ensures uniformity and facilitates comparison across different indicators.

The original numerical values of the categories to the desired rescaled range are linearly mapped. The mapping maintains the relative order and spacing of the original values.

For each category, its rescaled value is calculated using the determined maximum possible value and the chosen scaling range. The rescaling formula is consistently applied across all categories.

The rescaled values are normalized to ensure they fall within a standard range, such as 0 to 1. Normalization helps mitigate the influence of outliers and enhances the interpretability of the data. Min-Max normalization is used for normalization of these data types as well.

Indicator weightings

In the Compass, equal weightings are assigned to all pillars and sub-pillars.

Weighting at the indicator level is made in the following way:

1. Indicator specificity: Each indicator is assigned a score of 1 or 0.5 based on its relationship with digital development:
   • The score 0.5 is assigned to indicators identified as more weakly aligned and which only partially relate to digital development.
   • The score 1 is assigned to indicators that have a clear, direct connection to digital development.
   • A score of 0 is not included, as the data evaluation process already removed indicators that do not clearly relate to digital development (and which would have scored 0).
2. Indicator uniqueness or overlap: Although the Compass does not have any instances of the same indicator appearing more than one time (explicit double counting), in some cases, multiple indicators are measured with overlapping factors. For instance, ‘average fixed broadband download speeds’ and ‘average mobile broadband upload speeds’ are highly likely to be correlated, being dependent on the same infrastructure. These indicators needed to be scored lower to avoid counting the same metric more than once.
   • To quantitatively capture overlapping, indicators are scored by dividing 1 by the number of overlapping indicators.
   • In this way, an indicator that does not overlap with others scores 1; an indicator that overlaps with only one other indicator scores 1/2 = 0.5; an indicator that overlaps with two other indicators scores 1/3 = 0.33, and so on.
   • This process allows us to assign a lower score to those indicators that measure similar aspects of digital development.
3. Indicator number: Indicator scores are divided by the total number of indicator scores within a sub-pillar.

Imputing scores

For indicators where there are gaps in country data, or there is no collected data from a particular country, imputing scores is an option. While imputed data is not included into the scoring in the Digital Development Compass, users can still visualise the data with or without imputed scores using a toggle switch.

The following imputation methods are selected depending on their suitability within a given context:

1. Linear interpolation: Historical information can be used before implementing a modelling-based approach. If data are missing but there are values in preceding and subsequent years, linear interpolation is used.
2. Extrapolation: If there is no data from preceding and subsequent years, data are extrapolated as a constant value of the nearest reported data, or as a value on a trend line/curve.
3. Neighbouring/similar countries’ data: A mean average is taken based on a sample of regionally neighbouring countries and/or GDP figures.
4. Expectation-maximisation with a bootstrapping (EMB): The remaining missing data are imputed with an expectation-maximisation with a bootstrapping (EMB) multiple imputation algorithm.

The EMB method generates imputed values through an iterative procedure that uses other variables to impute a value (expectation) and then asserts whether the value is most likely to fit the data (maximisation). To account for variation caused by missing data, the model is run 10 times. The average of these 10 imputations is then used to impute the missing value.

Definitions and scales

Data that has been retrieved are then matched with a UN-defined list of countries, regions, sub-regions, income groups and territorial borders.

Raw data and data from other indexes are converted to values within the Compass’ 0 - 5 range. The method used to perform this conversion depends on the type of indicator. Through rescaling all scores between 0 and 5 (no zero scores), the countries are clearly within one of the five maturity stages of digital development.