I wanted to learn more about techniques for developing and measuring indicators and chose to explore several techniques for a project to satisfy a requirement to earn my Master of Foresight degree. While searching for indicator methods, I came across the following thesis paper titled “How Should Indicators be Found for Scenario Monitoring?,” written by Zheng He in 2013. I was intrigued by the process of assigning each driver of change a value for the purpose of monitoring the change in the strength of drivers that contribute to particular scenarios. The author found scenario monitoring methods to be lacking and so created a hybrid technique through a synthesis of existing indicator methods.
The scenario used in which to find and measure indicators is featured in a video by Bloomberg titled “Here’s How Electric Cars Will Cause the Next Oil Crisis,” published in 2016. I call this scenario “Here Comes the Sun” as it describes the demand for oil quickly diminishing based on the interplay of cheap solar energy and an increase in electric vehicles.
This method combines the 10-step indicator method developed by OECD (Organization for Economic Cooperation and Development) in 2008 with data/frame theory of sensemaking developed by Klein, Moon, and Hoffman (2006). It monitors scenario development by using “composite indicators” that measure the strength of driving forces which contribute to particular scenarios in the starting year and compares the scores to the monitoring year to determine the change to each driving force. Measuring the change to each driver helps to better identify which scenario is unfolding. I found this to be a very valuable technique if you have the time resources to invest.
Step 1: Develop a Theoretical Framework
A theoretical framework needs to be developed in order for proper individual indicators to be chosen and for the data analysis to make sense. The composite indicator represents the driving force and will include the variables of dimension, individual indicators, and the objective of the individual indicators.
Step 2: Selecting Variables/Individual Indicators
Use STEEP categories to find potential indicators that describe the driver. Choose potential indicators based on relevance and availability. Then, choose individual indicators from potential indicators based on non-repetition, coverage, and data-availability.
Example – Driver #1: Electric Vehicle Demand
Composite Indicator: EV Demand
Dimension 1: EV Attractiveness. Objective: Maximize
Dimension 2: Internal Combustion Engine (ICE) Attractiveness. Objective: Minimize
Step 3: Imputation of Missing Data
If indicators have years with missing data, use mean/median/mode, trend regression, or simply ignore the missing data, which is not recommended but may become necessary in certain situations.
Step 4: Normalization of Data
Several methods could be used, but the author recommends normalizing data to the base year by dividing the monitoring year data by the base year.
Step 5: Multivariate Analysis
This step is helps to eliminate human bias when choosing indicators. There are several methods of multivariate analysis but the author chose factor analysis because it assigns a weight to each component of the composite indicator according to its contribution to the total data variance. (For this indicator method exploration, I did not use step 5 or weighting in 6 as they are an extra layer of statistical analysis which is not needed for my trial purposes.)
Step 6: Weighting and Aggregation
Weighting Method: The author suggests using rotated component matrix from the factor analysis.
Aggregation Method: Linear or non-linear can be used but linear is much easier and recommended.
Aggregated Composite Indicator Scores (Without Weighting)
Step 7: Scenario Monitoring
The final step is to use the composite indicator scores to measure the strength of the drivers to monitor proposed scenarios. The method is to choose boundary values for your final indicator scores. If the score is above this boundary then the driver strength would be assigned “high,” below the boundary assigns “low,” and anything in between remains “uncertain.”
In this example, all of the drivers contributing to the scenario have grown in strength from 2016 to 2019. This result indicates that the scenario is on track and more likely as of the first monitoring year. Normally, the monitoring dashboard would have multiple scenarios instead of just one.
Final Thoughts:
Despite the hard data, there still remains several elements of guesswork which can skew the output, particularly in determining boundary/threshold values and dimensions for the composite indicators. The more I tried to properly label and identify composite drivers and dimensions, the more I came to the realization that there is no perfect way for it to be accomplished. This entire process is very similar to the process of framing a domain. What seems reasonable during a first attempt usually ends up being edited many times over in order to cover all that needs to be covered without ending up with redundancies. The author does address this issue, but perhaps not to the level of importance to which it deserves. I think a majority of work performed with this process would be in revising the framework.
“…the first theoretical framework should be tested by the data. If the data are inconsistent with it, new framework should be proposed until no evidence is contradicted with it.”
All in all, I found this to be a very good method for developing and measuring composite indicators to measure the strength of driving forces in monitoring years.
Comments