How is an indicator developed?

Indicator development cycle

CIHI indicators are developed in 4 stages after an information need or knowledge gap has been identified. Following is an overview of each stage and some of the activities included.

Source
Canadian Institute for Health Information. CIHI’s Indicator Development Cycle and Indicator Lifecycle Toolkit [unpublished document]. 2023. 
 

Stage 1: Initiate/evaluate

An idea undergoes initial investigations and scoping, involving

• Literature reviews and environmental scans to understand what currently exists about the identified gap 
• Consultation with potential experts and users to clarify what needs to be measured, why and how often

Stage 2: Develop

We develop and iteratively refine the indicator’s methodology based on input from experts who have a variety of perspectives. This is an essential stage that ensures we develop reliable and comparable indicators that support sound decision-making. Activities during this stage include

• Developing the indicator definition
• Establishing the inclusion/exclusion criteria for case and population selection
• Assessing/confirming the quality of the data
• Seeking expert advisory input (including clinical, research, policy and user perspectives)
• Developing a method to risk-adjust the indicator to make it comparable
• Sharing the overall methodology and preliminary results with experts and stakeholders, including conducting validation exercises with interested parties
• Refining the methodology based on expert and stakeholder feedback after review of preliminary results and (possibly) validation exercises

We also conduct in-depth analyses, including those focused on data quality assessments, to gain a fuller understanding of results (e.g., differences across age groups, males versus females and rural versus urban; outlier anomalies; large percentage changes across data years).

Stage 3: Calculate

This is the number-crunching stage once the methodology has been refined and finalized. We apply the methodology to the full scope of data to be included and prepare the results for release according to the agreed-upon specification and in the agreed-upon format (e.g., data tables, visuals). To ensure that no errors are introduced, we develop standard and repeatable approaches to calculations and complete specific checks (e.g., double runs of results, external auditing of calculations).

Stage 4: Release

Indicator results are released according to the agreed-upon specifications, either privately or publicly. For public release, a preview period in which stakeholders can review their results a final time before they are made public is often used to help stakeholders prepare. During this stage, we conduct final rounds of engagement to gain support for the release, which could include outreach to indicator-relevant stakeholders, system leaders and advocacy groups. Ensuring there are no surprises is CIHI’s goal when releasing new information. An embargo period precedes a public release to further ensure that stakeholders are prepared.

After we have released a new indicator, we return to Stage 1 and evaluate it, beginning the development cycle again. Evaluating an indicator follows a similar process to initiating a new one: we re-confirm what needs to be measured, why and how often. We evaluate our indicators on a regular basis to determine what we should keep reporting, what we should redevelop and what we should retire.