Context of blog – European Payer perspective
While developing this blog, I realized how important it is to clarify the perspective with which you review a certain question. My perspective, even though initially not conscious, is how oncology real world evidence can be used by payers and HTA in Europe to support decision making. From this perspective, the difference in the quality of the survival data is a significant issue.
Trying to address the question from an US regulatory perspective, where seemingly there has been most considerations about using oncology RWE, the below referenced papers show a very valuable methodological development enabling the use of the data for this purpose.
Importance of quality data
In this blog we hope to trigger a discussion about the importance of the quality of the data and we are looking forward to having your valuable thoughts and inputs on the same. Please join our LinkedIn group to participate in the discussion.
When evaluating the evidence obtained from real world data, it is obviously crucial to understand the quality of the data used for the specific purpose. The first step in the process is understanding and classifying critical data that is valid for a particular research question including relevant confounders and modifiers. This blog-post focusses on the comparison of question of mortality and cause of death.
Cause of death might be an important variable
Death is very crucial variable in the assessment of mortality benefits, for which the quality surprisingly varies from one dataset to another.
Flatiron data-set
In the publication “Development and Validation of a High-Quality Composite Real-World Mortality Endpoint” from 2018, a method to improve the sensitivity of the mortality information has been shown. It improves from 66% in the original data EHR – data-set (flatiron data-set) to:
- Sensitivity of 91 % in the sub-group of advanced non-small-cell lung cancer
- Sensitivity of 85-88 % in the sub-groups for advanced melanoma, metastatic colorectal cancer, and metastatic breast cancer
It is stated in the publication that, it is assumed that the comparative data set, National Death Index (NDI), has 100 % completeness. We tried to establish clarity about the quality of the golden standard, National Death index (NDI) used as a comparison in this trial and had difficulties to find data, but found values in the range of 91-93%. (See below list of publications we reviewed.)
The way the sensitivity is reported is different in the above publication.. Hence, we believe it would be fair to say that the completeness of the above-mentioned publication by Curtis et al would rather be 87% than the 91% as stated. Please correct us if this is an inappropriate extrapolation.
In a later publication, “An evaluation of the impact of missing deaths on overall survival analyses of advanced non–small cell lung cancer patients conducted in an electronic health records database” concludes that ‘the impact on survival analysis is minimal with high mortality sensitivity with only modest impact associated within external control arm applications.
Furthermore, the discussion also highlights the following:
- Imperfect data collection systems, workflows not designed to capture mortality data, and patients lost to follow up affects the completeness of the data
- Despite the minimal impact on most conclusions observed in a NSCLC (advanced Non Small Cell Lung Carcinoma, it is unclear how this will expand to other cancer types with longer mOS (Median of Survival)
If there would be a consideration to use this data for European decision makers, there appears to be important limitations in the ability to do survival follow up in several areas of cancer. Is this a fair observation or are we missing something?
Sweden
The Swedish cause of death registry, which was initiated 1749, and the data from statistics Sweden gives another level of quality in this variable.
- 100 % of deaths are reported within 30 days to Statistics Sweden.
- 96% of the individuals in the cause of death registry have a specific underlying cause of death recorded.
The ability to link the cause of death registry to any other data-set virtually provides a 100 % coverage.
Future blogs
In the future blogs we are going to continue discussing strengths and weaknesses of the datasets, including the issues of timing and accessibility of the Swedish data-sets.
Questions for debate:
- How important is the difference the quality of the data?
- Why is there so limited transparency about the quality of the data if we want decision makers (European payers / HTA) to use this?
- Why are these differences not discussed and highlighted?
- What is the quality of the mortality variable in other countries?
Flatiron publications:
- Carrigan et al., “An Evaluation of the Impact of Missing Deaths on Overall Survival Analyses of Advanced Non-Small Cell Lung Cancer Patients Conducted in an Electronic Health Records Database.”
- Curtis et al., “Development and Validation of a High-Quality Composite Real-World Mortality Endpoint.”
US death registry publications reviewed for the blog:
- Min-Woong, Elissa Oh, “Completeness and Accuracy of Death Dates and the Implications for Studying Disease Burdens: Focus on Alternative Data Sources”
- Lash and Silliman, “A Comparison of the National Death Index and Social Security Administration Databases to Ascertain Vital Status”
- Sohn et al., “Accuracy and Completeness of Mortality Data in the Department of Veterans Affairs”
- Wang, “Commentary in Response to Carrigan et Al”; Morales, McClellan, and Jacobs, “Empowering a Database with National Long-Term Data about Mortality”
- Wojcik, Huebner, and Jorgensen, “Strategies for Using the National Death Index and the Social Security Administration for Death Ascertainment in Large Occupational Cohort Mortality Studies”
- Williams, Demitrack, and Fries, “The Accuracy of the National Death Index When Personal Identifiers Other than Social Security Number Are Used.”
- Hermansen, Leitzmann, and Schatzkin, “The Impact on National Death Index Ascertainment of Limiting Submissions to Social Security Administration Death Master File Matches in Epidemiologic Studies of Mortality”
- Schisterman and Whitcomb, “Use of the Social Security Administration Death Master File for Ascertainment of Mortality Status.”
Swedish data:
- Brooke et al, “The Swedish Cause of Death Register.”
- Ludvigsson et al, “The Swedish Personal Identity Number.”