Skip to main content

Indicators of guideline-concordant care in lung cancer defined with a modified Delphi method and piloted in a cohort of over 5,800 cases



To identify indicators of guideline-concordant care in lung cancer, to implement such indicators with cancer registry data linked to health databases, and to pilot them in a cohort of patients from the cancer registry of the Milan Province.


Thirty-four indicators were selected by revision of main guidelines by cancer epidemiologists, and then evaluated by a multidisciplinary panel of clinicians involved in lung cancer care and working on the pathway of lung cancer diagnosis and treatment in the Lombardy region, Italy. With a modified Delphi method, they assessed for each indicator the content validity as a quality measure of the care pathway, the degree of modifiability from the health professional, and the relevance to the health professional. Feasibility was assessed using the cancer registry and the routine health records of the Lombardy region. Feasible indicators were then calculated in the cohort of lung cancer patients diagnosed in 2007–2012 derived from the cancer registry of the Milan Province. Criterion validity was assessed reviewing clinical records of a random sample of 114 patients (threshold for acceptable discordance ≤20%). Finally, reliability was evaluated at the provider level.


Initially, 34 indicators were proposed for evaluation in the first Delphi round. Of the finally 22 selected indicators, 3 were not feasible because the required information was actually not available. The remaining 19 were calculated on the pilot cohort. After assessment of criterion validity (3 eliminated), 16 indicators were retained in the final set and evaluated for reliability.


The developed and piloted set of indicators is now available to implement and monitor, over time, quality initiatives for lung cancer care in the studied health system.

Peer Review reports


Lung cancer remains a killer disease being the leading cause of cancer death for both genders in countries with a high Human Development Index (HDI), with world age-standardized mortality rates of 36.4 per 100,000 for males and 14.6 per 100,000 for females, accounting for approximately 22% of all cancer deaths in 2018 [1]. This despite the discontinuation in smoking habit occurred in high HDI countries that led to a reduction of incidence rates [2].

There is accumulating evidence that patients not receiving guideline concordant treatments have worse outcomes [3, 4]. Hence, it is important to define and implement a set of indicators to verify concordance between guidelines and care delivered in clinical practice. Previous studies defining indicators to evaluate guideline-concordance in lung cancer care have been carried out in The Netherlands [5, 6] and Ontario [7]. However, to serve the scope of quality of care improvement, the whole process from definition, to operationalization and pilot testing needs to be carried out in a specific health service [8, 9]. Also, indicators have to be timely, reliable and economically computable on a large scale to inform quality improvement activities [10, 11]. In the oncologic setting, this is impractical if indicators are retrospectively computed using data abstracted ad hoc from clinical records, while it is expected to be feasible using current health database linked to modern population cancer registers [12,13,14].

The aim of this project was to develop, with a multidisciplinary panel, a set of adherence to guidelines indicators concerning aspect of diagnosis, treatment and follow-up of lung cancer care and to pilot them in a cohort of lung cancer patients from the Cancer Registry of the Milan Province, a territory covered by the Agency for Health Protection of Milan, Italy.


Identification and selection of the indicators

We first identified relevant evidence-based practice guidelines and potential indicators from the literature and then, with a modified Delphi method, used clinical experts’ knowledge to select the final set of indicators in two rounds. The whole process is summarized in Supplementary Figure 1. On 07 May 2015 we interrogated the SAGE clinical guidelines database [15] (Supplementary Material), that collects and evaluates according to the scheme AGREE II all published guidelines in English [16]. From the retrieved list, we selected guidelines with a score ≥ 50% for the ‘rigour’ and ≥ 30% for the ‘applicability’ domain. Additionally, we used the 2013 Lung Cancer Guidelines of the Italian Association of Medical Oncology (AIOM), which are published in Italian language [17]. Those guidelines are developed according to the Scottish Intercollegiate Guidelines Network (SIGN) and Grading of Recommendations Assessment, Development and Evaluation (GRADE) methodologies [18, 19]. Systematic evaluation and selection of recommendations from the identified guidelines was carried out by one of the authors, and reviewed by a senior author. We included all recommendations whose compliance could be evaluated by process or short term outcomes indicators, and that could be theoretically calculated with data available from the cancer register and current health databases of the Lombardy Region. We excluded all recommendations that required data unavailable in the health databases to be verified (e.g. results of patient performance status assessment, advices given to patients). To identify previously developed indicators for lung cancer care evaluation, we searched the Pubmed database with no language and date restrictions on 20 April 2015 (Supplementary Material – Search strategy). We also searched the National Quality Measures Clearinghouse database on 05 May 2015 [20]. After screening on title and abstract, full-text of the relevant publications were examined. One of the author extracted all the indicators and matched them with one of the selected recommendation. If more than one indicator matched with a recommendation, we combined them into a single indicator. If there were minor differences in time windows or definition of the denominator, we maintained the more suitable for the database and health system under scrutiny. If a recommendation had no corresponding published indicator, a new one was proposed. This process was reviewed by other authors. The provisional list of indicators was then specified unambiguously and exhaustively in term of numerator, denominator, and data sources, and then organized in a questionnaire using the freely available survey tool eSurv ( The survey consisted of five sections evaluating the following dimensions: organization, diagnosis, surgery, medical treatment and follow-up. Each indicator was presented with a title and a description of how the indicator would have been calculated from the register and the administrative data. Below it was asked to rate, using a seven-point Likert scale, the validity of the indicator as a quality of care measure, the possibility to modify the value of the indicator at the patient level by the health professional, and the usefulness of the indicator for self-assessment. Definitions provided to the panel are provided in English translation in the Supplementary Material. A free text field allowed to suggest changes to the indicator wording or to propose an alternative indicator. Panel designation is described in Supplementary Material. Participants were chosen on agreement to participate basis. Results of the first round were evaluated, both in terms of obtained scores and comments expressed in the free-text field. An indicator was retained if at least 75% of the panel members completing the questionnaire scored the validity question equal or greater than five, and at least 50% of them scored the modifiability and utility question equal or greater than four. The reduced list of indicators was also updated according to relevant comments in the free-text field. A summary of the results of the first round was sent to the clinicians who completed the questionnaire, including rates and proposed modifications. The second round was conducted during an in-person meeting, run as an informal discussion, with representatives of the panel.

Identification of the cohort and pilot calculation of the indicators

We calculated all the finally selected indicators on a pilot cohort derived from Cancer Registry of the Agency for Health Protection of Milan, Italy including all patients developing a lung cancer (International Classification of Diseases for Oncology version 3, first revision (ICDO-3) topographic codes C33–34) in the period 2007–2012 and registered with the Lombardy Regional Health Service. A subset of the area covered by the Agency was used, corresponding to fourteen municipalities around Milan with a total population of 1.546.237 inhabitants at 01/01/2013 [21]. The cancer register is accredited at the national level and included in the Volume XI of Cancer Incidence in Five Continents [22]. It is semi-automated, using multiple sources of information (i.e. inpatient, histopathology and death certificate databases) and a record linkage algorithm to match all information at the individual level [23] with the review of all cases for the assignment of morphology and stage. We calculated a modified version of the Charlson comorbidity index using both in and outpatient databases [24, 25]. The date of incidence was defined, according to international cancer registration rules [26]. Exclusion criteria were: tumors identified only through death certificate and mesenchymal histology. For indicators applying to either small cell (SCLC) or non-small cell lung cancer (NSCLC), patients were classified as SCLC if the histology ICD-O-3 code was 8041–8045 and NSCLC in all other cases, including nonspecific histology codes and not cyto-histologically confirmed cases. To calculate the indicators at patient level we used the register and all available computerized sources of health information from January 2006 to December 2016, including outpatient diagnostic and therapeutic procedures, inpatient, prescription, and emergency access databases. We derived gender, age and stage at diagnosis from the register. First therapy was defined as described in Supplementary material. For each indicator, we assigned the patient to the hospital where he/she received the relevant procedure for the first time. We assessed feasibility of each indicator verifying that the data needed to calculate the indicators were actually available for the whole population in the administrative data. We assessed construct validity in a stratified (main histological type, stage, type of first treatment) random subsample of 114 patients, verifying that the required coding was reported in the administrative databases when the procedure had actually been performed according to the full clinical record, and reported number and percentage of discordances. The threshold of acceptable discordances to retain the indicator was set at equal or lower than 20%. We first computed the indicators overall, and in relevant subgroups of patients, and the assessed their clinimetric properties. Potential improvement was evaluated analyzing median value. We also computed number of providers scoring either 0 (flooring effect) or 100% (ceiling effect). Finally, we evaluated the reliability of the indicators, that is a measure that can vary between 0 and 1 and describes how well one can confidently distinguish the performance of one provider from another measuring the signal (variability of the indicator that can be attributed to real differences in performance) to noise (random variability of the indicator value) ratio [27]. We calculated the reliability of each indicator for all providers, transformed it to percentage to interpret as the percentage of variability that can be attributed to real difference in performances, and reported the number of providers with a reliability equal or greater than 70%, which is an accepted cut-off for good reliability [28, 29].

Statistical analysis

Descriptive statistics including quartiles were calculated for the validity, modifiability and utility scores received by each indicator in the first Delphi round. Differences in the distribution of covariates across gender, stage or years were assessed using the Manthel-Haenszel test, analysis of variance or Wilcoxon test, as appropriate. All tests were two-sided with a significance level of 0.05. We calculated each indicator as the proportion of patients who received the procedure in the defined time window among those defined as eligible. Indicators were calculated also stratified by age (≤60, 61–70, ≥70 years), sex, Charlson index (0, 1–2, ≥3) and stage (I to IV). To assess reliability, we measured noise as the variance of a proportion, and signal using the variance of the random effect of a hierarchical logistic model with the indicator value as the outcome, age, gender, stage and Charlson index as first level predictors, and provider as the only second level variable. All analyses were performed with SAS software (v.9.4, SAS Institute, Cary NC).


Review of the literature and identification of candidate indicators

The search of the SAGE clinical guidelines database retrieved 45 records (Supplementary Figure 1). Ten guidelines did not include NSCLC or SCLC, 11 had no AGREE II evaluation, and 18 guidelines had a rigour < 50% or an applicability < 30%. The six remaining guidelines were used to extract recommendations [30,31,32,33,34,35] .The literature search for lung cancer indicators retrieved 557 records. Based on title and/or abstract, n = 532 were excluded as not relevant and n = 8 because they were guidelines already included in the SAGE database search results. Three manuscripts were excluded on the basis of the full-text evaluation [36,37,38]. Indicators were extracted from the 14 remaining papers [7, 9, 39,40,41] and matched with a guidance. A set of n = 34 indicators, each assessing a recommendation, was included in the web-based questionnaire for the first round of the Delphi process.

Results of the Delphi process

A total of 225 physicians were invited to participate to the Delphi process. The number of questionnaires compiled at least partially was 95 (42% of respondents); 85 questionnaires were completed in full. Thirteen indicators did not satisfy the selection criteria and were excluded from the second round. One additional indicator evaluating time from thorax computed tomography (CT) to surgery was proposed by panel members (O4). Four indicators were partially modified based on the comments received in round 1. The meeting constituting the second round led to approval of the changes in the indicator formulation but did not modify the number of indicators. Table 1 reports the finally 22 selected indicators with first round scores.

Table 1 Indicators (No. 22) to measure guideline-concordant care in lung cancer patients selected with a modified Delphi process

Pilot cohort

In the 2007–2012 period there were 5860 cases of lung cancer (International Statistical Classification of Diseases and Related Health Problems 10th Revision (ICD10) topographic codes C33–34) in the resident population of the Milan province. Death certificate only (n = 88) and mesenchymal histology (n = 26) cases were excluded. This left n = 5746 patients, 91% NSCLC and 9% SCLC (Supplementary figure 2). Seventy-five percent were males and 55% of cases were in the 66–80 years range (Table 2). A significant increase in the proportion of female was observed over time from 24% in 2007 to 28% in 2012 (p = 0.009), while there were no significant changes in age distribution (p = 0.8). Overall 43% of patients had a Charlson comorbidity index greater than zero, with males having more comorbidities (mean Charlson index 1.5 for male vs. 1.2 for females, p < 0.001). No differences in Charlson index distribution were observed with stage (p = 0.5).

Table 2 Lung cancer patient cohort in the Milan province, Italy (n = 5746 subjects)

Clinimetric properties of the indicators

Indicators were calculated on the pilot cohort and their values are reported in Table 3. Regarding feasibility, the indicator ‘palliative care before death’ (M3) was actually not feasible because the required administrative datasets were not available for the entire population, i.e. available for some municipalities but not others. The same was true for ‘Multidisciplinary evaluation’ (O3), as the code was used only from 2 out of 92 providers in the analyzed years, and ‘Functional evaluation before surgery’ (S3) as the required exams were systematically never billed in the pre-admission consultation to surgery.

Table 3 Value of the indicators to measure guideline-concordant care in lung cancer patients of the 2007–12 pilot cohort of the Milan province (n = 5746 subjects), overall and in subgroups of patients

Three of the indicators did not meet the criterion validity threshold (i.e. 20% of discordances) and were eliminated (Table 4): ‘Treatment with curative intent preceded by PET’ (D3), ‘SCLC patients fully staged’ (D6), and ‘Patients with a thorax CT ≤30 days before surgery’ (S2). The indicator ‘SCLC patients undergoing medical oncologic therapy or radiotherapy’ (M2) did not have a high reliability (≥ 70%) in none of the 56 evaluated providers (Table 5). Also, the indicators O1, O4, D1 and S5 had a high reliability in less than 50% of the evaluated providers.

Table 4 Summary of the evaluation of construct validity, through clinical records evaluation, of a set of indicators to measure guideline-concordant care in lung cancer patients in a subsample of 114 patients from a pilot population of 5746 subjects
Table 5 Summary of the evaluation of reliability, ceiling effect assessment and distribution across providers of a set of indicators to measure guideline-concordant care in lung cancer patients in a pilot population of 5746 subjects

Calculation of the organizational indicators showed that the proportion of patients having rapid access to treatment after first contact or diagnostic procedures did not differ between men and women and was higher in younger, healthier and more advanced stage patients (O1-O4, Table 3). Half of the hospitals had values of the indicator ‘First contact to first therapy ≤60 days’ (O1, Table 4) equal or lower that 68%. Only 4% of stage II-III NSCLC patients received chemoradiation (M1). ‘Pain management before death’ (M4) had an overall value of 92% but a low improvement potential (median value across providers 98%). Overall, 72% of patients surviving to the end of the second year had a follow-up contact (F1).


This article describes the development, based on the evidence based guidelines and expert consensus, of a set of indicators aimed at monitoring concordance of lung cancer therapy to guidelines and its piloting on 5746 lung cancer patients diagnosed over a 6-year period.

One of the major strengths of this work is the participation of a high number of clinicians involved in the different steps of lung cancer care, guaranteeing that the identified reference guidelines were endorsed and perceived as relevant. Secondly, identifying the cohort with a cancer register allows to adequately define the denominators of the indicators, solving the problem of accurately identify the target population that has been encountered using administrative records only [42]. Also, lung cancer guidelines differ by main histological type (SCLC vs. NSCLC) and by tumor stage. The definition of the cohort from the cancer register also permits to reliably identify the histology.

Lung tumors still have a poor overall prognosis, which is certainly related to the often advanced stage at diagnosis [43]. However, there is evidence that care not concordant with evidence-based guidelines worsen survival and that actual delivered care is often not adherent to those guidelines, especially for advanced stages and SCLC [3, 4, 6, 44, 45]. The report from Nadpara et al., examining a large cohort of lung cancer cases from the Surveillance, Epidemiology, and End Results register linked with Medicare database, investigated variations in guideline-concordant care among the elderly, highlighting that a lower income, increasing age and being non-white reduced the probability to receive guideline-concordant care [4]. The results of our study also highlight differences in adherence to standards with age: older patients received curative surgery less frequently and this was independent from the comorbidity burden.

With the exception of S1 and S6, which are short term outcome-indicators, it was decided to focus on process indicators measuring adherence to guidelines. Process indicators have been criticized as they do not directly measure the outcome [46, 47], which is the main interest both in a patient-centered and public health perspective. A small set of outcome indicators has been developed also for lung cancer [48]. However, process indicators have the advantage to be more sensitive to quality of care differences, not heavily depending on external factors which are not directly under the control of care providers as it happens for outcome indicators [46]. Especially in a setting such not very early lung cancer, where prognosis is still dismal even when receiving the best standard of care, we think that process indicators are fundamental to assess heterogeneity between providers and implement quality improvement actions [49]. Of note, some of the indicators had a low reliability in a large percentage of the providers, suggesting caution in the interpretation of the value of indicators for those providers.

The use of the current administrative databases offers the possibility to implement the indicator on a population scale and to repeat the measurements in time at a reduced cost. However, it introduces some limitations. The first one is that we were not able to calculate three of the indicators produced by the Delphi consensus process because the investigated procedures were not coded in the necessary databases. We also had to drop three indicators having more than 20% discordances between reviewed clinical records and indicators calculated from the administrative databases. However, this is an important information to plan quality improvement initiatives for the future, raising the problem to consistently employ the codes with the clinicians involved in lung cancer care. Also, when using administrative data to calculate health indicators and define case-mix variables (e.g. comorbidity index), concerns about data quality are always present [25, 50, 51], particularly with regard to lack of accuracy in coding and coding variability across professionals. However, they are the larger, more systematic and continuous in time source of health information. Also, specifically to our study, Lombardy health databases are quality checked for reimbursement purposes and have also been found of good quality in several studies [52, 53]. It is also important to acknowledge that not all case-mix variables that are important to fairly compare providers [48] were available such as smoking status and performance status. Finally, the second round of Delphi method was an informal face to face criterion process, that could have been biased by strong opinion leaders. However, after the first round, a high agreement rate was obtained and minor changes were made on the basis of the second round.


The developed and piloted set of indicators is now available to implement and monitor, over time, quality initiatives for lung cancer care in the studied area.

Availability of data and materials

The dataset from this study is held securely at the ATS of Milan, Epidemiology Unit. Data sharing agreements prohibit the ATS of Milan from making the dataset publicly available. The full dataset creation plan and underlying analytic code are available from the authors upon request.



Italian Association of Medical Oncology


Computed tomography


Grading of Recommendations Assessment, Development and Evaluation


Human Development Index


International Statistical Classification of Diseases and Related Health Problems 10th Revision


International Classification of Diseases for Oncology version 3


Non-small cell lung cancer


Positron Emission Tomography


Small cell lung cancer


Scottish Intercollegiate Guidelines Network


  1. 1.

    International Agency for Research on Cancer. Cancer today [Internet]. [cited 2020 Apr 9]. Available from:

  2. 2.

    Islami F, Torre LA, Jemal A. Global trends of lung cancer mortality and smoking prevalence. Transl Lung Cancer Res. 2015;4:327–38.

    CAS  PubMed  PubMed Central  Google Scholar 

  3. 3.

    Powell HA, Tata LJ, Baldwin DR, Potter VA, Stanley RA, Khakwani A, et al. Treatment decisions and survival for people with small-cell lung cancer. Br J Cancer. 2014;110:908–15.

    CAS  PubMed  PubMed Central  Article  Google Scholar 

  4. 4.

    Nadpara PA, Madhavan SS, Tworek C, Sambamoorthi U, Hendryx M, Almubarak M. Guideline-concordant lung cancer care and associated health outcomes among elderly patients in the United States. J Geriatr Oncol. 2015;6:101–10.

    PubMed  PubMed Central  Article  Google Scholar 

  5. 5.

    Hermens RPMG, Ouwens MMTJ, Vonk-Okhuijsen SY, van der Wel Y, Tjan-Heijnen VCG, van den Broek LD, et al. Development of quality indicators for diagnosis and treatment of patients with non-small cell lung cancer: a first step toward implementing a multidisciplinary, evidence-based guideline. Lung Cancer Amst Neth. 2006;54:117–24.

    CAS  Article  Google Scholar 

  6. 6.

    Ouwens MMMTJ, Hermens RRPMG, Termeer RAR, Vonk-Okhuijsen SY, Tjan-Heijnen VCG, Verhagen AFTM, et al. Quality of integrated care for patients with nonsmall cell lung cancer: variations and determinants of care. Cancer. 2007;110:1782–90.

    PubMed  Article  PubMed Central  Google Scholar 

  7. 7.

    Darling G, Malthaner R, Dickie J, McKnight L, Nhan C, Hunter A, et al. Quality indicators for non-small cell lung cancer operations with use of a modified Delphi consensus process. Ann Thorac Surg. 2014;98:183–90.

    PubMed  Article  PubMed Central  Google Scholar 

  8. 8.

    Marshall MN, Shekelle PG, McGlynn EA, Campbell S, Brook RH, Roland MO. Can health care quality indicators be transferred between countries? Qual Saf Health Care. 2003;12:8–12.

    CAS  PubMed  PubMed Central  Article  Google Scholar 

  9. 9.

    Lennes IT, Lynch TJ. Quality indicators in cancer care: development and implementation for improved health outcomes in non-small-cell lung cancer. Clin Lung Cancer. 2009;10:341–6.

    PubMed  Article  PubMed Central  Google Scholar 

  10. 10.

    Freeman T. Using performance indicators to improve health care quality in the public sector: a review of the literature. Health Serv Manag Res. 2002;15:126–37.

    Article  Google Scholar 

  11. 11.

    on behalf of the Guidelines International Network (G-I-N) Performance Measures Working Group, Nothacker M, Stokes T, Shaw B, Lindsay P, Sipilä R, et al. Reporting standards for guideline-based performance measures. Implement Sci. 2015;11 [cited 2017 Sep 15]. Available from:

  12. 12.

    Jouhet V, Defossez G, CRISAP, CoRIM, Ingrand P. Automated selection of relevant information for notification of incident cancer cases within a multisource cancer registry. Methods Inf Med. 2013;52:411–21.

    CAS  PubMed  Article  Google Scholar 

  13. 13.

    Tognazzo S, Andolfo A, Bovo E, Fiore AR, Greco A, Guzzinati S, et al. Quality control of automatically defined cancer cases by the automated registration system of the venetian tumour RegistryQuality control of cancer cases automatically registered. Eur J Pub Health. 2005;15:657–64.

    CAS  Article  Google Scholar 

  14. 14.

    Andreano A, Anghinoni E, Autelitano M, Bellini A, Bersani M, Bizzoco S, et al. Indicators based on registers and administrative data for breast cancer: routine evaluation of oncologic care pathway can be implemented. J Eval Clin Pract. 2016;22:62–70.

    PubMed  Article  Google Scholar 

  15. 15.

    Search SAGE. [cited 2017 Sep 15]. Available from:

  16. 16.

    Brouwers MC, Kho ME, Browman GP, Burgers JS, Cluzeau F, Feder G, et al. AGREE II: advancing guideline development, reporting and evaluation in health care. CMAJ Can Med Assoc J. 2010;182:E839–42.

    Article  Google Scholar 

  17. 17.

    AIOM - Associazione Italiana di Oncologia Medica. Linee guida - Tumore polmonare. 2013. Available from:

    Google Scholar 

  18. 18.

    Harbour R, Miller J. A new system for grading recommendations in evidence based guidelines. BMJ. 2001;323:334–6.

    CAS  PubMed  PubMed Central  Article  Google Scholar 

  19. 19.

    Guyatt GH, Oxman AD, Kunz R, Falck-Ytter Y, Vist GE, Liberati A, et al. Going from evidence to recommendations. BMJ. 2008;336:1049–51.

    PubMed  PubMed Central  Article  Google Scholar 

  20. 20.

    Home | National Quality Measures Clearinghouse. [cited 2015 May 5]. Available from:

  21. 21.

    Istituto Nazionale di Statistica. Demo-Geodemo. Mappe, Popolazione, Statistiche Demografiche dell’ISTAT. 2013. Available from:

    Google Scholar 

  22. 22.

    Bray F, Colombet M, Mery L, Piñeros M, Znaor A. Cancer Incidence in Five Continents, Vol. XI. Lyon: International Agency for Research on Cancer. [cited 2017 Dec 11]. Available from:

  23. 23.

    Registro tumori ATS della Città metropolitana di Milano | AIRTUM [Internet]. [cited 2017 Mar 23]. Available from:

  24. 24.

    Deyo RA, Cherkin DC, Ciol MA. Adapting a clinical comorbidity index for use with ICD-9-CM administrative databases. J Clin Epidemiol. 1992;45:613–9.

    CAS  Article  Google Scholar 

  25. 25.

    Romano PS, Roos LL, Jollis JG. Presentation adapting a clinical comorbidity index for use with ICD-9-CM administrative data: differing perspectives. J Clin Epidemiol. 1993;46:1075–9.

    CAS  PubMed  Article  PubMed Central  Google Scholar 

  26. 26.

    Jensen OM. International Agency for Research on Cancer, World Health Organization, International Association of Cancer Registries, editors. Cancer registration: principles and methods. Lyon, France: New York: International Agency for Research on Cancer ; Distributed in the USA by Oxford University Press; 1991.

    Google Scholar 

  27. 27.

    Adams JL, Mehrotra A, Thomas JW, McGlynn EA. Physician cost profiling — reliability and risk of misclassification. N Engl J Med. 2010;362:1014–21.

    CAS  PubMed  PubMed Central  Article  Google Scholar 

  28. 28.

    Scholle SH, Roski J, Adams JL, Dunn DL, Kerr EA, Dugan DP, et al. Benchmarking physician performance: reliability of individual and composite measures. Am J Manag Care. 2008;14:833.

    PubMed  PubMed Central  Google Scholar 

  29. 29.

    Adams JL. The Reliability of Provider Profiling: A Tutorial. 2009 [cited 2020 Mar 5]. Available from:

    Google Scholar 

  30. 30.

    Lung cancer, NICE clinical guideline 121 [Internet]. [cited 2015 May 6]. Available from:

  31. 31.

    Goffin J, Lacchetti C, Ellis PM, Ung YC, Evans WK. Lung Cancer disease site Group of Cancer Care Ontario’s program in evidence-based care. First-line systemic chemotherapy in the treatment of advanced non-small cell lung cancer: a systematic review. J Thorac Oncol Off Publ Int Assoc Study Lung Cancer. 2010;5:260–74.

    Google Scholar 

  32. 32.

    Azzoli CG, Temin S, Aliff T, Baker S, Brahmer J, Johnson DH, et al. 2011 focused update of 2009 American Society of Clinical Oncology clinical practice guideline update on chemotherapy for stage IV non–small-cell lung Cancer. J Clin Oncol. 2011;29:3825–31.

    PubMed  PubMed Central  Article  Google Scholar 

  33. 33.

    Gefitinib for the first-line treatment of locally advanced or metastatic non-small-cell lung cancer | Guidance and guidelines | NICE. [cited 2017 Sep 18]. Available from:

  34. 34.

    Erlotinib for the first-line treatment of locally advanced or metastatic EGFR-TK mutation-positive non-small-cell lung cancer | Guidance and guidelines | NICE. [cited 2017 Sep 18]. Available from:

  35. 35.

    Pemetrexed for the maintenance treatment of non-small-cell lung cancer | Guidance and guidelines | NICE [Internet]. [cited 2017 Sep 18]. Available from:

  36. 36.

    Sleeman D, Moss L, Gyftodimos E, Nicolson M, Devereux G. A comparison between clinical decisions made about lung cancer patients and those inherent in the corresponding Scottish intercollegiate guidelines network (SIGN) guideline. Health Informatics J. 2010;16:260–73.

    PubMed  Article  Google Scholar 

  37. 37.

    Evans WK, Ung YC, Assouad N, Chyjek A, Sawka C. Improving the quality of lung cancer care in Ontario: the lung cancer disease pathway initiative. J Thorac Oncol Off Publ Int Assoc Study Lung Cancer. 2013;8:876–82.

    Google Scholar 

  38. 38.

    Al-Shamsi HO, Al Farsi A, Ellis PM. Stage III non-small-cell lung cancer: establishing a benchmark for the proportion of patients suitable for radical treatment. Clin Lung Cancer. 2014;15:274–80.

    PubMed  Article  Google Scholar 

  39. 39.

    Fasola G, Rizzato S, Merlo V, Aita M, Ceschia T, Giacomuzzi F, et al. Adopting integrated care pathways in non-small-cell lung cancer: from theory to practice. J Thorac Oncol Off Publ Int Assoc Study Lung Cancer. 2012;7:1283–90.

    Google Scholar 

  40. 40.

    Stirling RG, Evans SM, McLaughlin P, Senthuren M, Millar J, Gooi J, et al. The Victorian lung Cancer registry pilot: improving the quality of lung cancer care through the use of a disease quality registry. Lung. 2014;192:749–58.

    PubMed  Article  PubMed Central  Google Scholar 

  41. 41.

    Jakobsen E, Green A, Oesterlind K, Rasmussen TR, Iachina M, Palshof T. Nationwide quality improvement in lung cancer care: the role of the Danish lung Cancer group and registry. J Thorac Oncol. 2013;8:1238–47.

    PubMed  Article  PubMed Central  Google Scholar 

  42. 42.

    Tang PC, Ralston M, Arrigotti MF, Qureshi L, Graham J. Comparison of methodologies for calculating quality measures based on administrative data versus clinical data from an electronic health record system: implications for performance measures. J Am Med Inform Assoc. 2007;14:10–5.

    CAS  PubMed  PubMed Central  Article  Google Scholar 

  43. 43.

    Francisci S, Minicozzi P, Pierannunzio D, Ardanaz E, Eberle A, Grimsrud TK, et al. Survival patterns in lung and pleural cancer in Europe 1999–2007: results from the EUROCARE-5 study. Eur J Cancer. 2015;51:2242–53.

    PubMed  Article  PubMed Central  Google Scholar 

  44. 44.

    Conron M, Phuah S, Steinfort D, Dabscheck E, Wright G, Hart D. Analysis of multidisciplinary lung cancer practice. Intern Med J. 2007;37:18–25.

    CAS  PubMed  Article  PubMed Central  Google Scholar 

  45. 45.

    Fang P, He W, Gomez DR, Hoffman KE, Smith BD, Giordano SH, et al. Influence of age on guideline-concordant Cancer Care for Elderly Patients in the United States. Int J Radiat Oncol Biol Phys. 2017;98:748–57.

    PubMed  PubMed Central  Article  Google Scholar 

  46. 46.

    Mant J. Process versus outcome indicators in the assessment of quality of health care. Int J Qual Health Care. 2001;13:475–80.

    CAS  PubMed  Article  PubMed Central  Google Scholar 

  47. 47.

    Porter ME, Teisberg E. Redefining health care: creating value-based competition on results. 1 edizione. Boston, Mass: Harvard Business School Press; 2006.

    Google Scholar 

  48. 48.

    Mak KS, van Bommel ACM, Stowell C, Abrahm JL, Baker M, Baldotto CS, et al. Defining a standard set of patient-centred outcomes for lung cancer. Eur Respir J. 2016;48:852–60.

    PubMed  PubMed Central  Article  Google Scholar 

  49. 49.

    Rubin HR, Pronovost P, Diette GB. The advantages and disadvantages of process-based measures of health care quality. Int J Qual Health Care. 2001;13:469–74.

    CAS  PubMed  Article  PubMed Central  Google Scholar 

  50. 50.

    Romano PS, Mark DH. Bias in the coding of hospital discharge data and its implications for quality assessment. Med Care. 1994;32:81–90.

    CAS  PubMed  Article  PubMed Central  Google Scholar 

  51. 51.

    Gavrielov-Yusim N, Friger M. Use of administrative medical databases in population-based research. J Epidemiol Community Health. 2014;68:283–7.

    PubMed  Article  PubMed Central  Google Scholar 

  52. 52.

    Mazzali C, Paganoni AM, Ieva F, Masella C, Maistrello M, Agostoni O, et al. Methodological issues on the use of administrative data in healthcare research: the case of heart failure hospitalizations in Lombardy region, 2000 to 2012. BMC Health Serv Res. 2016;16:234.

    PubMed  PubMed Central  Article  Google Scholar 

  53. 53.

    Beghi E, Logroscino G, Micheli A, Millul A, Perini M, Riva R, et al. Validity of hospital discharge diagnoses for the assessment of the prevalence and incidence of amyotrophic lateral sclerosis. Amyotroph Lateral Scler Other Motor Neuron Disord. 2001;2:99–104.

    CAS  PubMed  Article  PubMed Central  Google Scholar 

Download references


Lombardy Lung Cancer Working Group, authors and institutions who contributed to the assessment of candidate indicators: Milena Andruccioli (U.O. Medicina Diagnostica e dei Servizi - Disciplina di Medicina Nucleare - ASST Lariana Ospedale Sant’Anna - Como); Antonio Ardizzoia (U.O. Oncologia Medica – ASST Lecco); Agnese Assi (U.O. Anatomia Patologica - ASST Ovest Milanese); Alessandro Baisi (U.O. Chirurgia Toracica - ASST Santi Paolo e Carlo - Milano), Amedeo Vittorio Bedini (U.O. Chirurgia Toracica – Asst Monza); Giordano Beretta (U.O. Oncologia Medica - Humanitas Gavazzeni); Maurizio Bersani (Struttura Piani e progetti, UO Innovazione e Sviluppo, Direzione Generale Welfare, Lombardy Region), Francesco Bertagna (U.O. Chirurgia - Università degli Studi di Brescia); Elisabetta Berti (Servizio Anatomia Patologica - ASST Santi Paolo e Carlo - Milano); Alessandro Bertolini (U.O. Oncologia Medica – ASST Valtellina e Alto Lario - Ospedale di Sondrio); Anna Bettini (U.O. Oncologia Medica- ASST Ospedale Papa Giovanni XXIII - Bergamo); Mario Bignardi (U.O. Radioterapia Oncologica - Fondazione Poliambulanza - Brescia); Francesco Bini (U.O. Pneumologia - ASST Rhodense); Claudio Bnà (U.O. Radiologia - Fondazione Poliambulanza - Brescia); Francesca Bono (U.O. Anatomia Patologica – Asst Monza); Daniele Bonora Ottoni (U.O. Chirurgia Toracica - Fondazione Poliambulanza - Brescia); Andrea Borghesi (U.O. Radiologia Diagnostica 2 – Università degli Studi di Brescia - ASST Spedali Civili di Brescia); Paolo Borghetti (U.O. Radioterapia Oncologica - ASST Brescia); Antonio Brunelli (U.O. Chirurgia Generale - ASST Melegnano Martesana); Alberto Buffoli (U.O. Radioterapia Oncologica - Istituto Clinico S. Anna BS); Carlo Cacioppo (U.O. Cure Palliative - Fondazione Don Gnocchi); Angelomaria Calati (U.O Chirurgia Toracica – ASST Lariana - Ospedale Sant’Anna - Como); Anna Calcagno (U.O. Oncologia Legnano - ASST Ovest Milanese); Sabrina Casagrande (U.O. Medicina Nucleare - ASST Settelaghi); Clelia Casartelli (U.O. Oncologia - Ospedale Valduce - Como); Massimo Castellani (U.O. Medicina Nucleare - Fondazione IRCCS Cà Granda - Ospedale Maggiore Policlinico - Milano); Antonino Castro (U.O. Oncologia - ASST Franciacorta); Gianpiero Catalano (U.O. Radioterapia – IRCCS Multimedica - Sesto S. Giovanni); Laura Catena (U.O. Oncologia - Policlinico di Monza); Giovanni Luca Ceresoli (U.O. Oncologia Pneumologia e Urologica - Humanitas Gavazzeni); Massimiliano Cergnul (U.O. Oncologia - ASST Ovest Milanese - Legnano); Gianluigi Ciocia (u.O. Medicina Nucleare - Cliniche Gavazzeni Humanitas - Bergamo); Annamaria Colombo (U.O. Cure Palliative - Policlinico di Monza); Barbara Conti (U.O. Chirurgia Toracica - ASST Bergamo Ovest); Daniela Corti (U.O. Anatomia Patologica - ASST Bergamo Ovest); Diego Cortinovis, (U.O. Terapia Medica – Asst Monza); Claudio Cozzi (U.O. Oncologia Medica - ASST Rhodense); Flavio Crippa (U.O. Medicina Nucleare - Fondazione IRCCS Istituto Nazionale Tumori - Milano); Andrea De Monte (U.O. Oncologia - Ospedale di Melegnano); Daniela De Candis (U.O. Oncologia - Clinica San Carlo - Paderno Dugnano); Domenico De Toma (U.O. Oncologia - Policlinico San Donato); Filippo De Marinis (U.O. Oncologia Toracica - Istituto Europeo di Oncologia - Milano); Biancamaria DiMarco (U.O. Anatomia Patologica - ASST Ovest Milanese); Franca Di Nuovo (U.O. Anatomia e Istologia Patologica - ASST Rhodense); Claudio Doglioni (U.O. Anatomia Patologica – IRCCS Ospedale San Raffaele - Milano); Valentina Fabbris (U.O. Anatomia Patologica - ASST Ovest Milanese); Paolo Faccioli (U.O. Radiologia - ASST Lecco - Ospedale “A. Manzoni”); Daniele Fagnani (U.O. Oncologia - ASST Vimercate); Gabriella Farina (U.O. Oncologia FBF - ASST Fatebenefratelli Sacco - Milano); Laura Fariselli (U.O. Radioterapia - Fondazione IRCCS - Istituto Neurologico Besta - Milano); Virginio Filipazzi (U.O. Oncologia Sacco - ASST Fatebenefratelli Sacco - Milano); Francesca Franzi (U.O. Anatomia Patologica – ASST Settelaghi); Paolo Frata (U.O. Radioterapia e Dipartimento Oncologico - ASST Valcamonica); Ana Maria Samanes Gajate (U.O. Medicina Nucleare - IRCCS Ospedale San Raffaele - Milano); Gianstefano Gardani (U.O. Radioterapia - ASST Monza); Massimo Gasparini (U.O. Medicina Nucleare - Multimedica Sesto S. Giovanni); Monica Giordano (U.O. Oncologia – ASST Lariana), Fabrizio Grignani (U.O. Chirurgia - ASST Ovest Milanese); Claudio Landoni (U.O. Medicina Nucleare - ASST Monza – Università Milano - Bicocca); Carlo Lombardi (U.O. Geriatria - Fondazione Poliambulanza - Brescia); Fabrizio Lombardi (U.O. Radioterapia - Policlinico San Donato); Andrea Luciani (U.O. Oncologia Medica - ASST Santi Paolo e Carlo); Lorenzo Stefano Maffioli (U.O. Medicina Nucleare – ASST Ovest Milanese); Roberta Marchione (U.O. Anatomia Patologica - ASST Valcamonica); Paola Mariani (Servizio Radiologia - ASST Santi Paolo e Carlo); Anna Maspero (U.O. Pneumologia - ASST Lariana - Ospedale Sant’Anna - Como); Fausto Meriggi (U.O. Oncologia - Fondazione Poliambulanza - Brescia); Aurora Miedico (U.O. Oncologia Medica - ASST Ovest Milanese); Patrizia Morbini (U.O. Anatomia Patologica - Policlinico San Matteo - Pavia); Giovanni Muriana (U.O. Chirurgia Toracica - ASST Mantova); Giulio Orlandoni (U.O. Chirurgia Toracica - Policlinico San Matteo - Pavia); Enrica Paris (U.O. Radiologia Diagnostica - ASST Valcamonica); Paolo Pedrazzoli (U.O. Oncologia - IRCCS Policlinico S. Matteo di Pavia e A.O. della Prov. di Pavia); Giuseppe Pepe (U. O. Pneumologia - ASST Rhodense); Ezio Pezzica (U.O. Anatomia e Istologia Patologica - ASST Bergamo Ovest); Fabio Piccoli (U.O. Radioterapia - ASST Ospedale Papa Giovanni XXIII); Gianpaolo Pinotti (U.O. Radiodiagnostica - ASST Garda); Antonello Quadri (U.O. Oncologia Medica - Policlinico San Pietro/San Marco GDS); Andrea Ravasio (U.O. Radioterapia - Humanitas Gavazzeni); Franco Ravenna (U.O. Pneumologia - ASST Mantova); Elisabetta Ricchiuti (U.O. Pneumologia - ASST Ovest Milanese); Davide Rinaldo (U.O. Pneumologia - ASST Garda); Gianni Roda’ (U.O. Oncologia - ASST Bergamo Est); Gerolamo Rossi (U.O. Chirurgia Toracica – Ospedale Valduce); Armando Santoro (U.O. Oncologia Medica ed. Oncologia - IRCCS Humanitas Rozzano); Carlo Pietro Soatti (U.O. Radioterapia - ASST Lecco - Ospedale “A. Manzoni”); Francesco Stiglich (Medicina Nucleare e Radioterapia Oncologica - ASST Valtellina e Alto Lario); Luca Tagliabue (U.O. Medicina Nucleare -ASST Santi Paolo e Carlo); Valeria Tirelli (U.O. Cure Palliative - ASST Garda); Andrea Tironi (U.O. Anatomia Patologica - ASST Spedali Civili di Brescia); Maurizio Tomirotti (U.O. Oncologia Medica - Fondazione IRCCS Policlinico (Osp. Maggiore) Mangiagalli); Laura Lavinia Travaini (U.O. Medicina Nucleare - Istituto Europeo di Oncologia - Milano); Giovanni Ucci (U.O. Oncologia - Ospedale Provincia di Lodi - Ospedale Maggiore); Adriano Vaghi (U.O. Pneumologia - H Garbagnate - ASST Rhodense); Angelo Vanzulli (U.O. Radiologia - ASST Grande Ospedale Metropolitano Niguarda e Dipartimento di Oncologia ed. Emato-Oncologia, Università degli Studi di Milano); Giulia Veronesi (U.O. Chirurgia Toracica - Humanitas - Rozzano); Isabella Vittimberga (U.O. Oncologia - Ospedale Provinciale di Lecco); Claudia Zambelli (U.O. Anatomia Patologica – ASST Spedali Civili Brescia).


This work was supported by the Italian Ministry of Health [RF 2011–02348859 to A.G.R].

Author information





Concept and design: AGR, AA. Acquisition, analysis, or interpretation of data: AA, MGV, AGR, SS. Drafting of the manuscript: AA, AGR. Critical revision of the manuscript for important intellectual content: MGV, SS. Statistical analysis: AA. Administrative, technical, or material support: All authors. AA had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis. AGR, the corresponding author, accepts full responsibility for the work, had access to the data, and controlled the decision to publish and attests that all listed authors meet authorship criteria and that no others meeting the criteria have been omitted. The author(s) read and approved the final manuscript.

Corresponding author

Correspondence to Antonio Giampiero Russo.

Ethics declarations

Ethics approval and consent to participate

Ethics approval and consent to participate were not required, as this is an observational study based on data routinely collected by the Agency for Health Protection (ATS) of Milan, a public body of the Regional Health Service – Lombardy Region. The ATS has among its institutional functions, established by the Lombardy Region legislation (R.L. 23/2015), the government of the care pathway at the individual level in the regional social and healthcare system, the evaluation of the services provided to, and the outcomes of, patients residing in the covered area. This study is also ethically compliant with the National Law (D.Lgs. 101/2018) and the “General Authorisation to Process Personal Data for Scientific Research Purposes” (n.8 and 9/2016, referred to in the Data Protection Authority action of 13/12/2018).

Consent for publication

Not applicable.

Competing interests

The authors declare that they have no competing interests.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1: Supplementary methods.

Supplementary Figure 1. Process from literature review to indicator calculation and evaluation. Supplementary Figure 2. Selection of the pilot cohort of lung cancer patients from the Cancer Register of the Province of XX.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit The Creative Commons Public Domain Dedication waiver ( applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Andreano, A., Valsecchi, M.G., Russo, A.G. et al. Indicators of guideline-concordant care in lung cancer defined with a modified Delphi method and piloted in a cohort of over 5,800 cases. Arch Public Health 79, 12 (2021).

Download citation


  • Lung neoplasms
  • Quality indicators, health care
  • Guideline adherence
  • Quality of health care
  • Health information systems