Skip to main content
Download PDF
Download ePub

Is the online ‘creating healthy eating and active environments survey' (CHEERS) tool reliable for early childhood educators in Alberta, Canada: a randomized crossover trial

Archives of Public Health volume 81, Article number: 22 (2023) Cite this article

Abstract

Background

The creating healthy eating and active environments survey (CHEERS) is an audit tool used to assess the nutrition and physical activity environment in early childhood education and care (ECEC) centres. Availability of the tool has been limited to paper-based versions. Digital health initiatives offer improved reach and immediacy of support for community-based clients through novel technology products. In order to provide increased access to the CHEERS tool, an online version was developed. The objective of this study was to assess the reliability of an online version of CHEERS.

Methods

Utilizing a randomized crossover design, ECEC educators completed either a paper-based or online-based survey and then the opposite mode with a two to three-week interval. The intraclass correlation coefficient (ICC, with 95% confidence interval) was used to determine the reliability between test and retest. Absolute index of reliability in the original measurement was assessed through the standard error of measurement (SEM = SD × √1-ICC). The smallest amount of change not due to inherent variation was assessed by determining minimal detectable change at the 95% confidence level (MDC95 = SEM × 1.96 ×√2; MDC95% = MDC95/mean ×100).

Results

Test–retest reliability was good to excellent for the online-based CHEERS total score (ICC = 0.86) and for each of the four subscales: food served (ICC = 0.82), healthy eating environment (ICC = 0.76), program planning (ICC = 0.76), and physical activity environment (ICC = 0.79). The SEM, MDC95, and MDC95% for the CHEERS overall score were 0.79, 2.19, and 9.6%, respectively.

Conclusions

The results of this study demonstrate that the online-based and paper-based versions of the CHEERS audit tool share comparable accuracy. The CHEERS tool can be reliably implemented in an online environment and this provides users an alternative means to complete the centre-based health assessment. The advantage of the online-based version includes user accessibility and the potential to develop a feedback response for participants using digitally collected data.

Peer Review reports

Background

The early years are a critical period in child development where the origins of lifestyle health behaviours can be identified. Patterns of eating and activity, as early as one to two years of age, have been demonstrated to track through childhood and into adulthood [1,2,3,4,5,6]. Research suggests that Canadian preschool-aged children consume less than one-third of the vegetable and fruit intake recommended combined with low levels of physical activity [7,8,9,10]. It is estimated that close to 60% of Canadian children 0–5 years participate in some type of childcare arrangement with over half of these attending centre-based care [11]. This demonstrates that early childhood education and care (ECEC) environments represent rich opportunities for interventions that promote healthy eating and reduced sedentary behaviours aimed at preventing childhood obesity and the reduction of the long-term non-communicable chronic disease [12,13,14].

Supporting child health and well-being is a public health priority and is an essential global health initiative. The nurturing care framework is an organizing principal that structures key components of high quality care in ECEC environments that is composed of five elements: health, nutrition, security and safety, responsive caregiving, and early learning opportunities [15]. In order to provide this high quality care, educators and public health professionals require access to valid and reliable tools. The creating healthy eating and active environments survey (CHEERS) tool is a community-based, educator-administered audit tool designed to offer ECEC centres an evaluative measure to assess their centres healthy eating and physical activity environment. The constructs measured include the quality of food served, the environment where children are eating, the curriculum connections for nutrition education, and the physical activity opportunities provided. This will also be a valuable tool for public health providers and researchers that can be used to measure the environmental context in a child care setting. The CHEERS tool can facilitate and empower ECEC centres to enhance nutrition and activity environments that support child health. While currently aligned with Canadian best practices, the tool would be adaptable to many international childcare contexts.

The tool has been assessed for reliability and validity with early childhood experts and educators. The CHEERS tool was content validated following a structured process involving a panel of ten nutrition, physical activity, early childhood, and public health professionals with a readability of grade 8.1 using Flesch–Kincaid [16] with evidence of good inter-rater reliability, intra-rater reliability, internal consistency, and concurrent validity [17].

The CHEERS tool was developed as a paper instrument that could be completed by an educator and returned by ground mail for evaluation. Paper-based questionnaires are human resource intensive and introduce the possibility of data omission and errors in data transcription to digital format. On the other hand, there are disadvantages of online-based questionnaires such as inaccurate responses due to misinterpretation of questions and low response rates. The internet and its applications are a continuous area of innovation and an exponentially increasing component of every day life. Electronic data collection is becoming more extensively used in healthcare research that bring with it many advantages such as access and service provision for remote communities, facilitated data entry time, reduced processing cost, and faster survey response return to stakeholder [18, 19]. Moreover, a key message from the early childhood community is the need for reciprocation. While data is collected from the ECEC centre, how can the centre benefit from completing the questionnaire? Educator awareness of policies and practices that support or impair healthy environments for children have been identified as a method of informing or reaffirming health promoting professional practice activities [20].  An online-based survey can increase response time and support just-in-time learning for educators.

An electronic version of the CHEERS tool was developed to align with increasing use of electronic public health outreach and resources provided through website in addition to the desire of educators to access the tool in an online context. While reliability and validity of the CHEERS tool have been demonstrated, it is not known if administration using an online-based interface is consistent with paper and pencil versions. The aim of this study was to assess the reliability and utility of an electronic version of the CHEERS tool.

Methods

This study was conducted in Alberta, Canada throughout provincial health zones. Ethical approval to conduct the study was obtained from the Mount Royal University Human Research Ethics Board (no. 100016). Participants were fully informed about the purpose and procedures of the study and provided written consent for paper first participation or digital consent for electronic first participation. No minors were involved in this study.

Participants

The intended users of the CHEERS tool, ECEC educators, were recruited from licensed ECEC facility-based centres that care for infants, toddlers and pre-school-aged children in Alberta, Canada. They typically provide care throughout the day, from the morning to early evening. To be eligible for the study, ECEC educators had to be employed full time in licensed ECEC centres that provided care for a minimum of 15 preschool aged (3 – 5 years) children with the classification of day care program. Inclusion criteria was access to an electronic device (smartphone/tablet/laptop), no previous experience with the CHEERS survey, and fluent in English. Exclusion criteria were exclusive employment in family day home or after school care programs or employment in unlicensed ECEC centres.

CHEER audit tool

The CHEERS audit tool includes 59 items divided into four subscales: food served (23 items), healthy eating environment (18 items), healthy eating program planning (6 items), and physical activity environment (12 items). Response options for each item range from always to never which are then recorded from 1 to 7. Scores are averaged for each subscale and summed for the CHEERS total score (range 4–28). A copy of the questionnaire is available [21] and included as a supplemental file (see Supplemental file 1).

Study design

This study is a randomized crossover trial design comparing paper and electronic survey formats. A convenience sample of ECEC educators were recruited from ECEC centres throughout Alberta. Flyer advertisements were distributed provincially through email, inviting ECEC educators to contact the project coordinator through email. A $20 gift card was provided to respondents as an incentive. Interested individuals were contacted via email or telephone and given detailed instructions about the study to ensure sample validity and data integrity [22]. Educators were randomly assigned by random generated number to complete the CHEERS survey either in paper version or electronic survey platform. The two study groups were reversed according to the randomized crossover design and the alternate mode completed within a two to three-week interval. The interval period being sufficiently short to avoid real change and sufficiently long enough to minimize recall with 2 weeks being a common interval in the development of health measurement scales [23].

Study methods

Participants assigned to paper-first administration received the CHEERS survey in a sealed envelope with instructions on how to complete the survey as well as a prepaid pre-addressed envelope to use to return the paper survey to the research team. An email reminder was sent to the participant ten days after mailing to ensure participants received the survey and to encourage return of the survey. In three cases, the paper surveys had not arrived and an additional package had to be resent. Upon return, surveys were coded and data entered into a Microsoft Excel 2016 spreadsheet. In the case of missing data, the input was left blank. If a page of survey responses were missed the survey was considered spoiled and the participant considered lost to follow up. To implement the washout period in the paper-first group, the participant was sent a personalized email link to the online version of the CHEERS survey one week after receiving the paper survey (three business day delivery within province). A reminder email was sent one week after the first email to those individuals where no recorded survey was evident in the online platform. Participants assigned to the online-first administration received a personalized the link to the online version of CHEERS which contained instructions on how to complete the survey and was followed by a reminder email one week later. To implement the washout period in the online-first group, the paper survey package was mailed to participants one week after the survey was verified as received in the online platform. An email reminder was sent to participants ten days later to check for paper-survey receipt and encourage return. In two cases, paper surveys had to be resent and in three cases participants indicated surveys were mailed back however these were never received by the research team.

Online interface

The electronic questionnaire system for the CHEERS administration was built in Qualtrics® online-based survey platform. This survey system is compatible across computer and mobile smartphone platforms (e.g. Apple Safari, Google Chrome, Internet Explorer, Mozilla Firefox). The survey is optimized to the mobile experience to adjust for screen size and formatting. The online version of the CHEERS questionnaire was developed to replicate the paper-based questionnaire experience as closely as possible. The online questionnaire presents three questions per page with a progress tracking bar at the top of the survey. The survey platform also records the duration (time to complete the survey). A sample screenshot of the online-based interface for the survey tool is presented in Fig. 1. Participants were provided unique email link once identity and eligibility verified.

Fig. 1
figure 1

Screenshot of the electronic CHEERS tool as it appears in a computer browser (explorer, left) or a mobile device (right)

Full size image

Sample size

Sample size needed for the reliability analysis was calculated using the sample size calculator [24] using the parameters of required number of participants when β = 0.20, α = 0.05, the number of repetitions per participant as two, with a potential dropout rate of 30%. We chose an ICC = 0.70 as minimally acceptable and an ICC = 0.90 as desired based on the recommended minimum reliability of 0.70 when the scale is used in research [23]. This calculation returned a required sample size of 23 participants per group (paper 1st or online 1st) or 46 participants estimated to be sufficient for testing reliability [23,24,25].

Statistical analysis

The results of the paper questionnaire were manually inputted into Microsoft Excel 2016 by the research assistant and data checking completed by the PI. Online data was captured by the Qualtrics software program and derived directly from the system database. Statistical analysis was performed using SPSS statistical package version 26 (SPSS Inc, Chicago, IL), with alpha level of p < 0.05. Descriptive statistics were used to report means and variation between trials for the overall CHEERS score and subscales. Group comparisons were performed using independent t-test and Pearson chi-square tests for numerical and categorical demographic variables, respectively, to assess heterogeneity between ECECs. Interquartile range (IQR) and Mann–Whitney U test was used to assess survey duration (time to complete the survey) in the online-based environment.

To assess the magnitude of the association between electronic and paper data, the intraclass correlation coefficient (ICC) was calculated by SPSS for each administration and administration time points along with their 95% confidence interval. The ICC takes into account the selection of raters as well as the correlation and agreement between raters [26,27,28]. The intraclass correlation coefficient estimates and their 95% confident intervals for test–retest reliability were based on a single rater/measurement, absolute-agreement, 2-way mixed-effects model [26, 29]. In this study, interpretations of ICC results followed Koo and Li’s categorizations: poor (less than 0.5), moderate (0.5–0.75), good (0.75–0.9), and excellent (greater than 0.90) [21]. To further assess response stability, the standard error of measurement (SEM) was calculated using the following formula: SEM = SD × √(1 – ICC) [30, 31]. The SEM provides an index of the “trial-to-trial noise in the data” which provides an absolute index of reliability in the original measurement unit without the influence of variance among participants [30]. Smaller SEM values, relative to the unit of measure of the score, demonstrate lower response variation from test to retest. The SEM value might be considered an estimation of the expected random variation in scores when no real change has taken place. Absolute reliability provides additional information about a change in score and what change would be required to provide a measure of true change beyond expected measurement error and individual variabilities. This can be evaluated using the minimal detectable change (MDC95) which provides a value of reliability in the original units of the measure reflecting the amount of change required to demonstrate a difference not due to chance at the 95% confidence level [32]. MDC95 was calculated using the following formula: MDC95 = SEM X 1.96 X √2 [33]. This change can be expressed as relative change (MDC95%) and was calculated in this study using the following formula: MDC95% = MDC95/mean X 100 [33]. An MDC95% less than 30 has been interpreted in the literature as acceptable random measurement error and under 10 as excellent [34, 35]. Utility was evaluated by documenting the length of time taken to complete the CHEERS tool as recorded in the survey platform housed in Qualtrics [36].

Results

Participants

A total of 80 educators aged between 26 and 55 + years volunteered to participate in the study (Fig. 2). Eight educators did not meet inclusion criteria or did not respond after initial interest and were excluded from randomization and the final 72 randomly assigned to a survey platform. Eleven participants did not continue in the paper-first survey (group A) and twelve in the online-first survey (group B). Two surveys in group B were excluded due to missing multiple survey questions on double sided page. Missing cases were excluded when the questionnaire scores were compared.

Fig. 2
figure 2

Study design and recruitment

Full size image

Participant demographics

Demographic characteristics of the child care centres are presented in Table 1. ECEC centres were representative of both not-for-profit and profit-based programs. Half of participating ECEC centres were from large urban centres with the remaining a mix of medium and small centres. No statistically significant differences among demographic characteristics of child care type (χ2 [1] = 1.647, p = 0.199), city size (χ2 [2] = 3.163, p = 0.206), or number of preschoolers enrolled (t [47] = 0.452, p = 0.145) between the two groups were noted.

Table 1 Licensed early childhood education centre characteristics, Alberta, Canada
Full size table

Demographic characteristics of the child care educators are presented in Table 2. All respondents were female with representation throughout educator training levels. Educator demographics were similar between the groups in terms of age (χ2 [4] = 3.667, p = 0.453), level of education (χ2 [2] = 0.091, p = 0.956), or years of experience (t [47] = 0.278, p = 0.951).

Table 2 Early childhood educator characteristics, Alberta, Canada
Full size table

Reliability

Overall test–retest reliability of form (paper versus online) and sequence (paper first versus online first) was assessed using the ICC two-way mixed-effects model and absolute-agreement. The ICC values for overall CHEERS score and subscales were good to excellent (0.77—0.92) regardless of form (paper vs online) or administration sequence (Table 3).

Table 3 CHEERS tool scores and intraclass correlation coefficient measures of test–retest reliability between form (online vs paper) and sequence (online first or paper first)
Full size table

Evaluation between time 1 and time 2 shows good to excellent reliability (Table 4). The ICC value between administration for the overall CHEERS score was 0.92 (95% CI: 0.84–0.96). ICC values for the subdomains ranged 0.85–0.90.

Table 4 CHEERS tool scores and intraclass correlation coefficient measures of test–retest reliability between Allocation Time 1 and Time 2
Full size table

Agreement

The SEM, MDC95 and MDC95% of the CHEERS scores and subdomain tests are shown in Table 5. The SEM between administration of surveys (time 1 versus time 2) were relatively small for the CHEERS overall score (0.79) and subdomains (0.21 – 0.45) demonstrating an acceptable level of agreement between time points.

Table 5 Reliability and minimal detectable change values for overall CHEERS tool scores
Full size table

The MDC95 of the overall CHEERS score, food served, healthy eating environment, program planning, and physical activity were 2.19, 0.57, 0.69, 1.26, and 0.72, respectively. The MDC95% in the overall CHEERS score, food served, healthy eating environment, program planning, and physical activity were 9.6%, 9.5%, 11.1%, 25.8%, and 12.6%, respectively.

Utility measure

The median time taken to complete the CHEERS tool for all educators, regardless of sequence, in the online environment was 16.6 min (IQR 12.7–29.4). The median time taken to complete the survey in an online-first sequence (Group B) was 12.3 min (IQR 12.3–33.8). The median time taken to complete the survey in an online-second sequence (Group A) was 16.1 min (IQR 12.7–19.6). A Mann–Whitney U test revealed that time taken to complete the study was not significantly different between online-first (Group B, Md = 12.3, n = 24) compared to the online-second sequence (Group A, Md = 16.1, n = 25), U = 254.5, p = 0.363.

Discussion

The results of this study demonstrate that the online version of the CHEERS tool presents good reliability, agreement, and utility to its paper version. Previous research demonstrates that the paper-based, educator-administered CHEERS survey is a reliable and valid audit tool for evaluating child care centre eating and activity environments [16, 17]. Electronic health (eHealth) and mobile health (mHealth) innovations offer new avenues for health promotion [37]. Online methods of health assessment and interventions have the advantages of convenience, ease of operation, adaptability, and accessibility. However, evidence of online reliability, validity, and effectiveness are an important component of building comprehensive evidence for the eHealth initiative [38].

The test–retest reliability of the online-based CHEERS overall score was strongly aligned with the paper-based administration (ICC = 0.91).

These results align well with previously reported paper-based reliability data for the overall score (ICC = 0.81) [17]. Psychometric properties of “reliability and agreement are estimates that vary based on interactions between a tool, its user, and the context of the assessment” [39]. Considering that the current study was done with a new cohort of early childhood educators who had not previously participated and similar intra-rater results were found contributes to the overall psychometric evidence demonstrating reliability of the CHEERS audit tool.

The outcomes of this study are in accordance with outcomes in similar paper to online validation studies of child related health tools. For example, the NutriSTEP® questionnaire screens nutrition risk in preschoolers through a survey completed by primary caregiver/parents. An ICC of 0.91 for CHEERS overall score is in line with the ICC of 0.91 found in the NutriSTEP® adaptation from paper-based to online-based administration [40]. There was similar alignment of ICC scores for the MiniPAQLQ, a quality of life questionnaire completed by caregiver’s of children with asthma, ICC of 0.89 [41]. This is further supported by the results of a meta-analysis suggesting that computer and paper-based questionnaire administrations result in equivalent scores when comparing electronic and paper-based administrations [42].

The standard error of measurement (SEM) is a measure of stability, reflecting the precision of individual scores expressed in the same units as the original scores [23, 30]. While ICC reflects reliability among people, SEM quantifies precision within individuals and permits the calculation of the minimal detectable difference that can reliably meaningful difference to measure change that can be used in future research to determine if an intervention has a significant impact on creating change. The current study has provided evidence of stability as demonstrated by the SEM, MDC95, and MDC95% from paper-based to online-based administration.

Tool reliability and validity are important aspects of health measurement scales. However, a measure must also have utility and ease of use for respondents thus an important component of psychometric tool assessment is the usability, functionality, and availability of the tool [43, 44]. The literature clearly indicates a preference for digital over paper questionnaire completion [40, 41, 45]. eHealth technologies can provide improved efficiency and accessibility for health promotion services for stakeholders. This study adds to the reliability and utility of the CHEERS tool through the addition of a user-friendly electronic version.

Potential limitations exist with this research. First, all participants in this study were women which brings into question the representativeness of the study sample. However, the fact that the ECE workforce is comprised primarily of women, 96% of all ECEs in Canada are women [46], this sampling does reflect the population of users intended for this tool. Second, findings may also be limited because this study used a convenience sample. While a random sampling frame would be a better approach, the early childhood educator profession in Canada does not have a central professional community where all members are listed. As a result, there is no mechanism from which to randomly select participants. The provincial government website has a listing of all licensed centres throughout the province by region so emails to centres provide the best option to contact a variety of educators in different cities and towns of various sizes. Thirdly, there is a possibility that participants remembered their responses from the first administration. A recommended interval between administrations appropriate for reliability studies assessing the development of health measurement scales is approximately 2 weeks [23]. Sufficiently short to ensure minimal opportunity for real behaviour change while sufficiently long enough to minimize recall and bias. However, some individuals may have remembered their responses and which may have allowed them to report similar answers in the second administration. Lastly, a limitation is the dropout rate. Thirty-six participants were assigned to arms of the crossover study, however, approximately ten were lost to follow-up resulting in a 27% dropout rate which within the 30% dropout rate [23]. This loss may have been due to characteristics of the survey (time demand) or high job turnover in the early childhood educator workforce [47, 48]. Five paper surveys were lost to lack of postal delivery (lost mail) and two lost due to incomplete data (missed backside of paper-survey). However, the reliability measures of the CHEERS adaptation from paper-based to electronic-based administration were similar the NutriSTEP® and MiniPAQLQ paper-to-electronic adaptations within a similar population and it is reasonable to assume the missing data would not have meaningfully impacted the study outcome.

Conclusions

This study provides evidence of good test-rest reliability of the online-based CHEERS tool relative to the paper-based version. Health interventions are increasingly being made available via multiple platforms (computer or mobile versions) and demonstrating that these digital options reliable improves the reach and immediacy of support for community-based clients through novel technology products. eHealth administration of the CHEERS tool can be used in public health initiatives within community settings and the outcomes can be interpreted in accordance with findings of the paper version of the tool. Future opportunities would be to investigate immediate feedback of the survey to respondents as well as educational outreach.

Availability of data and materials

The raw datasets are not publicly available as permission for sharing this dataset or making it publicly available was not requested as part of the application to the Research Ethics Board and participant consent for public sharing of their data not secured.

Abbreviations

CHEERS:

Creating healthy eating and activity environments survey

CI:

Confidence interval

eHealth:

Electronic health

ECEC:

Early learning and childcare

FS:

Food served

HEE:

Healthy eating environment

HPP:

Healthy eating program planning

ICC:

Intraclass correlation coefficient

MDC95 :

Minimal detectable change at 95% confidence interval

MDC95%:

Percent minimal detectable change at 95% confidence interval

PA:

Physical activity environment

SEM:

Standard error of measurement

References

  1. Tammelin R, Yang X, Leskinen E, Kankaanpaa A, Hirvensalo M, Tammelin T, et al. Tracking of physical activity from early childhood through youth into adulthood. Med Sci Sports Exerc. 2014;46(5):955–62.

    Article  PubMed  Google Scholar 

  2. Jones RA, Hinkley T, Okely AD, Salmon J. Tracking Physical Activity and Sedentary Behavior in Childhood: A Systematic Review. Am J Prev Med. 2013;44(6):651–8.

    Article  PubMed  Google Scholar 

  3. Lioret S, Campbell KJ, McNaughton SA, Cameron AJ, Salmon J, Abbott G, et al. Lifestyle patterns begin in early childhood, persist and are socioeconomically patterned, confirming the importance of early life interventions. Nutrients. 2020;12(3):E724.

    Article  Google Scholar 

  4. Luque V, Escribano J, Closa-Monasterolo R, Zaragoza-Jordana M, Ferré N, Grote V, et al. Unhealthy Dietary Patterns Established in Infancy Track to Mid-Childhood: The EU Childhood Obesity Project. J Nutr. 2018;148(5):752–9.

    Article  PubMed  Google Scholar 

  5. EDEN Mother-Child Cohort Study Group. Dietary Patterns Track from Infancy to Preschool Age: Cross-Sectional and Longitudinal Perspectives. J Nutr. 2015;145(4):775–82.

    Article  Google Scholar 

  6. Feinstein L, Sabates R, Sorhaindo A, Rogers I, Herrick D, Northstone K, et al. Dietary patterns related to attainment in school: the importance of early eating patterns. J Epidemiol Community Health. 2008;62(8):734–9.

    Article  CAS  PubMed  Google Scholar 

  7. Pabayo R, Spence JC, Casey L, Storey K. Food consumption patterns: In preschool children. Canadian Journal of Dietetic Practice and Research. 2012;

  8. Kuzik N, Clark D, Ogden N, Harber V, Carson V. Physical activity and sedentary behaviour of toddlers and preschoolers in child care centres in Alberta, Canada. Can J Public Health. 2015;106(4). Available from: http://journal.cpha.ca/index.php/cjph/article/view/4794

  9. Temple VA, Naylor PJ, Rhodes RE, Higgins JW. Physical activity of children in family child care. Appl Physiol Nutr Metab. 2009;34(4):794–8.

    Article  PubMed  Google Scholar 

  10. Chaput JP, Colley RC, Aubert S, Carson V, Janssen I, Roberts KC, et al. Proportion of preschool-aged children meeting the Canadian 24-Hour Movement Guidelines and associations with adiposity: results from the Canadian Health Measures Survey. BMC Public Health. 2017;17(5):829.

    Article  PubMed  PubMed Central  Google Scholar 

  11. Findlay LC. Early learning and child care for children aged 0 to 5 years: A provincial/territorial portrait [Internet]. Ottawa: Statistics Canada; 2019 Jul [cited 2021 Oct 8]. Report No.: Report No. 099. Cat No. 11–626-X. Available from: https://www150.statcan.gc.ca/n1/pub/11-626-x/11-626-x2019013-eng.htm

  12. Mazarello Paes V, Ong KK, Lakshman R. Factors influencing obesogenic dietary intake in young children (0–6 years): systematic review of qualitative evidence. BMJ Open. 2015;5(9):e007396.

    Article  PubMed  PubMed Central  Google Scholar 

  13. Larson N, Ward DS, Neelon SB, Story M. What Role Can Child-Care Settings Play in Obesity Prevention? A Review of the Evidence and Call for Research Efforts. J Am Diet Assoc. 2011;111(9):1343–62.

    Article  PubMed  Google Scholar 

  14. van de Kolk I, Verjans-Janssen SRB, Gubbels JS, Kremers SPJ, Gerards SMPL. Systematic review of interventions in the childcare setting with direct parental involvement: effectiveness on child weight status and energy balance-related behaviours. Int J Behav Nutr Phys Act. 2019;16(1):110.

    Article  PubMed  PubMed Central  Google Scholar 

  15. Wertlieb D. Nurturing care framework for inclusive early childhood development: opportunities and challenges. Dev Med Child Neurol. 2019;61(11):1275–80.

    Article  PubMed  Google Scholar 

  16. Lafave L, Tyminski S, Riege T, Hoy D, Dexter B. Content Validity for a Child Care Self-assessment Tool: Creating Healthy Eating Environments Scale (CHEERS). Can J Diet Pract Res. 2016;77:89–92.

    Article  PubMed  Google Scholar 

  17. Lafave LMZ. Creating a healthy eating and active environment survey (CHEERS) for childcare: an inter-rater, intra-rater reliability and validity study. BMC Public Health. 2019;19(1):1–9.

    Article  Google Scholar 

  18. Evans JR, Link to external site this link will open in a new window, Anil M. The value of online surveys: a look back and a look ahead. Internet Res. 2018;28(4):854–87.

    Article  Google Scholar 

  19. Zeleke AA, Naziyok T, Fritz F, Christianson L, Röhrig R. Data Quality and Cost-effectiveness Analyses of Electronic and Paper-Based Interviewer-Administered Public Health Surveys: Systematic Review. J Med Internet Res. 2021;23(1):e21382.

    Article  PubMed  PubMed Central  Google Scholar 

  20. Benjamin Neelon SE, Briley ME. Position of the American Dietetic Association: Benchmarks for Nutrition in Child Care. J Am Diet Assoc. 2011;111(4):607–15.

    Article  PubMed  Google Scholar 

  21. Lafave L. MOESM1 of Creating a healthy eating and active environment survey (CHEERS) for childcare: an inter-rater, intra-rater reliability and validity study. 2019 Oct; Available from: https://doi.org/10.6084/m9.figshare.10071512.v1

  22. Pozzar R, Hammer MJ, Underhill-Blazey M, Wright AA, Tulsky JA, Hong F, et al. Threats of Bots and Other Bad Actors to Data Quality Following Research Participant Recruitment Through Social Media: Cross-Sectional Questionnaire. J Med Internet Res. 2020;22(10):e23021.

    Article  PubMed  PubMed Central  Google Scholar 

  23. Streiner DL, Norman GR, Cairney J. Health Measurement Scales: a practical guide to their development and use. USA: Oxford University Press; 2015.

    Book  Google Scholar 

  24. Arifin WN. A Web-based Sample Size Calculator for Reliability Studies. Education in medicine journal. 2018 Sep 28;10(3).

  25. Walter SD, Eliasziw M, Donner A. Sample size and optimal designs for reliability studies. Stat Med. 1998;17(1):101–10.

    Article  CAS  PubMed  Google Scholar 

  26. Koo TK, Li MY. A Guideline of Selecting and Reporting Intraclass Correlation Coefficients for Reliability Research. J Chiropr Med. 2016;15(2):155–63.

    Article  PubMed  PubMed Central  Google Scholar 

  27. Shrout PE, Fleiss JL. Intraclass correlations: Uses in assessing rater reliability. Psychol Bull. 1979;86(2):420–8.

    Article  CAS  PubMed  Google Scholar 

  28. Trevethan R. Intraclass correlation coefficients: clearing the air, extending some cautions, and making some requests. Health Serv Outcomes Res Method. 2017;17(2):127–43.

    Article  Google Scholar 

  29. Qin S, Nelson L, McLeod L, Eremenco S, Coons SJ. Assessing test–retest reliability of patient-reported outcome measures using intraclass correlation coefficients: recommendations for selecting and documenting the analytical formula. Qual Life Res. 2019;28(4):1029–33.

    Article  PubMed  Google Scholar 

  30. Weir JP. Quantifying test-retest reliability using the intraclass correlation coefficient and the SEM. J Strength Con Res. 2005;19(1):231–40.

    Google Scholar 

  31. de Vet HCW, Terwee CB, Knol DL, Bouter LM. When to use agreement versus reliability measures. J Clin Epidemiol. 2006;59(10):1033–9.

    Article  PubMed  Google Scholar 

  32. de Vet HC, Terwee CB, Ostelo RW, Beckerman H, Knol DL, Bouter LM. Minimal changes in health status questionnaires: distinction between minimally detectable change and minimally important change. Health Qual Life Outcomes. 2006;22(4):54.

    Article  Google Scholar 

  33. Beckerman H, Roebroeck ME, Lankhorst GJ, Becher JG, Bezemer PD, Verbeek ALM. Smallest real difference, a link between reproducibility and responsiveness. Qual Life Res. 2001;10(7):571–8.

    Article  CAS  PubMed  Google Scholar 

  34. Huang SL, Hsieh CL, Wu RM, Tai CH, Lin CH, Lu WS. Minimal Detectable Change of the Timed “Up & Go” Test and the Dynamic Gait Index in People With Parkinson Disease. Phys Ther. 2011;91(1):114–21.

    Article  PubMed  Google Scholar 

  35. Kather F, Hadzic M, Hehle T, Eichler S, Klein J, Völler H, et al. Test-retest reliability of the Mini Nutritional Assessment-Short Form (MNA-SF) in older patients undergoing cardiac rehabilitation. J Geriatr Cardiol. 2020;17(9):574–9.

    PubMed  PubMed Central  Google Scholar 

  36. Anderson J, Manias E, Kusljic S, Finch S. Testing the validity, reliability and utility of the Self-Administration of Medication (SAM) tool in patients undergoing rehabilitation. Res Social Adm Pharm. 2014;10(1):204–16.

    Article  PubMed  Google Scholar 

  37. Grady A, Yoong S, Sutherland R, Lee H, Nathan N, Wolfenden L. Improving the public health impact of eHealth and mHealth interventions. Aust N Z J Public Health. 2018;42(2):118–9.

    Article  PubMed  Google Scholar 

  38. Enam A, Torres-Bonilla J, Eriksson H. Evidence-Based Evaluation of eHealth Interventions: Systematic Literature Review. J Med Internet Res. 2018;20(11):e10971.

    Article  PubMed  PubMed Central  Google Scholar 

  39. Kottner J, Audige L, Brorson S, Donner A, Gajewski BJ, Hroóbjartsson A, et al. Guidelines for Reporting Reliability and Agreement Studies (GRRAS) were proposed. Int J Nurs Stud. 2011;48(6):661–71.

    Article  PubMed  Google Scholar 

  40. Carducci B, Reesor M, Haresign H, Rysdale L, Keller H, Beyers J, et al. NutriSTEP® is Reliable for Internet and Onscreen Use. Can J Diet Pract Res. 2015;76(1):9–14.

    Article  PubMed  Google Scholar 

  41. Minard JP, Thomas NJ, Olajos-Clow JG, Wasilewski NV, Jenkins B, Taite AK, et al. Assessing the burden of childhood asthma: validation of electronic versions of the Mini Pediatric and Pediatric Asthma Caregiver’s Quality of Life Questionnaires. Qual Life Res. 2016;25(1):63–9.

    Article  PubMed  Google Scholar 

  42. Gwaltney CJ, Shields AL, Shiffman S. Equivalence of Electronic and Paper-and-Pencil Administration of Patient-Reported Outcome Measures: A Meta-Analytic Review. Value in Health. 2008;11(2):322–33.

    Article  PubMed  Google Scholar 

  43. Wakefield BJ, Turvey CL, Nazi KM, Holman JE, Hogan TP, Shimada SL, et al. Psychometric Properties of Patient-Facing eHealth Evaluation Measures: Systematic Review and Analysis. J Med Internet Res. 2017;19(10):e346.

    Article  PubMed  PubMed Central  Google Scholar 

  44. Keszei AP, Novak M, Streiner DL. Introduction to health measurement scales. J Psychosom Res. 2010;68(4):319–23.

    Article  PubMed  Google Scholar 

  45. van Kollenburg RA, de Bruin DM, Wijkstra H. Validation of the Electronic Version of the International Index of Erectile Function (IIEF-5 and IIEF-15): A Crossover Study. J Med Internet Res. 2019;21(7):e13490.

    Article  PubMed  PubMed Central  Google Scholar 

  46. Statistics Canada. Patterns of participation in early learning and child care among families with potential socioeconomic disadvantages in Canada. 2021 [cited 2021 Oct 6]; Available from: https://www150.statcan.gc.ca/n1/pub/36-28-0001/2021008/article/00002-eng.htm

  47. Cumming T, Logan H, Wong S. A critique of the discursive landscape: Challenging the invisibility of early childhood educators’ well-being. Contemp Issues Early Child. 2020;21(2):96–110.

    Article  Google Scholar 

  48. Nislin MA, Sajaniemi NK, Sims M, Suhonen E, Maldonado Montero EF, Hirvonen A, et al. Pedagogical work, stress regulation and work-related well-being among early childhood professionals in integrated special day-care groups. Eur J Spec Needs Educ. 2016;31(1):27–43.

    Article  Google Scholar 

Download references

Acknowledgements

Appreciation to Alberta Health Services, Population and Public Nutrition for their support of this project. Thanks to all the directors and educators in childcare centres throughout Alberta and Mount Royal University student research assistants.

Funding

The CHEERS study was funded by the Mount Royal University Research Grant Fund and the Faculty of Health, Community, and Education Innovation Fund. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Author information

Authors and Affiliations

  1. Department of Health and Physical Education, Mount Royal University, Calgary, AB, T3E 6K6, Canada

    Lynne M. Z. Lafave

Authors
  1. Lynne M. Z. Lafave
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

LL conceptualized the project idea; research design; primary data collection; executed reliability and validation analyses; drafted the manuscript, edited, read, and approved the final manuscript.

Corresponding author

Correspondence to Lynne M. Z. Lafave.

Ethics declarations

Ethics approval and consent to participate

This study was approved by the Mount Royal University Human Research Ethics Board (#100016). ECEC educators and directors were the participants in this study. ECEC educators and directors were fully informed about the purpose and procedures of the study and provided written or online consent for participation. No minors were involved in this study.

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Lafave, L.M.Z. Is the online ‘creating healthy eating and active environments survey' (CHEERS) tool reliable for early childhood educators in Alberta, Canada: a randomized crossover trial. Arch Public Health 81, 22 (2023). https://doi.org/10.1186/s13690-023-01036-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s13690-023-01036-z

Keywords

Archives of Public Health

ISSN: 2049-3258