Barriers and facilitators to answering clinical questions in the Americas: a cross-sectional study of surgical trauma care providers
====================================================================================================================================

* Helen Elizabeth Noble
* Felipe Vega Rivera
* Lacey LaGrone

## Abstract

**Background** We aimed to understand how surgical trauma providers in the Americas acquire answers to clinical questions and what barriers and facilitators they face in efforts to practice according to recommendations for common surgical cases. We hypothesized that increased English proficiency and country income improved providers’ acquisition and application of clinical knowledge.

**Methods** A 23-question survey evaluated reported confidence in interpretation of evidence, perceived language fluency, and access to and application of recommendations on sepsis and appendicitis. Electronic surveys were distributed across the Americas to Pan American Trauma Society members.

**Results** 108 participants from 21 countries completed this survey. 59% had ≥21 years of provider experience. 38% reported their English reading comprehension as less than or equal to “limited working proficiency.” 44% endorsed using Google Translate; 35% reported they did not need translation tools to evaluate medical literature. 59% felt uncertainty regarding clinical care at least weekly. 65% reported inability to answer their clinical questions at least once per month. 86% felt confident in their ability to interpret and apply evidence for their practice. To answer clinical questions, participants listed guidelines (76%), full-text peer-reviewed journal articles (61%), and meta-analyses (49%) as their most used resources. 25% answered all five clinical questions correctly, whereas 43% answered three or fewer correctly. 79% felt they had adequate access to resources to answer the five clinical questions. When controlling for individual demographic characteristics, decreased age (p<0.01) and increased country income level (p=0.03) positively impacted correct answers to questions.

**Discussion** Uncertainties in clinical care are unavoidable. Language, age, and country income level impacted provider acquisition and application of knowledge relevant to select clinical scenarios. These findings highlight disparities in access and training and add urgency to the movement for improved dissemination and implementation approaches for evidence-based practice in surgery.

**Level of evidence** IV.

*   education
*   medical
*   practice guideline

## Introduction

Nearly one-third of all deaths worldwide occur due to conditions potentially treated with surgical care.1 Many surgical systems across resource settings lack the necessary tools, including those required to advance human resources through continuous education, to achieve the best outcomes.2 The Lancet Commission on Global Surgery prioritizes high-quality, safe care2 as a crucial part of surgical systems, yet one USA-based analysis measuring quality of care in various healthcare settings found that only 50% to 60% of surgical patients receive quality, evidence-based care.3

Clinical practice that is based on evidence may be described to follow the sequence “ask, acquire, appraise, apply, assess.”4 Barriers arise from the very start of this process. The proven method to frame vague questions into researchable phrases using the mnemonic “population, intervention, comparison, outcome” (PICO) is not widespread in provider groups globally.5 Providers across resource settings also feel stuck at the “acquire” stage in evidence-based practice.6 To improve practical acquisition of evidence-based answers to clinical questions, the Lancet Commission on Global Surgery promoted standardized clinical protocols using the most updated evidence.2 However, cited reasons why providers do not use guidelines include lack of adequate dissemination,7 awareness, familiarity, or agreement on guidelines.8 Some practitioners feel that guidelines slow their practice or do not apply to low-resource settings.8 Institutions pay millions of dollars per year for access to research behind paywalls, and individuals pay on average $30 per paper.9 Even for publicly funded research, paywalls exist for the first year after publication, slowing the already sluggish pace of scientific progress.10 To push for faster access to scientific results, the US government drafted policies requiring immediate public access to papers funded by taxpayers’ dollars, but academic publishers effectively lobbied against the proposed policies.11 In 2016, only 19% of published articles were immediately available on publication.12 A new plan in 11 European countries, known as “Plan S,” aims to increase the number of articles immediately available on publication by requiring that all research with grants from European research councils or funding bodies publish directly to a public-access forum.12 13

Even as the scientific community pushes for public access, such as that provided through “Plan S,” many providers receive no training in the “appraise” portion of the evidence-based framework. The “appraise” stage of evidence-based practice requires financial resources, expertise, and time. In a 2010 study, only 33% of surgeons used primary literature to inform their practice; lack of education in appraising primary literature was described as a primary barrier.14 Most initiatives for education on evidence-based practice require hours of training,15 16 further preventing already time-strapped surgeons from acquiring the skills required to appraise literature.17 As one attempt to overcome these barriers, evidence-based search engines may provide predigested answers, providing an effective “short-cut” when primary literature is inaccessible or too dense to quickly appraise. The American College of Surgeons promotes the global use of these search engines,18 yet many surgeons still lack access to all papers referenced by the search engines and instead use abstracts to guide their practice.19 Aside from one study reporting that cost and language were barriers to practice of evidence-based medicine (EBM),6 very few data exist related to sociodemographic factors related to EBM or using these search engines.

Without adequate support for “acquiring” and “appraising,” the flow of evidence-based practice ends before “application” and “assessment,” never reaching the patient. This study aimed to better understand influencing factors on surgical trauma providers in the Americas as they acquire answers to clinical questions and to assess their level of confidence in appraising that information. We hypothesized that increased English proficiency and country income level would improve providers’ acquisition and application of clinical knowledge.

## Patients and methods

For this cross-sectional study we partnered with the Pan American Trauma Society (PTS), an international professional society of trauma providers across the full spectrum of education and clinical care, who collaborate to improve surgical trauma systems throughout the Americas.20 We used their listserv and Facebook account to electronically invite all 619 current members to anonymously participate in this survey (online supplemental appendix A). We partnered with a surgeon (FV) with a history of leadership within the PTS to design the approach and promote participation. The survey was distributed to participants to choose between English, Spanish, and Portuguese. The 23 multiple-choice question survey asked about inperson, paper, and electronic resources most often used when answering clinical questions (online supplemental file 1). It queried participant perception of language barriers including English reading comprehension and quality of translation resources (online supplemental file 2). Finally, the survey included five knowledge questions designed to assess participants’ ability to acquire information on common and life-saving clinical scenarios found in international guidelines for sepsis and appendicitis, relevant across resource settings and updated in the past 4 years (table 1, online supplemental appendix A).21 The survey link was available through REDCap (Research Electronic Data Capture) until over 100 participants completed the survey.

### Supplementary data

[[tsaco-2021-000774supp003.pdf]](pending:yes)

### Supplementary data

[[tsaco-2021-000774supp001.pdf]](pending:yes)

### Supplementary data

[[tsaco-2021-000774supp002.pdf]](pending:yes)

View this table:
[Table 1](http://tsaco.bmj.com/content/6/1/e000774/T1)

Table 1 
Knowledge questions included in the survey

Each participant’s knowledge score was the sum of their correct answers to knowledge questions. STATA, version 7 (StataCorp, 2015, College Station, TX) was used for data analysis. All variables of interest were ordinal categorical variables. For simplicity in reporting and interpreting our results, we used linear regression with robust SE to test for association between variables.22 In a post-hoc analysis, ordinal logistic regression resulted in the same statistical inference. In univariate analyses, a single independent variable was included in the regression model. In the multivariate analysis, the dependent variable was knowledge score, and the independent variables were selected a priori: age, country income level, English reading comprehension, translation quality, frequency of unanswerable clinical questions, frequency of uncertainty regarding clinical practice, and access to effective resources. For each variable of interest, we report a crude (univariate) and adjusted (multivariate) regression coefficient and 95% CI. For analyses related to knowledge score, the regression coefficient can be interpreted as the average change in knowledge score (ie, 1–5) associated with a one-unit increase in the independent variable. Statistical significance was set at p<0.05.

## Results

### Demographics

One hundred and eight participants from 21 countries finished the survey (online supplemental file 1). Most participants were general surgeons over 45 years old (67%) in upper-middle-income countries (U-MICs) (74%)—as compared with 44% of countries in Latin America and the Caribbean listed as U-MICs (World Bank)—working at urban public hospitals (78%) with fewer than 300 beds (69%)—as a reference, an urban tertiary public hospital in Lima has 452 beds (table 2).

View this table:
[Table 2](http://tsaco.bmj.com/content/6/1/e000774/T2)

Table 2 
Demographic information

### Language

Of the participants, 38% reported their English reading comprehension as less than or equal to “limited working proficiency,” 51% reported “professional proficiency,” and 11% were native English speakers. Lower English reading comprehension was associated with a decreased self-reported confidence in interpreting and applying medical literature to clinical practice (p<0.01). In selecting multiple ways to translate clinical resources, 44% of the participants used Google Translate, whereas 31% reported they did not need translation services and 35% reported understanding the language of the published research. Other participants (8%) used colleagues or other translation services for help. Of the participants, 19% reported language competency as a barrier to evidence-based clinical practice.

### Resources

Of the participants, 91% self-reported having uncertainty at least monthly about any aspect of clinical care they provide, with the remainder denying ever having clinical uncertainty (table 3).

View this table:
[Table 3](http://tsaco.bmj.com/content/6/1/e000774/T3)

Table 3 
Comparison of demographics and knowledge scores for participants with and without clinical uncertainty

Most participants (66%) reported they were unable to answer a clinical question at least monthly using all available resources. When asked to select their top three resources for addressing uncertainty in clinical care, 76% chose guidelines, 61% chose peer-reviewed journal articles, 49% chose meta-analyses, and 42% chose point-of-care information summaries.

For the clinical questions tested within the survey instrument, 79% felt they had adequate resources to answer correctly. When participants selected multiple options, the top three resources used to answer the clinical questions imbedded in the survey instrument were similar for sepsis and appendicitis: guidelines (45%, 44%), peer-reviewed journal articles (37%, 35%), and clinical point-of-care summaries (26%, 28%).

### Confidence and knowledge

Although most participants (86%) felt adequately confident in applying literature to their clinical practice, most (75%) answered at least one out of five clinical questions incorrectly on the survey (online supplemental appendix A). Participants reported their average weekly patient load, and all those who answered at least one survey knowledge question incorrectly may be estimated to collectively care for anywhere from 4795 to >7672 patients per week (table 4).

View this table:
[Table 4](http://tsaco.bmj.com/content/6/1/e000774/T4)

Table 4 
Knowledge questions and percent of participants who answered each individually correctly

In univariate regression analysis, higher country income level and higher reported English reading comprehension were significantly associated with higher knowledge scores (p=0.002 and p=0.003, respectively). A higher self-reported ability to interpret and apply evidence to clinical practice was significantly associated with a decreased knowledge score (p=0.014). Other variables of interest were not statistically significant (table 5).

View this table:
[Table 5](http://tsaco.bmj.com/content/6/1/e000774/T5)

Table 5 
Univariate analysis of variables of interest in association with knowledge score

After adjusting for the variables listed in table 6, age and country income level were significantly associated with lower and higher knowledge score, respectively. As age increases between predefined categories (online supplemental appendix A), knowledge scores decreased by 0.31 of the total 5 points (p<0.006). Practicing in a high-income country as compared with a low-middle-income country was associated with knowledge scores that were higher by 0.45 (p=0.026) on average (table 6). Feelings of adequate access to resources, English reading comprehension, quality of translation services, and confidence in interpreting and applying evidence to clinical practice were not significantly associated with changes in knowledge score in multivariate analysis.

View this table:
[Table 6](http://tsaco.bmj.com/content/6/1/e000774/T6)

Table 6 
Multivariate analysis of variables of interest in association with knowledge score

### Future learning

Many participants reported interest in learning to conduct research to answer clinical questions (67%) and in learning to appraise and apply existing evidence to their clinical practice (73%).

## Discussion

This study set out to understand how surgical trauma providers in the Americas acquire and appraise information to properly inform their clinical practice. Overall, these providers felt they had adequate resources to address their clinical uncertainties and were confident in their ability to interpret and apply those resources to their clinical practice. However, most reported difficulty resolving clinical questions even when using all resources available to them and answered at least one survey knowledge question incorrectly. Although there will always be room for growth in clarification of clinical issues, and the available literature may not provide direction in complex individual patient scenarios, we also captured a significant degree of uncertainty regarding more discrete issues which do have relevant, publicly available, recommendations based on a strong evidence base.

Interestingly, higher reported confidence in interpreting and applying literature to clinical practice correlated to lower knowledge score in univariate but not multivariate analysis. Previous studies parallel our findings, underlining the Dunning-Kruger effect describing the cognitive bias in which people overestimate their medical knowledge and ability.23 24

Participants perceived their translation services, when required, to be sufficient for clinical practice. This mirrors prior research on Google Translate working well in medical settings.25 However, participants with lower reported English reading comprehension also reported lower confidence in interpreting and applying literature to their clinical practice, showing that even with the use of translation services, language persists as a barrier to answering life-saving clinical questions. One in five providers felt that language prevented them from practicing medicine using evidence. Language competency has been previously reported to be an explicit barrier to evidence-based care6 and is here seen to undermine provider confidence in evidence interpretation.

In this analysis, when holding other variables constant, increased age correlated with decreased score on the included knowledge assessment. Although experience likely improves various other skills relevant to the provider, previous research also connects more years in practice with decreased adherence to current standards of practice and therapies and decreased knowledge.26 More senior providers are further from medical education and may not be exposed to practice updates outside of the academic setting. In some reports, age correlates with lower rates of technological literacy and thus potentially difficulty accessing guidelines related to our survey knowledge questions.27 The authors recognize that endorsement of and adherence to EBM is only one part of clinical care. These data do not capture other important components of clinical decision-making, including judgment, cognitive biases, collaborative decision-making, and other features of clinical care that may have different trends throughout providers’ careers. Providers may depend on their early formal clinical training years to acquire clinical knowledge, and then have limited and compromised options for new knowledge acquisition throughout their career. Further research is indicated on customized interventions to address barriers to EBM unique to various groups.

Country income level also significantly influenced knowledge scores, when holding other variables constant, yet all answers to the survey knowledge questions were available in well-established and public-access online guidelines. This confirms the complexity of the socioeconomic barriers to evidence-based practice. These data do not capture reasons for higher knowledge scores in high-income countries, and future research could focus on underlying factors. Most participants reported interest in learning more about research and evidence-based practice, indicating that providers recognize the value of, and would appreciate more focus on, these topics.

### Limitations

This voluntary study may over-represent surgical trauma providers from a specific demographic engaged in an academic society who are interested in participating in electronic surveys and the topic of evidence-based practice. Also, our data may not fully represent Latin America and the Caribbean, as more participants reported working in U-MICs as compared with the World Bank distribution of Latin American and Caribbean country income level. Our survey was not previously validated, as the authors did not find existing validated tools to measure our outcomes. Participants were not asked about their level of experience in clinical scenarios for each knowledge question; we thus cannot comment on the relationship between experience with a particular clinical phenomenon and knowledge scores. Although instructed, some participants may not have used all resources available to them. Finally, and likely most importantly, although we wrote survey knowledge questions for applicability across resource settings and included input from local coauthors, our results may reflect uncaptured distinctions in acceptable practice pattern that varies by country income level. This approach mirrors and highlights the well-described inequity not only in research consumption, but in production. Most guidelines are based on data gathered from populations that do not represent global diversity, and then those data are further collated and reviewed by practitioners who similarly do not represent the global population of providers.

## Conclusions

Most surveyed surgical trauma providers felt confident in their evidence-based practice yet do not answer all clinical questions correctly even in an “open book” assessment. Lower country income level and higher age of provider corresponded to fewer clinical questions answered in congruence with international guidelines. Almost one out of every five participants felt their English competency was a barrier to evidence-based practice. These data highlight the broken, and even sometimes absent, dissemination and implementation schemes for new, effective interventions—many of these interventions only available in English.

Our participants were motivated to expand their knowledge of EBM and research, indicating a disconnect between opportunity and desire for practicing using the latest recommendations. Equity in healthcare necessitates overcoming linguistic barriers so all providers can use evidence-based research written in any language to drive their clinical practice. Furthermore, the logistical, resource, and behavioral barriers that contribute to an age and resource gap in clinical knowledge scores merit further research and ultimately customized intervention.

Every single patient depends on their provider to adequately acquire and appraise new data and practice EBM. In surgical trauma settings, lives are often at stake. These data underscore the urgency to holistically improve global trauma systems by investing in continuous education and building accurate, rapid language translation to improve dissemination and implementation of evidence-based practice.

## Data availability statement

Data are available upon reasonable request. All data relevant to the study are included in the article or uploaded as supplementary information. Data availability is not applicable as this is not a clinical trial.

## Ethics statements

### Patient consent for publication

Not required.

### Ethics approval

The international review boards at the University of Washington approved this study under study numbers 00009303 and 00008216 and Universidad Peruana Cayetano Heredia under SIDISI 201558. Participants provided verbal informed consent prior to their participation.

## Acknowledgments

Bruna Dantos Lobato translated the survey from English to Spanish and Portuguese. Administrators at the Pan American Trauma Society, Ms Gladys Shanklin and Dr Pablo Ottolino, helped distribute the survey. Dr Megan Branda and Dr Ravi Sood advised on statistical analysis.

## Footnotes

*   Contributors HEN and LL each contributed substantially to the conception, planning, and analysis of this work. HEN also conducted the study and drafted the report. FVR contributed significantly to planning and reporting this work.

*   Funding The research reported in this publication was supported by the Fogarty International Center (FIC) of the National Institutes of Health under grant number D43TW009345 awarded to the Northern Pacific Global Health Fellows Program.

*   Disclaimer The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.

*   Competing interests None declared.

*   Provenance and peer review Not commissioned; externally peer reviewed.

[http://creativecommons.org/licenses/by-nc/4.0/](http://creativecommons.org/licenses/by-nc/4.0/)
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: [http://creativecommons.org/licenses/by-nc/4.0/](http://creativecommons.org/licenses/by-nc/4.0/).

## References

1.  Lozano R, Naghavi M, Foreman K, Lim S, Shibuya K, Aboyans V, Abraham J, Adair T, Aggarwal R, Ahn SY, et al. Global and regional mortality from 235 causes of death for 20 age groups in 1990 and 2010: a systematic analysis for the global burden of disease study 2010. Lancet 2012;380:2095–128.[doi:10.1016/S0140-6736(12)61728-0](http://dx.doi.org/10.1016/S0140-6736(12)61728-0)
    
    [CrossRef](http://tsaco.bmj.com/lookup/external-ref?access_num=10.1016/S0140-6736(12)61728-0&link_type=DOI) 
    
    [PubMed](http://tsaco.bmj.com/lookup/external-ref?access_num=23245604&link_type=MED&atom=%2Ftsaco%2F6%2F1%2Fe000774.atom) 
    
    [Web of Science](http://tsaco.bmj.com/lookup/external-ref?access_num=000312387000012&link_type=ISI) 

2.  Meara JG, Leather AJM, Hagander L, Alkire BC, Alonso N, Ameh EA, Bickler SW, Conteh L, Dare AJ, Davies J, et al. Global surgery 2030: evidence and solutions for achieving health, welfare, and economic development. The Lancet 2015;386:569–624.[doi:10.1016/S0140-6736(15)60160-X](http://dx.doi.org/10.1016/S0140-6736(15)60160-X)
    
    

3.  Schuster MA, McGlynn EA, Brook RH. How good is the quality of health care in the United States. The Milbank Quarterly 1998;4:517–63 [https://onlinelibrary.wiley.com/toc/14680009/76/4](https://onlinelibrary.wiley.com/toc/14680009/76/4).[doi:10.1111/1468-0009.00105](http://dx.doi.org/10.1111/1468-0009.00105)
    
    

4.  Rousseau DM, Gunia BC. Evidence-Based practice: the psychology of EBP implementation. Annu Rev Psychol 2016;67:667–92.[doi:10.1146/annurev-psych-122414-033336](http://dx.doi.org/10.1146/annurev-psych-122414-033336)
    
    

5.  Stillwell SB, Fineout-Overholt E, Melnyk BM, Williamson KB. Asking the clinical question: a key step in evidence-based practice. Am J Nurs 2010;110:58–61.[doi:10.1097/01.NAJ.0000368959.11129.79](http://dx.doi.org/10.1097/01.NAJ.0000368959.11129.79)
    
    [PubMed](http://tsaco.bmj.com/lookup/external-ref?access_num=20736714&link_type=MED&atom=%2Ftsaco%2F6%2F1%2Fe000774.atom) 

6.  LaGrone LN, Fuhs AK, Egoavil EH, Langdale LA, Fuangworawong P, Hamasaki JL, Gyedu A, Mock CN. A global assessment of access to and use of medical information: the state of evidence-based surgery. World J Surg 2018;42:521–31.[doi:10.1007/s00268-017-4175-4](http://dx.doi.org/10.1007/s00268-017-4175-4)
    
    

7.  Pang P. Clinical practice guideline dissemination and a new approach using Haddon matrix as a conceptual framework of evidence-based implementation strategies. World J Emerg Med 2010;1:6–11.
    
    

8.  Cabana MD, Rand CS, Powe NR, Wu AW, Wilson MH, Abboud PA, Rubin HR. Why don't physicians follow clinical practice guidelines? A framework for improvement. JAMA 1999;282:1458–67.[doi:10.1001/jama.282.15.1458](http://dx.doi.org/10.1001/jama.282.15.1458)
    
    [CrossRef](http://tsaco.bmj.com/lookup/external-ref?access_num=10.1001/jama.282.15.1458&link_type=DOI) 
    
    [PubMed](http://tsaco.bmj.com/lookup/external-ref?access_num=10535437&link_type=MED&atom=%2Ftsaco%2F6%2F1%2Fe000774.atom) 
    
    [Web of Science](http://tsaco.bmj.com/lookup/external-ref?access_num=000083111000032&link_type=ISI) 

9.  Zhang S. The Real Cost of Knowledge. The Atlantic. 2020. [https://www.theatlantic.com/science/archive/2019/03/uc-elsevier-publisher/583909/](https://www.theatlantic.com/science/archive/2019/03/uc-elsevier-publisher/583909/).
    
    

10. Morris ZS, Wooding S, Grant J. The answer is 17 years, what is the question: understanding time lags in translational research. J R Soc Med 2011;104:510–20.[doi:10.1258/jrsm.2011.110180](http://dx.doi.org/10.1258/jrsm.2011.110180)
    
    [CrossRef](http://tsaco.bmj.com/lookup/external-ref?access_num=10.1258/jrsm.2011.110180&link_type=DOI) 
    
    [PubMed](http://tsaco.bmj.com/lookup/external-ref?access_num=22179294&link_type=MED&atom=%2Ftsaco%2F6%2F1%2Fe000774.atom) 

11. Dyer O. Academic publishers urge Trump not to demand open access for federally funded research. BMJ 2019;367:l7064.[doi:10.1136/bmj.l7064](http://dx.doi.org/10.1136/bmj.l7064)
    
    [FREE Full Text](http://tsaco.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiRlVMTCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjE3OiIzNjcvZGVjMjBfNi9sNzA2NCI7czo0OiJhdG9tIjtzOjIzOiIvdHNhY28vNi8xL2UwMDA3NzQuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

12. Haug CJ. No Free Lunch - What Price Plan S for Scientific Publishing? N Engl J Med 2019;380:1181–5.[doi:10.1056/NEJMms1900864](http://dx.doi.org/10.1056/NEJMms1900864)
    
    

13. Plan S. Making full and immediate Open Access. 2020. [coalition-s.org](http://coalition-s.org).
    
    

14. Mittal R, Perakath B. Evidence-Based surgery: knowledge, attitudes, and perceived barriers among surgical trainees. J Surg Educ 2010;67:278–82.[doi:10.1016/j.jsurg.2010.06.012](http://dx.doi.org/10.1016/j.jsurg.2010.06.012)
    
    [CrossRef](http://tsaco.bmj.com/lookup/external-ref?access_num=10.1016/j.jsurg.2010.06.012&link_type=DOI) 
    
    [PubMed](http://tsaco.bmj.com/lookup/external-ref?access_num=21035766&link_type=MED&atom=%2Ftsaco%2F6%2F1%2Fe000774.atom) 

15. Ubbink DT, Legemate DA, Koelemay MJ. The merits of a two-day evidence-based medicine course for surgical residents. World J Surg 2016;40:1809–14.[doi:10.1007/s00268-016-3495-0](http://dx.doi.org/10.1007/s00268-016-3495-0)
    
    

16. Fritsche L, Greenhalgh T, Falck-Ytter Y, Neumayer H-H, Kunz R. Do short courses in evidence based medicine improve knowledge and skills? validation of Berlin questionnaire and before and after study of courses in evidence based medicine. BMJ 2002;325:1338–41.[doi:10.1136/bmj.325.7376.1338](http://dx.doi.org/10.1136/bmj.325.7376.1338)
    
    [Abstract/FREE Full Text](http://tsaco.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjEzOiIzMjUvNzM3Ni8xMzM4IjtzOjQ6ImF0b20iO3M6MjM6Ii90c2Fjby82LzEvZTAwMDc3NC5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

17. Kyriakoulis K, Patelarou A, Laliotis A, Wan AC, Matalliotakis M, Tsiou C, Patelarou E. Educational strategies for teaching evidence-based practice to undergraduate health students: systematic review. J Educ Eval Health Prof 2016;13:34. [doi:10.3352/jeehp.2016.13.34](http://dx.doi.org/10.3352/jeehp.2016.13.34)
    
    

18. American College of Surgeons. Evidence-Based Medicine. 2020. [https://www.facs.org/quality-programs/about/cqi/internetresources/ebm](https://www.facs.org/quality-programs/about/cqi/internetresources/ebm).
    
    

19. Waltho D, Kaur MN, Haynes RB, Farrokhyar F, Thoma A. Users' guide to the surgical literature: how to perform a high-quality literature search. Can J Surg 2015;58:349–58.[doi:10.1503/cjs.017314](http://dx.doi.org/10.1503/cjs.017314)
    
    

20. Pan American Trauma Society. To improve trauma care in the Americas. 2020. [http://www.panamtrauma.org/history-PTS](http://www.panamtrauma.org/history-PTS).
    
    

21. Rhodes A, Evans LE, Alhazzani W, Levy MM, Antonelli M, Ferrer R, Kumar A, Sevransky JE, Sprung CL, Nunnally ME, et al. Surviving sepsis campaign: international guidelines for management of sepsis and septic shock: 2016. Crit Care Med 2017;45:486–552.[doi:10.1097/CCM.0000000000002255](http://dx.doi.org/10.1097/CCM.0000000000002255)
    
    [CrossRef](http://tsaco.bmj.com/lookup/external-ref?access_num=10.1097/CCM.0000000000002255&link_type=DOI) 
    
    [PubMed](http://tsaco.bmj.com/lookup/external-ref?access_num=28098591&link_type=MED&atom=%2Ftsaco%2F6%2F1%2Fe000774.atom) 

22. Norris CM, Ghali WA, Saunders LD, Brant R, Galbraith D, Faris P. APPROACH Investigators. Ordinal regression model and the linear regression model were superior to the logistic regression models. J Clin Epidemiol 2006;59:448–56.
    
    [CrossRef](http://tsaco.bmj.com/lookup/external-ref?access_num=10.1016/j.jclinepi.2005.09.007&link_type=DOI) 
    
    [PubMed](http://tsaco.bmj.com/lookup/external-ref?access_num=16632132&link_type=MED&atom=%2Ftsaco%2F6%2F1%2Fe000774.atom) 
    
    [Web of Science](http://tsaco.bmj.com/lookup/external-ref?access_num=000237481600002&link_type=ISI) 

23. Kruger J, Dunning D. Unskilled and unaware of it: how difficulties in recognizing one's own incompetence lead to inflated self-assessments. J Pers Soc Psychol 1999;77:1121–34.[doi:10.1037/0022-3514.77.6.1121](http://dx.doi.org/10.1037/0022-3514.77.6.1121)
    
    [CrossRef](http://tsaco.bmj.com/lookup/external-ref?access_num=10.1037/0022-3514.77.6.1121&link_type=DOI) 
    
    [PubMed](http://tsaco.bmj.com/lookup/external-ref?access_num=10626367&link_type=MED&atom=%2Ftsaco%2F6%2F1%2Fe000774.atom) 
    
    [Web of Science](http://tsaco.bmj.com/lookup/external-ref?access_num=000084208700002&link_type=ISI) 

24. Rahmani M. Medical Trainees and the Dunning-Kruger Effect: When They Don't Know What They Don't Know. J Grad Med Educ 2020;12:532–4.[doi:10.4300/JGME-D-20-00134.1](http://dx.doi.org/10.4300/JGME-D-20-00134.1)
    
    [PubMed](http://tsaco.bmj.com/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Ftsaco%2F6%2F1%2Fe000774.atom) 

25. Balk EM, Mei C, Chen ML, Trikalinos TA, Chang LKW. Assessing the accuracy of Google translate to allow data extraction from trials published in non-English languages. AHRQ 2013.[doi:12(13)-EHC145-EF](http://dx.doi.org/12(13)-EHC145-EF)
    
    

26. Choudhry NK, Fletcher RH, Soumerai SB. Systematic review: the relationship between clinical experience and quality of health care. Ann Intern Med 2005;142:260–73.[doi:10.7326/0003-4819-142-4-200502150-00008](http://dx.doi.org/10.7326/0003-4819-142-4-200502150-00008)
    
    [CrossRef](http://tsaco.bmj.com/lookup/external-ref?access_num=10.7326/0003-4819-142-4-200502150-00008&link_type=DOI) 
    
    [PubMed](http://tsaco.bmj.com/lookup/external-ref?access_num=15710959&link_type=MED&atom=%2Ftsaco%2F6%2F1%2Fe000774.atom) 
    
    [Web of Science](http://tsaco.bmj.com/lookup/external-ref?access_num=000227239500004&link_type=ISI) 

27. Smith A. Older adults and technology use. Pew Research Center: Internet & Technology. 2020. [https://www.pewresearch.org/internet/2014/04/03/older-adults-and-technology-use/](https://www.pewresearch.org/internet/2014/04/03/older-adults-and-technology-use/).
    
    

28. Dünser MW, Takala J, Ulmer H, Mayr VD, Luckner G, Jochberger S, Daudel F, Lepper P, Hasibeder WR, Jakob SM. Arterial blood pressure during early sepsis and outcome. Intensive Care Med 2009;35:1225–33.[doi:10.1007/s00134-009-1427-2](http://dx.doi.org/10.1007/s00134-009-1427-2)
    
    [CrossRef](http://tsaco.bmj.com/lookup/external-ref?access_num=10.1007/s00134-009-1427-2&link_type=DOI) 
    
    [PubMed](http://tsaco.bmj.com/lookup/external-ref?access_num=19189077&link_type=MED&atom=%2Ftsaco%2F6%2F1%2Fe000774.atom) 

29. Di Saverio S, Birindelli A, Kelly MD, Catena F, Weber DG, Sartelli M, Sugrue M, De Moya M, Gomes CA, Bhangu A, et al. WSES Jerusalem guidelines for diagnosis and treatment of acute appendicitis. World J Emerg Surg 2016;11.[doi:10.1186/s13017-016-0090-5](http://dx.doi.org/10.1186/s13017-016-0090-5)
    
    

30. Abdulhamid AK, Sarker S-J. Is abdominal drainage after open emergency appendectomy for complicated appendicitis beneficial or waste of money? a single centre retrospective cohort study. Ann Med Surg 2018;36:168–72.[doi:10.1016/j.amsu.2018.10.040](http://dx.doi.org/10.1016/j.amsu.2018.10.040)