ABSTRACT

The COVID-19 pandemic has created an abrupt shift in the traditional delivery of Emergency Medicine (EM) residency conferences. The use of digital platforms, like Academic Life in Emergency Medicine (ALiEM) Remote, has eased the transition to a virtual platform for medical education. AliEM Remote supports educators by offering a curated speaker list, launch resources, and a shared conference calendar to help programs coordinate virtual sessions. In this descriptive observational study, we used the Airtable platform to create a form that collected speaker and residency conference information, and Google Analytics to collect and store data on website traffic between March 10, 2020 and April 28, 2020 which coincided with the transition from in-person to virtual educational engagement. During this period, the website received a total of 1,852 unique users, with the majority being located in the United States (79%). Ultimately, ALiEM compiled a list of 162 speakers from 54 institutions across the globe with a wide range of expertise in the areas of public policy, critical care, and pediatrics, among others. Engagement data suggest early utilization of the speaker list for conference planning with the residency program hosting 11 speakers sourced from the remote platform. However, no learner-level or educational outcome data were collected. This repository of virtual speakers and educational resources offers an early-stage model for recruiting speakers for future virtual conferences.

BACKGROUND

The COVID-19 pandemic disrupted in-person Emergency Medicine (EM) residency conferences, as shelter-in-place and social distancing policies led to widespread cancellation, postponement, or restructuring of scheduled sessions ^{[1]Using Technology to Maintain the Education of Residents During the COVID-19 Pandemic. J Surg Educ. 2020 Jul-Aug;77(4):729-732. doi: 10.1016/j.jsurg.2020.03.018. Epub 2020 Apr 3. PMID: 32253133; PMCID: PMC7270491.,[2]The #ASRASpring20 conference was canceled due to COVID-19 but the science survives and thrives. Reg Anesth Pain Med. 2021 Apr;46(4):374-375. doi: 10.1136/rapm-2020-101645. Epub 2020 May 13. PMID: 32409515.,[3]Letter: Academic Neurosurgery Department Response to COVID-19 Pandemic: The University of Miami/Jackson Memorial Hospital Model. Neurosurgery. 2020 Jul 1;87(1):E63-E65. doi: 10.1093/neuros/nyaa118. PMID: 32277754; PMCID: PMC7184381.}. Residency programs shifted to distance learning to comply with these constraints, requiring educators to figure out how to provide content remotely, including filling conference time previously reserved for in-person teaching such as group simulation and ultrasound sessions ^{[4]Experimenting with new paradigms for medical education and the emergence of a distance learning degree using the internet: teaching evidence-based medicine. Med Inform Internet Med. 2002 Mar;27(1):1-11. doi: 10.1080/14639230110105301. PMID: 12509118.,[5]Virtual Learning During the COVID-19 Pandemic: A Disruptive Technology in Graduate Medical Education. J Am Coll Cardiol. 2020 May 26;75(20):2635-2638. doi: 10.1016/j.jacc.2020.04.015. Epub 2020 Apr 15. PMID: 32304797; PMCID: PMC7159871.,[6]Mitigating the impact of conference and travel cancellations on researchers' futures. Elife. 2020 Mar 27;9:e57032. doi: 10.7554/eLife.57032. PMID: 32216872; PMCID: PMC7101232.}. As our knowledge of remote learning expands, adaptations should be made to support this growing new methodology in medical education.^{[4]Experimenting with new paradigms for medical education and the emergence of a distance learning degree using the internet: teaching evidence-based medicine. Med Inform Internet Med. 2002 Mar;27(1):1-11. doi: 10.1080/14639230110105301. PMID: 12509118.,[7]The impact of E-learning in medical education. Acad Med. 2006 Mar;81(3):207-12. doi: 10.1097/00001888-200603000-00002. PMID: 16501260.,[8]Integrating evidence based medicine into undergraduate medical education: combining online instruction with clinical clerkships. Teach Learn Med. 2010 Jul;22(3):219-23. doi: 10.1080/10401334.2010.488460. PMID: 20563945.,[9]Problem-based learning in distance education: a first exploration in continuing medical education. Med Educ. 1992 Sep;26(5):389-401. doi: 10.1111/j.1365-2923.1992.tb00192.x. PMID: 1435380.} This can be done by minimizing barriers for those unaccustomed to virtual education and tasked with creating conferences through instructional content and a tool to network in a post-geographic virtual didactic world.^{[10]Barriers and solutions to online learning in medical education - an integrative review. BMC Med Educ. 2018 Jun 7;18(1):130. doi: 10.1186/s12909-018-1240-0. PMID: 29880045; PMCID: PMC5992716.,[11]Internet-based medical education: a realist review of what works, for whom and in what circumstances. BMC Med Educ. 2010 Feb 2;10:12. doi: 10.1186/1472-6920-10-12. PMID: 20122253; PMCID: PMC2825237.,[12]Remote e-Work and Distance Learning for Academic Medicine: Best Practices and Opportunities for the Future. J Grad Med Educ. 2020 Jun;12(3):256-263. doi: 10.4300/JGME-D-20-00242.1. PMID: 32595840; PMCID: PMC7301927.}

Existing distance-learning literature has mainly focused on curricular content and technology platforms within individual programs, leaving a gap in tools that connect speakers, content, and conference scheduling across institutions (CITE). ALiEM Remote was developed to address this gap by providing a publicly-accessible platform that offers centralized speaker recruitment and conference scheduling.

OBJECTIVES

Our primary objective in this study was to describe the development and early utilization of ALiEM Remote in residency programs. Secondarily, we aimed to assess its feasibility as a model for future virtual speaker recruitment.

CURRICULAR DESIGN

The ALiEM team created an Airtable form hosted on a public website to collect speaker information. Recruitment of speakers and institutions occurred organically with the form being shared on twitter; participation was voluntary. Airtable is a cloud collaboration service whose virtual, spreadsheet-database hybrid has the features of a database but applied to a spreadsheet. Airtable also allows for the creation of forms, which consist of simple interfaces with questions that recipients respond to via answer fields. These responses then populate the Airtable view of speakers.

Unlike static speaker directories or single-institution conference calendars, ALiEM Remote combines real-time, crowdsourced speaker recruitment with a shared, cross-institutional conference schedule. This allows programs to identify speakers and coordinate co-hosted sessions in one platform.

Using Airtable enabled ease-of-use by both speakers, who could easily access and fill in their information via the ALiEM public website, and by the ALiEM team, who could easily view speaker information in the Airtable. Moreover, this approach enables us to utilize Google Analytics (GA), a free web analytics service, to assess user engagement with the form between March 10, 2020 and April 28, 2020. This window was selected because it captured the earliest and most concentrated period of pandemic-driven transition to virtual, based on the WHO’s March 11, 2020 pandemic declaration and the U.S’s shift to widespread stay-at-home orders and telehealth adoption by April 2020 ^{[14]WHO Director-General's opening remarks at the media briefing on COVID-19 — 11 March 2020. World Health Organization. Published March 11, 2020. https://www.who.int/director-general/speeches/detail/who-director-general-s-opening-remarks-at-the-media-briefing-on-covid-19---11-march-2020}.

Speakers filled out the following information: name, title, Twitter handle, category of lecture (e.g. leadership, clinical pearls, gender equity, etc.), time zone, length of talk, preference for offering a recorded or live talk, and program or hospital affiliation. We did not place limits on the number of topics submitted and did not require that speakers practice emergency medicine. To preserve the privacy of speakers, their contact information (e.g. email address, phone number, etc.) was not collected. We selected Twitter due to its wide use by medical professionals and the Free Open Access Medical education (FOAMed) community. This platform was more effective to facilitate rapid dissemination and professional networking for our intended audience. Twitter handles were collected to aid with recruitment of speakers, publicize the educational resources, and facilitate rapid connectivity.

Duplicate Tweets were removed prior to analysis to ensure each dissemination instance was reported once. Using GA, we assessed three areas: (1) Overall usage, (2) Platform, (3) Location. For the purpose of this study, engagement was defined as users who utilized ALiEM Remote website and its associated services. For overall usage, we assessed the total number of unique users, number of sessions, and average session duration. A session refers to a group of interactions, e.g. browse pages, download resources, purchase products, that one user takes within a given time frame on a site; in GA, that time frame defaults to 30 minutes. The platform looks at users who access the site via desktop versus mobile versus tablet. Geography refers to the country from which a session originates.

IMPACT/ EFFECTIVENESS

During the study window, the Airtable form was filled out by 162 speakers from 54 institutions internationally with expertise across a wide range of fields including public policy, critical care, and pediatrics. Twitter (mentions and hashtags) were used to disseminate the webpage and its content with strategic tagging of physicians with high followers. Table 1 represents data from relevant Tweets using #ALiEMRemote and #RemoteLearning. It demonstrates the scale and reach of virtual advertisements.

During the study window, the website received 1,852 unique users, representing a 9% “conversion rate,” the percentage of website visitors that complete a desired goal, in this case, signing up as a speaker. General digital marketing benchmarks for landing pages (industry average 2.35%, top 25% ≥5.31%, top 10% ≥ 11.45%)^{[15]What's a Good Conversion Rate? (It's Higher Than You Think). WordStream. Published on Feb 26, 2020. https://www.wordstream.com/blog/ws/2014/03/17/what-is-a-good-conversion-rate#:~:text=going%20to%20go%3F-,But%20what%20is%20a%20good%20conversion%20rate%3F,rates%20of%2011.45%25%20or%20higher.} are not designed for academic educational platforms and may not be directly comparable to ALiEM Remote. This figure is reported as one indicator of visitor interest during a period of unusually concentrated demand for virtual education resources, rather than as evidence of educational effectiveness. The conversion rate did not fluctuate significantly across the study period. Social engagement indicators such as impressions and link clicks served as additional behavioral measures of audience exposure and interest, complementing speaker sign-ups. However, these tools do not reflect confirmed educational participation or outcomes.

Users during this period, including new and returning users, engaged in a total of 1,900 sessions and an average session duration of 1 minute and 45 seconds. Industry wide, the average time spent on a site is 15 seconds^{[13]The 15 Second Rule: 3 Reasons Why Users Leave A Website. The Daily Egg, 2020.}. Almost all users used either mobile (43.79%) or desktop (55.53%) devices to access the site. The top country by session was the United States, with 1,461 sessions. The next top country was Australia with 79 sessions, followed by Canada, with 74, and the United Kingdom with 64. One residency program invited and hosted eleven speakers identified through the platform for their conference during the study window. This serves as an early indicator of downstream utilization of the platform. However, broader evaluation of recruitment outcomes across programs was not performed.

Several limitations should be considered when interpreting these findings. As this was a descriptive, engagement-based study, raw social media data were not organized and engagement metrics (likes, retweets, replies) were used as proxies for audience participation and visibility. This is not a validated measure of educational engagement. Form responses were entered manually by speakers and sometimes required grammatical editing and supplementation of missed fields by the ALiEM team. If this method were to be scaled, it would continue to require routine maintenance of minor errors. Moreover, communication between the speakers who filled out the form and educators recruiting speakers occurred primarily on Twitter. We received feedback from one user that this reliance on Twitter for communication between potential speakers and educators was challenging and resulted in delays. Moving forward, the team will need different avenues for communication, such as email.

The GA results demonstrate early interest and engagement with the ALiEM Remote platform. These findings should not be interpreted as evidence of the platform’s educational effectiveness because no learner-level or educational outcome data were collected. ALiEM Remote is best characterized as a feasibility demonstration and descriptive resource for the EM community, and as an early model for recruiting speakers for future virtual conferences. Further work is needed to evaluate its educational impact. This approach illustrates the feasibility of remote collaboration in a setting that is global in scope. Future iterations of the ALiEM Remote platform may include speaker topic requests, international speaking engagements, feedback/voting systems for speakers, referrals for speakers, and YouTube links of prior educational engagements so educators can learn asynchronously, alongside a formal evaluation of learner and program level outcomes.

REFERENCES

1. Chick RC, Clifton GT, Peace KM, Propper BW, Hale DF, Alseidi AA, Vreeland TJ. Using Technology to Maintain the Education of Residents During the COVID-19 Pandemic. J Surg Educ. 2020 Jul-Aug;77(4):729-732. doi: 10.1016/j.jsurg.2020.03.018. Epub 2020 Apr 3. PMID: 32253133; PMCID: PMC7270491.
2. Gupta RK, Mariano ER, Narouze S, Elkassabany NM. The #ASRASpring20 conference was canceled due to COVID-19 but the science survives and thrives. Reg Anesth Pain Med. 2021 Apr;46(4):374-375. doi: 10.1136/rapm-2020-101645. Epub 2020 May 13. PMID: 32409515.
3. Eichberg DG, Shah AH, Luther EM, Menendez I, Jimenez A, Perez-Dickens M, O'Phelan KH, Ivan ME, Komotar RJ, Levi AD. Letter: Academic Neurosurgery Department Response to COVID-19 Pandemic: The University of Miami/Jackson Memorial Hospital Model. Neurosurgery. 2020 Jul 1;87(1):E63-E65. doi: 10.1093/neuros/nyaa118. PMID: 32277754; PMCID: PMC7184381.
4. Fieschi M, Soula G, Giorgi R, Gouvernet J, Fieschi D, Botti G, Volot F, Berland Y. Experimenting with new paradigms for medical education and the emergence of a distance learning degree using the internet: teaching evidence-based medicine. Med Inform Internet Med. 2002 Mar;27(1):1-11. doi: 10.1080/14639230110105301. PMID: 12509118.
5. Almarzooq ZI, Lopes M, Kochar A. Virtual Learning During the COVID-19 Pandemic: A Disruptive Technology in Graduate Medical Education. J Am Coll Cardiol. 2020 May 26;75(20):2635-2638. doi: 10.1016/j.jacc.2020.04.015. Epub 2020 Apr 15. PMID: 32304797; PMCID: PMC7159871.
6. Weissgerber T, Bediako Y, de Winde CM, Ebrahimi H, Fernández-Chiappe F, Ilangovan V, Mehta D, Paz Quezada C, Riley JL, Saladi SM, Sarabipour S, Tay A. Mitigating the impact of conference and travel cancellations on researchers' futures. Elife. 2020 Mar 27;9:e57032. doi: 10.7554/eLife.57032. PMID: 32216872; PMCID: PMC7101232.
7. Ruiz JG, Mintzer MJ, Leipzig RM. The impact of E-learning in medical education. Acad Med. 2006 Mar;81(3):207-12. doi: 10.1097/00001888-200603000-00002. PMID: 16501260.
8. Aronoff SC, Evans B, Fleece D, Lyons P, Kaplan L, Rojas R. Integrating evidence based medicine into undergraduate medical education: combining online instruction with clinical clerkships. Teach Learn Med. 2010 Jul;22(3):219-23. doi: 10.1080/10401334.2010.488460. PMID: 20563945.
9. Engel CE, Browne E, Nyarango P, Akor S, Khwaja A, Karim AA, Towle A. Problem-based learning in distance education: a first exploration in continuing medical education. Med Educ. 1992 Sep;26(5):389-401. doi: 10.1111/j.1365-2923.1992.tb00192.x. PMID: 1435380.
10. O'Doherty D, Dromey M, Lougheed J, Hannigan A, Last J, McGrath D. Barriers and solutions to online learning in medical education - an integrative review. BMC Med Educ. 2018 Jun 7;18(1):130. doi: 10.1186/s12909-018-1240-0. PMID: 29880045; PMCID: PMC5992716.
11. Wong G, Greenhalgh T, Pawson R. Internet-based medical education: a realist review of what works, for whom and in what circumstances. BMC Med Educ. 2010 Feb 2;10:12. doi: 10.1186/1472-6920-10-12. PMID: 20122253; PMCID: PMC2825237.
12. He S, Lai D, Mott S, Little A, Grock A, Haas MRC, Chan TM. Remote e-Work and Distance Learning for Academic Medicine: Best Practices and Opportunities for the Future. J Grad Med Educ. 2020 Jun;12(3):256-263. doi: 10.4300/JGME-D-20-00242.1. PMID: 32595840; PMCID: PMC7301927.
13. "The 15 Second Rule: 3 Reasons Why Users Leave A Website". The Daily Egg, 2020, https://www.crazyegg.com/blog/why-users-leave-a-website/#:~:text=How%20long%20do%20users%20stay,someone's%20attention%20on%20your%20website.
14. Ghebreyesus TA. WHO Director-General's opening remarks at the media briefing on COVID-19 — 11 March 2020. World Health Organization. Published March 11, 2020. https://www.who.int/director-general/speeches/detail/who-director-general-s-opening-remarks-at-the-media-briefing-on-covid-19---11-march-2020
15. Kim L. “What's a Good Conversion Rate? (It's Higher Than You Think)” WordStream. Published on Feb 26, 2020. https://www.wordstream.com/blog/ws/2014/03/17/what-is-a-good-conversion-rate#:~:text=going%20to%20go%3F-,But%20what%20is%20a%20good%20conversion%20rate%3F,rates%20of%2011.45%25%20or%20higher.