1932
0
  1. Home
  2. A-Z Publications
  3. Annual Review of Biomedical Data Science
  4. Volume 2, 2019
  5. Article

Annual Review of Biomedical Data Science

Volume 2, 2019

Scientific Discovery Games for Biomedical Research

Abstract

Over the past decade, scientific discovery games (SDGs) have emerged as a viable approach for biomedical research, engaging hundreds of thousands of volunteer players and resulting in numerous scientific publications. After describing the origins of this novel research approach, we review the scientific output of SDGs across molecular modeling, sequence alignment, neuroscience, pathology, cellular biology, genomics, and human cognition. We find compelling results and technical innovations arising in problem-oriented games such as Foldit and Eterna and in data-oriented games such as EyeWire and Project Discovery. We discuss emergent properties of player communities shared across different projects, including the diversity of communities and the extraordinary contributions of some volunteers, such as paper writing. Finally, we highlight connections to artificial intelligence, biological cloud laboratories, new game genres, science education, and open science that may drive the next generation of SDGs.

Keyword(s): citizen science, cloud laboratory, crowdsourcing, interactive media, scientific discovery games, video games
Loading

Article metrics loading...

/content/journals/10.1146/annurev-biodatasci-072018-021139
2019-07-20
2024-04-23
Loading full text...

Full text loading...

/deliver/fulltext/biodatasci/2/1/annurev-biodatasci-072018-021139.html?itemId=/content/journals/10.1146/annurev-biodatasci-072018-021139&mimeType=html&fmt=ahah

Literature Cited

  1. 1. 
    Good BM, Su AI 2013. Crowdsourcing for bioinformatics. Bioinformatics 29:1925–33
     [Google Scholar]
  2. 2. 
    Treuille A, Das R 2014. Scientific rigor through videogames. Trends Biochem. Sci. 39:507–9
     [Google Scholar]
  3. 3. 
    Curtis V 2014. Online citizen science games: opportunities for the biological sciences. Appl. Transl. Genom. 3:90–94
     [Google Scholar]
  4. 4. 
    Roskams J, Popović Z 2016. Power to the people: addressing big data challenges in neuroscience by creating a new cadre of citizen neuroscientists. Neuron 92:658–64
     [Google Scholar]
  5. 5. 
    Lee J, Kladwang W, Lee M, Cantu D, Azizyan M et al. 2014. RNA design rules from a massive open laboratory. PNAS 111:2122–27
     [Google Scholar]
  6. 6. 
    Riedel-Kruse IH, Chung AM, Dura B, Hamilton AL, Lee BC 2011. Design, engineering and utility of biotic games. Lab Chip 11:14–22
     [Google Scholar]
  7. 7. 
    Mayer A 2010. Phenology and citizen science: Volunteers have documented seasonal events for more than a century, and scientific studies are benefiting from the data. BioScience 60:172–75
     [Google Scholar]
  8. 8. 
    Beberg AL, Ensign DL, Jayachandran G, Khaliq S, Pande VS 2009. Folding@ home: lessons from eight years of volunteer distributed computing. 2009 IEEE International Symposium on Parallel & Distributed Processing Los Alamitos, CA: IEEE Comput. Soc.
     [Google Scholar]
  9. 9. 
    Korpela EJ 2012. Seti@ home, BOINC, and volunteer distributed computing. Annu. Rev. Earth Planet. Sci. 40:69–87
     [Google Scholar]
  10. 10. 
    Cole H, Griffiths MD 2007. Social interactions in massively multiplayer online role-playing gamers. Cyberpsychol. Behav. 10:575–83
     [Google Scholar]
  11. 11. 
    Das R, Qian B, Raman S, Vernon R, Thompson J et al. 2007. Structure prediction for CASP7 targets using extensive all-atom refinement with Rosetta@ home. Proteins 69:118–28
     [Google Scholar]
  12. 12. 
    Lintott CJ, Schawinski K, Slosar A, Land K, Bamford S et al. 2008. Galaxy Zoo: morphologies derived from visual inspection of galaxies from the Sloan Digital Sky Survey. Mon. Not. R. Astron. Soc. 389:1179–89
     [Google Scholar]
  13. 13. 
    Cooper S, Khatib F, Treuille A, Barbero J, Lee J et al. 2010. Predicting protein structures with a multiplayer online game. Nature 466:756–60
     [Google Scholar]
  14. 14. 
    Schmidt M 2008. Diffusion of synthetic biology: a challenge to biosafety. Syst. Synth. Biol. 2:1–6
     [Google Scholar]
  15. 15. 
    Simpson R, Page KR, De Roure D 2014. Zooniverse: observing the world's largest citizen science platform. Proceedings of the 23rd International Conference on World Wide Web1049–54 New York: Assoc. Comput. Mach.
     [Google Scholar]
  16. 16. 
    Khatib F, DiMaio F, Cooper S, Kazmierczyk M, Gilski M et al. 2011a. Crystal structure of a monomeric retroviral protease solved by protein folding game players. Nat. Struct. Mol. Biol. 18:1175–77
     [Google Scholar]
  17. 17. 
    Luengo-Oroz MA, Arranz A, Frean J 2012. Crowdsourcing malaria parasite quantification: an online game for analyzing images of infected thick blood smears. J. Med. Internet Res. 14:e167
     [Google Scholar]
  18. 18. 
    Mavandadi S, Dimitrov S, Feng S, Yu F, Sikora U et al. 2012a. Distributed medical image analysis and diagnosis through crowd-sourced games: a malaria case study. PLOS ONE 7:e37245
     [Google Scholar]
  19. 19. 
    Kwak D, Kam A, Becerra D, Zhou Q, Hops A et al. 2013. Open-Phylo: a customizable crowd-computing platform for multiple sequence alignment. Genome Biol. 14:R116
     [Google Scholar]
  20. 20. 
    Kim JS, Greene MJ, Zlateski A, Lee K, Richardson M et al. 2014. Space–time wiring specificity supports direction selectivity in the retina. Nature 509:331–36
     [Google Scholar]
  21. 21. 
    Hayden EC 2014. The automated lab. Nature 516:131–32
     [Google Scholar]
  22. 22. 
    Hossain Z, Jin X, Bumbacher EW, Chung AM, Koo S et al. 2015. Interactive cloud experimentation for biology: an online education case study. Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems3681–90 New York: Assoc. Comput. Mach.
     [Google Scholar]
  23. 23. 
    Hossain Z, Bumbacher EW, Chung AM, Kim H, Litton C et al. 2016. Interactive and scalable biology cloud experimentation for scientific inquiry and education. Nat. Biotechnol. 34:1293–98
     [Google Scholar]
  24. 24. 
    Mnih V, Kavukcuoglu K, Silver D, Rusu AA, Veness J et al. 2015. Human-level control through deep reinforcement learning. Nature 518:529–33
     [Google Scholar]
  25. 25. 
    Sullivan DP, Winsnes CF, Åkesson L, Hjelmare M, Wiking M et al. 2018. Deep learning is combined with massive-scale citizen science to improve large-scale image classification. Nat. Biotechnol. 36:820–28
     [Google Scholar]
  26. 26. 
    Silver D, Huang A, Maddison CJ, Guez A, Sifre L et al. 2016. Mastering the game of Go with deep neural networks and tree search. Nature 529:484–89
     [Google Scholar]
  27. 27. 
    Silver D, Schrittwieser J, Simonyan K, Antonoglou I, Huang A et al. 2017. Mastering the game of Go without human knowledge. Nature 550:354–59
     [Google Scholar]
  28. 28. 
    Hossain Z, Bumbacher E, Brauneis A, Diaz M, Saltarelli A et al. 2017a. Design guidelines and empirical case study for scaling authentic inquiry-based science learning via open online courses and interactive biology cloud labs. Int. J. Artif. Intell. Educ. 28:478–507
     [Google Scholar]
  29. 29. 
    Wellington-Oguri R, Fisker E, Wiley M, Zada M 2018. Evidence of an unusual poly(A) RNA signature detected by high-throughput chemical mapping. bioRxiv 281147. https://doi.org/10.1101/281147
    [Crossref] [Google Scholar]
  30. 30. 
    Nat. Audubon Soc 2019. Annual summaries of the Christmas Bird Count, 1901–present https://www.audubon.org/content/american-birds-annual-summary-christmas-bird-count
  31. 31. 
    Reid P, Colebrook J, Matthews J, Aiken J, Contin. Plankton Rec. Team 2003. The Continuous Plankton Recorder: concepts and history, from plankton indicator to undulating recorders. Prog. Oceanogr. 58:117–73
     [Google Scholar]
  32. 32. 
    Vezzulli L, Reid P 2003. The CPR survey (1948–1997): a gridded database browser of plankton abundance in the North Sea. Prog. Oceanogr. 58:327–36
     [Google Scholar]
  33. 33. 
    Sullivan BL, Wood CL, Iliff MJ, Bonney RE, Fink D, Kelling S 2009. eBird: a citizen-based bird observation network in the biological sciences. Biol. Conserv. 142:2282–92
     [Google Scholar]
  34. 34. 
    Robinson KL, Luo JY, Sponaugle S, Guigand C, Cowen RK 2017. A tale of two crowds: public engagement in plankton classification. Front. Mar. Sci. 4:82
     [Google Scholar]
  35. 35. 
    Kanefsky B, Barlow NG, Gulick VC 2001. Can distributed volunteers accomplish massive data analysis tasks?Paper presented at the 32nd Annual Lunar and Planetary Science Conference, Houston, Texas, March 12–16, Abstr. 1272
  36. 36. 
    Anderson DP 2004. BOINC: a system for public-resource computing and storage. Fifth IEEE/ACM International Workshop on Grid Computing R Buyya4–10 Los Alamitos, CA: IEEE Comput. Soc.
     [Google Scholar]
  37. 37. 
    Howe J 2006. The rise of crowdsourcing. Wired Magazine 14: https://www.wired.com/2006/06/crowds/
     [Google Scholar]
  38. 38. 
    Ipeirotis PG 2010. Analyzing the Amazon Mechanical Turk marketplace. XRDS: Crossroads, The ACM Magazine for Students 17:16–21
     [Google Scholar]
  39. 39. 
    Good BM, Clarke EL, de Alfaro L, Su AI 2011. The Gene Wiki in 2011: community intelligence applied to human gene annotation. Nucleic Acids Res. 40:D1255–61
     [Google Scholar]
  40. 40. 
    Gardner PP, Daub J, Tate J, Moore BL, Osuch IH et al. 2010. Rfam: Wikipedia, clans and the “decimal” release. Nucleic Acids Res. 39:D141–45
     [Google Scholar]
  41. 41. 
    Schuster SC 2007. Next-generation sequencing transforms today's biology. Nat. Methods 5:16–18
     [Google Scholar]
  42. 42. 
    Pham DL, Xu C, Prince JL 2000. Current methods in medical image segmentation. Annu. Rev. Biomed. Eng. 2:315–37
     [Google Scholar]
  43. 43. 
    Dogangil G, Davies B, Rodriguez y Baena F 2010. A review of medical robotics for minimally invasive soft tissue surgery. Proc. Inst. Mech. Eng. H 224:653–79
     [Google Scholar]
  44. 44. 
    Thul PJ, Åkesson L, Wiking M, Mahdessian D, Geladaki A et al. 2017. A subcellular map of the human proteome. Science 356:eaal3321
     [Google Scholar]
  45. 45. 
    Prill RJ, Saez-Rodriguez J, Alexopoulos LG, Sorger PK, Stolovitzky G 2011. Crowdsourcing network inference: the DREAM predictive signaling network challenge. Sci. Signal. 4:mr7
     [Google Scholar]
  46. 46. 
    Moult J 2005. A decade of CASP: progress, bottlenecks and prognosis in protein structure prediction. Curr. Opin. Struct. Biol. 15:285–89
     [Google Scholar]
  47. 47. 
    Daneshjou R, Wang Y, Bromberg Y, Bovo S, Martelli PL et al. 2017. Working toward precision medicine: predicting phenotypes from exomes in the critical assessment of genome interpretation (CAGI) challenges. Hum. Mutat. 38:1182–92
     [Google Scholar]
  48. 48. 
    Saez-Rodriguez J, Costello JC, Friend SH, Kellen MR, Mangravite L et al. 2016. Crowdsourcing biomedical research: leveraging communities as innovation engines. Nat. Rev. Genet. 17:470–86
     [Google Scholar]
  49. 49. 
    Marchand A, Hennig-Thurau T 2013. Value creation in the video game industry: industry economics, consumer benefits, and research opportunities. J. Interact. Mark. 27:141–57
     [Google Scholar]
  50. 50. 
    Aarseth E 2001. Computer game studies, year one. Game Stud. 1: http://www.gamestudies.org/0101/editorial.html
    [Google Scholar]
  51. 51. 
    Von Ahn L 2006. Games with a purpose. Computer 39:92–94
     [Google Scholar]
  52. 52. 
    Das R, Baker D 2008. Macromolecular modeling with Rosetta. Annu. Rev. Biochem. 77:363–82
     [Google Scholar]
  53. 53. 
    Pierce NA, Winfree E 2002. Protein design is NP-hard. Protein Eng. 15:779–82
     [Google Scholar]
  54. 54. 
    Khatib F, Cooper S, Tyka MD, Xu K, Makedon I et al. 2011b. Algorithm discovery by protein folding game players. PNAS 108:18949–53
     [Google Scholar]
  55. 55. 
    Horowitz S, Koepnick B, Martin R, Tymieniecki A, Winburn AA et al. 2016. Determining crystal structures through crowdsourcing and coursework. Nat. Commun. 7:12549
     [Google Scholar]
  56. 56. 
    Khoury GA, Liwo A, Khatib F, Zhou H, Chopra G et al. 2014. Wefold: a coopetition for protein structure prediction. Proteins 82:1850–68
     [Google Scholar]
  57. 57. 
    Jorda J, Sawaya MR, Yeates TO 2014. Crowdphase: crowdsourcing the phase problem. Acta Crystallogr. D 70:1538–48
     [Google Scholar]
  58. 58. 
    Mazzanti L, Doutreligne S, Gageat C, Derreumaux P, Taly A et al. 2017. What can human-guided simulations bring to RNA folding?. Biophys. J. 113:302–12
     [Google Scholar]
  59. 59. 
    Eiben CB, Siegel JB, Bale JB, Cooper S, Khatib F et al. 2012. Increased Diels-Alderase activity through backbone remodeling guided by Foldit players. Nat. Biotechnol. 30:190–92
     [Google Scholar]
  60. 60. 
    Bonnet É, Rzążewski P, Sikora F 2018. Designing RNA secondary structures is hard. Proceedings of the 22nd Annual International Conference on Research in Computational Molecular Biology (RECOMB) BJ Raphael248–50 Cham, Switz.: Springer Int.
     [Google Scholar]
  61. 61. 
    Barone J, Bayer C, Copley R, Barlow N, Burns M et al. 2015. Nanocrafter: design and evaluation of a DNA nanotechnology game. Proceedings of the 10th International Conference on the Foundations of Digital Games (FDG 2015) http://www.fdg2015.org/papers/fdg2015_paper_74.pdf
    [Google Scholar]
  62. 62. 
    Wang L, Jiang T 1994. On the complexity of multiple sequence alignment. J. Comput. Biol. 1:337–48
     [Google Scholar]
  63. 63. 
    Rallapalli G, Saunders DG, Yoshida K, Edwards A, Lugo CA et al. 2015. Cutting edge: lessons from Fraxinus, a crowd-sourced citizen science game in genomics. eLife 4:e07460
     [Google Scholar]
  64. 64. 
    Kawrykow A, Roumanis G, Kam A, Kwak D, Leung C et al. 2012. Phylo: a citizen science approach for improving multiple sequence alignment. PLOS ONE 7:e31362
     [Google Scholar]
  65. 65. 
    Cannone JJ, Subramanian S, Schnare MN, Collett JR, D'Souza LM et al. 2002. The Comparative RNA Web (CRW) Site: an online database of comparative sequence and structure information for ribosomal, intron, and other RNAs. BMC Bioinform. 3:2
     [Google Scholar]
  66. 66. 
    Waldispühl J, Kam A, Gardner PP 2014. Crowdsourcing RNA structural alignments with an online computer game. Pac. Symp. Biocomput. 20:330–41
     [Google Scholar]
  67. 67. 
    Tacchella A, Romano S, Ferraldeschi M, Salvetti M, Zaccaria A et al. 2017. Collaboration between a human group and artificial intelligence can improve prediction of multiple sclerosis course: a proof-of-principle study. F1000Research 6:2172
     [Google Scholar]
  68. 68. 
    Harvey H, Havard M, Magnus D, Cho MK, Riedel-Kruse IH 2014. Innocent fun or “microslavery”? An ethical analysis of biotic games. Hastings Center Rep. 44:38–46
     [Google Scholar]
  69. 69. 
    Tsueng G, Kumar A, Nanis M, Su AI 2018.Aligning interests: integrating citizen science efforts into schools through service requirements. bioRxiv 304766. https://doi.org/10.1101/304766
     [Crossref]
  70. 70. 
    Hughes AJ, Mornin JD, Biswas SK, Beck LE, Bauer DP et al. 2018. Quanti.us: a tool for rapid, flexible, crowd-based annotation of images. Nat. Methods 15:587–90
     [Google Scholar]
  71. 71. 
    Alkhatib A, Bernstein MS, Levi M 2017. Examining crowd work and gig work through the historical lens of piecework. Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems4599–616 New York: Assoc. Comput. Mach.
     [Google Scholar]
  72. 72. 
    Guerrini CJ, Majumder MA, Lewellyn MJ, McGuire AL 2018. Citizen science, public policy. Science 361:134–36
     [Google Scholar]
  73. 73. 
    Scassa T, Chung H 2015. Typology of citizen science projects from an intellectual property perspective Policy Memo., Commons Lab, Wilson Cent., Washington, DC. https://www.wilsoncenter.org/publication/typology-citizen-science-projects-intellectual-property-perspective
  74. 74. 
    Kuss DJ, Griffiths MD, Karila L, Billieux J 2014. Internet addiction: a systematic review of epidemiological research for the last decade. Curr. Pharm. Des. 20:4026–52
     [Google Scholar]
  75. 75. 
    Anderson EL, Steen E, Stavropoulos V 2017. Internet use and problematic Internet use: a systematic review of longitudinal research trends in adolescence and emergent adulthood. Int. J. Adolesc. Youth 22:430–54
     [Google Scholar]
  76. 76. 
    Mortensen TE 2016. Anger, fear, and games: the long event of #GamerGate. Games Cult. 13:8787–806
     [Google Scholar]
  77. 77. 
    Newman L, Browne-Yung K, Raghavendra P, Wood D, Grace E 2017. Applying a critical approach to investigate barriers to digital inclusion and online social networking among young people with disabilities. Inf. Syst. J. 27:559–88
     [Google Scholar]
  78. 78. 
    Martínez-Pérez B, De La Torre-Díez I, López-Coronado M 2015. Privacy and security in mobile health apps: a review and recommendations. J. Med. Syst. 39:181
     [Google Scholar]
  79. 79. 
    Ortiz-Ruiz A, Postigo M, Gil-Casanova S, Cuadrado D, Bautista JM et al. 2018. Plasmodium species differentiation by non-expert on-line volunteers for remote malaria field diagnosis. Malaria J. 17:54
     [Google Scholar]
  80. 80. 
    Mavandadi S, Dimitrov S, Feng S, Yu F, Yu R et al. 2012b. Crowd-sourced biogames: managing the big data problem for next-generation lab-on-a-chip platforms. Lab Chip 12:4102–6
     [Google Scholar]
  81. 81. 
    Poostchi M, Silamut K, Maude R, Jaeger S, Thoma G 2018. Image analysis and machine learning for detecting malaria. Transl. Res. 194:36–55
     [Google Scholar]
  82. 82. 
    dos Reis FJC, Lynn S, Ali HR, Eccles D, Hanby A et al. 2015. Crowdsourcing the general public for large scale molecular pathology studies in cancer. EBioMedicine 2:681–89
     [Google Scholar]
  83. 83. 
    Irshad H, Oh EY, Schmolze D, Quintana LM, Collins L et al. 2017. Crowdsourcing scoring of immunohistochemistry images: evaluating performance of the crowd and an automated computational method. Sci. Rep. 7:43286
     [Google Scholar]
  84. 84. 
    Lawson J, Robinson-Vyas RJ, McQuillan JP, Paterson A, Christie S et al. 2017. Crowdsourcing for translational research: analysis of biomarker expression using cancer microarrays. Br. J. Cancer 116:237–45
     [Google Scholar]
  85. 85. 
    LeCun Y, Bengio Y, Hinton G 2015. Deep learning. Nature 521:436–44
     [Google Scholar]
  86. 86. 
    Loguercio S, Good BM, Su AI 2013. Dizeez: an online game for human gene-disease annotation. PLOS ONE 8:e71171
     [Google Scholar]
  87. 87. 
    Tsueng G, Nanis M, Fouquier J, Good B, Su A 2016. Citizen science for mining the biomedical literature. Citiz. Sci. 1:14
     [Google Scholar]
  88. 88. 
    Ratnasingham S, Hebert PD 2007. bold: the Barcode of Life Data System ( www.barcodinglife.org). Mol. Ecol. Notes 7:355–64
     [Google Scholar]
  89. 89. 
    Hatfull GF 2015. Innovations in undergraduate science education: going viral. J. Virol. 89:8111–13
     [Google Scholar]
  90. 90. 
    Coutrot A, Silva R, Manley E, de Cothi W, Sami S et al. 2018. Global determinants of navigation ability. Curr. Biol. 28:2861–66
     [Google Scholar]
  91. 91. 
    Cochrane G, Akhtar R, Bonfield J, Bower L, Demiralp F et al. 2008. Petabyte-scale innovations at the European Nucleotide Archive. Nucleic Acids Res. 37:D19–25
     [Google Scholar]
  92. 92. 
    McDonald D, Hyde E, Debelius JW, Morton JT, Gonzalez A et al. 2018. American Gut: an open platform for citizen science microbiome research. mSystems 3:e00031–18
     [Google Scholar]
  93. 93. 
    Wang Z, Monteiro CD, Jagodnik KM, Fernandez NF, Gundersen GW et al. 2016. Extraction and analysis of signatures from the Gene Expression Omnibus by the crowd. Nat. Commun. 7:12846
     [Google Scholar]
  94. 94. 
    Hadley D, Pan J, El-Sayed O, Aljabban J, Aljabban I et al. 2017. Precision annotation of digital samples in NCBI's Gene Expression Omnibus. Sci. Data 4:170125
     [Google Scholar]
  95. 95. 
    Franzoni C, Sauermann H 2014. Crowd science: the organization of scientific research in open collaborative projects. Res. Policy 43:1–20
     [Google Scholar]
  96. 96. 
    Anderson-Lee J, Fisker E, Kosaraju V, Wu M, Kong J et al. 2016. Principles for predicting RNA secondary structure design difficulty. J. Mol. Biol. 428:748–57
     [Google Scholar]
  97. 97. 
    Portela F 2018.An unexpectedly effective Monte Carlo technique for the RNA inverse folding problem. bioRxiv 345587. https://doi.org/10.1101/345587
     [Crossref]
  98. 98. 
    Tinati R, Luczak-Roesch M, Simperl E, Hall W 2016. Because science is awesome: studying participation in a citizen science game. Proceedings of the 8th ACM Conference on Web Science (WebSci '16)45–54 New York: Assoc. Comput. Mach.
     [Google Scholar]
  99. 99. 
    Raddick JM, Bracey G, Gay PL, Lintott CJ, Murray P et al. 2010. Galaxy Zoo: exploring the motivations of citizen scientist volunteers. Astron. Educ. Rev. 9:1010103–1
     [Google Scholar]
  100. 100. 
    Entertain. Softw. Assoc 2017. Essential facts about the computer and video game industry Surv. Rep., Entertain. Softw. Assoc. Washington, DC:
  101. 101. 
    Rowles TA 2013. Power to the people: Does Eterna signal the arrival of a new wave of crowd-sourced projects?. BMC Biochemistry 14:26
     [Google Scholar]
  102. 102. 
    Curtis V 2015. Motivation to participate in an online citizen science game: a study of Foldit. Sci. Commun. 37:723–46
     [Google Scholar]
  103. 103. 
    Cooper C 2013. Zen in the art of citizen science: apps for collective discovery and the 1 percent rule of the web. Scientific American Blog https://blogs.scientificamerican.com/guest-blog/zen-in-the-art-of-citizen-science-apps-for-collective-discovery-and-the-1-percent-rule-of-the-web/
    [Google Scholar]
  104. 104. 
    Feng WC, Brandt D, Saha D 2007. A long-term study of a popular MMORPG. Proceedings of the 6th ACM SIGCOMM Workshop on Network and System Support for Games19–24 New York: Assoc. Comput. Mach.
     [Google Scholar]
  105. 105. 
    Cooper S, Khatib F, Makedon I, Lu H, Barbero J et al. 2011. Analysis of social gameplay macros in the Foldit cookbook. Proceedings of the 6th International Conference on Foundations of Digital Games New York: Assoc. Comput. Mach.
     [Google Scholar]
  106. 106. 
    Eastman P, Shi J, Ramsundar B, Pande VS 2018. Solving the RNA design problem with reinforcement learning. PLOS Comput. Biol. 14:e1006176
     [Google Scholar]
  107. 107. 
    Cox J, Oh EY, Simmons B, Lintott C, Masters K et al. 2015. Defining and measuring success in online citizen science: a case study of Zooniverse projects. Comput. Sci. Eng. 17:28–41
     [Google Scholar]
  108. 108. 
    Gaston J, Cooper S 2017. To three or not to three: improving human computation game onboarding with a three-star system. Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems5034–39 New York: Assoc. Comput. Mach.
     [Google Scholar]
  109. 109. 
    Tinati R, Luczak-Roesch M, Simperl E, Shadbolt N, Hall W 2015. ‘/Command’ and conquer: analysing discussion in a citizen science game. Proceedings of the ACM Web Science Conference Pap. 26 New York: Assoc. Comput. Mach.
     [Google Scholar]
  110. 110. 
    Baaden M, Delalande O, Ferey N, Pasquali S, Waldispühl J, Taly A 2018. Ten simple rules to create a serious game, illustrated with examples from structural biology. PLOS Comput. Biol. 14:e1005955
     [Google Scholar]
  111. 111. 
    Bruggemann J, Lander GC, Su AI 2018. Exploring applications of crowdsourcing to cryo-EM. J. Struct. Biol. 203:37–45
     [Google Scholar]
  112. 112. 
    Haas A, Rossberg A, Schuff DL, Titzer BL, Holman M et al. 2017. Bringing the web up to speed with WebAssembly. Proceedings of the 38th ACM SIGPLAN Conference on Programming Language Design and Implementation185–200 New York: Assoc. Comput. Mach.
     [Google Scholar]
  113. 113. 
    Good B, Santini S, Wallace M, Fortugno N, Szeder J et al. 2017. Science Game Lab: tool for the unification of biomedical games with a purpose. bioRxiv 156141. https://doi.org/10.1101/156141
    [Crossref] [Google Scholar]
  114. 114. 
    Hossain Z, Riedel-Kruse IH 2018. Life-science experiments online: technological frameworks and educational use cases. Cyber-Physical Laboratories in Engineering and Science Education ME Auer, A Azad, A Edwards, T de Jong271–304 Cham, Switz.: Springer Int.
     [Google Scholar]
  115. 115. 
    Hossain Z, Bumbacher E, Blikstein P, Riedel-Kruse I 2017b. Authentic science inquiry learning at scale enabled by an interactive biology cloud experimentation lab. Proceedings of the Fourth (2017) ACM Conference on Learning @ Scale237–40 New York: Assoc. Comput. Mach.
     [Google Scholar]
  116. 116. 
    Riedel-Kruse IH 2018. Incorporating a commercial biology cloud lab into online education. Online Engineering & Internet of Things331–43 Cham, Switz.: Springer
     [Google Scholar]
  117. 117. 
    Tsang AC, Lam AT, Riedel-Kruse IH 2018. Polygonal motion and adaptable phototaxis via flagellar beat switching in the microswimmer Euglena gracilis. Nat. Phys. 14:1216–22
     [Google Scholar]
  118. 118. 
    Corbató FJ, Merwin-Daggett M, Daley RC 1962. An experimental time-sharing system. Proceedings of the May 1–3, 1962, Spring Joint Computer Conference335–44 New York: Assoc. Comput. Mach.
     [Google Scholar]
  119. 119. 
    Murty J 2008. Programming Amazon web services: S3, EC2, SQS, FPS, and SimpleDB. Sebastopol, CA: O'Reilly Media
     [Google Scholar]
  120. 120. 
    Zahariev A 2009. Google app engine Paper presented at TKKT-110.5190 Seminar on Internetworking, Helsinki Univ. Technol. http://www.cse.hut.fi/en/publications/B/5/papers/1Zahariev_final.pdf
     [Google Scholar]
  121. 121. 
    Wilder B 2012. Cloud Architecture Patterns: Using Microsoft Azure Sebastopol, CA: O'Reilly Media
  122. 122. 
    Kim H, Gerber LC, Chiu D, Lee SA, Cira NJ et al. 2016. LudusScope: accessible interactive smartphone microscopy for life-science education. PLOS ONE 11:e0162602
     [Google Scholar]
  123. 123. 
    Gerber LC, Kim H, Riedel-Kruse IH 2016. Interactive biotechnology: design rules for integrating biological matter into digital games. Proceedings of the First International Joint Conference of DiGRA and FDG http://www.digra.org/digital-library/publications/interactive-biotechnology-design-rules-for-integrating-biological-matter-into-digital-games/
    [Google Scholar]
  124. 124. 
    Lee SA, Chung AM, Cira N, Riedel-Kruse IH 2015. Tangible interactive microbiology for informal science education. Proceedings of the Ninth International Conference on Tangible, Embedded, and Embodied Interaction273–80 New York: Assoc. Comput. Mach.
     [Google Scholar]
  125. 125. 
    Lee SA, Bumbacher E, Chung AM, Cira N, Walker B et al. 2015. Trap it!: a playful human-biology interaction for a museum installation. Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems2593–602 New York: Assoc. Comput. Mach.
     [Google Scholar]
  126. 126. 
    Cira NJ, Chung AM, Denisin AK, Rensi S, Sanchez GN et al. 2015. A biotic game design project for integrated life science and engineering education. PLOS Biol. 13:e1002110
     [Google Scholar]
  127. 127. 
    Gerber LC, Doshi MC, Kim H, Riedel-Kruse IH 2016. Biographr: science games on a biotic computer. Proceedings of the First International Joint Conference of DiGRA and FDG http://www.digra.org/digital-library/publications/biographr-science-games-on-a-biotic-computer/
    [Google Scholar]
  128. 128. 
    Lam AT, Samuel-Gama KG, Griffin J, Loeun M, Gerber LC et al. 2017. Device and programming abstractions for spatiotemporal control of active micro-particle swarms. Lab Chip 17:1442–51
     [Google Scholar]
  129. 129. 
    Washington P, Samuel-Gamaa KG, Goyala S, Ramaswamia A, Riedel-Kruse IH 2019. Interactive programming paradigm for real-time experimentation with remote living matter. PNAS 116:125411–19
     [Google Scholar]
  130. 130. 
    Chambers C 2011. How virtual are virtual economies? An exploration into the legal, social and economic nature of virtual world economies. Comput. Law Secur. Rev. 27:377–84
     [Google Scholar]
  131. 131. 
    Buro M 2003. Real-time strategy games: a new AI research challenge. Proceedings of the 18th International Joint Conference on Artificial Intelligence1534–35 San Francisco: Morgan Kaufmann
     [Google Scholar]
  132. 132. 
    van der Maaten L, Hinton G 2008. Visualizing data using t-SNE. J. Mach. Learn. Res. 9:2579–605
     [Google Scholar]
  133. 133. 
    Marcus G 2018.Deep learning: a critical appraisal. arXiv:1801.00631 [cs.AI]
  134. 134. 
    Chi MT 2009. Active-constructive-interactive: a conceptual framework for differentiating learning activities. Top. Cogn. Sci. 1:73–105
     [Google Scholar]
  135. 135. 
    Comm. Concept. Framew. New K-12 Sci. Educ. Stand 2012. A Framework for K-12 Science Education: Practices, Crosscutting Concepts, and Core Ideas Washington, DC: Natl. Acad. Press
  136. 136. 
    Williams JJ 2013. Improving learning in MOOCs with cognitive science. Proceedings of the Workshops at the 16th International Conference on Artificial Intelligence in Education (AIED 2013) E Walker, C-K Looi49–54 http://ceur-ws.org/Vol-1009/
     [Google Scholar]
  137. 137. 
    Pedaste M, Mäeots M, Siiman LA, De Jong T, Van Riesen SA et al. 2015. Phases of inquiry-based learning: definitions and the inquiry cycle. Educ. Res. Rev. 14:47–61
     [Google Scholar]
  138. 138. 
    Stock A 2015. Guide to Implementing the Next Generation Science Standards by National Research Council. Sci. Scope 39:84–85
     [Google Scholar]
  139. 139. 
    Qian M, Clark KR 2016. Game-based learning and 21st century skills: a review of recent research. Comput. Hum. Behav. 63:50–58
     [Google Scholar]
  140. 140. 
    Boyle EA, Hainey T, Connolly TM, Gray G, Earp J et al. 2016. An update to the systematic literature review of empirical evidence of the impacts and outcomes of computer games and serious games. Comput. Educ. 94:178–92
     [Google Scholar]
  141. 141. 
    McKiernan EC 2017. Imagining the “open” university: sharing scholarship to improve research and education. PLOS Biol. 15:e1002614
     [Google Scholar]
  142. 142. 
    Baden T, Chagas AM, Gage G, Marzullo T, Prieto-Godino LL, Euler T 2015. Open labware: 3-D printing your own lab equipment. PLOS Biol. 13:e1002086
     [Google Scholar]
  143. 143. 
    Kuiken T 2016. Learn from DIY biologists: the citizen-science community has a responsible, proactive attitude that is well suited to gene-editing. Nature 531:167–69
     [Google Scholar]
  144. 144. 
    Scheifele LZ, Burkett T 2016. The first three years of a community lab: lessons learned and ways forward. J. Microbiol. Biol. Educ. 17:81–85
     [Google Scholar]
/content/journals/10.1146/annurev-biodatasci-072018-021139
Loading
Scientific Discovery Games for Biomedical Research
Annual Review of Biomedical Data Science 2, 253 (2019); https://doi.org/10.1146/annurev-biodatasci-072018-021139
/content/journals/10.1146/annurev-biodatasci-072018-021139
/content/journals/10.1146/annurev-biodatasci-072018-021139
Loading

Data & Media loading...

  • Article Type: Review Article

Most Cited Most Cited RSS feed

This is a required field
Please enter a valid email address
Approval was a Success
Invalid data
An Error Occurred
Approval was partially successful, following selected items could not be processed due to error