MLCommons

Research Working Group

Medical Working Group

Mission

Design benchmarks and propose best practices to accelerate the development of AI and machine learning for healthcare, catalyze new markets, and ultimately improve Patient outcome and improve Providers’ experience.

Purpose

Medical AI has tremendous potential to advance healthcare by supporting the evidence-based practice of medicine, personalizing patient treatment, reducing costs, and improving provider and patient experience. We believe that unlocking this potential requires a systematic way to measure the performance of medical AI models on large-scale heterogeneous data. To meet this need, we are building MedPerf, an open framework for benchmarking machine learning in the medical domain. MedPerf will enable federated evaluation in which models are securely distributed to different facilities for evaluation, thereby empowering healthcare organizations to assess and verify the performance of AI models in an efficient and human-supervised process, while prioritizing privacy. In our paper titled "MedPerf: Open Benchmarking Platform for Medical Artificial Intelligence using Federated Evaluation" we describe the current challenges healthcare and AI communities face, the need for an open platform, the design philosophy of MedPerf, its current implementation status, and our roadmap.

Within our working group we strongly believe that these efforts will give key stakeholders the confidence to trust models and the data/processes they relied on, therefore accelerating ML adoption in the clinical settings, possibly improving patient outcomes, optimizing healthcare costs, and improving provider experiences.

Deliverables

  1. Initial efforts will focus on developing a Proof-of-Concept (PoC):
        - Build confidence for discussions of participation with real clinical data
        - Validate our technological approach
        - Enhance our understanding of system requirements
        - Serve as a public example of federated learning in the medical space
        - Publish white papers with findings

  2. Long Term efforts will focus on:
        - Extending initial PoC features with benchmarks for methods in data utility, model evaluation, data privacy
        - Improving technical integration with multiple Federated Learning frameworks
        - Building partnerships with medical organizations
        - Facilitate clinical data access
        - Research on various open challenges: noisy labels, data utility, privacy

Meeting Schedule

Weekly on Monday from 10:00-11:00AM Pacific.

Mailing List

medical@mlcommons.org

Working Group Resources

Google Drive

Working Group Chair Emails

Alexandros Karargyris (alexandros.karargyris@ihu-strasbourg.eu)

Renato Umeton (renato_umeton@mail.dfci.harvard.edu)

Micah Sheller (micah.j.sheller@intel.com), Vice Chair

Working Group Chair Bios

Alexandros Karargyris is a senior researcher at the Institute of Image-Guided Surgery of Strasbourg, a unique place for translative clinical research. He is leading projects related to  applications in the intersection of surgery and artificial intelligence (AI).  Previously, he worked as a researcher at IBM and NIH for more than 10 years. His research interests lie in the space of medical imaging, machine learning and mobile health. He has contributed to healthcare commercial products and imaging solutions deployed in under-resourced areas. His work has been published in peer-reviewed journals and conferences.

LinkedIn

Renato Umeton heads artificial intelligence (AI) and data science in the Informatics & Analytics department of Dana-Farber Cancer Institute, a teaching-affiliate of Harvard Medical School. His work focuses on Operations, translating AI from the research realm into the Clinic and into Enterprise software offerings that benefit patients and cancer researchers, at scale. Renato started working on artificial intelligence, data science and big data in 2007, when these areas were yet to be well defined; since then he has published in several scientific fields, he has worked in academia, in consulting, in hospital settings, in biotech, and is currently affiliated also with Harvard School of Public Health, Massachusetts Institute of Technology, and Weill Cornell Medicine.

LinkedIn

Micah Sheller currently works as a senior research scientist in Intel's Security and Privacy Research Labs, where he leads secure federated learning research. He developed the first version of the OpenFL open-source federated learning platform and is the technical federated learning lead for the FeTS initiative, which recently trained a 3DResUNet across 53 hospitals. Micah has had the pleasure of working on a wide range of projects since his first Intel internship in 1999, when he worked on the Intel Web Tablet. Work in USB 3.0, the prototype Intel SGX software runtime, passive and continuous biometrics and more has kept Micah happily learning and making friends throughout his career.

Intel Bio

Technical Leads

Johnu George is a staff engineer at Nutanix with a wealth of experience in building production grade cloud native platforms. He has a strong distributed systems background and has led efforts in building large scale hybrid data pipelines. He holds multiple patents in this area and has been an invited speaker at various conferences like Kubecon, Apache Big Data etc. He is an active open source contributor and has steered several industry collaborations on projects like Kubeflow, Apache Mnemonic and Knative. His current research interests include machine learning system design, distributed learning infrastructure improvements and ML workloads characterization. He is an Apache PMC member and currently chairing Kubeflow Training and AutoML Working groups.

Alejandro Aristizabal is a machine learing engineer with Factored. He has more than 4 years of experience in Software Development. He holds a bachelors degree in sound engineering. He is very interested in AI Safety, Security and Privacy, as well as human-machine interaction.