Free Public Service for Algorithm Audits

Machine learning algorithms should be easy to evaluate for performance and bias. But for most algorithm developers it is difficult to answer the simple question: does this algorithm do what it’s supposed to do? And does it do it fairly — for everyone?

Dandelion is proud to offer a free public service to solve this problem, starting with ECG and Chest CT algorithms. We will run your algorithm against a large representative holdout dataset and provide a confidential performance report.

This effort is generously funded by the Gordon and Betty Moore Foundation and The SCAN Foundation.

For additional detail, please see our press coverage in Forbes and STAT.

How it works

  • Dandelion makes it easy to audit your algorithm while protecting valuable IP.

  • Dandelion's free service supports a range of outcome labels.

  • Dandelion's confidential and free audit report includes overall performance metrics.

  • Dandelion's confidential and free audit report includes valuable performance details.

  • Dandelion's confidential and free audit report includes subgroup performance.

  • Dandelion's confidential and free audit report includes calibration performance.

  • Contact us to get a free and confidential algorithm audit report.

How to get your free and confidential report

Step 1: Sign a contract to protect your IP

Complete our Contact Form. Share the contact information of the person who should receive the contract. We will email you the contract and detailed instructions.

Step 2: Prepare and submit your algorithm for audit

We will provide detailed instructions and templates to pre-process your algorithm, wrap it (~10 lines of code), place it into a secure container, and submit your container for audit.

Step 3: Receive your confidential performance report

We will email your confidential performance report as soon as possible to your appointed recipient. Delivery times may vary depending on demand.

Program leadership

  • Sendhil Mullainathan

    Roman Family University Professor of Computation and Behavioral Science at the University of Chicago Booth School of Business

    Co-Founder of Dandelion Health, Nightingale Open Science, and ideas42

  • Ziad Obermeyer

    Blue Cross of California Distinguished Associate Professor of Health Policy and Management at the University of California, Berkeley School of Public Health

    Co-Founder of Dandelion Health and Nightingale Open Science

Select co-authored publications

Frequently Asked Questions

  • Dandelion Health is a healthcare data platform that focuses on AI and precision analytics. It is offering a free validation service as part of its product suite.

  • Dandelion is the first company to offer a free public service for auditing algorithms, starting with ECG algorithms. Any researcher or company can upload an ECG algorithm to Dandelion’s secure computing environment, if they have a legal right to do so. The participant will specify performance metrics. Dandelion will then run the algorithm on its dataset—which spans 10 million patients, representing a range of geographies, ethnic and racial groups, care settings, etc.—to produce performance metrics, overall and for pre-specified groups of interest.

  • This program is open to any researcher or company with an algorithm that takes an ECG waveform or Chest CT scan as input and predicts a number of outcome labels as its output.

    Our full list of outcome labels is available here: link.

  • We protect participants’ intellectual property by:

    1. Signing a contract that protects your IP rights

    2. Requiring the use of containerized algorithms so we can never see your IP

    3. Automating the entire validation process, including deletion of the container 30 days after completion of the confidential report

  • Depending on demand and capacity limitations, we reserve the right to limit the number of algorithms we accept from any one company or organization.

  • The goal of our pilot program is to make it easy for developers to gain insight into how well their algorithms perform using a scientifically rigorous validation process that protects patient privacy and is aligned with the public good. We believe that patients, Dandelion, and the industry as a whole will benefit from greater rigor and transparency around algorithm performance and bias.

  • Patient privacy is a central priority of everything we do at Dandelion. All the data that are used for this program are rigorously de-identified based on industry best practices and in accordance with Dandelion’s ethics statement.

  • We are starting with ECG and Chest CT algorithms because there are a wide range of interesting use cases already available in the market, and many more are in development. Dandelion has access to data for a wide range of other modalities as well and may expand the program to include other modalities based on market feedback.

  • Through our consortium of three major healthcare systems — Sharp HealthCare (San Diego, CA), Sanford Health (Sioux Falls, SD) and Texas Health Resources (Arlington, TX) — Dandelion has aggregated de-identified clinical data for roughly 10 million patients throughout the United States across a range of geographies, ethnic and racial groups, care settings and more. The breadth and depth of the resulting raw de-identified clinical data and Dandelion’s expertise in creating high-quality clinical datasets uniquely positions the company to validate algorithms using data that have not previously been available at this scale.

  • Ground truth is defined by you: you choose which metrics you’d like your algorithm to be evaluated on, from a menu of available data. Right now, the menu includes a range of possible labels, sourced from the ECG interpretation (by a cardiologist) and the EMR data:

    A full list of available outcome labels is available here: link.

  • The names of participating researchers and organizations will never be made public in any form by Dandelion. You may make your results public if you wish.

  • Complete and submit our Contact Form.