diff --git a/.gitignore b/.gitignore index 6e843bfaa..d0f33f313 100644 --- a/.gitignore +++ b/.gitignore @@ -9,6 +9,8 @@ mimic-iv/docker-postgres/.env # data *.csv.gz +# reference data distributed with the repository +!mimic-iii/concepts/diagnosis/ccs_multi_dx.csv.gz # credentials key.json diff --git a/README.md b/README.md index 826fb1ec1..53fe9f9f3 100644 --- a/README.md +++ b/README.md @@ -6,7 +6,7 @@ You can read more about the code repository in the following open access paper: ## Cloud access to datasets -The various MIMIC databases are available on Google Cloud Platform (GCP) and Amazon Web Services (AWS). To access the data on the cloud, simply add the relevant cloud identifier to your PhysioNet profile. Then request access to the dataset for the particular cloud platform via the PhysioNet project page. Further instructions are available on [the MIMIC website](https://mimic.mit.edu/iv/access/cloud/). +The various MIMIC databases are available on Google Cloud Platform (GCP) and Amazon Web Services (AWS). To access the data on the cloud, simply add the relevant cloud identifier to your PhysioNet profile. Then request access to the dataset for the particular cloud platform via the PhysioNet project page. Further instructions are available on [the MIMIC website](https://mimic.mit.edu/docs/gettingstarted/cloud/). ## Navigating this repository @@ -71,7 +71,7 @@ pytest tests/ ## Other useful tools -* [Bloatectomy](https://github.com/MIT-LCP/bloatectomy) ([paper](https://github.com/MIT-LCP/bloatectomy/blob/master/paper/paper.md)) - A python based package for removing duplicate text in clinical notes +* [Bloatectomy](https://github.com/MIT-LCP/bloatectomy) ([paper](https://github.com/MIT-LCP/bloatectomy/blob/master/paper/bloatectomy_paper.md)) - A python based package for removing duplicate text in clinical notes * [Medication categories](https://github.com/mghassem/medicationCategories) - Python script for extracting medications from free-text notes * [MIMIC Extract](https://github.com/MLforHealth/MIMIC_Extract) ([paper](https://doi.org/10.1145/3368555.3384469)) - A python based package for transforming MIMIC-III data into a machine learning friendly format * [FIDDLE](https://github.com/MLD3/FIDDLE) ([paper](https://doi.org/10.1093/jamia/ocaa139)) - A python based package for a FlexIble Data-Driven pipeLinE (FIDDLE), transforming structured EHR data into a machine learning friendly format diff --git a/mimic-iii/Makefile.md b/mimic-iii/Makefile.md index 404b2497e..65c04b57d 100644 --- a/mimic-iii/Makefile.md +++ b/mimic-iii/Makefile.md @@ -7,10 +7,13 @@ Starting from a fresh system which has GNU Make installed, PostgreSQL installed, ```sh export datadir="/path/to/data" +make create-user make mimic-download physionetuser= datadir=$datadir make mimic-gz datadir=$datadir ``` +The `create-user` step creates the database, schema, and user (by default the database is named `mimic` and the user is `postgres`); skip it if these already exist. + Note that if you have already downloaded the data, you can skip the `make mimic-download`, just be sure to set `datadir` appropriately. If you have already decompressed the data into `.csv` files, call `make mimic` instead of `make mimic-gz`, e.g. `make mimic datadir=/path/to/data`. Optionally, additional contributed materialized views can be created afterward by running: @@ -19,7 +22,7 @@ Optionally, additional contributed materialized views can be created afterward b make concepts ``` -Note that you may want to modify parameters at the top of the Makefile - e.g. the username (see below "non-standard username or database name"). +Note that you may want to modify parameters at the top of `buildmimic/postgres/Makefile` - e.g. the username (see below "non-standard username or database name"). ### Authentication In order to avoid the prompts for your database password each time, you may create a file in your home directory called .pgpass containing the following: @@ -33,12 +36,12 @@ Replace ```mimic``` with your username and ```password``` with your password. No Alternatively you can configure the database to use operating system level authentication. See the Postgres manual for more detail (in particular, the section(s) on "Peer Authentication"): https://www.postgresql.org/docs/9.6/static/auth-methods.html ### Non-standard username or database name -If you need to use a username or database name other than ```mimic```, then you will need to specify this by modifying the top-level Makefile: +If you need to use a database name other than ```mimic``` or a username other than ```postgres```, then you will need to specify this by modifying the defaults at the top of `buildmimic/postgres/Makefile`: ``` -DBNAME=mimic -DBUSER=mimic +DBNAME := mimic +DBUSER := postgres ``` ## Contributing -If you would like to contribute code to create a materialized view to the `concepts` folder, simply add a command which calls the script to the `concepts/make-concepts.sql` file. The format is fairly straightforward: e.g. adding the `\i sepsis/angus.sql` line informs the script to call the `concepts/sepsis/angus.sql` file. +If you would like to contribute a new concept, add the script (written in the BigQuery dialect) to the [concepts](/mimic-iii/concepts) folder and add a corresponding line to [concepts/make-concepts.sh](/mimic-iii/concepts/make-concepts.sh). The PostgreSQL and DuckDB versions in [concepts_postgres](/mimic-iii/concepts_postgres) and [concepts_duckdb](/mimic-iii/concepts_duckdb) are then regenerated by transpilation, and the new script should be added to `concepts_postgres/postgres-make-concepts.sql` and `concepts_duckdb/duckdb.sql`. See the [concepts README](/mimic-iii/concepts/README.md) for details. diff --git a/mimic-iii/benchmark/postgres-benchmark-4.sql b/mimic-iii/benchmark/postgres-benchmark-4.sql index c7c398eae..d914868d1 100644 --- a/mimic-iii/benchmark/postgres-benchmark-4.sql +++ b/mimic-iii/benchmark/postgres-benchmark-4.sql @@ -18,7 +18,7 @@ STOPPED VARCHAR(50) ) ; --- PARTITIONNING +-- PARTITIONING -- CREATE CHARTEVENTS_PARTITIONED TABLE CREATE TABLE CHARTEVENTS_PARTITIONED_0 ( CHECK ( itemid in (211, 220045, 51, 442, 455, 6701, 220179, 220050, 8368, 8440, 8441, 8555, 220180, 220051, 456, 52, 6702, 443, 220052, 220181, 225312, 618, 615, 220210, 224690, 646, 220277, 223762, 676, 223761, 678 ) )) INHERITS (CHARTEVENTS_PARTITIONED); diff --git a/mimic-iii/buildmimic/README.md b/mimic-iii/buildmimic/README.md index 92929c4de..ce11732e9 100644 --- a/mimic-iii/buildmimic/README.md +++ b/mimic-iii/buildmimic/README.md @@ -1,6 +1,6 @@ # Building the MIMIC database -This directory contains scripts that can be used to create a new instance of the MIMIC Critical Care Database. Tutorials for building a local PostgreSQL database using these scripts are available on the MIMIC website, under the "Tutorials" item in the menu: http://mimic.physionet.org/about/mimic/ +This directory contains scripts that can be used to create a new instance of the MIMIC Critical Care Database. Tutorials for building a local PostgreSQL database using these scripts are available on the MIMIC website, under the "Tutorials" item in the menu: https://mimic.mit.edu/docs/III/about/ ## Automated tests (PostgreSQL scripts only) diff --git a/mimic-iii/buildmimic/apache-drill/README.md b/mimic-iii/buildmimic/apache-drill/README.md index ae0db324f..a3c46aa46 100644 --- a/mimic-iii/buildmimic/apache-drill/README.md +++ b/mimic-iii/buildmimic/apache-drill/README.md @@ -66,7 +66,7 @@ In DBeaver, connect and copy/paste: 1. drill_create_data.sql 1. create alias for all csv in temp (dfs.tmp = /tmp on linux) ln -s /path/to/mimic/csv/\* /tmp -1. create table one by one. Otherwize, it crashes when multi query are made in one run +1. create table one by one. Otherwise, it crashes when multi query are made in one run ## Notes diff --git a/mimic-iii/buildmimic/aws-athena/README.md b/mimic-iii/buildmimic/aws-athena/README.md index 0e7fb9133..5c2e5d0dc 100644 --- a/mimic-iii/buildmimic/aws-athena/README.md +++ b/mimic-iii/buildmimic/aws-athena/README.md @@ -1,17 +1,17 @@ # Study MIMIC-III data using AWS Athena -The MIMIC-III dataset is available in the AWS cloud through our [Open Data on AWS program](https://registry.opendata.aws/). This allows researchers to use the MIMIC-III dataset without having to download it, make a copy of it, or pay to store it. They can simply analyze MIMIC-III using AWS services like Amazon EC2, Amazon Athena, AWS Lambda, or Amazon EMR by pointing to its address in the AWS cloud. This makes it faster and less expensive to perform research studies. In order to gain access, please login to the [MIMIC PhysioNet website](https://mimic.physionet.org/), [input your AWS account number](https://physionet.org/settings/cloud/), and [request access to the MIMIC-III Clinical Database on AWS](https://physionet.org/projects/mimiciii/1.4/request_access/2). +The MIMIC-III dataset is available in the AWS cloud through our [Open Data on AWS program](https://registry.opendata.aws/). This allows researchers to use the MIMIC-III dataset without having to download it, make a copy of it, or pay to store it. They can simply analyze MIMIC-III using AWS services like Amazon EC2, Amazon Athena, AWS Lambda, or Amazon EMR by pointing to its address in the AWS cloud. This makes it faster and less expensive to perform research studies. In order to gain access, please login to the [MIMIC PhysioNet website](https://mimic.mit.edu/), [input your AWS account number](https://physionet.org/settings/cloud/), and [request access to the MIMIC-III Clinical Database on AWS](https://physionet.org/projects/mimiciii/1.4/request_access/2). Found here is some useful code to help you understand how to access the MIMIC-III dataset on AWS. ### mimic-iii-athena.yaml -This is an [AWS CloudFormation Template](https://aws.amazon.com/cloudformation/) that will deploy a database in the [AWS Glue](https://aws.amazon.com/glue/) Data Catalog that contains all of the MIMIC-III tables. It also deploys a Jupyter Notebook instance in [Amazon SageMaker](https://aws.amazon.com/sagemaker/) that contains the content of this [mimic-code](https://github.com/MIT-LCP/mimic-code/) GitHub repository and is set up to access the MIMIC-III data through AWS Glue. This repository contains a version of the [Aline Study](https://github.com/JamesSWiggins/mimic-code/tree/master/notebooks/aline/awsathena) that is configured to run it's SQL queries against the MIMIC-III data in AWS using [AWS Athena](https://aws.amazon.com/athena/). +This is an [AWS CloudFormation Template](https://aws.amazon.com/cloudformation/) that will deploy a database in the [AWS Glue](https://aws.amazon.com/glue/) Data Catalog that contains all of the MIMIC-III tables. It also deploys a Jupyter Notebook instance in [Amazon SageMaker](https://aws.amazon.com/sagemaker/) that contains the content of this [mimic-code](https://github.com/MIT-LCP/mimic-code/) GitHub repository and is set up to access the MIMIC-III data through AWS Glue. This repository contains a version of the [Aline Study](/mimic-iii/notebooks/aline-aws) that is configured to run it's SQL queries against the MIMIC-III data in AWS using [AWS Athena](https://aws.amazon.com/athena/). Use the below Launch Stack button to deploy this AWS CloudFormation template into your AWS account. On the first screen, the template link has already been specified, so just click next. On the second screen, provide a Stack name (letters and numbers) and click next, on the third screen, just click next. On the forth screen, at the bottom, there is a box that says **I acknowledge that AWS CloudFormation might create IAM resources.**. Check that box, and then click **Create**. Once the Stack has complete deploying, look at the **Outputs** tab of the AWS CloudFormation console for links to your Juypter Notebooks instance. [![cloudformation-launch-stack](cloudformation-launch-stack.png)](https://console.aws.amazon.com/cloudformation/home?region=us-east-1#/stacks/new?stackName=MIMIC&templateURL=https://aws-bigdata-blog.s3.amazonaws.com/artifacts/biomedical-informatics-studies/mimic-iii-athena.yaml) ### mimictoparquet_glue_job.py -The MIMIC-III dataset is offered on AWS in both the original gzipped CSV format as well as the Apache Parquet format. This Python script is run by the MIMIC team as an AWS Glue job to convert the CSV.gz files to Apache Parquet format. It uses Apache Spark to help conver the files. You don't need to use this script yourself to use the MIMIC-III data in Parquet format, but it is provided as a template that could be used to convert other data sets from CSV to Parquet. The benefits of querying this data in Parquet format are outlined in this blog post: +The MIMIC-III dataset is offered on AWS in both the original gzipped CSV format as well as the Apache Parquet format. This Python script is run by the MIMIC team as an AWS Glue job to convert the CSV.gz files to Apache Parquet format. It uses Apache Spark to help convert the files. You don't need to use this script yourself to use the MIMIC-III data in Parquet format, but it is provided as a template that could be used to convert other data sets from CSV to Parquet. The benefits of querying this data in Parquet format are outlined in this blog post: ### mimic-aws-athena-ddl.sql This is a SQL script that can be run directly within AWS Athena to define each of the MIMIC-III dataset tables from the Apache Parquet files provide in the AWS OpenData program. You do not need to execute these if you use the above ```mimic-iii-athena.yaml``` CloudFormation template. The table spaces are defined for you by that automation. However, if you wanted to define the tables yourself within your account, you can use these SQL statements. diff --git a/mimic-iii/buildmimic/bigquery/README.md b/mimic-iii/buildmimic/bigquery/README.md index b7ea10da5..8b03a48e9 100644 --- a/mimic-iii/buildmimic/bigquery/README.md +++ b/mimic-iii/buildmimic/bigquery/README.md @@ -2,7 +2,7 @@ Following are the steps to create the MIMIC-III dataset on BigQuery and load the source files (.csv.gz) downloaded from Physionet. -**IMPORTANT**: Only users with approved Physionet Data Use Agreement (DUA) should have access to the MIMIC dataset via BigQuery or Cloud Storage. If you don't have access to MIMIC, follow the instructions [here](https://mimic.physionet.org/gettingstarted/access/) to request access. +**IMPORTANT**: Only users with approved Physionet Data Use Agreement (DUA) should have access to the MIMIC dataset via BigQuery or Cloud Storage. If you don't have access to MIMIC, follow the instructions [here](https://mimic.mit.edu/docs/gettingstarted/) to request access. --- @@ -12,7 +12,7 @@ Following are the steps to create the MIMIC-III dataset on BigQuery and load the For this tutorial, we will proceed using the compressed files (.csv.gz) stored in a Google Cloud Storage (GCS) bucket. In order to use these files, you must have a Google account with access permission granted via PhysioNet. -You can read about being provisioned access to MIMIC-III on Google [on the cloud tutorial page](https://mimic.physionet.org/gettingstarted/cloud/). +You can read about being provisioned access to MIMIC-III on Google [on the cloud tutorial page](https://mimic.mit.edu/docs/gettingstarted/cloud/). Once you have configured your account on PhysioNet, go to the [MIMIC-III page on PhysioNet](https://physionet.org/content/mimiciii/) and scroll down to the Files section. @@ -139,12 +139,12 @@ BigQuery schemas are defined by JSON files. These files are an array of dictiona The information about the columns are compiled from several sources: 1. [The PostgreSQL build scripts](https://github.com/MIT-LCP/mimic-code/tree/main/mimic-iii/buildmimic/postgres) -2. [The MIMIC-III online documentation](https://mimic.physionet.org/about/mimic/) +2. [The MIMIC-III online documentation](https://mimic.mit.edu/docs/III/about/) 3. [A schematic of the MIMIC-III database](https://mit-lcp.github.io/mimic-schema-spy/) Currently all columns are set as NULLABLE as the tables will be used for searching/querying only and users with Viewer role should not be able to modify the tables (ex: insert/delete/update data). However, depending on your environment, you may need to set the mode REQUIRED for the appropriate columns. -We selected the type DATETIME for all columns with dates and times. However, you could also use TIMESTAMP. As [discussed in the MIMIC-III documentation](https://mimic.physionet.org/mimicdata/time), columns with suffix DATE (ex: CHARTDATE) "will always have 00:00:00 as the hour, minute, and second values. This does not mean it was recorded at midnight: it indicates that we do not have the exact time, only the date". This is also true for other columns such DOB and DOD (patients table). +We selected the type DATETIME for all columns with dates and times. However, you could also use TIMESTAMP. As [discussed in the MIMIC-III documentation](https://mimic.mit.edu/docs/III/about/time.html), columns with suffix DATE (ex: CHARTDATE) "will always have 00:00:00 as the hour, minute, and second values. This does not mean it was recorded at midnight: it indicates that we do not have the exact time, only the date". This is also true for other columns such DOB and DOD (patients table). --- diff --git a/mimic-iii/buildmimic/duckdb/README.md b/mimic-iii/buildmimic/duckdb/README.md index 04a0c38cf..5ab9021f1 100644 --- a/mimic-iii/buildmimic/duckdb/README.md +++ b/mimic-iii/buildmimic/duckdb/README.md @@ -85,7 +85,7 @@ $ Here's an example invocation that will make the database in the default "mimic3.db": ```sh -$ ./import_duckdb.sh physionet.org/files/mimiciii/1.4 +$ ./import_duckdb.sh mimiciii/1.4 ... output removed Successfully finished loading data into mimic3.db. diff --git a/mimic-iii/buildmimic/duckdb/import_duckdb.sh b/mimic-iii/buildmimic/duckdb/import_duckdb.sh index 15d4977fe..77b76c7d1 100755 --- a/mimic-iii/buildmimic/duckdb/import_duckdb.sh +++ b/mimic-iii/buildmimic/duckdb/import_duckdb.sh @@ -29,7 +29,7 @@ usage () { die " USAGE: ./import_duckdb.sh mimic_data_dir [output_db] WHERE: - mimic_data_dir directory that contains csv.tar.gz or csv files + mimic_data_dir directory that contains csv.gz or csv files output_db: optional filename for duckdb file (default: mimic3.db)\ " } diff --git a/mimic-iii/buildmimic/oracle/README.md b/mimic-iii/buildmimic/oracle/README.md index 62e19a791..f1ad6162d 100644 --- a/mimic-iii/buildmimic/oracle/README.md +++ b/mimic-iii/buildmimic/oracle/README.md @@ -31,11 +31,11 @@ Alternatively, you can follow this step by step guide: 3. Open up either `sqlplus` or SQLDeveloper 4. Convert NOTEEVENTS.csv into a workable format: `python add_oracle_rowdelimiter.py -d ',' -i 'NOTEEVENTS.csv' -r '><> diff --git a/mimic-iii/concepts/README.md b/mimic-iii/concepts/README.md index cedf3c476..039474bcc 100644 --- a/mimic-iii/concepts/README.md +++ b/mimic-iii/concepts/README.md @@ -5,7 +5,7 @@ The scripts are intended to be run against the BigQuery instantiation of MIMIC-I Concepts are categorized into folders if possible, otherwise they remain in the top-level directory. A table of contents is provided below: [List of concepts](#list-of-concepts). -You can read about cloud access to MIMIC-III, including via Google BigQuery, on the [cloud page](https://mimic.physionet.org/gettingstarted/cloud/). +You can read about cloud access to MIMIC-III, including via Google BigQuery, on the [cloud page](https://mimic.mit.edu/docs/gettingstarted/cloud/). The rest of this README describes: @@ -42,6 +42,7 @@ Folder | Table | Description --- | --- | --- . | [echo_data](echo_data.sql) | Text extracted from echocardiography reports using regular expressions. . | [code_status](code_status.sql) | Whether the patient has restrictions on life saving resuscitation. +. | [rrt](rrt.sql) | Whether the patient received any dialysis during their ICU stay. comorbidity | [elixhauser_ahrq_v37](comorbidity/elixhauser_ahrq_v37.sql) | Comorbidities in categories proposed by Elixhauser et al. AHRQ produced the mapping. comorbidity | [elixhauser_ahrq_v37_no_drg](comorbidity/elixhauser_ahrq_v37_no_drg.sql) | As above, but DRG codes are not used to exclude primary conditions. comorbidity | [elixhauser_quan](comorbidity/elixhauser_quan.sql) | Comorbidities in categories proposed by Elixhauser et al. using an algorithm by Quan et al. @@ -76,7 +77,7 @@ durations | [neuroblock_dose](durations/neuroblock_dose.sql) durations | [norepinephrine_dose](durations/norepinephrine_dose.sql) | Dose administered with start/stop times for norepinephrine. durations | [phenylephrine_dose](durations/phenylephrine_dose.sql) | Dose administered with start/stop times for phenylephrine. durations | [vasopressin_dose](durations/vasopressin_dose.sql) | Dose administered with start/stop times for vasopressin. -durations | [ventilation_classification](durations/ventilation_classification.sql) | Classifies patient settings as implying mechanical ventilation. +durations | [ventilation_classification](durations/ventilation_classification.sql) | Classifies ventilator settings (tidal volume, PEEP, etc.) as implying mechanical ventilation. durations | [ventilation_durations](durations/ventilation_durations.sql) | Start and stop times for mechanical ventilation. durations | [weight_durations](durations/weight_durations.sql) | Start and stop times for daily weight measurements. **firstday** | | The first day subfolder contains scripts to summarizes a patient's health on their first ICU day. @@ -93,6 +94,8 @@ firstday | [weight_first_day](firstday/weight_first_day.sql) **fluid_balance** | | Tables which track fluid input and output for the patient. fluid_balance | [colloid_bolus](fluid_balance/colloid_bolus.sql) | Times at which a patient received a bolus of colloidal fluid. fluid_balance | [crystalloid_bolus](fluid_balance/crystalloid_bolus.sql) | Times at which a patient received a bolus of crystalloid fluid. +fluid_balance | [ffp_transfusion](fluid_balance/ffp_transfusion.sql) | Times at which a patient received a transfusion of fresh frozen plasma. +fluid_balance | [rbc_transfusion](fluid_balance/rbc_transfusion.sql) | Times at which a patient received a transfusion of red blood cells. fluid_balance | [urine_output](fluid_balance/urine_output.sql) | Urine output for a patient with the time of documentation. **organfailure** | | Summarizations of the degree of organ failure for single organ systems. organfailure | [kdigo_creatinine](organfailure/kdigo_creatinine.sql) | Creatinine values with baseline creatinine as defined by KDIGO. @@ -103,6 +106,7 @@ organfailure | [kdigo_stages_7day](organfailure/kdigo_stages_7day.sql) organfailure | [meld](organfailure/meld.sql) | The MELD score, often used to assess health of liver transplant candidates. **pivot** | | Pivoted views contain the patient `icustay_id`, the `charttime`, and a number of variables. They are useful to acquiring a time series of values for patient stays. pivot | [pivoted_bg](pivot/pivoted_bg.sql) | Blood gas measurements. +pivot | [pivoted_bg_art](pivot/pivoted_bg_art.sql) | Arterial blood gas measurements, explicitly labelled or inferred from SpO2. pivot | [pivoted_fio2](pivot/pivoted_fio2.sql) | Fraction of inspired oxygen. pivot | [pivoted_gcs](pivot/pivoted_gcs.sql) | Glasgow Coma Scale. pivot | [pivoted_height](pivot/pivoted_height.sql) | Height. @@ -113,7 +117,6 @@ pivot | [pivoted_oasis](pivot/pivoted_oasis.sql) pivot | [pivoted_rrt](pivot/pivoted_rrt.sql) | Renal replacement therapy. pivot | [pivoted_sofa](pivot/pivoted_sofa.sql) | The Sequential Organ Failure Assessment (SOFA) scale. pivot | [pivoted_uo](pivot/pivoted_uo.sql) | Urine output. -pivot | [pivoted_vent_setting](pivot/pivoted_vent_setting.sql) | Ventilator settings (tidal volume, PEEP, etc). pivot | [pivoted_vital](pivot/pivoted_vital.sql) | Vital signs. **sepsis** | | Definitions of sepsis, a common cause of mortality for intensive care unit patients. sepsis | [angus](sepsis/angus.sql) | Sepsis defined using billing codes validated by Angus et al. diff --git a/mimic-iii/concepts/diagnosis/README.md b/mimic-iii/concepts/diagnosis/README.md index ae6d02301..b323d76e0 100644 --- a/mimic-iii/concepts/diagnosis/README.md +++ b/mimic-iii/concepts/diagnosis/README.md @@ -4,7 +4,7 @@ The Clinical Classification Software (CCS) categorizes ICD-9 coded diagnoses int The `ccs_multi_dx.csv.gz` data file must be uploaded to `physionet-data.mimiciii_derived.ccs_multi_dx`. -The BigQuery schema definition is available in this folder as [ccs_multi_dx.json](/ccs_multi_dx.json). +The BigQuery schema definition is available in this folder as [ccs_multi_dx.json](ccs_multi_dx.json). ## Creation of the ccs_multi_dx.csv.gz file @@ -24,7 +24,7 @@ Use Python to convert all apostrophes in `ccs_multi_dx_tool_2015.csv` into doubl ```python import pandas as pd -df = pd.read_csv('ccs_multi_dx_tool_2015.csv.gz') +df = pd.read_csv('ccs_multi_dx_tool_2015.csv') # remove apostrophes from header names and relabel df.rename(columns={"'ICD-9-CM CODE'": "icd9_code", "'CCS LVL 1'": "ccs_level1", "'CCS LVL 1 LABEL'": "ccs_group1", "'CCS LVL 2'": "ccs_level2", "'CCS LVL 2 LABEL'": "ccs_group2", "'CCS LVL 3'": "ccs_level3", "'CCS LVL 3 LABEL'": "ccs_group3", "'CCS LVL 4'": "ccs_level4", "'CCS LVL 4 LABEL'": "ccs_group4", }, inplace=True) diff --git a/mimic-iii/concepts/diagnosis/ccs_dx.sql b/mimic-iii/concepts/diagnosis/ccs_dx.sql index d67062d15..24781e525 100644 --- a/mimic-iii/concepts/diagnosis/ccs_dx.sql +++ b/mimic-iii/concepts/diagnosis/ccs_dx.sql @@ -1,5 +1,5 @@ -- add in matched ID, name, and ccs_id --- matched id (ccs_mid): the ccs ID with the hierachy, e.g. 7.1.2.1 +-- matched id (ccs_mid): the ccs ID with the hierarchy, e.g. 7.1.2.1 -- name (ccs_name): the most granular CCS category the diagnosis is in -- ID (ccs_id): the CCS identifier for the ICD-9 code (integer) SELECT diff --git a/mimic-iii/concepts/diagnosis/ccs_multi_dx.csv.gz b/mimic-iii/concepts/diagnosis/ccs_multi_dx.csv.gz new file mode 100644 index 000000000..5e13857a6 Binary files /dev/null and b/mimic-iii/concepts/diagnosis/ccs_multi_dx.csv.gz differ diff --git a/mimic-iii/concepts/severityscores/README.md b/mimic-iii/concepts/severityscores/README.md index 9ede5159e..e407cbbc8 100644 --- a/mimic-iii/concepts/severityscores/README.md +++ b/mimic-iii/concepts/severityscores/README.md @@ -3,19 +3,24 @@ This folder generates materialized views for a variety of severity of illness scores (detailed below). The queries make use of materialized views which aggregate data from the first day of a patient's ICU stay. To run the code, it is necessary to: -1. Run the scripts which generate the constituent materialized views - * echodata - generated by /concepts/echo-data.sql - * ventdurations - generated by /concepts/durations/ventilation-durations.sql - (needed for subsequent view) - * vitalsfirstday - generated by /concepts/firstday/vitals-first-day.sql - * uofirstday - generated by /concepts/firstday/urine-output-first-day.sql - * ventfirstday - generated by /concepts/firstday/ventilated-first-day.sql - * gcsfirstday - generated by /concepts/firstday/gcs-first-day.sql - * labsfirstday - generated by /concepts/firstday/labs-first-day.sql - * bloodgasfirstday - generated by /concepts/firstday/blood-gas-first-day.sql - (needed for subsequent view) - * bloodgasfirstdayarterial - generated by /concepts/firstday/blood-gas-first-day-arterial.sql +1. Run the scripts which generate the constituent tables + * echo_data - generated by [/mimic-iii/concepts/echo_data.sql](/mimic-iii/concepts/echo_data.sql) + * ventilation_durations - generated by [/mimic-iii/concepts/durations/ventilation_durations.sql](/mimic-iii/concepts/durations/ventilation_durations.sql) - (needed for subsequent table) + * vitals_first_day - generated by [/mimic-iii/concepts/firstday/vitals_first_day.sql](/mimic-iii/concepts/firstday/vitals_first_day.sql) + * urine_output_first_day - generated by [/mimic-iii/concepts/firstday/urine_output_first_day.sql](/mimic-iii/concepts/firstday/urine_output_first_day.sql) + * ventilation_first_day - generated by [/mimic-iii/concepts/firstday/ventilation_first_day.sql](/mimic-iii/concepts/firstday/ventilation_first_day.sql) + * gcs_first_day - generated by [/mimic-iii/concepts/firstday/gcs_first_day.sql](/mimic-iii/concepts/firstday/gcs_first_day.sql) + * labs_first_day - generated by [/mimic-iii/concepts/firstday/labs_first_day.sql](/mimic-iii/concepts/firstday/labs_first_day.sql) + * blood_gas_first_day - generated by [/mimic-iii/concepts/firstday/blood_gas_first_day.sql](/mimic-iii/concepts/firstday/blood_gas_first_day.sql) - (needed for subsequent table) + * blood_gas_first_day_arterial - generated by [/mimic-iii/concepts/firstday/blood_gas_first_day_arterial.sql](/mimic-iii/concepts/firstday/blood_gas_first_day_arterial.sql) 2. Run the script for the severity of illness score you are interested in + * APS III - apsiii.sql + * LODS - lods.sql (and a modified version, mlods.sql) * OASIS - oasis.sql + * qSOFA - qsofa.sql * SAPS - saps.sql + * SAPS II - sapsii.sql + * SIRS - sirs.sql * SOFA - sofa.sql The scripts were written and tested in PostgreSQL 9.4.4. The scripts are written to follow the SQL standard as close as possible, and there should be minimal changes necessary for them to operate under different languages. Key changes likely involve: @@ -48,11 +53,7 @@ Note that the organ-specific morbidity scores are highlighted as a useful compli The following severity scores have yet to be implemented, but are desirable, and contributions are welcome. -## SAPS II - -The Simplified Acute Physiology Score II (SAPS II) published in 1993 (Le Gall, 1993), aimed to rectify two issues with SAPS. First, the variable selection process in SAPS was done by clinical judgement, whereas SAPS II utilised univariate feature selection to filter out features uncorrelated with hospital mortality. Second, there was no model for calculating a probability of mortality from SAPS. SAPS II was published with calibration coefficients which allowed conversion of the integer score into a risk of mortality which ranged between zero and one. - -### SAPS III +## SAPS III The Simplified Acute Physiology Score III (SAPS III) is the latest model published and attempts to account for poor calibration of SAPS II found in later studies (Metnitz, 2005). While SAPS II was developed on ICUs in western Europe, SAPS III included ICUs worldwide. diff --git a/mimic-iii/concepts_duckdb/README.md b/mimic-iii/concepts_duckdb/README.md index eb23446e2..f6649b9f9 100644 --- a/mimic-iii/concepts_duckdb/README.md +++ b/mimic-iii/concepts_duckdb/README.md @@ -4,7 +4,7 @@ This folder contains scripts to generate useful abstractions of raw MIMIC-III da Almost all of these scripts were generated automatically from the BigQuery SQL dialect using the [sqlglot](https://github.com/tobymao/sqlglot) package. If you would like to contribute a correction, do not make it here. Instead, make your correction in the [concepts folder](/mimic-iii/concepts/) using the BigQuery SQL syntax, and regenerate this folder as described in that folder's [README](/mimic-iii/concepts/README.md). -One file is hand-written for DuckDB and has no BigQuery source: [diagnosis/ccs_multi_dx.sql](diagnosis/ccs_multi_dx.sql) loads the ICD-9 to CCS mapping from [diagnosis/ccs_multi_dx.csv.gz](diagnosis/ccs_multi_dx.csv.gz). +One file is hand-written for DuckDB and has no BigQuery source: [diagnosis/ccs_multi_dx.sql](diagnosis/ccs_multi_dx.sql) loads the ICD-9 to CCS mapping from [ccs_multi_dx.csv.gz in the concepts folder](/mimic-iii/concepts/diagnosis/ccs_multi_dx.csv.gz). ## Using these concepts diff --git a/mimic-iii/concepts_duckdb/diagnosis/ccs_multi_dx.sql b/mimic-iii/concepts_duckdb/diagnosis/ccs_multi_dx.sql index a67d5b810..89174eb33 100644 --- a/mimic-iii/concepts_duckdb/diagnosis/ccs_multi_dx.sql +++ b/mimic-iii/concepts_duckdb/diagnosis/ccs_multi_dx.sql @@ -1,13 +1,14 @@ -- THIS SCRIPT IS HAND-WRITTEN FOR DUCKDB (there is no BigQuery source). --- Loads the CCS multi-level diagnosis mapping from the CSV distributed --- alongside this script. On BigQuery the same table is loaded with: +-- Loads the CCS multi-level diagnosis mapping from the CSV stored in the +-- concepts folder (../concepts/diagnosis/ccs_multi_dx.csv.gz). On BigQuery +-- the same table is loaded with: -- bq load mimiciii_derived.ccs_multi_dx diagnosis/ccs_multi_dx.csv.gz diagnosis/ccs_multi_dx.json -- Run from the concepts_duckdb directory so the relative path resolves. DROP TABLE IF EXISTS mimiciii_derived.ccs_multi_dx; CREATE TABLE mimiciii_derived.ccs_multi_dx AS SELECT * FROM read_csv( - 'diagnosis/ccs_multi_dx.csv.gz', + '../concepts/diagnosis/ccs_multi_dx.csv.gz', header = true, columns = { 'icd9_code': 'VARCHAR(5)', diff --git a/mimic-iii/concepts_duckdb/duckdb.sql b/mimic-iii/concepts_duckdb/duckdb.sql index dc49fd86f..c74342eb9 100644 --- a/mimic-iii/concepts_duckdb/duckdb.sql +++ b/mimic-iii/concepts_duckdb/duckdb.sql @@ -95,6 +95,8 @@ .read pivot/pivoted_bg_art.sql .print 'pivot/pivoted_sofa.sql' .read pivot/pivoted_sofa.sql +.print 'pivot/pivoted_oasis.sql' +.read pivot/pivoted_oasis.sql -- comorbidity .print 'comorbidity/elixhauser_ahrq_v37.sql' diff --git a/mimic-iii/concepts_postgres/README.md b/mimic-iii/concepts_postgres/README.md index 250bf012c..0510e718c 100644 --- a/mimic-iii/concepts_postgres/README.md +++ b/mimic-iii/concepts_postgres/README.md @@ -6,7 +6,7 @@ If you would like to contribute a correction, do not make it here. Instead, make Two files are hand-written for PostgreSQL and have no BigQuery source: -* [diagnosis/ccs_multi_dx.sql](diagnosis/ccs_multi_dx.sql) loads the ICD-9 to CCS mapping from [diagnosis/ccs_multi_dx.csv.gz](diagnosis/ccs_multi_dx.csv.gz). +* [diagnosis/ccs_multi_dx.sql](diagnosis/ccs_multi_dx.sql) loads the ICD-9 to CCS mapping from [ccs_multi_dx.csv.gz in the concepts folder](/mimic-iii/concepts/diagnosis/ccs_multi_dx.csv.gz). * [demographics/note_counts.sql](demographics/note_counts.sql) is an optional PostgreSQL-only concept which summarizes note counts per hospital admission (it is not run by the make script). ## Using these concepts diff --git a/mimic-iii/concepts_postgres/diagnosis/ccs_dx.sql b/mimic-iii/concepts_postgres/diagnosis/ccs_dx.sql index 84215817e..19c352b05 100644 --- a/mimic-iii/concepts_postgres/diagnosis/ccs_dx.sql +++ b/mimic-iii/concepts_postgres/diagnosis/ccs_dx.sql @@ -1,6 +1,6 @@ -- THIS SCRIPT IS AUTOMATICALLY GENERATED. DO NOT EDIT IT DIRECTLY. DROP TABLE IF EXISTS mimiciii_derived.ccs_dx; CREATE TABLE mimiciii_derived.ccs_dx AS -/* add in matched ID, name, and ccs_id */ /* matched id (ccs_mid): the ccs ID with the hierachy, e.g. 7.1.2.1 */ /* name (ccs_name): the most granular CCS category the diagnosis is in */ /* ID (ccs_id): the CCS identifier for the ICD-9 code (integer) */ +/* add in matched ID, name, and ccs_id */ /* matched id (ccs_mid): the ccs ID with the hierarchy, e.g. 7.1.2.1 */ /* name (ccs_name): the most granular CCS category the diagnosis is in */ /* ID (ccs_id): the CCS identifier for the ICD-9 code (integer) */ SELECT icd9_code, COALESCE(ccs_level4, ccs_level3, ccs_level2, ccs_level1) AS ccs_matched_id, /* remove the trailing ccs_id from name column, i.e. "Burns [240.]" -> "Burns" */ diff --git a/mimic-iii/concepts_postgres/diagnosis/ccs_multi_dx.sql b/mimic-iii/concepts_postgres/diagnosis/ccs_multi_dx.sql index dd30e9c4c..22a09d48a 100644 --- a/mimic-iii/concepts_postgres/diagnosis/ccs_multi_dx.sql +++ b/mimic-iii/concepts_postgres/diagnosis/ccs_multi_dx.sql @@ -1,6 +1,7 @@ -- There is no BigQuery source for this script. --- Loads the CCS multi-level diagnosis mapping from the CSV distributed --- alongside this script. On BigQuery the same table is loaded with: +-- Loads the CCS multi-level diagnosis mapping from the CSV stored in the +-- concepts folder (../concepts/diagnosis/ccs_multi_dx.csv.gz). On BigQuery +-- the same table is loaded with: -- bq load mimiciii_derived.ccs_multi_dx diagnosis/ccs_multi_dx.csv.gz diagnosis/ccs_multi_dx.json -- Run from the concepts_postgres directory so the relative path resolves. DROP TABLE IF EXISTS mimiciii_derived.ccs_multi_dx; @@ -17,4 +18,4 @@ CREATE TABLE mimiciii_derived.ccs_multi_dx ccs_level4 VARCHAR(10), ccs_group4 VARCHAR(100) ); -\COPY mimiciii_derived.ccs_multi_dx FROM PROGRAM 'gzip -dc diagnosis/ccs_multi_dx.csv.gz' CSV HEADER; +\COPY mimiciii_derived.ccs_multi_dx FROM PROGRAM 'gzip -dc ../concepts/diagnosis/ccs_multi_dx.csv.gz' CSV HEADER; diff --git a/mimic-iii/concepts_postgres/postgres-make-concepts.sql b/mimic-iii/concepts_postgres/postgres-make-concepts.sql index 3779bef6c..40fe88413 100644 --- a/mimic-iii/concepts_postgres/postgres-make-concepts.sql +++ b/mimic-iii/concepts_postgres/postgres-make-concepts.sql @@ -62,6 +62,7 @@ SET search_path TO mimiciii_derived, mimiciii; \i pivot/pivoted_vital.sql \i pivot/pivoted_bg_art.sql \i pivot/pivoted_sofa.sql +\i pivot/pivoted_oasis.sql -- comorbidity \i comorbidity/elixhauser_ahrq_v37.sql diff --git a/mimic-iii/notebooks/aline-aws/README.md b/mimic-iii/notebooks/aline-aws/README.md index ecfc40834..526126d35 100644 --- a/mimic-iii/notebooks/aline-aws/README.md +++ b/mimic-iii/notebooks/aline-aws/README.md @@ -17,7 +17,7 @@ You can learn more about the details of this modification and see a performance The study can be reproduced by: -1. Use the below Launch Stack button to deploy access to the MIMIC-III dataset into your AWS account. This will give you real-time access to the MIMIC-III data in your AWS account without having to download a copy of the MIMIC-III dataset. It will also deploy a Jupyter Notebook with access to the content of this GitHub repository in your AWS account. Prior to launching this, please login to the [MIMIC PhysioNet website](https://mimic.physionet.org/), [input your AWS account number](https://physionet.org/settings/cloud/), and [request access to the MIMIC-III Clinical Database on AWS](https://physionet.org/projects/mimiciii/1.4/request_access/2). +1. Use the below Launch Stack button to deploy access to the MIMIC-III dataset into your AWS account. This will give you real-time access to the MIMIC-III data in your AWS account without having to download a copy of the MIMIC-III dataset. It will also deploy a Jupyter Notebook with access to the content of this GitHub repository in your AWS account. Prior to launching this, please login to the [MIMIC PhysioNet website](https://mimic.mit.edu/), [input your AWS account number](https://physionet.org/settings/cloud/), and [request access to the MIMIC-III Clinical Database on AWS](https://physionet.org/projects/mimiciii/1.4/request_access/2). To start this deployment, click the Launch Stack button. On the first screen, the template link has already been specified, so just click next. On the second screen, provide a Stack name (letters and numbers) and click next, on the third screen, just click next. On the forth screen, at the bottom, there is a box that says **I acknowledge that AWS CloudFormation might create IAM resources.**. Check that box, and then click **Create**. Once the Stack has complete deploying, look at the **Outputs** tab of the AWS CloudFormation console for links to your Juypter Notebooks instance. diff --git a/mimic-iii/notebooks/aline-aws/aline-awsathena.ipynb b/mimic-iii/notebooks/aline-aws/aline-awsathena.ipynb index f2c8d0a72..68cf11005 100644 --- a/mimic-iii/notebooks/aline-aws/aline-awsathena.ipynb +++ b/mimic-iii/notebooks/aline-aws/aline-awsathena.ipynb @@ -1122,7 +1122,7 @@ " -- severity of illness just before ventilation\n", " , so.sofa as sofa_first\n", "\n", - " -- vital sign value just preceeding ventilation\n", + " -- vital sign value just preceding ventilation\n", " , vi.map as map_first\n", " , vi.heartrate as hr_first\n", " , vi.temperature as temp_first\n", diff --git a/mimic-iii/notebooks/aline-aws/aline_labs-awsathena.sql b/mimic-iii/notebooks/aline-aws/aline_labs-awsathena.sql index 3273c0c0e..ddf11e169 100644 --- a/mimic-iii/notebooks/aline-aws/aline_labs-awsathena.sql +++ b/mimic-iii/notebooks/aline-aws/aline_labs-awsathena.sql @@ -1,6 +1,6 @@ CREATE TABLE DATABASE.ALINE_LABS as -with labs_preceeding as +with labs_preceding as ( select co.icustay_id , l.valuenum, l.charttime @@ -49,7 +49,7 @@ with labs_preceeding as select icustay_id, valuenum, label, obs_after_vent , ROW_NUMBER() over (partition by icustay_id, label, obs_after_vent order by charttime DESC) as rn - from labs_preceeding + from labs_preceding ) , labs_grp as ( diff --git a/mimic-iii/notebooks/aline-aws/aline_vitals-awsathena.sql b/mimic-iii/notebooks/aline-aws/aline_vitals-awsathena.sql index 6d30869ca..bd0ebcab5 100644 --- a/mimic-iii/notebooks/aline-aws/aline_vitals-awsathena.sql +++ b/mimic-iii/notebooks/aline-aws/aline_vitals-awsathena.sql @@ -37,7 +37,7 @@ with vitals_stg0 as and valuenum is not null and coalesce(error,0) != 1 ) --- next, assign an integer where rn=1 is the vital sign just preceeding vent +-- next, assign an integer where rn=1 is the vital sign just preceding vent , vitals_stg1 as ( select diff --git a/mimic-iii/notebooks/aline/README.md b/mimic-iii/notebooks/aline/README.md index 1017a1e2f..b817cea66 100644 --- a/mimic-iii/notebooks/aline/README.md +++ b/mimic-iii/notebooks/aline/README.md @@ -10,7 +10,7 @@ The code here reproduces this study in the MIMIC-III database. This involved man # Requirements -There are a number of prerequesites to running this code: +There are a number of prerequisites to running this code: * an installation of MIMIC-III in a PostgreSQL database * Python 2.7 with the numpy, pandas, matplotlib, and psycopg2 packages diff --git a/mimic-iii/notebooks/aline/aline.ipynb b/mimic-iii/notebooks/aline/aline.ipynb index a0c1df3f8..bdc92212d 100644 --- a/mimic-iii/notebooks/aline/aline.ipynb +++ b/mimic-iii/notebooks/aline/aline.ipynb @@ -1062,7 +1062,7 @@ " -- severity of illness just before ventilation\n", " , so.sofa as sofa_first\n", "\n", - " -- vital sign value just preceeding ventilation\n", + " -- vital sign value just preceding ventilation\n", " , vi.map as map_first\n", " , vi.heartrate as hr_first\n", " , vi.temperature as temp_first\n", diff --git a/mimic-iii/notebooks/aline/aline_labs.sql b/mimic-iii/notebooks/aline/aline_labs.sql index f7efec0a9..1f07ac4b8 100644 --- a/mimic-iii/notebooks/aline/aline_labs.sql +++ b/mimic-iii/notebooks/aline/aline_labs.sql @@ -3,7 +3,7 @@ DROP MATERIALIZED VIEW IF EXISTS ALINE_LABS CASCADE; CREATE MATERIALIZED VIEW ALINE_LABS as -with labs_preceeding as +with labs_preceding as ( select co.icustay_id , l.valuenum, l.charttime @@ -52,7 +52,7 @@ with labs_preceeding as select icustay_id, valuenum, label, obs_after_vent , ROW_NUMBER() over (partition by icustay_id, label, obs_after_vent order by charttime DESC) as rn - from labs_preceeding + from labs_preceding ) , labs_grp as ( diff --git a/mimic-iii/notebooks/aline/aline_vitals.sql b/mimic-iii/notebooks/aline/aline_vitals.sql index 9ee60b797..f3f93a2c0 100644 --- a/mimic-iii/notebooks/aline/aline_vitals.sql +++ b/mimic-iii/notebooks/aline/aline_vitals.sql @@ -38,7 +38,7 @@ with vitals_stg0 as and valuenum is not null and coalesce(error,0) != 1 ) --- next, assign an integer where rn=1 is the vital sign just preceeding vent +-- next, assign an integer where rn=1 is the vital sign just preceding vent , vitals_stg1 as ( select diff --git a/mimic-iii/tutorials/cohort-selection.ipynb b/mimic-iii/tutorials/cohort-selection.ipynb index c16cc53eb..43cfb338c 100644 --- a/mimic-iii/tutorials/cohort-selection.ipynb +++ b/mimic-iii/tutorials/cohort-selection.ipynb @@ -18,7 +18,7 @@ "* Python packages installable with: \n", " * `pip install numpy pandas matplotlib psycopg2 jupyter`\n", " \n", - "First, as always, we open a connection to a local copy of the database. If you don't have a local copy of the database in PostgreSQL, follow the tutorial online (http://mimic.physionet.org) to install one.\n", + "First, as always, we open a connection to a local copy of the database. If you don't have a local copy of the database in PostgreSQL, follow the tutorial online (https://mimic.mit.edu) to install one.\n", "\n", "Let's begin!" ] @@ -1825,7 +1825,7 @@ "Above we can see that the `curr_service` column gives an abbreviation for the current service. The `prev_service` column is null, *unless* the patient had a transfer of service, in which case it identifies the previous service. For example, we can see `subject_id = 471` has had at least two service changes: once from TSURG to MED and once from MED back to TSURG (note: there may be more as we have limited this query using `LIMIT 10`, and you could examine this patient in detail using `SELECT * FROM services WHERE subject_id = 471` if you like).\n", "\n", "A list of the unique services and their descriptions can be found at:\n", - "http://mimic.physionet.org/mimictables/services/\n", + "https://mimic.mit.edu/docs/III/tables/services.html\n", "\n", "In particular, if we want to exclude surgery, we should exclude patients who were admitted under:\n", "\n", diff --git a/mimic-iii/tutorials/data_viz/01_data_viz_basic.ipynb b/mimic-iii/tutorials/data_viz/01_data_viz_basic.ipynb index be2fb0623..6a0b7056b 100644 --- a/mimic-iii/tutorials/data_viz/01_data_viz_basic.ipynb +++ b/mimic-iii/tutorials/data_viz/01_data_viz_basic.ipynb @@ -65,8 +65,8 @@ } }, "source": [ - "* Database description: https://mimic.physionet.org/gettingstarted/overview/\n", - "* Table description: https://mimic.physionet.org/mimictables/admissions/\n", + "* Database description: https://mimic.mit.edu/docs/gettingstarted/\n", + "* Table description: https://mimic.mit.edu/docs/III/tables/admissions.html\n", "* ER-Diagram: https://mit-lcp.github.io/mimic-schema-spy/relationships.html" ] }, diff --git a/mimic-iii/tutorials/data_viz/README.md b/mimic-iii/tutorials/data_viz/README.md index ecc6f6e3a..7a4c82872 100644 --- a/mimic-iii/tutorials/data_viz/README.md +++ b/mimic-iii/tutorials/data_viz/README.md @@ -13,6 +13,6 @@ conda activate dataviz please change the path if MIMIC dataset is located at different directory > ex: "`YOUR_PATH`/D_LABITEMS.csv" -* This is made for the Data analysis and visualization tutorial at [TMF summer school 2019](http://tmf-ev.de/Termine/ctl/Details/Mid/785/ItemID/1466.aspx) +* This is made for the Data analysis and visualization tutorial at the TMF summer school 2019 * Authors: Prof. med. Thomas Ganslandt
* and Kim Hee diff --git a/mimic-iii/tutorials/dplyr-frontend/intro.html b/mimic-iii/tutorials/dplyr-frontend/intro.html index 90d36d2f7..6538bd7bd 100755 --- a/mimic-iii/tutorials/dplyr-frontend/intro.html +++ b/mimic-iii/tutorials/dplyr-frontend/intro.html @@ -163,7 +163,7 @@

Attach R packages

Instantiate the MIMIC-III database

-

I followed the Unix/Mac instructions at PhysioNet to create an instance of MIMIC-III.2 In particular, i used the user name (mimicuser), database name (mimic), and schema name (mimiciii) suggested there, with the password mimic. If you make different choices, then you’ll need to change the corresponding parameter values in the dbConnect() call below. While the queries in this notebook can be performed on the entire database, a user new to MIMIC, Postgres, or R may want to install the demo data set instead, following the same process except for the CSV files used.3

+

I followed the Unix/Mac instructions at PhysioNet to create an instance of MIMIC-III.2 In particular, i used the user name (mimicuser), database name (mimic), and schema name (mimiciii) suggested there, with the password mimic. If you make different choices, then you’ll need to change the corresponding parameter values in the dbConnect() call below. While the queries in this notebook can be performed on the entire database, a user new to MIMIC, Postgres, or R may want to install the demo data set instead, following the same process except for the CSV files used.3

Connect to MIMIC-III

@@ -305,7 +305,7 @@

Subset and join query tables

## 9 18340 1626 117062 1 41001 ## 10 30343 2700 100335 1 41001 ## # ... with more rows -

MI may not be listed as the principal diagnosis; as explained in the documentation for the patients table, the seq_num field is a priority ranking for the diagnoses generated at the end of stay. In order to focus on patients for whom MI was central to their hospitalization, i will include records with MI in any of the first five diagnosis positions, according to the "seq_num" field. To avoid duplicate admissions, i use group_by() and top_n() to limit the query to the first MI diagnosis for each admission.

+

MI may not be listed as the principal diagnosis; as explained in the documentation for the patients table, the seq_num field is a priority ranking for the diagnoses generated at the end of stay. In order to focus on patients for whom MI was central to their hospitalization, i will include records with MI in any of the first five diagnosis positions, according to the "seq_num" field. To avoid duplicate admissions, i use group_by() and top_n() to limit the query to the first MI diagnosis for each admission.

mi_admissions %>%
   filter(seq_num <= 5) %>%
   group_by(subject_id, hadm_id) %>%
@@ -392,7 +392,7 @@ 
Transform and augment query tables

 ##  9        158  169433            1 412       FALSE       
 ## 10        160  161672            1 41041     TRUE        
 ## # ... with more rows
-

Finally, i adopt a common outcome measure for critical care: 30-day mortality. I’m interested in survival after discharge, so i must restrict to patients who did not die in hospital. This information is recorded in the "hospital_expire_flag" field (though not yet described in the admissions table documentation; see the tutorial on querying MIMIC-III). I also require the dates (admission and discharge) of each stay from the admissions table and the date of death (where available) of each patient from the patients table. While i’m working with dates, i’ll also calculate each patient’s age on the day of admission.

+

Finally, i adopt a common outcome measure for critical care: 30-day mortality. I’m interested in survival after discharge, so i must restrict to patients who did not die in hospital. This information is recorded in the "hospital_expire_flag" field (though not yet described in the admissions table documentation; see the tutorial on querying MIMIC-III). I also require the dates (admission and discharge) of each stay from the admissions table and the date of death (where available) of each patient from the patients table. While i’m working with dates, i’ll also calculate each patient’s age on the day of admission.

I first join the necessary date fields into study_admissions. The syntax gets a bit cluttered here in order to keep the query to one pipeline. This is my own preference; you may prefer, especially while familiarizing yourself with dplyr, to cut these into smaller chunks.

study_admissions %>%
   left_join(
@@ -425,7 +425,7 @@ 
Transform and augment query tables

 ## 10        160  161672            1 41041     TRUE        
 ## # ... with more rows, and 4 more variables: admittime <dttm>,
 ## #   dischtime <dttm>, dob <dttm>, dod <dttm>
-

Functionality for working with dates and times is not yet implemented in dbplyr, but an invaluable feature of its SQL translation is that unrecognized functions pass through verbatim, where Postgres will attempt to interpret them. This allows to use date_part() below to extract components of timestamp fields as numbers. (Postgres also has a convenient age() function that would simplify the code chunk below, but this produces a character string that doesn’t lend itself to analysis purposes.) The documentation for the patients table explains that patients of 90 years and older had their ages artificially inflated, so i’ve removed these patients from my analysis. I reorder the fields toward the end in order to show the results of the date calculations. In the last transformation step, everything() adds in all the fields i don’t explicitly select.

+

Functionality for working with dates and times is not yet implemented in dbplyr, but an invaluable feature of its SQL translation is that unrecognized functions pass through verbatim, where Postgres will attempt to interpret them. This allows to use date_part() below to extract components of timestamp fields as numbers. (Postgres also has a convenient age() function that would simplify the code chunk below, but this produces a character string that doesn’t lend itself to analysis purposes.) The documentation for the patients table explains that patients of 90 years and older had their ages artificially inflated, so i’ve removed these patients from my analysis. I reorder the fields toward the end in order to show the results of the date calculations. In the last transformation step, everything() adds in all the fields i don’t explicitly select.

study_admissions %>%
   mutate(tt_death = date_part("day", dod) - date_part("day", dischtime)) %>%
   mutate(mortality = tt_death <= 30) %>%
diff --git a/mimic-iii/tutorials/dplyr-frontend/intro.md b/mimic-iii/tutorials/dplyr-frontend/intro.md
index 2c33ab6ef..908f3f5b6 100755
--- a/mimic-iii/tutorials/dplyr-frontend/intro.md
+++ b/mimic-iii/tutorials/dplyr-frontend/intro.md
@@ -76,7 +76,7 @@ in a few queries.
 ### Instantiate the MIMIC-III database
 
 I followed [the Unix/Mac instructions at
-PhysioNet](https://mimic.physionet.org/tutorials/install-mimic-locally-ubuntu/)
+PhysioNet](https://mimic.mit.edu/docs/gettingstarted/local/install-mimic-locally-ubuntu.html)
 to create an instance of MIMIC-III.[2] In particular, i used the user
 name (`mimicuser`), database name (`mimic`), and schema name
 (`mimiciii`) suggested there, with the password `mimic`. If you make
@@ -84,7 +84,7 @@ different choices, then you'll need to change the corresponding
 parameter values in the `dbConnect()` call below. While the queries in
 this notebook can be performed on the entire database, a user new to
 MIMIC, Postgres, or R may want to install [the demo data
-set](https://mimic.physionet.org/gettingstarted/demo/) instead,
+set](https://mimic.mit.edu/docs/III/demo.html) instead,
 following the same process except for the CSV files used.[3]
 
 ### Connect to MIMIC-III
@@ -296,7 +296,7 @@ any fields from the codes table.
 
 MI may not be listed as the principal diagnosis; as explained in [the
 documentation for the `patients`
-table](https://mimic.physionet.org/mimictables/diagnoses_icd/), the
+table](https://mimic.mit.edu/docs/III/tables/diagnoses_icd.html), the
 `seq_num` field is a priority ranking for the diagnoses generated at the
 end of stay. In order to focus on patients for whom MI was central to
 their hospitalization, i will include records with MI in any of the
@@ -423,9 +423,9 @@ mortality. I'm interested in survival after discharge, so i must
 restrict to patients who did *not* die in hospital. This information is
 recorded in the `"hospital_expire_flag"` field (though not yet described
 in [the `admissions` table
-documentation](https://mimic.physionet.org/mimictables/admissions/); see
+documentation](https://mimic.mit.edu/docs/III/tables/admissions.html); see
 [the tutorial on querying
-MIMIC-III](https://mimic.physionet.org/tutorials/intro-to-mimic-iii/)).
+MIMIC-III](https://github.com/MIT-LCP/mimic-code/blob/main/mimic-iii/tutorials/sql-intro.md)).
 I also require the dates (admission and discharge) of each stay from the
 `admissions` table and the date of death (where available) of each
 patient from the `patients` table. While i'm working with dates, i'll
@@ -478,7 +478,7 @@ to extract components of timestamp fields as numbers. (Postgres also has
 a convenient `age()` function that would simplify the code chunk below,
 but this produces a character string that doesn't lend itself to
 analysis purposes.) [The documentation for the `patients`
-table](https://mimic.physionet.org/mimictables/patients/) explains that
+table](https://mimic.mit.edu/docs/III/tables/patients.html) explains that
 patients of 90 years and older had their ages artificially inflated, so
 i've removed these patients from my analysis. I reorder the fields
 toward the end in order to show the results of the date calculations. In
diff --git a/mimic-iii/tutorials/dplyr-frontend/intro.rmd b/mimic-iii/tutorials/dplyr-frontend/intro.rmd
index b3b804977..b5847f91f 100755
--- a/mimic-iii/tutorials/dplyr-frontend/intro.rmd
+++ b/mimic-iii/tutorials/dplyr-frontend/intro.rmd
@@ -47,7 +47,7 @@ library(stringr)
 
 ### Instantiate the MIMIC-III database
 
-I followed [the Unix/Mac instructions at PhysioNet](https://mimic.physionet.org/tutorials/install-mimic-locally-ubuntu/) to create an instance of MIMIC-III.[^corrections] In particular, i used the user name (`mimicuser`), database name (`mimic`), and schema name (`mimiciii`) suggested there, with the password `mimic`. If you make different choices, then you'll need to change the corresponding parameter values in the `dbConnect()` call below. While the queries in this notebook can be performed on the entire database, a user new to MIMIC, Postgres, or R may want to install [the demo data set](https://mimic.physionet.org/gettingstarted/demo/) instead, following the same process except for the CSV files used.[^etl]
+I followed [the Unix/Mac instructions at PhysioNet](https://mimic.mit.edu/docs/gettingstarted/local/install-mimic-locally-ubuntu.html) to create an instance of MIMIC-III.[^corrections] In particular, i used the user name (`mimicuser`), database name (`mimic`), and schema name (`mimiciii`) suggested there, with the password `mimic`. If you make different choices, then you'll need to change the corresponding parameter values in the `dbConnect()` call below. While the queries in this notebook can be performed on the entire database, a user new to MIMIC, Postgres, or R may want to install [the demo data set](https://mimic.mit.edu/docs/III/demo.html) instead, following the same process except for the CSV files used.[^etl]
 
 [^corrections]: The instantiation required some changes to the Postgres commands, e.g. `alter user mimic nosuperuser;` should in fact be `alter user mimicuser nosuperuser;`.
 
@@ -137,7 +137,7 @@ tbl_mimic(diagnoses_icd) %>%
   print() -> mi_admissions
 ```
 
-MI may not be listed as the principal diagnosis; as explained in [the documentation for the `patients` table](https://mimic.physionet.org/mimictables/diagnoses_icd/), the `seq_num` field is a priority ranking for the diagnoses generated at the end of stay. In order to focus on patients for whom MI was central to their hospitalization, i will include records with MI in any of the first five diagnosis positions, according to the `"seq_num"` field. To avoid duplicate admissions, i use `group_by()` and `top_n()` to limit the query to the first MI diagnosis for each admission.
+MI may not be listed as the principal diagnosis; as explained in [the documentation for the `patients` table](https://mimic.mit.edu/docs/III/tables/diagnoses_icd.html), the `seq_num` field is a priority ranking for the diagnoses generated at the end of stay. In order to focus on patients for whom MI was central to their hospitalization, i will include records with MI in any of the first five diagnosis positions, according to the `"seq_num"` field. To avoid duplicate admissions, i use `group_by()` and `top_n()` to limit the query to the first MI diagnosis for each admission.
 
 ```{r heart attack admissions}
 mi_admissions %>%
@@ -179,7 +179,7 @@ tbl_mimic(drgcodes) %>%
   print() -> study_admissions
 ```
 
-Finally, i adopt a common outcome measure for critical care: 30-day mortality. I'm interested in survival after discharge, so i must restrict to patients who did _not_ die in hospital. This information is recorded in the `"hospital_expire_flag"` field (though not yet described in [the `admissions` table documentation](https://mimic.physionet.org/mimictables/admissions/); see [the tutorial on querying MIMIC-III](https://mimic.physionet.org/tutorials/intro-to-mimic-iii/)).  I also require the dates (admission and discharge) of each stay from the `admissions` table and the date of death (where available) of each patient from the `patients` table.
+Finally, i adopt a common outcome measure for critical care: 30-day mortality. I'm interested in survival after discharge, so i must restrict to patients who did _not_ die in hospital. This information is recorded in the `"hospital_expire_flag"` field (though not yet described in [the `admissions` table documentation](https://mimic.mit.edu/docs/III/tables/admissions.html); see [the tutorial on querying MIMIC-III](https://github.com/MIT-LCP/mimic-code/blob/main/mimic-iii/tutorials/sql-intro.md)).  I also require the dates (admission and discharge) of each stay from the `admissions` table and the date of death (where available) of each patient from the `patients` table.
 While i'm working with dates, i'll also calculate each patient's age on the day of admission.
 
 I first join the necessary date fields into `study_admissions`.
@@ -203,7 +203,7 @@ study_admissions %>%
   print() -> study_admissions
 ```
 
-Functionality for working with dates and times is not yet implemented in **dbplyr**, but an invaluable feature of [its SQL translation](https://dbplyr.tidyverse.org/articles/sql-translation.html) is that unrecognized functions pass through verbatim, where Postgres will attempt to interpret them. This allows to use `date_part()` below to extract components of timestamp fields as numbers. (Postgres also has a convenient `age()` function that would simplify the code chunk below, but this produces a character string that doesn't lend itself to analysis purposes.) [The documentation for the `patients` table](https://mimic.physionet.org/mimictables/patients/) explains that patients of 90 years and older had their ages artificially inflated, so i've removed these patients from my analysis. I reorder the fields toward the end in order to show the results of the date calculations. In the last transformation step, `everything()` adds in all the fields i don't explicitly select.
+Functionality for working with dates and times is not yet implemented in **dbplyr**, but an invaluable feature of [its SQL translation](https://dbplyr.tidyverse.org/articles/sql-translation.html) is that unrecognized functions pass through verbatim, where Postgres will attempt to interpret them. This allows to use `date_part()` below to extract components of timestamp fields as numbers. (Postgres also has a convenient `age()` function that would simplify the code chunk below, but this produces a character string that doesn't lend itself to analysis purposes.) [The documentation for the `patients` table](https://mimic.mit.edu/docs/III/tables/patients.html) explains that patients of 90 years and older had their ages artificially inflated, so i've removed these patients from my analysis. I reorder the fields toward the end in order to show the results of the date calculations. In the last transformation step, `everything()` adds in all the fields i don't explicitly select.
 
 ```{r time differences}
 study_admissions %>%
diff --git a/mimic-iii/tutorials/dplyr-frontend/intro.tex b/mimic-iii/tutorials/dplyr-frontend/intro.tex
index 7b77a96d1..1e7d18759 100755
--- a/mimic-iii/tutorials/dplyr-frontend/intro.tex
+++ b/mimic-iii/tutorials/dplyr-frontend/intro.tex
@@ -238,7 +238,7 @@ \subsubsection{Instantiate the MIMIC-III
 database}\label{instantiate-the-mimic-iii-database}
 
 I followed
-\href{https://mimic.physionet.org/tutorials/install-mimic-locally-ubuntu/}{the
+\href{https://mimic.mit.edu/docs/gettingstarted/local/install-mimic-locally-ubuntu.html}{the
 Unix/Mac instructions at PhysioNet} to create an instance of
 MIMIC-III.\footnote{The instantiation required some changes to the
   Postgres commands, e.g. \texttt{alter\ user\ mimic\ nosuperuser;}
@@ -250,7 +250,7 @@ \subsubsection{Instantiate the MIMIC-III
 \texttt{dbConnect()} call below. While the queries in this notebook can
 be performed on the entire database, a user new to MIMIC, Postgres, or R
 may want to install
-\href{https://mimic.physionet.org/gettingstarted/demo/}{the demo data
+\href{https://mimic.mit.edu/docs/III/demo.html}{the demo data
 set} instead, following the same process except for the CSV files
 used.\footnote{When installing a new database, i find it much more
   efficient to wrap my steps in an R script that i can execute from the
@@ -512,7 +512,7 @@ \subsubsection{Subset and join query
 \end{verbatim}
 
 MI may not be listed as the principal diagnosis; as explained in
-\href{https://mimic.physionet.org/mimictables/diagnoses_icd/}{the
+\href{https://mimic.mit.edu/docs/III/tables/diagnoses_icd.html}{the
 documentation for the \texttt{patients} table}, the \texttt{seq\_num}
 field is a priority ranking for the diagnoses generated at the end of
 stay. In order to focus on patients for whom MI was central to their
@@ -668,9 +668,9 @@ \subsubsection{Transform and augment query
 restrict to patients who did \emph{not} die in hospital. This
 information is recorded in the \texttt{"hospital\_expire\_flag"} field
 (though not yet described in
-\href{https://mimic.physionet.org/mimictables/admissions/}{the
+\href{https://mimic.mit.edu/docs/III/tables/admissions.html}{the
 \texttt{admissions} table documentation}; see
-\href{https://mimic.physionet.org/tutorials/intro-to-mimic-iii/}{the
+\href{https://github.com/MIT-LCP/mimic-code/blob/main/mimic-iii/tutorials/sql-intro.md}{the
 tutorial on querying MIMIC-III}). I also require the dates (admission
 and discharge) of each stay from the \texttt{admissions} table and the
 date of death (where available) of each patient from the
@@ -731,7 +731,7 @@ \subsubsection{Transform and augment query
 numbers. (Postgres also has a convenient \texttt{age()} function that
 would simplify the code chunk below, but this produces a character
 string that doesn't lend itself to analysis purposes.)
-\href{https://mimic.physionet.org/mimictables/patients/}{The
+\href{https://mimic.mit.edu/docs/III/tables/patients.html}{The
 documentation for the \texttt{patients} table} explains that patients of
 90 years and older had their ages artificially inflated, so i've removed
 these patients from my analysis. I reorder the fields toward the end in
diff --git a/mimic-iii/tutorials/sql-intro.md b/mimic-iii/tutorials/sql-intro.md
index d74c91518..9fbcdaf6f 100644
--- a/mimic-iii/tutorials/sql-intro.md
+++ b/mimic-iii/tutorials/sql-intro.md
@@ -17,7 +17,7 @@ After this tutorial you should have:
 
 MIMIC-III is an openly available dataset developed by the MIT Lab for Computational Physiology, comprising deidentified health data associated with >60,000 hospital stays. Spanning 2001-2012, it includes demographics, vital signs, laboratory tests, medications, and more. A paper describing MIMIC-III is available from: http://www.nature.com/articles/sdata201635
 
-The dataset is provided as a collection of comma-separated value (CSV) files, which can be loaded into a database system such as PostgreSQL. A list of tables is provided on the MIMIC website: http://mimic.physionet.org/mimictables/admissions/
+The dataset is provided as a collection of comma-separated value (CSV) files, which can be loaded into a database system such as PostgreSQL. A list of tables is provided on the MIMIC website: https://mimic.mit.edu/docs/III/tables/admissions.html
 
 We have highlighted some of the key tables below:
 
@@ -32,8 +32,8 @@ We have highlighted some of the key tables below:
 
 ## Solution 1
 
-1. The patient's date of birth can be found in the *patients* table, as detailed in the documentation: http://mimic.physionet.org/mimictables/patients/
-2. The patient's hospital admission time can be found in the *admissions* table, which tracks hospital admission information, as detailed here: http://mimic.physionet.org/mimictables/admissions/
+1. The patient's date of birth can be found in the *patients* table, as detailed in the documentation: https://mimic.mit.edu/docs/III/tables/patients.html
+2. The patient's hospital admission time can be found in the *admissions* table, which tracks hospital admission information, as detailed here: https://mimic.mit.edu/docs/III/tables/admissions.html
 
 # Comma separated value files
 
@@ -41,8 +41,6 @@ Comma separated value (CSV) files are a plain text format used for storing data
 
 As there is no official standard for the CSV format, the term is used somewhat loosely, which can often cause issues when seeking to load the data into a data analysis package. A general recommendation is to follow the definition for CSVs set out by the Internet Engineering Task Force in the RFC 4180 specification document.
 
-![CSV file](./csvformat.png)
-
 Summarized briefly, RFC 4180 specifies that:
 
 - files may optionally begin with a header row, with each field separated by a comma;
@@ -89,8 +87,6 @@ Giving a simple example of a hospital database with four tables, it might compri
 
 The patients table lists unique patients. The admissions table lists unique hospital admissions. The chartevents table lists charted events such as heart rate measurements. The `d_items` table is a dictionary that lists `itemid`s and associated labels, as shown in the example query. pk is primary key. fk is foreign key.
 
-![Relational databases consist of multiple data tables linked by keys.](./relationaldb.png)
-
 # What is Structured Query Language (SQL)?
 
 Structured Query Language (SQL) is a programming language used to manage relational databases.
@@ -124,8 +120,8 @@ FROM patients;
 ## Solution 2
 
 1. If you have not installed the MIMIC-III database into a PostgreSQL server either locally or otherwise, you can follow the tutorial on installing MIMIC-III:
-    * OS X or Ubuntu: http://mimic.physionet.org/tutorials/install-mimic-locally-ubuntu/
-    * Windows: http://mimic.physionet.org/tutorials/install-mimic-locally-windows/
+    * OS X or Ubuntu: https://mimic.mit.edu/docs/gettingstarted/local/install-mimic-locally-ubuntu.html
+    * Windows: https://mimic.mit.edu/docs/gettingstarted/local/install-mimic-locally-windows.html
 2. `SELECT * FROM patients`
 3. `SELECT subject_id, dob, gender FROM patients`
 
@@ -270,7 +266,7 @@ Note that the `WHERE` clause is optional, and in the above query we have omitted
 
 Often we need information coming from multiple tables. This can be achieved using the SQL `JOIN` keyword. There are several types of join, including `INNER JOIN`, `LEFT JOIN`, and `RIGHT JOIN`. It is important to understand the difference between these joins because their usage can significantly impact query results. Detailed guidance on joins is widely available on the web.
 
-![SQL joins. Adapted from an image by Arbeck on Wikipedia: https://commons.wikimedia.org/wiki/File:SQL_Joins.svg](./sql-joins.png)
+![SQL joins. Image by Arbeck on Wikimedia Commons (CC BY 3.0): https://commons.wikimedia.org/wiki/File:SQL_Joins.svg](https://upload.wikimedia.org/wikipedia/commons/9/9d/SQL_Joins.svg)
 
 Using the `INNER JOIN` keyword, let’s select a list of patients from the *patients* table along with dates of birth, and join to the *admissions* table to get the admission time for each hospital admission. We use the `INNER JOIN` to indicate that two or more tables should be combined based on a common attribute, which in our case is `subject_id`:
 
diff --git a/mimic-iv-cxr/txt/README.md b/mimic-iv-cxr/txt/README.md
index 8e5ac7cc8..7a03df1b9 100644
--- a/mimic-iv-cxr/txt/README.md
+++ b/mimic-iv-cxr/txt/README.md
@@ -16,7 +16,7 @@ The script can be run (from this folder) as follows:
 
 `python create_section_files.py --reports_path /db/mimic-cxr/files --output_path /db/mimic-cxr/mimic-cxr-sections`
 
-... where you should replace `/db/mimic-cxr/mimic-cxr-reports/files` with the location of your MIMIC-CXR files folder.
+... where you should replace `/db/mimic-cxr/files` with the location of your MIMIC-CXR files folder.
 
 ## CheXpert
 
@@ -24,4 +24,4 @@ Instructions for generating CheXpert annotations from the reports are available
 
 ## NegBio
 
-Instructions for generating CheXpert annotations from the reports are available [in the negbio subfolder](/mimic-iv-cxr/txt/negbio).
\ No newline at end of file
+Instructions for generating NegBio annotations from the reports are available [in the negbio subfolder](/mimic-iv-cxr/txt/negbio).
\ No newline at end of file
diff --git a/mimic-iv-cxr/txt/chexpert/README.md b/mimic-iv-cxr/txt/chexpert/README.md
index fcc6ff80e..e9c345967 100644
--- a/mimic-iv-cxr/txt/chexpert/README.md
+++ b/mimic-iv-cxr/txt/chexpert/README.md
@@ -5,7 +5,7 @@ This folder provides code and instructions for running the CheXpert NLP tool on
 ## Requirements
 
 1. Sectioned report CSVs using the create_section_files.py script. See the [txt folder](/mimic-iv-cxr/txt/) for details.
-    * From this, note the path containing the CSVs, e.g. `/data/mimic-cxr/sections`. This folder should have 22 files, with filenames `mimic_cxr_000.csv`, `mimic_cxr_001.csv`, ...
+    * From this, note the path containing the CSVs, e.g. `/data/mimic-cxr/sections`. This folder should have 22 files, with filenames `mimic_cxr_00.csv`, `mimic_cxr_01.csv`, ...
 2. We use the `conda` manager to create a virtual environment to run the code in. To use `conda`, you will need to install [Miniconda](https://docs.conda.io/en/latest/miniconda.html) or [Anaconda](https://www.anaconda.com/) (Miniconda is a light-weight alternative of Anaconda).
 
 # Running CheXpert
@@ -29,10 +29,18 @@ Open up a terminal in this folder.
 `cd chexpert-labeler; git checkout 2f29daf2a7af97f494bc134e74600c05a7a5f987; cd ..`
 
 5. Run the bash script which calls CheXpert
-  * The first argument should be the location of the MIMIC-CXR sectioned files, output by `section_parser.py`
+  * The first argument should be the location of the MIMIC-CXR sectioned files, output by `create_section_files.py`
   * The second argument should be the location of the cloned chexpert repo
 
 `sh run_chexpert_on_files.sh /db/mimic-cxr/mimic-cxr-sections chexpert-labeler`
 
-6. Aggregate the labels together into a single file
+6. Aggregate the labels together into a single file. The script outputs one `mimic_cxr_XX_labeled.csv` file per input CSV in the current directory; these can be concatenated, e.g. with pandas:
+
+```python
+import glob
+import pandas as pd
+
+df = pd.concat(pd.read_csv(fn) for fn in sorted(glob.glob('mimic_cxr_*_labeled.csv')))
+df.to_csv('mimic-cxr-chexpert.csv', index=False)
+```
 
diff --git a/mimic-iv-cxr/txt/negbio/README.md b/mimic-iv-cxr/txt/negbio/README.md
index 74e624291..d464fd6cc 100644
--- a/mimic-iv-cxr/txt/negbio/README.md
+++ b/mimic-iv-cxr/txt/negbio/README.md
@@ -5,7 +5,7 @@ This folder provides code and instructions for running the NegBio NLP tool on MI
 ## Requirements
 
 1. Sectioned report CSVs using the create_section_files.py script. See the [txt folder](/mimic-iv-cxr/txt/) for details.
-    * From this, note the path containing the CSVs, e.g. `/data/mimic-cxr/sections`. This folder should have 22 files, with filenames `mimic_cxr_000.csv`, `mimic_cxr_001.csv`, ...
+    * From this, note the path containing the CSVs, e.g. `/data/mimic-cxr/sections`. This folder should have 22 files, with filenames `mimic_cxr_00.csv`, `mimic_cxr_01.csv`, ...
 2. We use the `conda` manager to create a virtual environment to run the code in. To use `conda`, you will need to install [Miniconda](https://docs.conda.io/en/latest/miniconda.html) or [Anaconda](https://www.anaconda.com/) (Miniconda is a light-weight alternative of Anaconda).
 
 ## Installation
@@ -32,5 +32,5 @@ Open up a terminal in this folder.
 **Warning: running this on all of MIMIC-CXR will generate ~15 GB of space on your hard drive**.
 
 ```
-bash run_negbio_on_files.sh /db/mimic-cxr/sections negbio-mimic-cxr
+bash run_negbio.sh /db/mimic-cxr/sections negbio-mimic-cxr
 ```
\ No newline at end of file
diff --git a/mimic-iv-ed/README.md b/mimic-iv-ed/README.md
index de7b8fcea..7e4df08a3 100644
--- a/mimic-iv-ed/README.md
+++ b/mimic-iv-ed/README.md
@@ -1,3 +1,3 @@
 # MIMIC-IV-ED
 
-MIMIC-IV-ED is a publicly available database of emergency department visit data. You can read more about the dataset on [the PhysioNet project page](https://mimic.mit.edu/iv/modules/ed/).
\ No newline at end of file
+MIMIC-IV-ED is a publicly available database of emergency department visit data. You can read more about the dataset on [the PhysioNet project page](https://mimic.mit.edu/docs/iv/modules/ed/).
\ No newline at end of file
diff --git a/mimic-iv-ed/buildmimic/duckdb/import_duckdb.sh b/mimic-iv-ed/buildmimic/duckdb/import_duckdb.sh
index 9142f7b40..88cf287b5 100644
--- a/mimic-iv-ed/buildmimic/duckdb/import_duckdb.sh
+++ b/mimic-iv-ed/buildmimic/duckdb/import_duckdb.sh
@@ -29,7 +29,7 @@ usage () {
     die "
 USAGE: ./import_duckdb.sh mimic_data_dir [output_db]
 WHERE:
-    mimic_data_dir        directory that contains csv.tar.gz or csv files
+    mimic_data_dir        directory that contains csv.gz or csv files
     output_db: optional   filename for duckdb file (default: mimic4_ed.db)\
 "
 }
diff --git a/mimic-iv-ed/buildmimic/postgres/README.md b/mimic-iv-ed/buildmimic/postgres/README.md
index 297f72fe5..62eb5a969 100644
--- a/mimic-iv-ed/buildmimic/postgres/README.md
+++ b/mimic-iv-ed/buildmimic/postgres/README.md
@@ -11,11 +11,11 @@ git clone https://github.com/MIT-LCP/mimic-code.git
 cd mimic-code
 # download data
 wget -r -N -c -np --user  --ask-password https://physionet.org/files/mimic-iv-ed/2.2/
-mv physionet.org/files/mimiciv-iv-ed mimiciv && rmdir physionet.org/files && rm physionet.org/robots.txt && rmdir physionet.org
+mv physionet.org/files/mimic-iv-ed mimiciv && rmdir physionet.org/files && rm physionet.org/robots.txt && rmdir physionet.org
 # if mimiciv not exists
 # createdb mimiciv
 psql -d mimiciv -f mimic-iv-ed/buildmimic/postgres/create.sql
-psql -d mimiciv -v ON_ERROR_STOP=1 -v mimic_data_dir=mimiciv/2.2/ed -f load_gz.sql
+psql -d mimiciv -v ON_ERROR_STOP=1 -v mimic_data_dir=mimiciv/2.2/ed -f mimic-iv-ed/buildmimic/postgres/load_gz.sql
 ```
 
 
@@ -46,7 +46,7 @@ psql -d mimic -v ON_ERROR_STOP=1 -v mimic_data_dir=
 If you encounter following error:
 
 ```bash
-psql "dbname=mimic user=postgres options=--search_path=mimic_ed" -v ON_ERROR_STOP=1 -f create.sql
+psql "dbname=mimic user=postgres options=--search_path=mimiciv_ed" -v ON_ERROR_STOP=1 -f create.sql
 psql: FATAL:  Peer authentication failed for user "postgres"
 ```
 
diff --git a/mimic-iv-note/buildmimic/duckdb/import_duckdb.sh b/mimic-iv-note/buildmimic/duckdb/import_duckdb.sh
index c0425fe6f..b194212fb 100644
--- a/mimic-iv-note/buildmimic/duckdb/import_duckdb.sh
+++ b/mimic-iv-note/buildmimic/duckdb/import_duckdb.sh
@@ -29,7 +29,7 @@ usage () {
     die "
 USAGE: ./import_duckdb.sh mimic_data_dir [output_db]
 WHERE:
-    mimic_data_dir        directory that contains csv.tar.gz or csv files
+    mimic_data_dir        directory that contains csv.gz or csv files
     output_db: optional   filename for duckdb file (default: mimic4_note.db)\
 "
 }
diff --git a/mimic-iv-note/buildmimic/postgres/load.sql b/mimic-iv-note/buildmimic/postgres/load.sql
index 38a79a82b..8c0f26d8b 100644
--- a/mimic-iv-note/buildmimic/postgres/load.sql
+++ b/mimic-iv-note/buildmimic/postgres/load.sql
@@ -6,7 +6,7 @@
 --  psql "dbname= user=" -v mimic_data_dir= -f load_gz.sql
 \cd :mimic_data_dir
 
--- making sure that all tables are emtpy and correct encoding is defined -utf8- 
+-- making sure that all tables are empty and correct encoding is defined -utf8- 
 SET CLIENT_ENCODING TO 'utf8';
 
 \COPY mimiciv_note.discharge FROM 'discharge.csv' DELIMITER ',' CSV HEADER NULL '';
diff --git a/mimic-iv-note/buildmimic/postgres/load_7z.sql b/mimic-iv-note/buildmimic/postgres/load_7z.sql
index 57c4dee65..0263e7ba4 100644
--- a/mimic-iv-note/buildmimic/postgres/load_7z.sql
+++ b/mimic-iv-note/buildmimic/postgres/load_7z.sql
@@ -6,7 +6,7 @@
 --  psql "dbname= user=" -v mimic_data_dir= -f load_gz.sql
 \cd :mimic_data_dir
 
--- making sure that all tables are emtpy and correct encoding is defined -utf8- 
+-- making sure that all tables are empty and correct encoding is defined -utf8- 
 SET CLIENT_ENCODING TO 'utf8';
 
 \COPY mimiciv_note.discharge FROM PROGRAM '7z e -so discharge.csv.gz' DELIMITER ',' CSV HEADER NULL '';
diff --git a/mimic-iv-note/buildmimic/postgres/load_gz.sql b/mimic-iv-note/buildmimic/postgres/load_gz.sql
index bbe9ba64b..ff8c97272 100644
--- a/mimic-iv-note/buildmimic/postgres/load_gz.sql
+++ b/mimic-iv-note/buildmimic/postgres/load_gz.sql
@@ -6,7 +6,7 @@
 --  psql "dbname= user=" -v mimic_data_dir= -f load_gz.sql
 \cd :mimic_data_dir
 
--- making sure that all tables are emtpy and correct encoding is defined -utf8- 
+-- making sure that all tables are empty and correct encoding is defined -utf8- 
 SET CLIENT_ENCODING TO 'utf8';
 
 \COPY mimiciv_note.discharge FROM PROGRAM 'gzip -dc discharge.csv.gz' DELIMITER ',' CSV HEADER NULL '';
diff --git a/mimic-iv/README.md b/mimic-iv/README.md
index 55b1af87c..919b196cc 100644
--- a/mimic-iv/README.md
+++ b/mimic-iv/README.md
@@ -12,9 +12,9 @@
 The [MIMIC-IV concepts](/mimic-iv/concepts) are written in an SQL syntax compatible with BigQuery.
 The BigQuery [physionet-data.mimiciv_derived](https://console.cloud.google.com/bigquery?ws=!1m4!1m3!3m2!1sphysionet-data!2smimiciv_derived) dataset contains the output of the SQL scripts present in the concepts folder.  These tables are generated using the code in the [latest release on GitHub](https://github.com/MIT-LCP/mimic-code/releases). Access to this dataset is available to MIMIC-IV approved users: see the [cloud instructions](https://mimic.mit.edu/docs/gettingstarted/cloud/).
 
-* [List of the concept folders and their content](#concept-index)
-* [Generating the concept tables on BigQuery](#generating-the-concepts-on-bigquery)
-* [Generating the concept tables on PostgreSQL](#generating-the-concepts-on-postgresql)
+* [List of the concept folders and their content](/mimic-iv/concepts/README.md#concept-index)
+* [Generating the concept tables on BigQuery](#bigquery)
+* [Generating the concept tables on PostgreSQL](#postgresql)
 
 ## Generating the concepts
 
diff --git a/mimic-iv/buildmimic/bigquery/README.md b/mimic-iv/buildmimic/bigquery/README.md
index b3da2a379..801f0b52b 100644
--- a/mimic-iv/buildmimic/bigquery/README.md
+++ b/mimic-iv/buildmimic/bigquery/README.md
@@ -211,10 +211,6 @@ OK....labevents
 Waiting on bqjob_r6a896e170257f60e_00000177f4b97a15_1 ... (65s) Current status: DONE   
 OK....microbiologyevents
 Waiting on bqjob_r1a722205e1dad7a9_00000177f4ba8456_1 ... (108s) Current status: DONE   
-OK....note
-Waiting on bqjob_r1c4ba80cb44c271d_00000177f4bc3685_1 ... (0s) Current status: DONE   
-OK....note_detail
-Waiting on bqjob_rf7b708a80accdd7_00000177f4bc402f_1 ... (224s) Current status: DONE   
 OK....pharmacy
 Waiting on bqjob_r624a5dcfb4d949ab_00000177f4bfb5f9_1 ... (254s) Current status: DONE   
 OK....poe
diff --git a/mimic-iv/buildmimic/duckdb/import_duckdb.sh b/mimic-iv/buildmimic/duckdb/import_duckdb.sh
index ab1d37fa3..8e72733c3 100755
--- a/mimic-iv/buildmimic/duckdb/import_duckdb.sh
+++ b/mimic-iv/buildmimic/duckdb/import_duckdb.sh
@@ -29,7 +29,7 @@ usage () {
     die "
 USAGE: ./import_duckdb.sh mimic_data_dir [output_db]
 WHERE:
-    mimic_data_dir        directory that contains csv.tar.gz or csv files
+    mimic_data_dir        directory that contains csv.gz or csv files
     output_db: optional   filename for duckdb file (default: mimic4.db)\
 "
 }
diff --git a/mimic-iv/buildmimic/postgres/README.md b/mimic-iv/buildmimic/postgres/README.md
index 78adb239b..5ab28c2c4 100644
--- a/mimic-iv/buildmimic/postgres/README.md
+++ b/mimic-iv/buildmimic/postgres/README.md
@@ -4,7 +4,7 @@ The scripts in this folder create the schema for MIMIC-IV and load the data into
 If you are having trouble, take a look at the common issues in the FAQ at the bottom of this page.