fix typos and dead links

.gitignore

@@ -9,6 +9,8 @@ mimic-iv/docker-postgres/.env
 
 # data
 *.csv.gz
+# reference data distributed with the repository
+!mimic-iii/concepts/diagnosis/ccs_multi_dx.csv.gz
 
 # credentials
 key.json

README.md

@@ -6,7 +6,7 @@ You can read more about the code repository in the following open access paper:
 
 ## Cloud access to datasets
 
-The various MIMIC databases are available on Google Cloud Platform (GCP) and Amazon Web Services (AWS). To access the data on the cloud, simply add the relevant cloud identifier to your PhysioNet profile. Then request access to the dataset for the particular cloud platform via the PhysioNet project page. Further instructions are available on [the MIMIC website](https://mimic.mit.edu/iv/access/cloud/).
+The various MIMIC databases are available on Google Cloud Platform (GCP) and Amazon Web Services (AWS). To access the data on the cloud, simply add the relevant cloud identifier to your PhysioNet profile. Then request access to the dataset for the particular cloud platform via the PhysioNet project page. Further instructions are available on [the MIMIC website](https://mimic.mit.edu/docs/gettingstarted/cloud/).
 
 ## Navigating this repository
 
@@ -71,7 +71,7 @@ pytest tests/
 
 ## Other useful tools
 
-* [Bloatectomy](https://github.com/MIT-LCP/bloatectomy) ([paper](https://github.com/MIT-LCP/bloatectomy/blob/master/paper/paper.md)) - A python based package for removing duplicate text in clinical notes
+* [Bloatectomy](https://github.com/MIT-LCP/bloatectomy) ([paper](https://github.com/MIT-LCP/bloatectomy/blob/master/paper/bloatectomy_paper.md)) - A python based package for removing duplicate text in clinical notes
 * [Medication categories](https://github.com/mghassem/medicationCategories) - Python script for extracting medications from free-text notes
 * [MIMIC Extract](https://github.com/MLforHealth/MIMIC_Extract) ([paper](https://doi.org/10.1145/3368555.3384469)) - A python based package for transforming MIMIC-III data into a machine learning friendly format
 * [FIDDLE](https://github.com/MLD3/FIDDLE) ([paper](https://doi.org/10.1093/jamia/ocaa139)) - A python based package for a FlexIble Data-Driven pipeLinE (FIDDLE), transforming structured EHR data into a machine learning friendly format

mimic-iii/Makefile.md

@@ -7,10 +7,13 @@ Starting from a fresh system which has GNU Make installed, PostgreSQL installed,
 
 ```sh
 export datadir="/path/to/data"
+make create-user
 make mimic-download physionetuser=<PHYSIONETWORKS_USERNAME> datadir=$datadir
 make mimic-gz datadir=$datadir
 ```
 
+The `create-user` step creates the database, schema, and user (by default the database is named `mimic` and the user is `postgres`); skip it if these already exist.
+
 Note that if you have already downloaded the data, you can skip the `make mimic-download`, just be sure to set `datadir` appropriately. If you have already decompressed the data into `.csv` files, call `make mimic` instead of `make mimic-gz`, e.g. `make mimic datadir=/path/to/data`.
 
 Optionally, additional contributed materialized views can be created afterward by running:
@@ -19,7 +22,7 @@ Optionally, additional contributed materialized views can be created afterward b
 make concepts
 ```
 
-Note that you may want to modify parameters at the top of the Makefile - e.g. the username (see below "non-standard username or database name").
+Note that you may want to modify parameters at the top of `buildmimic/postgres/Makefile` - e.g. the username (see below "non-standard username or database name").
 
 ### Authentication
 In order to avoid the prompts for your database password each time, you may create a file in your home directory called .pgpass containing the following:
@@ -33,12 +36,12 @@ Replace ```mimic``` with your username and ```password``` with your password. No
 Alternatively you can configure the database to use operating system level authentication. See the Postgres manual for more detail (in particular, the section(s) on "Peer Authentication"): https://www.postgresql.org/docs/9.6/static/auth-methods.html
 
 ### Non-standard username or database name
-If you need to use a username or database name other than ```mimic```, then you will need to specify this by modifying the top-level Makefile:
+If you need to use a database name other than ```mimic``` or a username other than ```postgres```, then you will need to specify this by modifying the defaults at the top of `buildmimic/postgres/Makefile`:
 
 ```
-DBNAME=mimic
-DBUSER=mimic
+DBNAME := mimic
+DBUSER := postgres
 ```
 
 ## Contributing
-If you would like to contribute code to create a materialized view to the `concepts` folder, simply add a command which calls the script to the `concepts/make-concepts.sql` file. The format is fairly straightforward: e.g. adding the `\i sepsis/angus.sql` line informs the script to call the `concepts/sepsis/angus.sql` file.
+If you would like to contribute a new concept, add the script (written in the BigQuery dialect) to the [concepts](/mimic-iii/concepts) folder and add a corresponding line to [concepts/make-concepts.sh](/mimic-iii/concepts/make-concepts.sh). The PostgreSQL and DuckDB versions in [concepts_postgres](/mimic-iii/concepts_postgres) and [concepts_duckdb](/mimic-iii/concepts_duckdb) are then regenerated by transpilation, and the new script should be added to `concepts_postgres/postgres-make-concepts.sql` and `concepts_duckdb/duckdb.sql`. See the [concepts README](/mimic-iii/concepts/README.md) for details.

mimic-iii/benchmark/postgres-benchmark-4.sql

@@ -18,7 +18,7 @@
 	STOPPED VARCHAR(50)
    ) ;
 
--- PARTITIONNING
+-- PARTITIONING
 -- CREATE CHARTEVENTS_PARTITIONED TABLE
 CREATE TABLE CHARTEVENTS_PARTITIONED_0 ( CHECK ( itemid in (211, 220045, 51, 442, 455, 6701, 220179, 220050, 8368, 8440, 8441, 8555, 220180, 220051, 456, 52, 6702, 443, 220052, 220181, 225312, 618, 615, 220210, 224690, 646, 220277, 223762, 676,	223761, 678 )
     )) INHERITS (CHARTEVENTS_PARTITIONED);

mimic-iii/buildmimic/README.md

@@ -1,6 +1,6 @@
 # Building the MIMIC database
 
-This directory contains scripts that can be used to create a new instance of the MIMIC Critical Care Database. Tutorials for building a local PostgreSQL database using these scripts are available on the MIMIC website, under the "Tutorials" item in the menu: http://mimic.physionet.org/about/mimic/
+This directory contains scripts that can be used to create a new instance of the MIMIC Critical Care Database. Tutorials for building a local PostgreSQL database using these scripts are available on the MIMIC website, under the "Tutorials" item in the menu: https://mimic.mit.edu/docs/III/about/
 
 ## Automated tests (PostgreSQL scripts only)
 

mimic-iii/buildmimic/apache-drill/README.md

@@ -66,7 +66,7 @@ In DBeaver, connect and copy/paste:
 
 1. drill_create_data.sql
 1. create alias for all csv in temp (dfs.tmp = /tmp on linux) ln -s /path/to/mimic/csv/\* /tmp 
-1. create table one by one. Otherwize, it crashes when multi query are made in one run
+1. create table one by one. Otherwise, it crashes when multi query are made in one run
 
 ## Notes
 

mimic-iii/buildmimic/aws-athena/README.md

@@ -1,17 +1,17 @@
 # Study MIMIC-III data using AWS Athena
-The MIMIC-III dataset is available in the AWS cloud through our [Open Data on AWS program](https://registry.opendata.aws/).  This allows researchers to use the MIMIC-III dataset without having to download it, make a copy of it, or pay to store it.  They can simply analyze MIMIC-III using AWS services like Amazon EC2, Amazon Athena, AWS Lambda, or Amazon EMR by pointing to its address in the AWS cloud. This makes it faster and less expensive to perform research studies.  In order to gain access, please login to the [MIMIC PhysioNet website](https://mimic.physionet.org/), [input your AWS account number](https://physionet.org/settings/cloud/), and [request access to the MIMIC-III Clinical Database on AWS](https://physionet.org/projects/mimiciii/1.4/request_access/2).
+The MIMIC-III dataset is available in the AWS cloud through our [Open Data on AWS program](https://registry.opendata.aws/).  This allows researchers to use the MIMIC-III dataset without having to download it, make a copy of it, or pay to store it.  They can simply analyze MIMIC-III using AWS services like Amazon EC2, Amazon Athena, AWS Lambda, or Amazon EMR by pointing to its address in the AWS cloud. This makes it faster and less expensive to perform research studies.  In order to gain access, please login to the [MIMIC PhysioNet website](https://mimic.mit.edu/), [input your AWS account number](https://physionet.org/settings/cloud/), and [request access to the MIMIC-III Clinical Database on AWS](https://physionet.org/projects/mimiciii/1.4/request_access/2).
 
 Found here is some useful code to help you understand how to access the MIMIC-III dataset on AWS.
 
 ### mimic-iii-athena.yaml
-This is an [AWS CloudFormation Template](https://aws.amazon.com/cloudformation/) that will deploy a database in the [AWS Glue](https://aws.amazon.com/glue/) Data Catalog that contains all of the MIMIC-III tables.  It also deploys a Jupyter Notebook instance in [Amazon SageMaker](https://aws.amazon.com/sagemaker/) that contains the content of this [mimic-code](https://github.com/MIT-LCP/mimic-code/) GitHub repository and is set up to access the MIMIC-III data through AWS Glue.  This repository contains a version of the [Aline Study](https://github.com/JamesSWiggins/mimic-code/tree/master/notebooks/aline/awsathena) that is configured to run it's SQL queries against the MIMIC-III data in AWS using [AWS Athena](https://aws.amazon.com/athena/).
+This is an [AWS CloudFormation Template](https://aws.amazon.com/cloudformation/) that will deploy a database in the [AWS Glue](https://aws.amazon.com/glue/) Data Catalog that contains all of the MIMIC-III tables.  It also deploys a Jupyter Notebook instance in [Amazon SageMaker](https://aws.amazon.com/sagemaker/) that contains the content of this [mimic-code](https://github.com/MIT-LCP/mimic-code/) GitHub repository and is set up to access the MIMIC-III data through AWS Glue.  This repository contains a version of the [Aline Study](/mimic-iii/notebooks/aline-aws) that is configured to run it's SQL queries against the MIMIC-III data in AWS using [AWS Athena](https://aws.amazon.com/athena/).
 
 Use the below Launch Stack button to deploy this AWS CloudFormation template into your AWS account.  On the first screen, the template link has already been specified, so just click next.  On the second screen, provide a Stack name (letters and numbers) and click next, on the third screen, just click next.  On the forth screen, at the bottom, there is a box that says **I acknowledge that AWS CloudFormation might create IAM resources.**.  Check that box, and then click **Create**.  Once the Stack has complete deploying, look at the **Outputs** tab of the AWS CloudFormation console for links to your Juypter Notebooks instance.
 
 [![cloudformation-launch-stack](cloudformation-launch-stack.png)](https://console.aws.amazon.com/cloudformation/home?region=us-east-1#/stacks/new?stackName=MIMIC&templateURL=https://aws-bigdata-blog.s3.amazonaws.com/artifacts/biomedical-informatics-studies/mimic-iii-athena.yaml)
 
 ### mimictoparquet_glue_job.py
-The MIMIC-III dataset is offered on AWS in both the original gzipped CSV format as well as the Apache Parquet format.  This Python script is run by the MIMIC team as an AWS Glue job to convert the CSV.gz files to Apache Parquet format.  It uses Apache Spark to help conver the files.  You don't need to use this script yourself to use the MIMIC-III data in Parquet format, but it is provided as a template that could be used to convert other data sets from CSV to Parquet.  The benefits of querying this data in Parquet format are outlined in this blog post:
+The MIMIC-III dataset is offered on AWS in both the original gzipped CSV format as well as the Apache Parquet format.  This Python script is run by the MIMIC team as an AWS Glue job to convert the CSV.gz files to Apache Parquet format.  It uses Apache Spark to help convert the files.  You don't need to use this script yourself to use the MIMIC-III data in Parquet format, but it is provided as a template that could be used to convert other data sets from CSV to Parquet.  The benefits of querying this data in Parquet format are outlined in this blog post:
 
 ### mimic-aws-athena-ddl.sql
 This is a SQL script that can be run directly within AWS Athena to define each of the MIMIC-III dataset tables from the Apache Parquet files provide in the AWS OpenData program.  You do not need to execute these if you use the above ```mimic-iii-athena.yaml``` CloudFormation template.  The table spaces are defined for you by that automation.  However, if you wanted to define the tables yourself within your account, you can use these SQL statements.

mimic-iii/buildmimic/bigquery/README.md

@@ -2,7 +2,7 @@
 
 Following are the steps to create the MIMIC-III dataset on BigQuery and load the source files (.csv.gz) downloaded from Physionet.
 
-**IMPORTANT**: Only users with approved Physionet Data Use Agreement (DUA) should have access to the MIMIC dataset via BigQuery or Cloud Storage. If you don't have access to MIMIC, follow the instructions [here](https://mimic.physionet.org/gettingstarted/access/) to request access.
+**IMPORTANT**: Only users with approved Physionet Data Use Agreement (DUA) should have access to the MIMIC dataset via BigQuery or Cloud Storage. If you don't have access to MIMIC, follow the instructions [here](https://mimic.mit.edu/docs/gettingstarted/) to request access.
 
 ---
 
@@ -12,7 +12,7 @@ Following are the steps to create the MIMIC-III dataset on BigQuery and load the
 
 For this tutorial, we will proceed using the compressed files (.csv.gz) stored in a Google Cloud Storage (GCS) bucket.
 In order to use these files, you must have a Google account with access permission granted via PhysioNet.
-You can read about being provisioned access to MIMIC-III on Google [on the cloud tutorial page](https://mimic.physionet.org/gettingstarted/cloud/).
+You can read about being provisioned access to MIMIC-III on Google [on the cloud tutorial page](https://mimic.mit.edu/docs/gettingstarted/cloud/).
 
 Once you have configured your account on PhysioNet, go to the [MIMIC-III page on PhysioNet](https://physionet.org/content/mimiciii/) and scroll down to the Files section.
 
@@ -139,12 +139,12 @@ BigQuery schemas are defined by JSON files. These files are an array of dictiona
 The information about the columns are compiled from several sources:
 
 1. [The PostgreSQL build scripts](https://github.com/MIT-LCP/mimic-code/tree/main/mimic-iii/buildmimic/postgres)
-2. [The MIMIC-III online documentation](https://mimic.physionet.org/about/mimic/)
+2. [The MIMIC-III online documentation](https://mimic.mit.edu/docs/III/about/)
 3. [A schematic of the MIMIC-III database](https://mit-lcp.github.io/mimic-schema-spy/)
 
 Currently all columns are set as NULLABLE as the tables will be used for searching/querying only and users with Viewer role should not be able to modify the tables (ex: insert/delete/update data). However, depending on your environment, you may need to set the mode REQUIRED for the appropriate columns.
 
-We selected the type DATETIME for all columns with dates and times. However, you could also use TIMESTAMP. As [discussed in the MIMIC-III documentation](https://mimic.physionet.org/mimicdata/time), columns with suffix DATE (ex: CHARTDATE) "will always have 00:00:00 as the hour, minute, and second values. This does not mean it was recorded at midnight: it indicates that we do not have the exact time, only the date". This is also true for other columns such DOB and DOD (patients table).
+We selected the type DATETIME for all columns with dates and times. However, you could also use TIMESTAMP. As [discussed in the MIMIC-III documentation](https://mimic.mit.edu/docs/III/about/time.html), columns with suffix DATE (ex: CHARTDATE) "will always have 00:00:00 as the hour, minute, and second values. This does not mean it was recorded at midnight: it indicates that we do not have the exact time, only the date". This is also true for other columns such DOB and DOD (patients table).
 
 ---
 

mimic-iii/buildmimic/duckdb/README.md

@@ -85,7 +85,7 @@ $
 Here's an example invocation that will make the database in the default "mimic3.db":
 
 ```sh
-$ ./import_duckdb.sh physionet.org/files/mimiciii/1.4
+$ ./import_duckdb.sh mimiciii/1.4
 
 ... output removed
 Successfully finished loading data into mimic3.db.

mimic-iii/buildmimic/duckdb/import_duckdb.sh

@@ -29,7 +29,7 @@ usage () {
     die "
 USAGE: ./import_duckdb.sh mimic_data_dir [output_db]
 WHERE:
-    mimic_data_dir        directory that contains csv.tar.gz or csv files
+    mimic_data_dir        directory that contains csv.gz or csv files
     output_db: optional   filename for duckdb file (default: mimic3.db)\
 "
 }

mimic-iii/buildmimic/oracle/README.md

@@ -31,11 +31,11 @@ Alternatively, you can follow this step by step guide:
 3. Open up either `sqlplus` or SQLDeveloper
 4. Convert NOTEEVENTS.csv into a workable format: `python add_oracle_rowdelimiter.py -d ',' -i 'NOTEEVENTS.csv' -r '><><?~`;;`'`
 5. Run `oracle_create_tables.sql`
-6. Add permissions to run the bash script: `chmod a+x load_data_oracle.sh`
-7. Run the bash script: `./load_data_oracle.sh`
+6. Add permissions to run the bash script: `chmod a+x build_mimic_oracle.sh`
+7. Run the bash script: `./build_mimic_oracle.sh`
     * It helps to be logged in as the Oracle user, as the script uses OS authentication to speed things up
     * If you do not know how to use OS authentication, remove '/ as SYSDBA' from the beginning of each line in the script
-8. Run the `oracle_add_indexes.sql` and `oracle_add_constrants.sql` files
+8. Run the `oracle_add_indexes.sql` and `oracle_add_constraints.sql` files
 
 These scripts have yet to be fully tested and we would welcome pull requests, bug reports, or suggestions via GitHub.
 

mimic-iii/buildmimic/postgres/Makefile

@@ -281,7 +281,7 @@ concepts:
 	@echo '---------------------'
 	@echo ''
 	@sleep 2
-	cd ../../concepts/ && psql "$(DBSTRING)" -v ON_ERROR_STOP=1 -f make-concepts.sql
+	cd ../../concepts_postgres/ && psql "$(DBSTRING)" -v ON_ERROR_STOP=1 -c "CREATE SCHEMA IF NOT EXISTS mimiciii_derived;" -f postgres-make-concepts.sql
 
 
 .PHONY: help mimic clean

mimic-iii/buildmimic/postgres/README.md

@@ -3,7 +3,7 @@
 The scripts in this folder create a database to host the MIMIC-III data. You can use these scripts in one of two ways:
 
 * On *nix systems (such as Ubuntu or Mac OS X), you can use the make file
-* You can follow the tutorial to run each file individually. Windows users can follow along [here](https://mimic.physionet.org/tutorials/install-mimic-locally-windows/), while *nix/Mac OS X users can follow along [here](https://mimic.physionet.org/tutorials/install-mimic-locally-ubuntu/)
+* You can follow the tutorial to run each file individually. Windows users can follow along [here](https://mimic.mit.edu/docs/gettingstarted/local/install-mimic-locally-windows.html), while *nix/Mac OS X users can follow along [here](https://mimic.mit.edu/docs/gettingstarted/local/install-mimic-locally-ubuntu.html)
 
 If following the tutorials, be sure to download the scripts locally and the MIMIC-III files locally. If you choose the makefile approach, see the below section.