Introductory teaching materials by Gaurav Sood on collecting data, working with databases, statistical learning, clustering, and presenting analyses. The repository includes six slide decks with editable LaTeX sources, a scraping assignment, and an R plotting tutorial.
Start with the introduction below and work through the lessons in order. The PDFs can be read without installing software. The lessons assume basic probability and linear regression, and some experience writing loops and functions. For a programming introduction, start with the Python workshop.
| Lesson | Topics | Materials |
|---|---|---|
| Introduction to Data Science | Data sources and scale, computational constraints, MapReduce, PageRank, and prerequisites | Slides · Source |
| Get Your Own Data | HTML and PDF extraction in Python, regular expressions, edit distance, tokenization, and stemming | Slides · Source · Assignment |
| Databases and SQL | Data models, XML and JSON, relational algebra, joins, set operations, views, and SQLite | Slides · Source |
| Introduction to Statistical Learning | Nearest neighbors, model flexibility, bias and variance, model evaluation, validation, and false discoveries | Slides · Source |
| Unsupervised Learning | k-means, distance and scaling, choosing the number of clusters, and computational cost | Slides · Source |
| Presenting Analyses | ggplot2 mappings, layers, scales, coordinates, facets, uncertainty, and saving plots | R tutorial |
| From Paper to Digital | OCR pipelines, recognition errors, error correction, and evaluating extracted text | Slides · Source |
For more on supervised learning, the companion Learning From Data materials cover loss functions and linear regression, bias, variance, and regularization, and model evaluation. They also include lessons on causal inference, computer vision, fairness, and interpretability. The unsupervised deck here concentrates on k-means; PCA is introduced briefly, and there is no separate correspondence analysis lesson.
Clone or download the repository. Run commands from its root directory so that the examples can find the included files.
Use Python 3.11 or newer. On macOS or Linux:
git clone https://github.com/finite-sample/data-science.git
cd data-science
python3 -m venv .venv
. .venv/bin/activate
python -m pip install -r requirements.txtOn Windows PowerShell, create the environment with py -3 -m venv .venv and
activate it with .venv\Scripts\Activate.ps1, then run the same installation
command.
Run the Python examples in the data-collection slides in order, in one Python session. They use Beautiful Soup, pypdf, and NLTK. The PDF example reads the included database slides; the HTML and text examples use small inline samples. These examples need neither API credentials nor NLTK corpus downloads. Standalone scraping and API scripts are in the Python workshop.
Install R, then install the plotting packages from an R session:
install.packages(c("ggplot2", "maps"))Run the chunks in ggplot2 in brief in order. They use datasets
included with R and ggplot2; the map example uses maps.
The scraping assignment offers two projects: build a database of NYC legislation, or analyze monthly New York Times coverage of budget deficits. Use R or Python. Define the collection scope, handle pagination, check the retrieved records, and submit code, permitted data, plots, and a short interpretation. The NYT project requires an API key; read it from an environment variable and keep it out of Git.
With the Python environment activated, install the development dependencies:
python -m pip install -r requirements-dev.txt
Rscript -e 'install.packages(c("knitr", "ggplot2", "maps", "cluster"))'
make checkmake check runs Black, isort, and flake8, executes the Python slide examples,
checks JSON, XML, SQL, and regular-expression examples, verifies mathematical
identities in R, and renders the plotting examples. The checks use local data.
Without Make, run the same checks directly:
python -m black --check tests
python -m isort --profile black --check-only tests
python -m flake8 --max-line-length=88 tests
python -m unittest discover -s tests -v
Rscript tests/check_math.RWith Docker and Make installed, make ci-docker runs the checks in standard
Python and R containers. It downloads the required images and dependencies.
The PDFs are included. To rebuild them, install Make and a LaTeX distribution with pdfLaTeX, Beamer, Helvetica, and cmbright, then run:
make slidesEach deck's source and figures are in its lesson directory. The build writes the
six PDFs linked above. To rebuild one deck, use its PDF path as the Make target,
for example make ds2/ds2_present_web.pdf.
These external projects illustrate applications of the lessons. Some use older APIs or services; check each project's setup and service availability before running it.
- OCR: Tesseract in Python, Abbyy FineReader Cloud OCR in R, and Captricity in R.
- Data collection: journalists, weather, cricket, congressional speeches, social-media counts, NYT civil-rights coverage, and social-network data.
- Text analysis: sentiment analysis, words and ideology, and a text classifier.
- Plotting: NYT civil-rights coverage (R code, plot) and military experience of UK prime ministers (R code, plot).
- Suggestions for writing.
- The Elements of Statistical Learning, by Trevor Hastie, Robert Tibshirani, and Jerome Friedman.
- Python Programming: An Introduction to Computer Science, by John Zelle.
- ggplot2: Elegant Graphics for Data Analysis, by Hadley Wickham, Danielle Navarro, and Thomas Lin Pedersen.