ML-USAS

This repository contains various DataTrove pipelines, filters, formatters, and helper functions for processing the HuggingFaceFW finewiki dataset, in various languages listed in the languages section, to create a synthetic (silver labelled) training dataset for USAS semantic tags and Multi Word Expression (MWE) identification for some languages.

For more information on the filtering and processing, see the filtering and processing section below and for more information about the data we use see the data section below.

Setup

You can either use the dev container with your favourite editor, e.g. VSCode. Or you can create your setup locally below we demonstrate both.

In both cases they share the same tools, of which these tools are:

• uv for Python packaging and development
• make (OPTIONAL) for automation of tasks, not strictly required but makes life easier.

Dev Container

A dev container uses a docker container to create the required development environment, the Dockerfile we use for this dev container can be found at ./.devcontainer/Dockerfile. To run it locally it requires docker to be installed, you can also run it in a cloud based code editor, for a list of supported editors/cloud editors see the following webpage.

To run for the first time on a local VSCode editor (a slightly more detailed and better guide on the VSCode website):

1. Ensure docker is running.
2. Ensure the VSCode Dev Containers extension is installed in your VSCode editor.
3. Open the command pallete CMD + SHIFT + P and then select Dev Containers: Rebuild and Reopen in Container

You should now have everything you need to develop, uv, make, for VSCode various extensions like Pylance, etc.

If you have any trouble see the VSCode website..

Local

To run locally first ensure you have the following tools installted locally:

• uv for Python packaging and development. (version 0.9.6)
• make (OPTIONAL) for automation of tasks, not strictly required but makes life easier.
  • Ubuntu: apt-get install make
  • Mac: Xcode command line tools includes make else you can use brew.
  • Windows: Various solutions proposed in this blog post on how to install on Windows, inclduing Cygwin, and Windows Subsystem for Linux.

When developing on the project you will want to install the Python package locally in editable format with all the extra requirements, this can be done like so:

uv sync --all-extras

Linting

Linting and formatting with ruff it is a replacement for tools like Flake8, isort, Black etc, and we us ty for type checking.

To run the linting:

make lint

Tests

To run the tests (uses pytest and coverage) and generate a coverage report:

make test

Example of using Flash Attention

from sentence_transformers import SentenceTransformer

model = SentenceTransformer(
    "sentence-transformers/all-MiniLM-L6-v2",
    model_kwargs={"attn_implementation": "flash_attention_2", "torch_dtype": "bfloat16"},
)

sentences = ["This is an example sentence", "Each sentence is converted"]
embeddings = model.encode(sentences)

Offline data download of the USAS silver labelled dataset

uv run processing_scripts/download_dataset.py ./data/usas_silver_data/

HuggingFace Authentication

Before processing or uploading to the HuggingFace hub please authenticate using a token from huggingface.co/settings/tokens;

hf auth login

or by using a token that is set within ./.env, read using dotenv, e.g.

HF_TOKEN="HUGGINGFACE_TOKEN_KEY_VALUE"

Set the relevant permissions, the minimum for this is repository is "read" only permission, if you want to upload the created synthetic silver labelled dataset to HuggingFace please ensure that you have allowed write permission to the namespace/repository you are going to upload too on HuggingFace.

Data

The data will be coming from HuggingFaceFW finewiki dataset and will be filtered so that each Wikipedia article is either rated as a "Good Articles" (GA) or "Featured Articles" (FA) by an editor, we hope that this will remove articles that might be incomplete or require additional editing. This filtering is inspired by Conia et al. 2024 whereby they found training on data from only "featured" and "good" articles performed similarly to training on the far larger Wikipedia articles that contained non-good and non-featured articles thus showing that training on smaller amounts of data is as affective and more efficient. The "Featured" and "Good" article can be defined differently for each Wikipedia language site as stated in the English site definition within the following article. The list of GA and FA can be found at the HuggingFace dataset ucrelnlp/wikipedia-ga-fa-ids.

Software

Train with HuggingFace through sentence-transformers, experimental tracking with trackio, and carbon emission tracking using carboncode.

Notes

• Data can be sampled so that we see a maximum number of tokens, $N$ from a given tag class, if more than $N$ occur for a given tag class then tokens assigned with that class are sub-sampled by $M/N$ where $M$ are the number of token samples for that given class.

Future improvements

• Adding un-pad token efficiency for flash-attention which cannot be used (flash-attention can just not un-padding) as build_model form @src/ml_usas/wsd/model.py contains unpad_inputs=False.
• Adding hard negative examples from a teacher model that finds difficult examples.
• I think that having to_train_ntuple_dataset within src/ml_usas/wsd/dataset.py as an interable streaming dataset would be better as it will dynamic hard negatives per epoch rather than the same hard negatives.
• I think token_row_list = list(token_rows) should be an iterable of some description within to_ir_eval_data which is in src/ml_usas/wsd/dataset.py
• Create an evaluation metric for top_k accuracy whereby the model has to predict all of the valid tokens, when it is a multi tag token the bi-encoder models will never get those samples correct.
• Logging with trackio

Languages

The languages that this repository covers and supports, of which this table is also available in machine readable format at ./data/languages.yaml (languages that have the value of True for the key training). These languages have been selected based on semantic tagging support for the given language whereby in some cases setting up the semantic tagger for a given language can be difficult within a large scale tagging pipeline in addition some languages have very few to none GA or FA articles.

Language	ISO 639-3
English	eng
Dutch	nld
Spanish	spa
Danish	dan
Italian	ita
Portuguese	por
Chinese	zho
Finnish	fin

License

The code is licensed under Apache License Version 2.0.

Claude settings

For those that use Anthropic's Claude we have shared some suggested settings, see ./.claude folder that are enforced within this project but can be easily adjusted or removed if you prefer to use your own settings or the default settings of Claude. The project level settings for Claude, can be found at ./.claude/settings.json are auto generated by running the following script;

cd .claude/hooks && uv run generate_settings.py > ../settings.json

This script creates a settings file with;

• Numerous Deny permissions that have come from the list of files, stated in ./.claude/hooks/sensitive_patterns.py, that you do not want Claude to write/edit/read.
• A pre-hook, ./.claude/hooks/block_sensitive_files.py, that catches any write/edit/read to the list of files that the Deny permissions might miss, e.g. a call to Python using Bash.

To note this pre-hook and Deny permissions would not stop Claude from write/edit/read if Claude requests the file through an unusual regex pattern like e*v to get the .env file, but this is a best effort try to reduce Claude's access to these more sensitive files. Generally speaking if you are using API keys reduce the scope as much as possible and limit the time and resource access while developing.