Proposal
Add support for TracrBench transformers
Motivation
I and @JeremyAlain recently wrote a paper in which we introduced a dataset of 121 tracr-transformers. Tracr transformers are meant to be used as test beds or "sanity-checks" in the development of novel interpretability methods. To make them as accessible as possible we convert from the deepmind-internal "haiku" framework to Hooked Transformers (following this template made by Neel). We would like and have been asked by multiple people to make these toy models available from within transformerlens.
Pitch
We have all the models uploaded to huggingface and i have code to load the models. It's a little different from the code used to load the typical LLMs. Since the model requires input and output encoders, we wrap the hooked transformer class in another simple class called "TracrModel".
My question is, whether this is possible and if so, where to put this code/the tracr_models.py file.
Alternatives
An alternative would be to integrate the code to download the tracr models for use within transformerlens in another repo.
Proposal
Add support for TracrBench transformers
Motivation
I and @JeremyAlain recently wrote a paper in which we introduced a dataset of 121 tracr-transformers. Tracr transformers are meant to be used as test beds or "sanity-checks" in the development of novel interpretability methods. To make them as accessible as possible we convert from the deepmind-internal "haiku" framework to Hooked Transformers (following this template made by Neel). We would like and have been asked by multiple people to make these toy models available from within transformerlens.
Pitch
We have all the models uploaded to huggingface and i have code to load the models. It's a little different from the code used to load the typical LLMs. Since the model requires input and output encoders, we wrap the hooked transformer class in another simple class called "TracrModel".
My question is, whether this is possible and if so, where to put this code/the tracr_models.py file.
Alternatives
An alternative would be to integrate the code to download the tracr models for use within transformerlens in another repo.