somaniarushi / tinymoe Public

Notifications You must be signed in to change notification settings
Fork 0
Star 1

💥 Tiny 9M MoEs with Dynamic Routing

1 star 0 forks Branches Tags Activity

Notifications

Name		Name	Last commit message	Last commit date
Latest commit History 6 Commits
data		data
docs		docs
experiments		experiments
src		src
.gitignore		.gitignore
.pre-commit-config.yaml		.pre-commit-config.yaml
README.md		README.md
requirements.txt		requirements.txt

Repository files navigation

TinyMoE: 9M Mixture of Experts Models

Heavily inspired by Karpathy's Makemore as well as AviSoori's MakeMoe.

✍️ Experiment Logbook: https://www.notion.so/dailyink/Dynamic-Routing-in-Tiny-9M-MoE-Models-4205e3283ef3479e8fcfdbc5514f0bdd

About

💥 Tiny 9M MoEs with Dynamic Routing

Report repository

Releases

No releases published

Packages

No packages published

Languages

Python 100.0%