This is my personal wiki for machine learning & engineering related topics: a living document of all code snippets, tutorials, step-by-step processes, and config files I recurrently find useful. It's also how I keep my cool.
The project is not intended to be 'run', nor do the various files fit together or function as a cohesive project. Everything in this repository is either a useful insight, piece of code, or config file that I find myself using again and again. The tutorials section is my effort to organise these snippets into more cohesive wholes.
I don't claim that anything in this repository represents my own insight and ability. Everything is a product of my having read countless blogs/forums/implementations, customized things as needed for myself, and there was no one place I would be able to go to get all the information I need the next I want to perform a particular task. This repository is first and foremost a toolshed for myself, but I try to keep it clean in the hope that you'll find something useful here too.
This README should be used as the index and entry point into everything the Toolshed has to offer.
- Tensorflow Serving and how to use it for real production settings.
The following notes are my attempt to write practical high-signal documentation for processes that usually require me to click through >5 pages of AWS docs. There are 7' (short) and LP (long) versions of each note as well as references to various pages of official documentation for further reading.
- How to increase EC2 storage
- How to setup an EC2 instance
- How to setup the Cloudwatch Agent for EC2
``
This section contains general tips on deployment and automated infrastructure setup, and some specific example walkthroughs of specific, real-life production pipelines.
- EC2 deployment w. Gitlab CI & Docker 🏗️ [Coming Soon] 🏗️
- Blue-Green Deployment with Terraform & AWS 🏗️ [Coming Soon] 🏗️
This section primarily contains the Dockerfiles and docker-compose yaml files used by tutorials in other sections of the README, as well as a few Docker-specific tutorials
- SSL w. certbot, nginx, & docker-compose 🏗️ [Coming Soon] 🏗️
- Files
- docker-compose.tf.yml (used by the Tensorflow Serving tutorial).
This contains a bunch of notebook specific functions or functions I use with Colab.
- images.ipynb contains helper functions for displaying & manipulating images
- stylegan.ipynb contains StyleGAN2 helper functions; I mostly use these w. Google Colab StyleGAN implementations
- MLFlow in Production 🏗️ [Coming Soon] 🏗️
- MyMLFlowClient contains the client I use for all programmatic MLFlow interaction
This section contains a number of code snippets & tutorials related to Tensorflow and the
Tensorflow-in-production ecosystem. All code snippets are available inside the tf directory,
though most of these are referenced in at least one tutorial
- Tutorials
- Tensorflow Serving (for real production workflows)
- Tensorflow Model Formats [Coming Soon] 🏗️
- Code (most of which is referenced by the tutorials)
callbacks.pycontains a collection of training callbacks for use with.fitfunctionmodels.pycontains a collection of utils for model loading/modifying/customizing/converting etc.model_formats.pyserving_predictions.pycontains code for getting predictions from a tensorflow serving ModelServer
And here are some utility functions that don't fit anywhere else
- programmatic GPU config & customization for tensorflow & pytorch.
- programmatic ssh, i.e. creating client connection in Python code
- programmatic scp, i.e. using the above ssh function to copy files from/to some remote machine