A microservice to periodically scrape content from freecycle.org and broadcast new posts as Twitter tweets. See the service in action here.
Disclaimer: This is a prototype and not considered production ready.
The freecycle.org site is outdated. In advance of any future improvements, it would be nice to provide the community with a more modern interface. A Twitter bot is one way to offer a pub-sub style mechanism for users while at the same time minimising operational overhead.
The service architecture is broadly composed of three elements.
- A scraper which parses
freecycle's public website usingcheerio. - A persistence layer which stores post history using a free managed
PostgreSQLinstance from Scaleway. - A "tweeter" which publishes new posts to
Twitterusing thetwitclient.
The above functionality is wrapped up in a long running NodeJS process executed in a Docker container. Travis CI is used for CI tooling.
To run in development:
- Clone the repo.
git clone https://github.com/hughmp/freecycle-twitter-bot.git- Build the docker image.
docker build --name freecycle-twitter-bot .- Set your configuration.
export POSTGRES_USERNAME='username'
export POSTGRES_PASSWORD='password'
export POSTGRES_DATABASE='database'
export POSTGRES_HOST='host'
export POSTGRES_PORT=1234
export TWITTER_CONSUMER_KEY='abcdefghijk'
export TWITTER_CONSUMER_SECRET='lmnopqrstuvw'
export TWITTER_ACCESS_TOKEN='abcdefghijk'
export TWITTER_ACCESS_SECRET='lmnopqrstuvw'- Run the container.
docker run -d \
--env POSTGRES_USERNAME \
--env POSTGRES_PASSWORD \
--env POSTGRES_DATABASE \
--env POSTGRES_HOST \
--env POSTGRES_PORT \
--env TWITTER_CONSUMER_KEY \
--env TWITTER_CONSUMER_SECRET \
--env TWITTER_ACCESS_TOKEN \
--env TWITTER_ACCESS_SECRET \
freecycle-twitter-bot- Profit!
- Improve test coverage
- Integrate continuous deployment
- Cosider implementing lightweight DB in a
Sidecarpattern - Remove secrets from build and publish public Docker image