Skip to content

Latest commit

 

History

History
235 lines (154 loc) · 8.77 KB

DEVELOPMENT.md

File metadata and controls

235 lines (154 loc) · 8.77 KB
Raw

Setting up a local development environment

Fork and clone the repository onto your computer. You can either run the project using a Docker container or by setting it up Manually.

The project is organized into the following directories:

  • common: Features that are shared between the frontend and backend of Common Voice, such as Typescript types.
  • locales: Information about activated languages on the site, automatically generated by yarn import-locales
  • maintenance: Static code for maintenance mode on Common Voice
  • scripts: Some scripts for managing data
  • server: The server-side app logic, written in TypeScript.
  • web: The Common Voice website files, written in TypeScript. We use React to build the website.

Docker

We provide a docker-compose setup to orchestrate the local development environment configuration using Docker. This is our recommended way of setting up your local development environment, as it will bypass the need to install each component separately.

Requirements

Configuration

You can find configurable options, like the port Common Voice is running on, in /server/src/config-helper.ts. If you want to modify any of these values, you will need to create a configuration file.

If you're using Docker, you should save this file as .env-local-docker (see .env-local-docker.example) in the root directory of the project, and it will be formatted like unix env values, with each key having a CV_ prefix. For example:

CV_DB_ROOT_PASS="root"
CV_MYSQLHOST="db"
CV_IMPORT_SENTENCES="true"

You can copy the example with cp .env-local-docker.example .env-local-docker.

Setup steps

Run the following commands:

> cd common-voice
> docker-compose up

This is going to:

  • Launch a mysql instance configured for common-voice
  • Launch an s3proxy instance to store files locally and avoid going through setting up AWS S3.
  • Mount the project using a Docker volume to allow reflecting changes to the codebase directly to the container.
  • Launch common-voice server

You can visit the website at http://localhost:9000.

Note: Docker can be a very memory-intensive process. If you notice intermittent failures, or if features like auto-rebuilding crash, try increasing Docker's available memory from within Docker's Preferences > Resources settings.**

Apple M1 Silicon

There is an outstanding issue where MySQL doesn't work in Docker on Apple Silicon (as of 17th January 2022). You can instead replace it with MariaDB in a docker-compose.override.yml file.

services:
  db:
    image: mariadb:10.4
    command: mysqld --sql_mode="STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_ENGINE_SUBSTITUTION"

Troubleshooting

If you get an error like the following when running native Docker (not Docker for Desktop),

ERROR: Couldn't connect to Docker daemon at http+docker://localhost - is it running?

You may need to build a new image. You can do that by issuing the following commands:

> cd docker/
> docker build .

Then after this you can:

> cd ..
> docker-compose up

You may have to run these commands as root/superuser.

Manual installation

You may want to set up each component manually. In which case you will need:

Requirements

Configuration

You can find configurable options, like the port Common Voice is running on, in /server/src/config-helper.ts. If you want to modify any of these values, you will need to create a configuration file.

If you installed the app manually, create a /config.json with the config you want to override in JSON format. The keys will not have a CV_ prefix. For example:

{
  "IMPORT_SENTENCES": false,
  "MYSQLDBNAME": "voice",
  "MYSQLHOST": "127.0.0.1",
}

Database preparation

Once the required components are installed, you need to prepare your database.

You can either create a MySQL superuser that that uses the default DB_ROOT_USER and DB_ROOT_PASS values from /server/src/config-helper.ts or create your own config as described above.

S3 configuration

The Common Voice project uses S3 for voice clip storage. This will be provided for you if you use the Docker installation, but if you are doing local development you will need to set up your own S3 instance. For detaield instructions on how to do that, see HOWTO_S3.md

Setup steps

Make sure your MySQL server is running. Then run the following commands:

> yarn
> yarn start

This will:

  1. Install all JavaScript dependencies.
  2. Build and serve files located in the web folder on localhost.
  3. Save uploaded voice clips onto Amazon's S3.
  4. Lint and rebuild all js files on every change.

You can then access the website at http://localhost:9000.

Linting

We're using ESLint (with Typescript) and Prettier to lint the project.

Install ESlint and Prettier extentions into your code editor or you can run this for all files with:

yarn lint

For now this is not automatically checked on Pull Requests or blocking while we fix existing issues.

Authentication

If you want to work with login-related features (Profile, Dashboard, Goals, ...) you'll need to set up authentication:

  1. Create an Auth0 account.
  2. Click "Applications" from the dashboard. Create a new one, or use the default application.
  3. On "Applications" still, next to your application, click the "three dots" icon, then Settings.
  4. Add http://localhost:9000/callback to the "Allowed Callback URLs" list.
  5. Copy the Auth0 application settings into your configuration file. These are found in the same Settings tab as the "Allowed Callback URLs".

For Docker, in .env-local-docker:

CV_AUTH0_DOMAIN="<domain_here>"
CV_AUTH0_CLIENT_ID="<client_id_here>"
CV_AUTH0_CLIENT_SECRET="<client_secret_here>"

For local development, in config.json:

"AUTH0": {
 "DOMAIN": "<domain_here>",
 "CLIENT_ID": "<client_id_here>",
 "CLIENT_SECRET": "<client_secret_here>"
}
  1. You can add more login options to your app from the "Connections" tab
  2. Restart your local Common Voice instance
  3. You will now be able to create a new user by clicking on "Log in" on your Common Voice instance and then switching over to the "Sign Up" tab on the login dialog.

Database migrations

We use db-migrate for running database migrations.

To add a migration run: yarn migrate create <MIGRATION_NAME>.

At the moment you manually have to change the migration file extension to .ts. A migration has to expose the following API:

export const up = async function (db: any): Promise<any> {
  return null;
};

export const down = async function (): Promise<any> {
  return null;
};

Migrations are always run when the server is started.

Localization

We're using Fluent to localize strings. You can find examples all over the frontend code. Strings that appear in the english message files, can then be translated on Pontoon. Some things to note regarding string changes are documented on MDN.

Updating languages

To update the list of locales run:

> yarn import-locales

This creates/updates files in /locales:

  • fetch locale codes & names from Pontoon and save them in all.json
  • based on Pontoon translated data and a threshold defined in the script, save "completed" locales to translated.json
  • add codes that have a sentence folder in /server/data and at least 5k sentences to contributable.json

Submitting an Issue

Did you notice a bug? Do you have a feature request? Please file an issue here on GitHub.

Something Else?

Want to talk about something but can't find a home for it here? Head to our Discourse Category to discuss everything from feedback and ideas to questions and random musings.