Skip to content

Latest commit

 

History

History
61 lines (43 loc) · 2.4 KB

TESTING.md

File metadata and controls

61 lines (43 loc) · 2.4 KB
Raw

Unit Tests

Unit tests are executed with npm run build and npm run test. GitHub Actions won't let PRs merge if the tests fail.

To run on Windows, use:

jest --testMatch */**/*.test.ts

Integration Test

We have one set of integration tests. It is a CDK app that creates a set of runners. Once deployed, the runners should be tested using the self-hosted.yml workflow. This should tell us GitHub integration and all runner provider types are working properly.

We keep a snapshot of the CloudFormation template generated by the integration test in test/default.integ.snapshot. During build time and PR validation, we validate the integration test still results in the same CloudFormation template as the snapshot. GitHub Actions won't let PRs merge if the tests fail.

If the snapshot changes, you should commit it as part of your PR. You should ideally deploy and test it. The PR should mention whether the integration test was actually deployed and tested.

To assert the snapshot hasn't changed, use:

npm run integ:default:assert

To deploy the integration test, use:

npm run integ:default:deploy

To update the snapshot, use:

npm run integ:default:snapshot

Manual Tests

Integration tests check the happy paths. We should also test the unhappy paths manually. This is a list of scenarios we should manually test before releasing a new version:

  • Setup page
    • GitHub app
    • Personal access token
    • GitHub Enterprise Server
  • Idle reaper
    • Confirm idle runner is stopped automatically
    • Confirm runner doesn't stay registered in GitHub
    • Confirm runner is not retried
    • Step function result is aborted and not failed
  • Retries
    • Confirm runner errors are retried
    • Confirm failed runner doesn't stay registered in GitHub

The last two scenarios can be tested with the following test cases:

  • Start step function without a job actually pending (e.g. by duplicating input from a previous job, or cancelling a job before a runner picks it up)
    • The step function should be aborted as an idle runner
    • No runner should be registered on GitHub at the end
  • Let Lambda runner timeout by starting a job that lasts longer than 15 minutes
    • The runner should be retried and eventually the step function should be aborted as an idle runner
    • No runner should be registered on GitHub at the end