Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

perf(build): reduce redundant OpenAPI generator executions in codegen #3403

Open
petermetz opened this issue Jul 11, 2024 · 0 comments
Open
Assignees
Labels
dependencies Pull requests that update a dependency file Developer_Experience P2 Priority 2: High Performance Everything related to how fast/efficient the software or it's tooling (e.g. build) is. Significant_Change Applying this label triggers the more stringent review of the maintainers and the 2+1 PR rule.
Milestone

Comments

@petermetz
Copy link
Member

petermetz commented Jul 11, 2024

Description

The codegen scripts should only call the openapi-generator-cli if the hash of the openapi.json has changed.

In theory the best way to achieve this is to save a hash of the openapi.json file with generated sources.
For example in a manifest.json file such that ./src/main/typescript/generated/openapi/typescript-axios/manifest.json
contains some useful metadata about the code generation process, like the hash of the openapi.json file and the openapi generator config, package.json, openapitools.json, etc. (so any file whose contents could modify what the generated sources look like)

An example of the manifest.json file could look like this:

{
  "inputFiles": [
    "packages/cactus-plugin-keychain-memory-wasm/package.json",
    "packages/cactus-plugin-keychain-memory-wasm/openapitools.json",
    "packages/cactus-plugin-keychain-memory-wasm/src/main/json/openapi.json"
  ],
  "inputFilesMd5": "1abcb33beeb811dca15f0ac3e47b88d9",
  "createdAt": "2024-07-18T02:29:48.058Z",
  "commitSha": "497ea3226631fdcad763e6281ee058d91ca01988"
}

Then before we run the code generation at build time, we examine if the hashes of the files listed in inputFiles is the same as in inputFilesMd5 and if yes, then we can skip the code generation because the generated sources would equal the current state anyway.

For the hashing, we can write a custom script in the ./tools/ directory which can be called by the npm scripts of the packages prior to them calling the generator to perform the examination of whether we need to re-run the codegen on that specific package or not.
Another script also under the ./tools/ directory can be made that generates the manifest.json after each successful code generation process, once for EACH invocation of the code generator. Note that there are some packages which have multiple scripts invoking the code generator so it's important to examine those separately to not miss out on performance gains.

Another Idea: Batching

Look into doing the code generation with a single JVM process (might speed up the process by itself enough that we don't worry about execution times anymore)

https://openapi-generator.tech/docs/usage#batch

@petermetz petermetz added dependencies Pull requests that update a dependency file Developer_Experience Performance Everything related to how fast/efficient the software or it's tooling (e.g. build) is. Significant_Change Applying this label triggers the more stringent review of the maintainers and the 2+1 PR rule. P2 Priority 2: High labels Jul 11, 2024
@petermetz petermetz added this to the v2.1.0 milestone Jul 11, 2024
@petermetz petermetz self-assigned this Jul 11, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
dependencies Pull requests that update a dependency file Developer_Experience P2 Priority 2: High Performance Everything related to how fast/efficient the software or it's tooling (e.g. build) is. Significant_Change Applying this label triggers the more stringent review of the maintainers and the 2+1 PR rule.
Projects
Development

No branches or pull requests

1 participant