Skip to content

Commit

Permalink
merge dev (#123)
Browse files Browse the repository at this point in the history
* fix: set cli as a es module

* fix: bump 0.3.0; minor missing imports for enums

* feat (EC2-demo) testing VMs

* feat(ec2) - continuing with poc

* fix(sybil) - add tests

* fix(sybil) - added prod test switch

* build: update external packages dependencies

* feat(public ceremonies) - allow to download ceremony data to everyone

* refactor: move auto-generated types to root

* refactor: improve package configuration; remove unused files

* refactor: complete configuration; lint and small fix

* fix: wrong prod Firebase project for CI

* refactor: update phase2cli package config; fixed bin overwrite when installing

* refactor: use meta import to optimize repository file read

* refactor: use ts-node instead of dist to execute phase2cli

* refactor: update .env and package.json imports

* fix: wrong path for environment config file

* fix(public-data) remove cloud function

* fix(public-data) remove function from index.ts

* fix(public data)  yarn.locl

* fix: missing skipLibCheck option was causing docs script to exit

* fix: add missing options to ts config

* chore(release): publish 0.3.0

* build: update yarn version

* fix: spinner does not stop correctly after contribution verification

* chore(release): publish 0.3.1

* fix: missing updates for current contributor fields when coordinating

* fix: duplicate messages when waiting for contribution; remove wrong listener

* chore(release): publish 0.3.5

* fix(sybil checks): amended env var name and added to the backend

* fix(s3): creating public bucket with object ACL enabled

configuring S3 to allow download of verification transcript

* fix(tests): fixed tests

* fix: unconsistency when resuming a contribution

* fix: add missing alternative case; fix wrong tests

* chore(release): publish 0.4.0

* fix: wrong ci env project

* fix: missing sort for ceremony circuits

* fix: wrong document property lead to error when checking github antisybil

* build: missing dependencies on actions package

* chore(release): publish 0.4.1

* chore(release): publish 0.4.2

* test: update github anti-sybil tests

* fix(ux): added more verbose messages to inform the user of waiting times and errors

* feat(ec2): added tests and ip function

* feat(ec2): merge udpates and add tests

* feat(vms): added ssm code and more tests + changes on backend

* feat(vms): refactoring

* fix: imports and package version

* feat(vms): added tests

* fix(vms): tests

* feat(ec2): fixed bug in running commands at deployment and refactored setup

* test(ec2): added setup e2e test

* fix(vms): refactoring and code fixes

* fix(vm): bug fix

* fix(vm): missing files

* feat(vms): implemented e2e test for contribution verification

* feat(vms): tests and refactoring

* fix(vms): remove redundant code

* test(vms): completed e2e tests with VMs

* fix(vms): cleanup test

* fix(vms): remove redundant parameter on VM startup - ssh keypair

* fix(vms): add tags on EC2 creation

* refactor: add missing hash computation; minor conflicts and changes

* fix(vms): fix VM commands

* test(vms): fix test to also delete the verification transcript

* chore: publish 0.4.7

* test(vms): refactoring of VM tests with extra debug options

* test(vms): fixed finalize test and cloud function. Adding extra debugging for verification

* fix(vms): fixed wrong path in blake3 bin command and various fixes on the verification CF

* chore: publish 0.4.8

* fix(vms): removed stdin/stdout redirection in VM command and added regex match for blake3 hash

* fix(vms): fixed cloud function for VM verification

* chore: publish 0.4.9

* fix(vms): fix cloud function bug related to the wrong verification transcript path

* test(vms): fix broken tests due to VM upgrade

* fix(vms): disable eslint rule for regex

* feat(vms): implement SNS topic command to trigger Lambda that stops the VM after initialization

* feat(vms): add marker file in userData to avoid running initialization code more than once

* fix(vms): fix userData commands

* refactor: add VM prices

* refactor: add selector for VM disk type

* test: update tests

* refactor: complete and remove old todos

* refactor: add retry mechanism for VM

* fix: wrong path to Verifier smart contract template

* feat(vms): implement terraform script to deploy AWS inf

* fix(vms): fix import error

* fix: correct failing tests

* fix(vms): fix cloud function multipartupload wrong parameter name

* chore: publish 0.4.10

* fix(vms): fix terraform and lambda config

* fix(vms): temp revert of retry feature in verifyContribution

* fix(vms): fix emulator tests

* fix(vms): fix prod tests

* refactor: improve verifycontribution function code; minor changes to code readability

* fix(vms): fix prod test - wrong parameter order in ec2 tests

* chore: publish 0.5.0

* fix: missing else statement for verifycontribution function; wrong return on custom promise

* fix(vms): revert part of the verifyContribution refactoring and update terraform

* fix(vms): retry mechanism for VM startup

* refactor: add options for contribute command

* refactor: make coordinator artifacts publicly downloadable by default

* chore: update backend firebase default prod project

* fix(tests): add extra sleep when setting custom claims to allow propagation

* fix(tests): skip a test that would fail due to environment not clean

* fix(tests): fix issues with user not being recognized as coordinator

* fix(tests): add extra sleep and coordinator checks within tests

* refactor: avoid mandatory memory check; add missing loader and messages to improve ux

* feat(setup): start to implement non interactive setup

* feat(setup): implement non interactive setup

* fix(setup): change test file

* fix(setup): remove non working test

* fix(setup): change setup test file

* fix(setup): amend after PR review

* feat(setup): add option to pass the authentication token as cli param

passing the cli token as param one could easily use p0tion non interactive setup in a CI/CD env like
GitHub actions.

* fix(tests): add extra coordinator user to resolve random test failure

* fix(tests): add extra sleep after user creation

* fix(auth): fix non interactive auth and force token refresh for coordinator

* fix(setup): fix local path to circuit artifacts + change error in security tests

* fix(setup): add missing circuit artifacts local path

allow circuit artifacts to be in a separate folder for setup so that it can be used with
definitelySetup

* fix(tests): fix tests

* refactor: improve ux ui of auth and logout commands

* feat(setup): add a cli command to validate the ceremony setup file and the constraints

* refactor: add spinners to avoid delays; add custom feedback messages for contributor

* chore: correct text

* feat(list ceremonies): add a command to list all ceremonies

* feat(list): exit with code 1 when calling showError with exit=true

* fix(tests and setup): fix tests and cleanup config file

* chore: publish 1.0.0

* fix(verifycontribution): fix issue with failing to deconstruct undefined object (vm)

* Revise workflow

* Fix a typo

* V1.1.1

* V1.1

* fix: set cli as a es module

* fix: bump 0.3.0; minor missing imports for enums

* feat (EC2-demo) testing VMs

* feat(ec2) - continuing with poc

* fix(sybil) - add tests

* fix(sybil) - added prod test switch

* build: update external packages dependencies

* feat(public ceremonies) - allow to download ceremony data to everyone

* refactor: move auto-generated types to root

* refactor: improve package configuration; remove unused files

* refactor: complete configuration; lint and small fix

* fix: wrong prod Firebase project for CI

* refactor: update phase2cli package config; fixed bin overwrite when installing

* refactor: use meta import to optimize repository file read

* refactor: use ts-node instead of dist to execute phase2cli

* refactor: update .env and package.json imports

* fix: wrong path for environment config file

* fix(public-data) remove cloud function

* fix(public-data) remove function from index.ts

* fix(public data)  yarn.locl

* fix: missing skipLibCheck option was causing docs script to exit

* fix: add missing options to ts config

* chore(release): publish 0.3.0

* build: update yarn version

* fix: spinner does not stop correctly after contribution verification

* chore(release): publish 0.3.1

* fix: missing updates for current contributor fields when coordinating

* fix: duplicate messages when waiting for contribution; remove wrong listener

* chore(release): publish 0.3.5

* fix(sybil checks): amended env var name and added to the backend

* fix(s3): creating public bucket with object ACL enabled

configuring S3 to allow download of verification transcript

* fix(tests): fixed tests

* fix: unconsistency when resuming a contribution

* fix: add missing alternative case; fix wrong tests

* chore(release): publish 0.4.0

* fix: wrong ci env project

* fix: missing sort for ceremony circuits

* fix: wrong document property lead to error when checking github antisybil

* build: missing dependencies on actions package

* chore(release): publish 0.4.1

* chore(release): publish 0.4.2

* test: update github anti-sybil tests

* fix(ux): added more verbose messages to inform the user of waiting times and errors

* feat(ec2): added tests and ip function

* feat(ec2): merge udpates and add tests

* feat(vms): added ssm code and more tests + changes on backend

* feat(vms): refactoring

* fix: imports and package version

* feat(vms): added tests

* fix(vms): tests

* feat(ec2): fixed bug in running commands at deployment and refactored setup

* test(ec2): added setup e2e test

* fix(vms): refactoring and code fixes

* fix(vm): bug fix

* fix(vm): missing files

* feat(vms): implemented e2e test for contribution verification

* feat(vms): tests and refactoring

* fix(vms): remove redundant code

* test(vms): completed e2e tests with VMs

* fix(vms): cleanup test

* fix(vms): remove redundant parameter on VM startup - ssh keypair

* fix(vms): add tags on EC2 creation

* refactor: add missing hash computation; minor conflicts and changes

* fix(vms): fix VM commands

* test(vms): fix test to also delete the verification transcript

* chore: publish 0.4.7

* test(vms): refactoring of VM tests with extra debug options

* test(vms): fixed finalize test and cloud function. Adding extra debugging for verification

* fix(vms): fixed wrong path in blake3 bin command and various fixes on the verification CF

* chore: publish 0.4.8

* fix(vms): removed stdin/stdout redirection in VM command and added regex match for blake3 hash

* fix(vms): fixed cloud function for VM verification

* chore: publish 0.4.9

* fix(vms): fix cloud function bug related to the wrong verification transcript path

* test(vms): fix broken tests due to VM upgrade

* fix(vms): disable eslint rule for regex

* feat(vms): implement SNS topic command to trigger Lambda that stops the VM after initialization

* feat(vms): add marker file in userData to avoid running initialization code more than once

* fix(vms): fix userData commands

* refactor: add VM prices

* refactor: add selector for VM disk type

* test: update tests

* refactor: complete and remove old todos

* refactor: add retry mechanism for VM

* fix: wrong path to Verifier smart contract template

* feat(vms): implement terraform script to deploy AWS inf

* fix(vms): fix import error

* fix: correct failing tests

* fix(vms): fix cloud function multipartupload wrong parameter name

* chore: publish 0.4.10

* fix(vms): fix terraform and lambda config

* fix(vms): temp revert of retry feature in verifyContribution

* fix(vms): fix emulator tests

* fix(vms): fix prod tests

* refactor: improve verifycontribution function code; minor changes to code readability

* fix(vms): fix prod test - wrong parameter order in ec2 tests

* chore: publish 0.5.0

* fix: missing else statement for verifycontribution function; wrong return on custom promise

* fix(vms): revert part of the verifyContribution refactoring and update terraform

* fix(vms): retry mechanism for VM startup

* refactor: add options for contribute command

* refactor: make coordinator artifacts publicly downloadable by default

* chore: update backend firebase default prod project

* fix(tests): add extra sleep when setting custom claims to allow propagation

* fix(tests): skip a test that would fail due to environment not clean

* fix(tests): fix issues with user not being recognized as coordinator

* fix(tests): add extra sleep and coordinator checks within tests

* refactor: avoid mandatory memory check; add missing loader and messages to improve ux

* feat(setup): start to implement non interactive setup

* feat(setup): implement non interactive setup

* fix(setup): change test file

* fix(setup): remove non working test

* fix(setup): change setup test file

* fix(setup): amend after PR review

* feat(setup): add option to pass the authentication token as cli param

passing the cli token as param one could easily use p0tion non interactive setup in a CI/CD env like
GitHub actions.

* fix(tests): add extra coordinator user to resolve random test failure

* fix(tests): add extra sleep after user creation

* fix(auth): fix non interactive auth and force token refresh for coordinator

* fix(setup): fix local path to circuit artifacts + change error in security tests

* fix(setup): add missing circuit artifacts local path

allow circuit artifacts to be in a separate folder for setup so that it can be used with
definitelySetup

* fix(tests): fix tests

* refactor: improve ux ui of auth and logout commands

* feat(setup): add a cli command to validate the ceremony setup file and the constraints

* refactor: add spinners to avoid delays; add custom feedback messages for contributor

* chore: correct text

* feat(list ceremonies): add a command to list all ceremonies

* feat(list): exit with code 1 when calling showError with exit=true

* fix(tests and setup): fix tests and cleanup config file

* chore: publish 1.0.0

* fix(verifycontribution): fix issue with failing to deconstruct undefined object (vm)

* Revise workflow

* Fix a typo

---------

Co-authored-by: ctrlc03 <93448202+ctrlc03@users.noreply.github.com>
Co-authored-by: cedoor <me@cedoor.dev>
Co-authored-by: Jeeiii <giacomo.corrias7@gmail.com>
Co-authored-by: Daehyun Paik <paik@a30a.dev>

* Separate deployment and testing

---------

Co-authored-by: daodesigner <58278045+daodesigner@users.noreply.github.com>
Co-authored-by: ctrlc03 <93448202+ctrlc03@users.noreply.github.com>
Co-authored-by: cedoor <me@cedoor.dev>
Co-authored-by: Jeeiii <giacomo.corrias7@gmail.com>

* Separate deploy and test (#99)

* feat(setup): non interactive setup with artifacts download

* feat(setup): add cloud function for file transfer and unit tests

* feat(setup): add transfer of object inside phase2cli

* feat(setup): remove return value and amend tests

* fix(setup): revert transfer of object and add region to config

* feat(setup) fix test template

* fix(setup): fix not waiting for file download

* fix(setup): remove redundant function

* chore(release): publish 1.0.1

* fix(vms): specify region in sns command

* fix(vms) - missing param

* fix(vms): missing doc

* fix(vms): fix default region

* chore(release): publish 1.0.2

* fix(setup): fix vm command due to differences in AMI config between regions

* chore(release): publish 1.0.4

* fix (ceremony) - fix various bugs (#116)

* fix(setup): add spinners

* fix(bugs): fix backend and cli bugs

* fix(regions): move cloud functions to same region as firestore and s3

* fix(tests): add new region on test utils

* Ensure all CFs are deployed into `europe-west1` region

* fix(verification): set concurrency to 1 and ensure that files are downloaded

* chore(release): publish 1.0.5 (#120)

* fix(verify): revert removing concurrency and add unique filenames (#122)

---------

Co-authored-by: cedoor <me@cedoor.dev>
Co-authored-by: Jeeiii <giacomo.corrias7@gmail.com>
Co-authored-by: Daehyun Paik <paik@a30a.dev>
Co-authored-by: daodesigner <58278045+daodesigner@users.noreply.github.com>
Co-authored-by: daodesigner <designerdao.eth@gmail.com>
  • Loading branch information
6 people authored Jul 24, 2023
1 parent 10f8fd6 commit e522cb4
Show file tree
Hide file tree
Showing 13 changed files with 186 additions and 88 deletions.
6 changes: 6 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,12 @@
All notable changes to this project will be documented in this file.
See [Conventional Commits](https://conventionalcommits.org) for commit guidelines.

## [1.0.5](https://github.com/privacy-scaling-explorations/p0tion/compare/v1.0.4...v1.0.5) (2023-07-24)

**Note:** Version bump only for package p0tion



## [1.0.4](https://github.com/privacy-scaling-explorations/p0tion/compare/v1.0.2...v1.0.4) (2023-07-20)


Expand Down
2 changes: 1 addition & 1 deletion lerna.json
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
],
"npmClient": "yarn",
"useWorkspaces": true,
"version": "1.0.4",
"version": "1.0.5",
"changelogPreset": {
"name": "conventionalcommits",
"issuePrefixes": [
Expand Down
7 changes: 7 additions & 0 deletions packages/actions/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,13 @@
All notable changes to this project will be documented in this file.
See [Conventional Commits](https://conventionalcommits.org) for commit guidelines.


## [1.0.5](https://github.com/privacy-scaling-explorations/p0tion/compare/v1.0.4...v1.0.5) (2023-07-24)

**Note:** Version bump only for package @p0tion/actions



## [1.0.4](https://github.com/privacy-scaling-explorations/p0tion/compare/v1.0.2...v1.0.4) (2023-07-20)


Expand Down
2 changes: 1 addition & 1 deletion packages/actions/package.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "@p0tion/actions",
"version": "1.0.4",
"version": "1.0.5",
"description": "A set of actions and helpers for CLI commands",
"repository": "git@github.com:privacy-scaling-explorations/p0tion.git",
"homepage": "https://github.com/privacy-scaling-explorations/p0tion",
Expand Down
64 changes: 44 additions & 20 deletions packages/actions/src/helpers/utils.ts
Original file line number Diff line number Diff line change
Expand Up @@ -185,39 +185,63 @@ export const parseCeremonyFile = async (path: string, cleanup: boolean = false):
// check that the timeout is provided for the correct configuration
let dynamicThreshold: number | undefined
let fixedTimeWindow: number | undefined

let circuit: CircuitDocument | CircuitInputData = {} as CircuitDocument | CircuitInputData

if (data.timeoutMechanismType === CeremonyTimeoutType.DYNAMIC) {
if (circuitData.dynamicThreshold <= 0)
throw new Error("The dynamic threshold should be > 0.")
dynamicThreshold = circuitData.dynamicThreshold

// the Circuit data for the ceremony setup
circuit = {
name: circuitData.name,
description: circuitData.description,
prefix: circuitPrefix,
sequencePosition: i+1,
metadata: metadata,
files: files,
template: template,
compiler: compiler,
verification: verification,
dynamicThreshold: dynamicThreshold,
avgTimings: {
contributionComputation: 0,
fullContribution: 0,
verifyCloudFunction: 0
},

}
}

if (data.timeoutMechanismType === CeremonyTimeoutType.FIXED) {
if (circuitData.fixedTimeWindow <= 0)
throw new Error("The fixed time window threshold should be > 0.")
fixedTimeWindow = circuitData.fixedTimeWindow
}

// the Circuit data for the ceremony setup
const circuit: CircuitDocument | CircuitInputData = {
name: circuitData.name,
description: circuitData.description,
prefix: circuitPrefix,
sequencePosition: i+1,
metadata: metadata,
files: files,
template: template,
compiler: compiler,
verification: verification,
fixedTimeWindow: fixedTimeWindow,
// dynamicThreshold: dynamicThreshold,
avgTimings: {
contributionComputation: 0,
fullContribution: 0,
verifyCloudFunction: 0
},


// the Circuit data for the ceremony setup
circuit = {
name: circuitData.name,
description: circuitData.description,
prefix: circuitPrefix,
sequencePosition: i+1,
metadata: metadata,
files: files,
template: template,
compiler: compiler,
verification: verification,
fixedTimeWindow: fixedTimeWindow,
avgTimings: {
contributionComputation: 0,
fullContribution: 0,
verifyCloudFunction: 0
},

}
}


circuits.push(circuit)

// remove the local r1cs download (if used for verifying the config only vs setup)
Expand Down
7 changes: 7 additions & 0 deletions packages/backend/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,13 @@
All notable changes to this project will be documented in this file.
See [Conventional Commits](https://conventionalcommits.org) for commit guidelines.


## [1.0.5](https://github.com/privacy-scaling-explorations/p0tion/compare/v1.0.4...v1.0.5) (2023-07-24)

**Note:** Version bump only for package @p0tion/backend



## [1.0.4](https://github.com/privacy-scaling-explorations/p0tion/compare/v1.0.2...v1.0.4) (2023-07-20)

**Note:** Version bump only for package @p0tion/backend
Expand Down
4 changes: 2 additions & 2 deletions packages/backend/package.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "@p0tion/backend",
"version": "1.0.4",
"version": "1.0.5",
"description": "MPC Phase 2 backend for Firebase services management",
"repository": "git@github.com:privacy-scaling-explorations/p0tion.git",
"homepage": "https://github.com/privacy-scaling-explorations/p0tion",
Expand Down Expand Up @@ -67,7 +67,7 @@
"@aws-sdk/client-ssm": "^3.357.0",
"@aws-sdk/middleware-endpoint": "^3.329.0",
"@aws-sdk/s3-request-presigner": "^3.329.0",
"@p0tion/actions": "^1.0.4",
"@p0tion/actions": "^1.0.5",
"blakejs": "^1.2.1",
"dotenv": "^16.0.3",
"ethers": "5.7.2",
Expand Down
155 changes: 98 additions & 57 deletions packages/backend/src/functions/circuit.ts
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ import { zKey } from "snarkjs"
import { CommandInvocationStatus, SSMClient } from "@aws-sdk/client-ssm"
import { FinalizeCircuitData, VerifyContributionData } from "../types/index"
import { LogLevel } from "../types/enums"
import { COMMON_ERRORS, logAndThrowError, printLog, SPECIFIC_ERRORS } from "../lib/errors"
import { COMMON_ERRORS, logAndThrowError, makeError, printLog, SPECIFIC_ERRORS } from "../lib/errors"
import {
createEC2Client,
createSSMClient,
Expand Down Expand Up @@ -236,9 +236,6 @@ const waitForVMCommandExecution = (
if (cmdStatus === CommandInvocationStatus.SUCCESS) {
printLog(`Command ${commandId} successfully completed`, LogLevel.DEBUG)

// Clear the interval.
clearInterval(interval)

// Resolve the promise.
resolve()
} else if (cmdStatus === CommandInvocationStatus.FAILED) {
Expand Down Expand Up @@ -268,6 +265,62 @@ const waitForVMCommandExecution = (
}, 60000) // 1 minute.
}

/**
* Wait until the artifacts have been downloaded.
* @param {any} resolve the promise.
* @param {any} reject the promise.
* @param {string} potTempFilePath the tmp path to the locally downloaded pot file.
* @param {string} firstZkeyTempFilePath the tmp path to the locally downloaded first zkey file.
* @param {string} lastZkeyTempFilePath the tmp path to the locally downloaded last zkey file.
*/
const waitForFileDownload = (
resolve: any,
reject: any,
potTempFilePath: string,
firstZkeyTempFilePath: string,
lastZkeyTempFilePath: string,
circuitId: string,
participantId: string
) => {
const maxWaitTime = 5 * 60 * 1000 // 5 minutes
// every second check if the file download was completed
const interval = setInterval(async () => {
printLog(`Verifying that the artifacts were downloaded for circuit ${circuitId} and participant ${participantId}`, LogLevel.DEBUG)
try {
// check if files have been downloaded
if (!fs.existsSync(potTempFilePath)) {
printLog(`Pot file not found at ${potTempFilePath}`, LogLevel.DEBUG)
}
if (!fs.existsSync(firstZkeyTempFilePath)) {
printLog(`First zkey file not found at ${firstZkeyTempFilePath}`, LogLevel.DEBUG)
}
if (!fs.existsSync(lastZkeyTempFilePath)) {
printLog(`Last zkey file not found at ${lastZkeyTempFilePath}`, LogLevel.DEBUG)
}

// if all files were downloaded
if (fs.existsSync(potTempFilePath) && fs.existsSync(firstZkeyTempFilePath) && fs.existsSync(lastZkeyTempFilePath)) {
printLog(`All required files are present on disk.`, LogLevel.INFO)
// resolve the promise
resolve()
}
} catch (error: any) {
// if we have an error then we print it as a warning and reject
printLog(`Error while downloading files: ${error}`, LogLevel.WARN)
reject()
} finally {
printLog(`Clearing the interval for file download. Circuit ${circuitId} and participant ${participantId}`, LogLevel.DEBUG)
clearInterval(interval)
}
}, 5000)

// we want to clean in 5 minutes in case
setTimeout(() => {
clearInterval(interval)
reject(new Error('Timeout exceeded while waiting for files to be downloaded.'))
}, maxWaitTime)
}

/**
* This method is used to coordinate the waiting queues of ceremony circuits.
* @dev this cloud function is triggered whenever an update of a document related to a participant of a ceremony occurs.
Expand Down Expand Up @@ -525,7 +578,7 @@ export const verifycontribution = functionsV2.https.onCall(
? `${contributorOrCoordinatorIdentifier}_${finalContributionIndex}_verification_transcript.log`
: `${lastZkeyIndex}_${contributorOrCoordinatorIdentifier}_verification_transcript.log`
}`
const firstZkeyFilename = `${prefix}_${genesisZkeyIndex}.zkey`

const lastZkeyFilename = `${prefix}_${isFinalizing ? finalContributionIndex : lastZkeyIndex}.zkey`

// Prepare state for VM verification (if needed).
Expand Down Expand Up @@ -555,7 +608,7 @@ export const verifycontribution = functionsV2.https.onCall(
if (isUsingVM) {
// Create temporary path.
verificationTranscriptTemporaryLocalPath = createTemporaryLocalPath(
verificationTranscriptCompleteFilename
`${circuitId}_${participantDoc.id}.log`
)

await sleep(1000) // wait 1s for file creation.
Expand Down Expand Up @@ -807,9 +860,9 @@ export const verifycontribution = functionsV2.https.onCall(
verificationTranscriptTemporaryLocalPath = createTemporaryLocalPath(
verificationTranscriptCompleteFilename
)
const potTempFilePath = createTemporaryLocalPath(files.potFilename)
const firstZkeyTempFilePath = createTemporaryLocalPath(firstZkeyFilename)
const lastZkeyTempFilePath = createTemporaryLocalPath(lastZkeyFilename)
const potTempFilePath = createTemporaryLocalPath(`${circuitId}_${participantDoc.id}.pot`)
const firstZkeyTempFilePath = createTemporaryLocalPath(`${circuitId}_${participantDoc.id}_genesis.zkey`)
const lastZkeyTempFilePath = createTemporaryLocalPath(`${circuitId}_${participantDoc.id}_last.zkey`)

// Create and populate transcript.
const transcriptLogger = createCustomLoggerForFile(verificationTranscriptTemporaryLocalPath)
Expand All @@ -826,57 +879,45 @@ export const verifycontribution = functionsV2.https.onCall(
await downloadArtifactFromS3Bucket(bucketName, firstZkeyStoragePath, firstZkeyTempFilePath)
await downloadArtifactFromS3Bucket(bucketName, lastZkeyStoragePath, lastZkeyTempFilePath)

await sleep(10000)
await sleep(6000)

// check if files have been downloaded
if (!fs.existsSync(potTempFilePath)) {
printLog(`Pot file not found at ${potTempFilePath}`, LogLevel.DEBUG)
// retry once
printLog(`Retrying to download pot file from ${potStoragePath} to ${potTempFilePath}`, LogLevel.DEBUG)
await downloadArtifactFromS3Bucket(bucketName, potStoragePath, potTempFilePath)
}
if (!fs.existsSync(firstZkeyTempFilePath)) {
printLog(`First zkey file not found at ${firstZkeyTempFilePath}`, LogLevel.DEBUG)
// retry once
printLog(`Retrying to download first zkey file from ${firstZkeyStoragePath} to ${firstZkeyTempFilePath}`, LogLevel.DEBUG)
await downloadArtifactFromS3Bucket(bucketName, firstZkeyStoragePath, firstZkeyTempFilePath)
}
if (!fs.existsSync(lastZkeyTempFilePath)) {
printLog(`Last zkey file not found at ${lastZkeyTempFilePath}`, LogLevel.DEBUG)
// retry once
printLog(`Retrying to download last zkey file from ${lastZkeyStoragePath} to ${lastZkeyTempFilePath}`, LogLevel.DEBUG)
await downloadArtifactFromS3Bucket(bucketName, lastZkeyStoragePath, lastZkeyTempFilePath)
}
// wait until the files are actually downloaded
return new Promise<void>((resolve, reject) =>
waitForFileDownload(resolve, reject, potTempFilePath, firstZkeyTempFilePath, lastZkeyTempFilePath, circuitId, participantDoc.id)
)
.then(async () => {
printLog(`Downloads from AWS S3 bucket completed - ceremony ${ceremonyId} circuit ${circuitId}`, LogLevel.DEBUG)

// Step (1.A.4).
isContributionValid = await zKey.verifyFromInit(
firstZkeyTempFilePath,
potTempFilePath,
lastZkeyTempFilePath,
transcriptLogger
)

// Compute contribution hash.
lastZkeyBlake2bHash = await blake512FromPath(lastZkeyTempFilePath)

// Free resources by unlinking temporary folders.
// Do not free-up verification transcript path here.
try {
fs.unlinkSync(potTempFilePath)
fs.unlinkSync(firstZkeyTempFilePath)
fs.unlinkSync(lastZkeyTempFilePath)
} catch (error: any) {
printLog(`Error while unlinking temporary files - Error ${error}`, LogLevel.WARN)
}

printLog(`Downloads from AWS S3 bucket completed - ceremony ${ceremonyId}`, LogLevel.DEBUG)
await completeVerification()
})
.catch((error: any) => {
// Throw the new error
const commonError = COMMON_ERRORS.CM_INVALID_REQUEST
const additionalDetails = error.toString()

// Step (1.A.4).
try {
isContributionValid = await zKey.verifyFromInit(
firstZkeyTempFilePath,
potTempFilePath,
lastZkeyTempFilePath,
transcriptLogger
)
} catch (error: any) {
printLog(`Error while verifying contribution - Error ${error}`, LogLevel.WARN)
isContributionValid = false
}

// Compute contribution hash.
lastZkeyBlake2bHash = await blake512FromPath(lastZkeyTempFilePath)

await completeVerification()

// Free resources by unlinking temporary folders.
// Do not free-up verification transcript path here.
try {
fs.unlinkSync(potTempFilePath)
fs.unlinkSync(firstZkeyTempFilePath)
fs.unlinkSync(lastZkeyTempFilePath)
} catch (error: any) {
printLog(`Error while unlinking temporary files - Error ${error}`, LogLevel.WARN)
}
logAndThrowError(makeError(commonError.code, commonError.message, additionalDetails))
})
}
}
}
Expand Down
1 change: 1 addition & 0 deletions packages/backend/src/functions/user.ts
Original file line number Diff line number Diff line change
Expand Up @@ -109,6 +109,7 @@ export const registerAuthUser = functions
* which uses the Firebase Authentication service.
*/
export const processSignUpWithCustomClaims = functions
.region("europe-west1")
.runWith({
memory: "512MB"
})
Expand Down
9 changes: 7 additions & 2 deletions packages/backend/src/lib/utils.ts
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ import admin from "firebase-admin"
import dotenv from "dotenv"
import { DeleteObjectCommand, GetObjectCommand, PutObjectCommand } from "@aws-sdk/client-s3"
import { getSignedUrl } from "@aws-sdk/s3-request-presigner"
import { createWriteStream } from "node:fs"
import { createWriteStream, fstat } from "node:fs"
import { pipeline } from "node:stream"
import { promisify } from "node:util"
import { readFileSync } from "fs"
Expand Down Expand Up @@ -213,8 +213,13 @@ export const downloadArtifactFromS3Bucket = async (bucketName: string, objectKey
if (response.status !== 200 || !response.ok) logAndThrowError(SPECIFIC_ERRORS.SE_STORAGE_DOWNLOAD_FAILED)

// Write the file locally using streams.
const writeStream = createWriteStream(localFilePath)
const streamPipeline = promisify(pipeline)
await streamPipeline(response.body, createWriteStream(localFilePath))
await streamPipeline(response.body, writeStream)

writeStream.on('finish', () => {
writeStream.end()
})
}

/**
Expand Down
Loading

0 comments on commit e522cb4

Please sign in to comment.