Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Geth new tracing system/hooks ported to Erigon #10757

Open
wants to merge 22 commits into
base: main
Choose a base branch
from

Conversation

maoueh
Copy link
Contributor

@maoueh maoueh commented Jun 14, 2024

In Geth 1.14.0 release, the overall tracing plugin system has been refactored and augmented quite significantly, here the relevant part from Geth release notes:

Geth v1.14.0 introduces a brand new live-tracing feature, where one or more transaction tracers might be injected into the block processing pipeline, ensuring that tracing and execution happen in lockstep (ethereum/go-ethereum#29189). Since Go does not have a cross platform, OS native plugin infrastructure, adding live tracers needs to be done at the Geth source code level, and Geth itself subsequently rebuilt. That said, the advantage is that such tracers have full execution flexibility to do whatever they like and however they like. Please see the live-tracer changelog and docs for details.

  • Live tracing runs in lockstep with block execution, one waiting for the other. You should never run a live tracer on a validating node, or any other where latency is important. The recommended practice is to have tracers collect and export the bare minimum data needed and do any post-processing in external systems where latency is not relevant.
  • The live tracer work required a number of breaking internal API changes. If you had your own native tracers implemented before this change, the changelog contains the necessary steps needed to update your old code for the new APIs.

This work has been done as a collaboration between the Geth team and StreamingFast team, which had similar tracer in production system. StreamingFast is a core-dev team of The Graph and received various asks from the indexer community to have this new tracing capabilities within Erigon too.

While working on getting the Geth PR merged upstream, we also started the work to port the same set of changes to Erigon. Today, we are ready to open the PR and get the initial feedback. Of course this changes how the overall tracing plugin looks like.

Here a excerpt of the PR description that highlights some of the changes:

It (the PR) refactors the whole logger infrastructure. Previously internally we were passing around the EVMLogger interface and tracers had to implement that interface. This PR changes it so that we pass pointer to a struct around internally. Tracer constructors will have to return an instance of the tracing.Hooks struct. This necessitated some changes to the API.

Outside of the overall architecture change, here other stuff this PR brings in:

  • Merged the two possible entry points for call tracing into a pair of tracing method OnEnter/OnExit
  • StateDB is now fully traced for nonce changes, balance changes, code changes, etc.
  • Gas change are recorded enabling fine-grained gas tracking
  • Reasons for gas change & balance change.

This PR is quite big since it touches a lof of places where transaction execution happens. I understand it could be daunting to review and we are open to see how we could make the review easier, splitting in smaller PR is one of the possibilities.

Happy to schedule call with any stakeholder on the Erigon side to see how we can move that PR forward if needed.

@@ -17,7 +17,7 @@ import (
"github.com/ledgerwatch/erigon/core/state"
"github.com/ledgerwatch/erigon/core/types/accounts"
"github.com/ledgerwatch/erigon/eth/stagedsync/stages"
"github.com/ledgerwatch/erigon/eth/tracers"
tracerConfig "github.com/ledgerwatch/erigon/eth/tracers/config"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

create separated 1 separated PR for such changes. easy to merge. it will help to reduce size of this PR. tnx.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

}

callTracer := calltracer.NewCallTracer()
vmConfig.Debug = true
vmConfig.Tracer = callTracer
if vmConfig.Tracer == nil {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

plz add it into into cmd/state/exec3/state.go:RunTxTaskNoLock

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maoueh added a commit to graphprotocol/erigon that referenced this pull request Jun 17, 2024
…ort cycle

This is extracted from erigontech#10757 to make the overall merge more manageable.
maoueh added a commit to graphprotocol/erigon that referenced this pull request Jun 17, 2024
…rding

This is a split PR off erigontech#10757. In this smaller PR, we introduced the new `core/tracing` package and propagates the `BalanceChange` and `GasChange` reason in the API without being actually used.

This is 100% backward compatible.
maoueh added a commit to graphprotocol/erigon that referenced this pull request Jun 17, 2024
…rding

This is a split PR off erigontech#10757. In this smaller PR, we introduced the new `core/tracing` package and propagates the `BalanceChange` and `GasChange` reason in the API without being actually used.

This is 100% backward compatible.
maoueh added a commit to graphprotocol/erigon that referenced this pull request Jun 17, 2024
…r interface

The 2 new methods are made available in `consensus.ChainHeaderReader` interface to future usage in tracing revamp.

This is a split PR off of erigontech#10757.
@maoueh
Copy link
Contributor Author

maoueh commented Jun 17, 2024

@AskAlexSharov I've spin-off 3 smaller PRs out of this one (listed below). Once they are merged, I'll update this overarching PR and check what could be split further for easier review/merge management.

  1. Moved tracers.TracerConfig to its own package to avoid (future) import cycle #10782
  2. Introduced core/tracing/hooks.go new API + balance/gas change forwarding #10783
  3. Added CurrentFinalizedHeader and CurrentSafeHeader in chain reader interface #10784

AskAlexSharov pushed a commit that referenced this pull request Jun 18, 2024
…r interface (#10784)

The 2 new methods are made available in `consensus.ChainHeaderReader`
interface to future usage in tracing revamp.

This is a split PR off of
#10757.
AskAlexSharov pushed a commit that referenced this pull request Jun 18, 2024
…ort cycle (#10782)

This is extracted from #10757
to make the overall merge more manageable.
…n-main

# Conflicts:
#	consensus/consensus.go
#	core/state/state_test.go
#	core/vm/interpreter.go
#	polygon/tracer/trace_bor_state_sync_txn.go
#	turbo/jsonrpc/debug_api.go
#	turbo/jsonrpc/debug_api_test.go
#	turbo/jsonrpc/gen_traces_test.go
#	turbo/jsonrpc/tracing.go
#	turbo/transactions/tracing.go
AskAlexSharov pushed a commit that referenced this pull request Jun 22, 2024
…rding (#10783)

This is a split PR off #10757.
In this smaller PR, we introduced the new `core/tracing` package and
propagates the `BalanceChange` and `GasChange` reason in the API without
being actually used.

This is 100% backward compatible.
AskAlexSharov pushed a commit that referenced this pull request Jul 3, 2024
- This PR is created to from the
#10757 to simplify it's review
and merge process.
- In this PR, we have plugged tracing hook framework to the consensus
engine.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants