can not query any pprof with profile_id while tracing contains profile_id

[zhzure]

i can record tracing in every root span like:

but i can not find anything when query with those profile_id like:

i am so confused, is there anyone meet the same?

[kolesnikovae]

Hello @zhzure, thank you for reaching out!

There are two issues that could potentially cause the described experience:
– Go CPU profiler probes stack traces 100 times per second, meaning that spans shorter than 10ms may be missing.
– We've added support for sub-second precision in queries only recently but haven't released otel-profiling-go yet. You can evaluate the most recent version by adding the replacement directive to go.mod file followed by go mod tidy command:

replace github.com/pyroscope-io/otelpyroscope => github.com/pyroscope-io/otel-profiling-go v0.4.0

[zhzure]

thanks very much, i will try it asap.

[zhzure]

i check my code and turns out that i am using github.com/pyroscope-io/otel-profiling-go v0.4.0 now
dose that means my go-agent in code and pyroscope service worked just fine if i can see profile_id in trace record?

i choose a request longer than 10ms,but i still got nothing

i wonder if its my code which is wrong, i follow this guid ,but the only diff is my exporter using otel:

exporter, err := otlptrace.New(
		context.Background(),
		otlptracegrpc.NewClient(
			secureOption,
			otlptracegrpc.WithEndpoint(config.Config.Trace.CollectorURL),
		),
	)

tp := sdktrace.NewTracerProvider(
		sdktrace.WithSampler(sdktrace.AlwaysSample()),
		// sdktrace.WithSampler(sdktrace.TraceIDRatioBased(1.0)),
		sdktrace.WithBatcher(exporter),
		sdktrace.WithResource(resources),
	)
	// We wrap the tracer provider to also annotate goroutines with Span ID so
	// that pprof would add corresponding labels to profiling samples.
	otel.SetTracerProvider(otelpyroscope.NewTracerProvider(tp,
		otelpyroscope.WithAppName("corplink-admin"),
		otelpyroscope.WithPyroscopeURL("http://127.0.0.1:4040"),
		otelpyroscope.WithRootSpanOnly(true),
		otelpyroscope.WithAddSpanName(true),
		otelpyroscope.WithProfileURL(true),
		otelpyroscope.WithProfileBaselineURL(true),
	))

[zhzure]

i noticed that if i choose tag span_name , this is a lot of request path i'm pretty sure i have request plenty of time, like api/v2/user/list,but this is nothing. i have already set both profile sample rate 100 and trace sample rate 100

i still can figure it out why it is

[kolesnikovae]

That's interesting. I'd like to learn a bit more about the deployment:

• pyroscope server version
• go client version

dose that means my go-agent in code and pyroscope service worked just fine if i can see profile_id in trace record?

Unfortunately, in the current implementation presence of pyroscope.profile.id does not mean that a profile has been captured for the span but only establishes a link between profiles and traces. We added these span attributes more for convenience reasons, so that users would be able to navigate to the Pyroscope UI by link.

i choose a request longer than 10ms,but i still got nothing

Oh, I have to clarify that 10s of CPU time (active execution) not wall time (that includes e.g. IO latencies).

i noticed that if i choose tag span_name , this is a lot of request path i'm pretty sure i have request plenty of time, like api/v2/user/list,but this is nothing. i have already set both profile sample rate 100 and trace sample rate 100

According to the table, pyroscope server has received samples only for the 3rd and the 6st series within the queried time range. Also, presence of span_name labels says that there was at least one sample (a stack with a value) in the server DB. May I ask you to try to extend the query time range in the Tag Explorer page? I'm wondering how many profiles/samples were captured.

[zhzure]

thanks very much, really appreciate

• pyroscope server version
  v0.25.1
  using wget https://dl.pyroscope.io/release/pyroscope_0.25.1_amd64.deb
• go client version
  github.com/pyroscope-io/otel-profiling-go v0.4.0
• last 1 year in Tag Explorer page
  
  there is seems only one Tage NameGorm.Query differ from others, it's a polling request, directly query mysql,
  and i will try to figure out if our project's most requests take less than 10ms in CPU time so pyroscope can't collect profile data asap
  

[kolesnikovae]

Thank you very much, this is super helpful! Out of curiosity, could you try to find a profile for one of the gorm.Query spans please?

[zhzure]

yes of cause, but i just got off work, couldn't access company‘s Intranet from home. i will post it here as soon as i go to work tomorrow

[zhzure]

i filter all slow query where latency greater than 10ms, but i still tried dozens of span id till finally find one in pyroscope server

turns out that pyroscope server do got profile data, but not all of them,
does that mean i have deployed pyroscope and go agent properly only because most of my request latency shorter than 10ms CPU time so they can not be collected?

[kolesnikovae]

Thank you @zhzure! Yes, I think both pyroscope server and agent are configured properly.

Go profiler just can't capture samples for so short spans – theoretically, we could increase sample rate but this is not a trivial tweak as it is hardcoded and, generally speaking, not recommended. We're continue working on improving the integration: first of all, I guess it would make more sense to not add attributes to a span if no samples were captured during the execution.

Along with this, pyroscope server should show you meaningful profiles by queries (including ones with span_id)