Performance measurement criteria

[mihakralj]

I started to work on performance tests and wanted to make sure we have the right criteria in place. Here are proposed test conditions:

• 10,000 bars in the initial dataset
• calculated EMA(200) on OHLC4 of the dataset
• incrementally add 1,000 bars (one at the time)
• calculate EMA(200) each time the bar is added

#!time
RandomGbm data = new(10000);
for (int i=0; i<1000; i++) {
    data.Add(DateTime.Now);
    data.GetBaseQuote(CandlePart.OHLC4).GetEma(200);
}
#### Wall time: 6026.18ms ####

The same scenario with another library:

#!time
GBM_Feed data = new(10000);
EMA_Series ema = new(data.OHLC4, 200);
for (int i=0; i<1000; i++) {
    data.Add(DateTime.Now);
}
#### Wall time: 34.60ms ####

[mihakralj]

Just to be clear where the performance hit is, I executed the loop without calculating EMA(200):

#!time
RandomGbm data = new(10000);
for (int i=0; i<1000; i++) {
    data.Add(DateTime.Now);
    //data.GetBaseQuote(CandlePart.OHLC4).GetEma(200);
}
#### Wall time: 85.17ms ####

So, generation of dataset consumes only 85ms and the remaining 5940ms is burned on EMA(200) calculation.

I did the same with QuanTAlib and there is no perceived difference between running empty loop and loop with EMA(200):

#!time
GBM_Feed data = new(10000);
//EMA_Series ema = new(data.OHLC4, 200);
for (int i=0; i<1000; i++) {
    data.Add(DateTime.Now);
}
#### Wall time: 33.10ms ####

[DaveSkender]

If you're just trying to come up with a way to help optimize your own code, doing these at the unit level to establish a baseline has been quite helpful for me. It allows me to quickly make tweaks to see if it helps or hurts performance. It's helped find certain syntax that sucks the life out of performance (notably LINQ queries are quite bad, overall).

[mihakralj]

1. I capture 1 second market signals on Oanda (or Binance) and typically look at 4+ hours of quotes That is 14,400 quotes. Lots of my indicators that look for long-term trend use period of 500+, so I can't have a dataset of only 500 quotes!

2. your helper class RandomGbm has .Add method - because I wrote it! :-) I am re-using RandomGbm (although I had to expose it as you declared it internal)

3. I did change the composite syntax from data.GetBaseQuote(CandlePart.OHLC4).GetEma(200) to data.Use(CandlePart.OHLC4).GetEma(200) - but I didn't see much difference...

I'd like to define a common scenario to do real use-case performance tests across different libraries. taking 500 bars and run a single pass over it is not realistic.

• 10,000 bars is too much, 500 is too little. How about 2,000?
• tiny periods for indicators are not realistic either. if EMA(200) is too large and EMA(10) is too small, how about EMA(100)?
• Do you agree that we should run multiple passess on additive quotes (where data is added to the series one item at the time)? Should we do that 60 times (one minute) or more?

[DaveSkender]

Lol. 😆 I think we have different definitions for "common scenarios". All I can say is it's quite easy to tune these criteria to make one library or use case look better than another. Everything you're suggesting is geared to benefit how you've optimized your personal library for your unique use case, so they would naturally make yours look better, but I suspect that's what you're really trying to do?

[DaveSkender]

Just ran my work-in-progress streaming use case with the above parameters for an apples-to-apples comparison. It's the second result below. The first one is raw quotes, using Close price; the third one is the same OHLC4 case, but starting with an empty set instead of 10k quotes. Not sure why the third one is 2× faster yet; I'm probably doing an unnecessary conversion somewhere.

|                     Method |     Mean |     Error |    StdDev |
|--------------------------- |---------:|----------:|----------:|
|       GetEmaStream11kQuote | 2.196 ms | 0.0280 ms | 0.0262 ms |
|   GetEmaStreamOHLC4base10k | 2.457 ms | 0.0481 ms | 0.0735 ms |
| GetEmaStreamOHLC4baseEmpty | 1.024 ms | 0.0048 ms | 0.0042 ms |

[DaveSkender]

I haven't added an event handler yet either, so that'll slow things down a bit. We'll see. I have a bunch of things to explore before I settle on a final approach to the streaming use case.

[mihakralj]

What I want to establish is a realistic performance testing scenario. What I saw so far (Tulip benchmark) and your performance benchmark are not relevant for any trading analysis scenario. What is a relevant benchmark setup that we could use?

[DaveSkender]

There'd need to be a whole range of use cases, not just your perspective or personal needs. I think "relevant" can mean a lot of things to a lot of people. Your use cases may be quite good for what you're trying to do, but quite bad for how other people want a library to work.

[DaveSkender]

I think we can come up with a list of use cases that cover all common scenarios. For example:

• static analysis of quotes
• small vs large quote histories
• streaming use cases
• composite / chaining
• etc

These are probably more like dimensions where there can be all sorts of permutations to test.

[mihakralj]

Can you verify that these two tests are identical? 1,000 bars, EMA(20), 10,000 iterations:

//Skender Stock Indicators test
RandomGbm data = new(1000);
for (int i=0; i<10000; i++) {
    data.GetEma(20);
}

and this one:

//QuanTAlib test
GBM_Feed data = new(1000);
for (int i=0; i<10000; i++) {
    EMA_Series ema = new(data.Close, 20);
}

What results are you getting for each?

[mihakralj]

what period do you use for EMA? And on what datapoint of a quote do you run a test (O/H/L/C or OHL3 or OHLC4?)
Also, if you are unwilling to re-run the test 1,000 times to get a better fidelity of results, are you willing to increase the length of the set? I would like to create test conditions that return perf in (multiples of) seconds, not miliseconds...

[mihakralj]

actually, can you post the static run code that you use?

[DaveSkender]

I'm using Benchmarks.NET, a tool that's purpose-built for comprehensive C# performance testing. It runs a statistically relevant number of iterations automatically on the unit test, so I'm already doing what you're trying to emulate. You can find it in the tests/performance folder. I also have a scant blurb on performance testing in the contributing guidelines. The default.csv quotes data I use is in the tests/../data folder, but there are a bunch of others there too.

[DaveSkender]

It's quite good, I like it.

// * Summary *

BenchmarkDotNet=v0.13.2, OS=Windows 10 (10.0.19045.2364)
Intel Core i7-7700HQ CPU 2.80GHz (Kaby Lake), 1 CPU, 8 logical and 4 physical cores
.NET SDK=7.0.101
  [Host]     : .NET 7.0.1 (7.0.122.56804), X64 RyuJIT AVX2
  DefaultJob : .NET 7.0.1 (7.0.122.56804), X64 RyuJIT AVX2

| Method |     Mean |    Error |   StdDev |
|------- |---------:|---------:|---------:|
| GetEma | 51.34 us | 1.042 us | 0.924 us |

// * Hints *
Outliers
  IndicatorPerformance.GetEma: Default -> 1 outlier  was  removed (59.76 us)

// * Legends *
  Mean   : Arithmetic mean of all measurements
  Error  : Half of 99.9% confidence interval
  StdDev : Standard deviation of all measurements
  1 us   : 1 Microsecond (0.000001 sec)

[DaveSkender]

Compared to those Tulip benchmarks (and TA-LIB), they're indicating about 340s to process one million quotes for EMA. Here's what I'm getting with this library, about 1400-2000× faster 🚀, but those Tulip numbers are probably done on an older processor with old code.

|         Method |     Mean |   Error |   StdDev |   Median |
|--------------- |---------:|--------:|---------:|---------:|
|       GetEma1M | 240.2 ms | 4.80 ms | 10.33 ms | 236.5 ms |
| GetEma1MStream | 175.0 ms | 2.93 ms |  2.60 ms | 175.2 ms |

[mihakralj]

This can't be right, can it?

BenchmarkDotNet=v0.13.3, OS=Windows 11 (10.0.25267.1000)
Intel Core i9-9900K CPU 3.60GHz (Coffee Lake), 1 CPU, 16 logical and 8 physical cores
.NET SDK=7.0.101
  [Host]     : .NET 7.0.1 (7.0.122.56804), X64 RyuJIT AVX2
  DefaultJob : .NET 7.0.1 (7.0.122.56804), X64 RyuJIT AVX2


|        Method |     Mean |    Error |   StdDev |
|-------------- |---------:|---------:|---------:|
| QuanTAlib_EMA | 10.83 us | 0.165 us | 0.138 us |
|   Skender_EMA | 32.64 us | 0.497 us | 0.465 us |

[DaveSkender]

Could be right, 32us sounds about right for i9 vs my i7 processor. I expect my incremental model to be ~10us on my processor. Is that using the 500 quotes?

[DaveSkender]

You were counting the quote generator before, which is probably the slow part.

[DaveSkender]

Considering that just fetching quotes from a fast REST API over HTTP is probably 2-4ms and sometime > 100ms, that’s quite good performance.

[mihakralj]

I really like this benchmark framework. Something to keep me occupied over the weekend!
I want to create a battery of tests for all four .NET TA frameworks across 500, 5,000 and 10,000 bars, for EMA(10), EMA(100) and EMA(200).

And then do the same for all key indicators...

[DaveSkender]

Nice. And yes, it’s a great tool that I like to run when trying different things to see what works better.

[mihakralj]

What is the cleanest/fastest method to CHANGE the last item in IEnumerable<Quote>? I can add to the list with .Append, but I also need a method/approach to update the last value of the still-forming quote...
Here is why (animated gif):

[DaveSkender]

It sounds like something that can be instrumented on a performance test! 😉

Generally speaking, to answer the first part more directly, internal to the library, I usually stick to List and Add over IEnum and Append. If you require an external interface, you might try ICollection and Add, it's probably faster than IEnumerable.

I'm actually taking another look at this right now, specifically for the streaming case. There's more than performance considerations; there's a tradeoff or aggregation consideration with managing out-of-date-sequence quotes coming in from a feed. For example, if you're feed is sub-minute bars, but you're collecting 5-minute bars, you'd have to do live aggregation; or if one shows up for an older bar, how do you go back to correct it (and presumably all dependent indicators)?

[DaveSkender]

Converting an IEnumerable to List, then doing operations on it is at least 2-3 times faster than trying to do most operations directly on IEnumerable. It's not intuitive but is an example of how performance benchmarking can help see these things. Run the test, it's surprising if you've not seen it before.

[mihakralj]

I will tidy up the tests, but here is what I found so far:

• providing a static input (list, iEnumerable or array), the ranking of EMA() indicators is clear: TA-LIB - Tulip - Skender - QuanTAlib
• There are very small perf differences between short/long datastets and if Period is short or long
• Most of indicators are very poorly prepared to ADD a quote (or update the last quote) - either using Array.Resize() or converting iEnumeralbe to List and then back to iEnumreable - the time required to add a quote is way larger than the time required to calculate EMA() on the new dataset.

When QuanTAlib is tested in completely static conditions with immutable dataset, it ranks dead last. But the moment there is a more realistic testing condition with just a couple of updates of the dataset and few additions of new bars to the end of the dataset, there is no comparison - QuanTAlib shoots to the front, followed by Skender Indicators.

[DaveSkender]

Yah, there's a real advantage to using some of these newer .NET frameworks, just out of the box they're 2-3x faster than older ones. I still have quite a bit of tuning and playing around to do on my incremental models, but expect to be comparable to what you're getting. We're basically on the same path, doing the same things.

I'd not originally designed for the incremental "add" use case, but always had it on my roadmap, so there are zero surprises that it's not top of the list in the streaming category. Still, it works fine for most people trying to use a new quote series every 15s or per minute (it's only a problem if trying to use tick frequency updates). I'm making progress on the streaming use case for people who want to venture into that world.

Even with the importance of performance, I'm also trying to balance other design tradeoffs. Lots of things to consider, just picking at them one at a time.