Random distribution of the first example

[qzcx]

Hey cool project. I'd love to make sure of it since I've got an environment which uses python to generate C test bench parameters.

I'm trying it out, but the first example isn't providing very good randomization.

The b variable is almost always one extreme or the other. About 1 in 10 times it will venture out away from the bounds. Likewise d seems to mostly pick 255 which is it's upper bound.

How should I fix this code to get better distribution? I'd expect a much better distribution from SV.

I tacked on this following print functions:

    def __str__(self):
        return f"a {self.a} b {self.b} c {self.c} d {self.d}"

Ran it for 30 iterations:

s = my_s()

for i in range(0,30):
    vsc.randomize(s)
    print(s)

This results in:

a 0 b 0 c 0 d 0
a 2 b 44 c 43 d 44
a 4 b 255 c 175 d 255
a 2 b 255 c 104 d 255
a 1 b 255 c 204 d 255
a 4 b 9 c 8 d 9
a 1 b 224 c 202 d 224
a 8 b 49 c 48 d 49
a 4 b 105 c 57 d 134
a 1 b 46 c 45 d 46
a 1 b 73 c 73 d 255
a 8 b 88 c 88 d 201
a 1 b 33 c 33 d 91
a 2 b 255 c 196 d 255
a 1 b 209 c 209 d 255
a 2 b 255 c 162 d 255
a 1 b 59 c 45 d 162
a 8 b 171 c 157 d 214
a 8 b 255 c 196 d 255
a 2 b 39 c 39 d 99
a 4 b 205 c 27 d 255
a 8 b 3 c 3 d 255
a 8 b 156 c 156 d 255
a 2 b 51 c 51 d 255
a 1 b 255 c 214 d 255
a 8 b 243 c 226 d 243
a 4 b 43 c 42 d 43
a 4 b 125 c 31 d 125
a 1 b 199 c 24 d 255
a 2 b 255 c 167 d 255
a 1 b 255 c 187 d 255

[mballance]

Hi @qzcx,
You're right that I would expect a bit better distribution in this example. When looking at distribution, it's important to collect data across a fairly large number of samples (10x domain is a good rule of thumb). I had a quick look using 1000 and 10000 samples, and can definitely see a bias on 'c' and 'c' toward the largest possible value.

One thing you can do is to add dist constraints, just like you would in SystemVerilog. That said, there does appear to be an issue worth investigating here, and I'll plan to dig into it over the next week.

Here's my best 'how the sausage is made' explanation of what's currently happening in case you're interested:

• The solver that PyVSC uses is an SMT solver, which provides a solution that satisfies a constraint system, but does not provide random results
• PyVSC adds additional constraints to bias the solver and produce a (hopefully) good random distribution
• Currently, PyVSC selects the target random values based on non-rand constraints. So, for 'a', it picks randomly picks one of [1 2 4 8] because of the 'in' constraint
• This can be a problem in a case like this where, depending on the order in which 'c' and 'd' are randomized and the values selected, only one of the randomly-selected values may applied (eg if we pick 20 for 'c' and 10 for 'd'). In this case, the SMT solver gets to pick the value for the field PyVSC wasn't able to, and it's answers are definitely non-random.

Again, thanks for pointing out this case. Random distribution is an area in which PyVSC continues to improve. For now, feel free to play around with dist weights, and I'll get back to you in a few days -- hopefully with news of an improvement.

Best Regards,
Matthew

[qzcx]

Thanks Matthew.

I'll keep playing with it from my end and see what I can do to get better results and I'll use larger samples in the future.

Thanks for the help. I think this could be very helpful with my upcoming work.

qzcx

[mballance]

Hi @qzcx,
I wanted to provide an update on this. I've been able to implement the approach which I mentioned earlier, which involves a randomized stepwise approach to selecting random values (ie we select a random subset of the available variables and process them in a random order) that adapts to previous selections. The results on the testcase above appear quite good -- not completely flat distribution, but without terribly obvious hot spots in the distribution. I'm seeing about a 3% hit on performance across the current test suite, which isn't too bad for a significant increase in distribution uniformity.

I have a couple detail bugs to tie up, and I'll get this into a release.

Best Regards,
Matthew

[qzcx]

Hey Matthew,

I came back to try this and I'm not seeing better results. I updated my version to pyvsc-0.5.8.20210912.1. Should I see the improved results on this testcase in this version? Are there any changes to the testcase needed?

Thanks,
qz

[mballance]

Hey Jon,
Apologies for the delay. At luck (?) would have it, I received a series of reports of non-uniform distribution around the time of the original report from you. Some responded well to how I planned to address this issue, and some not so much. I now believe I have a different approach that addresses all the issues. I just released that as 0.5.9 this morning. Please have a look and let me know of any feedback.

Best Regards,
Matthew

[qzcx]

Great thanks! I'll give it a try and report back.

[qzcx]

@mballance Wow! Way better distribution on my examples. Thanks for the improvement, Going to have to look closer at using this now for my project.