Generalise table merge with a monoid on values

[tranma]

This is a simple change to allow substituting the standard value union with another monoid, such that merge can be used to calculate the number of bytes used by each key. With this information we can filter out any abnormally large entities. Example:

→ ./dist/build/Zebra/zebra import input1.ztxt -s input.zschema -o input1.zbin && ./dist/build/Zebra/zebra merge input1.zbin --measure -o output1.zbin && ./dist/build/Zebra/zebra export output1.zbin                                                                                                                                                             
{"key":{"entity_hash":30,"entity_id":"lisa"},"value":19}
{"key":{"entity_hash":40,"entity_id":"homer"},"value":16}
{"key":{"entity_hash":50,"entity_id":"bart"},"value":111}
{"key":{"entity_hash":50,"entity_id":"millhouse"},"value":24}

→ cat input1.ztxt                                                                                                                                                                                                                                                                                                                                               
{ "key": { "entity_hash": 50, "entity_id": "millhouse" }, "value": { "cash": 19, "item": { "none": {} } } }
{ "key": { "entity_hash": 40, "entity_id": "homer" }, "value": { "cash": 5, "item": { "some": "" } } }
{ "key": { "entity_hash": 30, "entity_id": "lisa" }, "value": { "cash": 5, "item": { "some": "sax" } } }
{ "key": { "entity_hash": 50, "entity_id": "bart" }, "value": { "cash": 27, "item": { "some": "averylongstringaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" } } }

! @nhibberd @charleso

[nhibberd]

Will take a look when I get back

[tranma]

I've added an extraction step, so you can get a zebra file of the large entities only. What to do from here is up to you, I think the idea we had was to run this merge measure-greater-than-megabytes step before the actual merge, acquire the blacklist and tell merge to ignore these keys. I haven't put in anything to do that latter step, but it should be straightforward to do so.

→ ./dist/build/Zebra/zebra import input1.ztxt -s input.zschema -o input1.zbin && ./dist/build/Zebra/zebra merge input1.zbin --measure-greater-than-megabytes 0 -o output1.zbin && ./dist/build/Zebra/zebra export output1.zbin                                                                                                                                    [292bd7e]
{"key":{"entity_hash":30,"entity_id":"lisa"},"value":19}
{"key":{"entity_hash":40,"entity_id":"homer"},"value":16}
{"key":{"entity_hash":50,"entity_id":"bart"},"value":111}
{"key":{"entity_hash":50,"entity_id":"millhouse"},"value":24}

→ ./dist/build/Zebra/zebra import input1.ztxt -s input.zschema -o input1.zbin && ./dist/build/Zebra/zebra merge input1.zbin --measure-greater-than-megabytes 1 -o output1.zbin && ./dist/build/Zebra/zebra export output1.zbin                                                                                                                                    [292bd7e]

[erikd-ambiata]

I'm not familiar with the zebra code base, but this looks more than reasonable.