v7.7.0 - a Dashboard style home page

7.7.0 - 2023-08-31

We finally have a brand new homepage! Quepid started as a tool for relevancy engineers only, but today we have human judges using Quepid to rate documents, Search Managers who are keeping an eye on the offline metrics, and of course in more complex setups, networking types who configure the connection between Quepid and the Search Engine. In the past, regardless of who you were, we dropped you right into a Case. If you had never created one before, well, you got shoved into the Case creation wizard, whether you wanted it or not. It made for a unpleasant first experience for anyone other than the hard core relevancy engineer.

The new dashboard is an attempt to change that UX. It's just a first cut, and honestly, it probably doesn't quite make anyone perfectly happy. I'm hoping that we can get a lot of good feedback and learn from it. I could imagine in the future that based on what you do, we surface information you care about. For example, a network engineer would want to know that the Quepid can talk to each Search Engine. A Search Manager would be more interested in insights and summary progress information. Human judges want to know "what is my next task to do". So please share feedback!

Features

• New home page (and sidebar!) in #681 is added via #684 by @epugh.

Improvements

• The feedback message after creating an empty book is now smart about if there are related cases or not, and gives better next steps instructions. #796 by @OkkeKlein fixed by #797 by @epugh. Thanks Okke!

• The Cases listing page shows your cases from oldest to newest. @atarora opened #708 for sortable columns. Implementing this didn't go well, so as an alternative, changed the sorting to be from newest to oldest, taking into account the case update time as well as the last time you viewed a specific case. #795 by @epugh. We may want to redo the entire Cases listing page at some point.... Hopefully this is better @atarora ;-).

• Reworked the messageing (and error handling) when you attempt to open a case that hasn't been shared with you (or doesn't exist!). We had been showing the "You need to reload Quepid in the HTTPS" type message, when actually it was a "Make sure this case has been shared with you!" situation. Also reduced the amount of error messages in the browser and the server in this situation by being smarter. Thanks @cmorley for surfacing this. #792 by @epugh fixed by #793.

• Thumb images can use a prefix_url, just like regular images. #790 by @OkkeKlein is fixed in #805 by @epugh.

• Using Nginx and have huge documents being snapshotted etc. New configuration suggested by @OkkeKlein allows Nginx to be happy. #804 by @epugh.

Bugs

• Drop a link in Slack to team 3, and you see details about case 3;-). #733 by @epugh fixed by #794 by @epugh to only do unfurling for cases.

• Ace code editor insists on loading two javascript files from specific paths in Quepid. Despite best effort, couldn't change this. So #793 just mounts those javascript where Ace wants it to shut up some very loud error messages in the browser console. If you can't beat'em, join'em.

• When judging ratings using the Book of Judgements, you could sometimes get a situation where you create a judgement, and then tried to create a new one, hitting a constraint. #809 by @epugh makes this more robust.

• Guess what? 7 GB out of our 9 GB production database is scores for cases being inserted OVER AND OVER again! The AngularJS app uses event emitting to signal when scoring is completed, this causes extra events to be triggered, causing extra PUT of the case scores. This shouldn't matter as we have logic on the server side. However, due to a bug, that logic doesn't actually work. The unit tests for the front end and the back end each independently validate the logic, it's just when you put them together it doesn't work. This didn't matter for years, but the new Dashboard surfaces scores ;-). Fixed in #807 by @epugh.

7.6.1: Round Trip Human Ratings!

Lots has happened since the last release notes, so doing one big combined one below!

7.6.1 - 2023-07-06

• Chased down bug with showing you the previous judgement on the Human Judgement page. #779 by @epugh.

7.6.0 - 2023-07-05

Big overhaul on the human rating side of things. We now roundtrip ratings that you have made from the core Case screen into your Book of Judgements when you populate it. (We are also now tracking who the last person was to rate a doc in the Case screen). You can now also merge ratings from multiple Book's into a new Book, which lets you reuse your judgements in new combinations.

Additionally, to encourage your human judges, a progress screen has been added every 20 judgements showing progress and a leaderboard.

• #778 by @epugh introduces some "fun" into the rating process ;-).

• #766 introduced the synthesizing of a Book, and fixes #765, track who rates in the main quepid ui, #763, link from list of books in judgements to the book itself, and #761, Combining books loses judgments but still produces scores.

• #760 changes Quepid to use floats for ratings and judgements, which opens the door to implicit judgements.

7.5.0 - 2023-06-15

Need to interact with Quepid API's from outside of Quepid? We now support Personal Access Tokens! From your Profile page you can generate your own token and then use that to authenticate against Quepid. For example, you can programatically load a judgement directly into Quepid:

curl -X POST http://localhost:3000/api/books/2/judgements/ -H 'Authorization: Bearer 4a82040bf1b2d255c63833cb59fa9275' -H 'Content-Type: application/json' -d '{
  "judgement": {
    "query_doc_pair_id": 201
    "rating": 1
  }
}'

#759 and #756 by @epugh.

In support of this, we added new API's for judgements and ratings in #757 by @epugh.

Bugs

• Exporting Ratings in CSV format was returning the same rating for every single rater, regardless of what they did! Fixed in by @epugh with special help from @grahamwren. I randomly sat next to him on plane flight home and we paired on it ;-).

• Looking up the wrong info need for a Query Doc Pair! Fixed in #755 by @epugh.

• Missed a relative path for the tries history visualization. Fixed in #754.

7.4.1 - 2023-06-09

Lots of small things from working with Quepid using templates with OpenSearch. Also, if you haven't used https://github.com/o19s/agent_q, it's been updated to work with Quepid 7.x line. I use it to automate pulling my relevance metrics daily from Quepid cases ;-).

Features

• Rework how we handle OS/ES templates in Quepid so that you don't need to append /template to the url, instead, use the existence of the {"id":"my-template"} in the query parameters to decide what URL to use. This simplifies life when you are using a mix of templates and not since you don't need to change the url constantly. This was mostly in the Splainer-Search v2.22, v2.22.1, and v2.22.2 releases. https://github.com/o19s/splainer-search/

• Now, under Explain Query modal, you can see the rendered template query for a specific query! #751 by @epugh.

Improvements

• Kind of in the weeds, but early database migrations in Quepid back in Rails 4 days didn't have a version spec. Rails 7 complains about this. #739 fixed by #746 by @epugh.

• Introduced a new environment variable QUEPID_CONSIDER_ALL_REQUESTS_LOCAL that lets you bubble up errors message in a production Rails environment, which is helpful in troubleshooting deployments when you can't see the logs. #750 by @epugh.

• Package Jupyterlite to not require external network calls. #721 by @epugh fixed by #728 by @mkr.

Bugs

• A three point custom scorer should support keyboard shortcuts. #738 by @epugh fixed by #752 by @depahelix2021.

• Missing Documents Modal doesn't work when an ES/OS Query Template is defined. #747 fixed by #753 by @epugh.

• Populating a book of judgements would blow up. Plus the data model allowed a single user to rate a query/doc pair multiple times, which was icky. #734 fixed by #745 by @epugh.

• Use relative paths everywhere so when Quepid is behind proxies it works properly. #754 by @epugh.

7.3.3 - 2023-05-30

• When populating a book of judgements, if the title field wasn't title, then it wouldn't show up with the correct name. #737 by @epugh fixes this.

7.3.1 - 2023-05-26

• Fixed some additional API paths for use with a nested Quepid context. #736 by @epugh. Follow up to #719.

7.3.3

7.3.3 - 2023-05-30

• When populating a book of judgements, if the title field wasn't title, then it wouldn't show up with the correct name. #737 by @epugh fixes this.

7.3.0

7.3.0 - 2023-05-23

Improvements

• Importing Snapshots was a feature from the dawn of Quepid, but @epugh had never used it till now. Originally you had to put into the csv file the id of the Case you want to import into, which is awkward and error prone. Now you can import a snapshot via the Import modal for a Case. #727 by @depahelix2021 fixes #724 by @epugh.

Bugs

• OpenSearch and Elasticsearch have a _source field that is nested JSON. Turns out we don't support stringifying a JSON object in detailed exports or Snapshots. #732 by @depahelix2021 fixes #730 by @epugh.

• With the logic for matching Quepid TLS to the search engine TLS happening only via frontend logic (introduced in #719), we found the User Experience really rough. #731 by @epugh reworks the UX to be clear about when you need to reload your Quepid.

• We ship Jupyterlite for data manipulation, but maybe you don't have public network access? #728 by @mkr fixes #721 by @epugh.

• Cloning a Case loses the advanced settings. #729 by @epugh fixes #725 opened by @jvia. Thanks @jvia for reporting this!

7.2.1

7.2.1 - 2023-05-12

Well, that didn't take long... Wanted to use the "Import Snapshot" function, and discovered that the "Export Snapshot" function was busted.

Features

• We've had the ability to mark a Case as "Public" for a while, but we only supported it for the visualization of tries screen. As a spike, trial allowing public access to /api/cases/CASE_ID.json and /api/cases/CASE_ID/snapshots/SNAPSHOT_ID.json end points. #723 by @epugh.

Bugs

• Fix Export Snapshot. #722 by @depahelix2021.

7.2.0

7.2.0 - 2023-05-11

Features

• Interested in Rank Biased Overlap as a more powerful version of the venerable Jaccard metric? We have it now as an example Jupter notebook. Thanks to Tito Sierra at the 2023 US Haystack for introducing me to this metric. Thanks to @mkr for creating the draft notebook.

• Provide link to Team from Book of Judgements. #718 by @epugh.

Improvements

• Lots of fixes and cleanups for nesting Quepid under another domain in #719. Credit to @frankcelia for figuring out that we were too smart by half in letting the Rails app redirect traffic to try and match Quepid TLS to the search engine's TLS. Pulling that out has simplified our logic immensely. #719 by @epugh and @frankcelia.

• We're now on Jupyterlite 0.1.0 from our relatively old 0.1.0b14, and have a build process that will let us stay up to date with this rapidly evolving project. #709 by @mkr.

7.1.0

7.1.0 - 2023-05-05

Features

Some bigger organizations deploying Quepid want to nest it under another domain, like https://tools.bigcorp.com/quepid instead of the more typical https://quepid.bigcorp.com. @worleydl open a PR to support this eleven months ago, and now we finally have it over the finish line! Just specify the RAILS_RELATIVE_URL_ROOT for when you want to nest Quepid under another domain. #500 by @worleydl.

Improvements

• Custom scorers are sorted by the order from oldest to newest in the modal picker UI. Instead, lets sort them alphabetically. #717 by @epugh fixes #695.

• teamSvc unit test was failing a lot, and now has been dealt with... By ignoring the checks. :-(.

• When creating a new Book of judgements, seed the Scorer with the one from the Case you were using. #716 by @epugh fixes #705.

Bugs

• Using templates in OpenSearch (and Elasticsearch) clashes with how we display (or hide) the fieldSpec field. #706 by @mkr fixes #699.

• The "Find Missing Docs" UI doesn't actual work with OpenSearch. #707 by @mkr fixes #700. Nicer help text as well.

• Generated link to individual OpenSearch (and Elasticsearch) document changed, and didn't render properly. o19s/splainer-search#117 by @mkr fixes #701.

• Swapping from one Scorer to another Scorer would lose the labels in the popup window. #717 by @epugh fixes #704 and #696 by @epugh. It may also fix #613 by @atarora ;-).

7.0.0 Ready for Human Rating! And the Future.

7.0.0 - 2023-04-24

Are you ready to launch a Human Rating Program? Quepid is now finally able to support you and your fellow human judges in gathering explicit ratings on your search results.

Since the dawn of Quepid, we've suffered from an ahem sub optimal rating interface for gathering human judgements.

The rating interface failed most best practices for human rating, including suffering from position bias by listing documents in the order the search engine returned them, only supporting a single rating per query/doc pair, requiring lots of mouse movement and clicking.
All that, and the UI is a combination of features required for a Relevancy Engineer with those of a Human Judge.
It's enough to make you weep.
Just to make life harder, the rating interface requires a live connection to your search engine, which often required additional technical hoops to be jumped by your decidedly untechnical subject matter experts.

However that is no longer the case!

This rating interface features:

• You can have up to three independent ratings for every query/doc pair, opening the door to interesting measurements of rating quality.
• A static dataset for rating that is populated from your live search engine. Now your set of query/doc pairs won't change over time unless you want them to.
• Query/doc pair are randomly sampled, but with a bias in favour of higher positioned results over lower positioned so you can get to meaningful numbers quicker.
• Ability to mark a query/doc pair as "I can't Rate This Document" so we can find edge/confusing cases.
• You control what shows up on the card using the same display configuration as in your typical Quepid case.
• Supports thumbnails and hero images in the card.
• There are Scoring Guidelines for your Raters to understand what the scoring scale means.

To make the life of a Relevancy Engineer better, you can now import your queries and ratings from a Book of Judgments.
Indeed you can roundtrip from a Case with query/doc pairs to a Book of judgements, get them rated, and then import then back into your Case.

This work was inspired by the great work that the folks at the Wikimedia Foundation did on Discernatron, their human rating tool.

Quepid is now the big 7.0! There have been 98 PR's since release 6.14.0. We are now running on Rails 7, which is great for opening the door to future improvements and keeping us from accumulating tech debt. This also means we took the opportunity to bump the versions of Redis and MySQL we use, as well as Node, Ruby, and all the other libraries. We're now on a modern infrastructure everywhere except for the Relevance Engineer's UI which is still on AngularJS.

Do back up your data before doing this upgrade! Normally I'm pretty cavalier about updates in Quepid-land, but this changes a lot of things, including the MySQL version from 5.6 to 8.... So please back up your data, and for good measure, use those export options to export your precious ratings that you've collected.

Below are details on some selected PR's.

Features

• As mentioned above, the human rater interface is a big feature. There are too many PR's and Issues to call them out individually. However, I do want to thank everyone who contributed to that really important feature. You know who you are!

• Allow a Case to be marked Public to facilitate sharing analytics. Public cases don't require you to log in for certain screens. #595.

• Jupyter notebook for calculating Jaccard Similarity between Snapshots. #586 by @atarora.

• Add Reciprocal Rank as a default Scorer. #525 by @david-fisher.

Improvements

• Rails 7 Upgrade! This would be enough to move us from Quepid 6 to Quepid 7 by itself. Turned out to be pretty painless upgrade. Most of the work was in #627.

• We enabled RenovateBot to provide automatic PR's for dependencies. In the past three months we had a flood of dependency updates, which improves our security profile and helps us deal with tech debt by ensuring we aren't falling behind.

• Refactor to remove manualMaxScore and manualMaxScoreValue from custom scorers as not used. #609.

6.14.0

6.14.0 - 2022-11-22

Fixes inspired by folks working with the Chorus project.

Improvements

• Rework the selection of settings for a search engine so that we are smarter about if you pick a TMDB demo server, use great defaults, but if you are picking your own search engine, then we make fewer assumptions about query structure, which reduces the chance the first query will blow up! This was first identified by @macohen as part of querqy/chorus#129. Tracked as #580 by @epugh and fixed in #582,

Bugs

• Command line tool thor import:ratings didn't handle a csv file with a header row. Now filters off the header row. Thanks @wrigleyDan for spotting this problem. #581 by @epugh and fixed in #583.

Full Changelog: v6.13.0...v6.14.0

Megan's Birthday Edition

6.13.0 - 2022-10-28

It's my sister Megan's birthday 🎂 today, so I figured I would give her a release of Quepid. Happy birthday Megan!

There is so much to celebrate in this release, however I have to call out adding support for OpenSearch, enabling API keys to work with Elasticsearch, and the first release supporting Jupyter notebooks runing in Quepid!

Features

• Quepid now supports OpenSearch! #319 was opened 18 months ago by @flaxsearch, and contributed to by @DmitryKey. Huge thanks to @mkr for stepping up and adding the support to both splainer-search and Quepid in #559.

• Quepid now supports Elastic Cloud and Elasticsearch auth with API Keys! Thanks to the work by @aditya-kanekar in #563 and then extended by @worleydl in #566. @aditya-kanekar also wrote up the docs on https://github.com/o19s/quepid/wiki/Troubleshooting-Elastic-Cloud-and-Quepid for how to set up Elastic Cloud!

• Jupyter Notebooks In Quepid 🎊. Everyone wants to customize their analytics and have different visualizations, so let's make that easier. Quepid now integrates Jupyterlite, a in browser version of Jupyter. So you can write your notebooks using the Python you know and love, and not have to worry about installing dependencies, as Quepid ships all the typical ones. This is an area that I expect a lot of improvement and change as we get to know how to ship sample analytics as Jupyter notebooks. #544 by @epugh.

• Our first sample notebook let's you compare the scores of two snapshots using a histogram. Really drives home the "before and after" story of Relevance Tuning. To support this notebook, we needed to preserve the score and if all the docs are rated in the snapshot datastructure. #550 by @epugh.

• Snapshots now include the total number of results for a query, useful for analytics. Also expose the Quepid API for looking up snapshots in the snapshots modal UI. #553 by @epugh fixes #539 by @renekrie.

• Curious what version of Quepid you are running? So are the rest of us! Display the Quepid version number in the page footer, #570 by @jzonthemtn is fixed by #576 by @epugh.

Improvements

• We've moved away from the "master" terminology to "main" for the default code branch, and updated links for that.

• Moved to Ruby 3! Ruby 3.1.2 on Bullseye is apparently twice as fast as Ruby 2. These two changes let Quepid run on Apple Silicon. It also lets us simplify our Dockerfile setup for Chromium, which we use for testing our frontend application. Lastly, we ripped out Webpacker (Webpack). Webpacker was added for JS toolingduring our migration to Rails 6, however we never actually used it in our development tooling, and is no longer preferred as part of Rails 7. #558 by @epugh and @mkr.

• Integrate updating of database schema annotations into build processes. This leverages the annotations gem, which we've had for years, but wasn't documented and therefore wasn't being manually run. #555 by @epugh.

• When we first moved to Rails from python, we had various database table constraints that were not enforced. Over the years we've started enforcing them, but never cleaned up the old data, till now! #552 by @epugh. Mostly of interest to folks with a deployment of Quepid back to 2016 ;-).

• During the Rails 6 migration we found out Redis wasn't critical to running Quepid, and attempted to make it optional. However that turned out to be confusing, and future features will require Redis. So let's put it back in our default production docker compose setup. #557 by first time contributor @OkkeKlein. Thanks!

Bugs

• Number of results to show is not preserved when cloning a case. #578 by @epugh fixes #565 submitted by @MassimilianoUngheretti-TomTom.

• Occasionally the Case snapshot isn't successful and there is no snapshot status information displayed to the user. #569 by @atarora fixes #568, also by @atarora.

• The Snapshot Comparison Jupyter notebook doesn't show the number of results found. #571 fixes #572, both by @atarora.

• If your TC_URL isn't defined, then that causes Quepid to blow up. #573 by @jzonthemtn is fixed by #575 by @epugh.

• docker-compose.prod.yml is out of date and needs Redis and Nginx. #557 by @OkkeKlein fixes #554.

• Changing Search Engines doesn't reset the api_method, so you could have ES attmepting to use JSONP, which is Solr specific. #560 opened by @epugh fixed by #559 by @mkr and @epugh.

New Contributors

• @aditya-kanekar made their first contribution in #563
• @OkkeKlein made their first contribution in #557
• @mkr made their first contribution in #559

Full Changelog: v6.12.1...v6.13.0