[RFC] How do we want to handle tracking?

[GentlemanHal]

Question

How can we help?

How do we want to handle tracking projects for display on the Monitor page going forward?

What is the purpose of this issue?

We already have two issues related to this raised, #63 and #129, and it recently came up again on chat.

So the purpose of this issue is to highlight my current thinking and to get a discussion going about what people think is best.

What is the background?

Nevergreen was started on a project that had a single CI server for most teams and easily over 200 projects and our team only wanted to show a small subset of those projects on our build monitor.

Nevergreen also replaced an existing build monitor (which was written in Ruby and I think is on GitHub somewhere but unfortunately I can't remember the name) which had a simple server side configuration file with an allowed projects list.

Both of these facts very strongly influenced how Nevergreen was built and that is why today it has the concept of selecting projects to include for monitoring.

What is the problem with this approach?

As highlighted in #63 and #129, the biggest issue is people adding or renaming projects on the CI server and forgetting to include them in Nevergreen for monitoring. This can result in builds being unexpectedly broken, potentially for a while as (awesomely) people have come to rely on Nevergreen showing the build status.

What alternatives are there?

#63 to automatically include projects Nevergreen has never seen before, but this is quite complicated technically.

#129 to just include everything, but I'm not sure how many teams would actually want to include everything without any filtering.

Use an exclude list rather than an include list. This would be much simpler technically but might be more confusing for the user as conceptually its easier to select the things you want to monitor rather than selecting all the things you don't.

What are you proposing?

I think we can implement #129 for teams that own the CI server or use something like Jenkins views.

For now we should ignore #63 as I think it adds a lot of technical complexity. We create a new issue for showing a message about Nevergreen needing to be manually updated if projects are added or renamed.

We create a new issue for adding an exclude list option, which would be useful for teams that want to monitor most projects but not all.

This would effectively make the UX something like the following:

• The user adds a new tray
• The tray is added and by default a single question "what do you want to monitor?" is present with the following options (either radio buttons or a drop down)
  • everything
  • explicitly exclude projects
  • explicitly include projects
• if the user selects "everything" then nothing else needs to happen
• if the user selects "exclude" or "include" a similar project view as we have now would be shown
• if they selected "include" we could also show a message that Nevergreen will need to be manually updated if projects are added or renamed

I'm not suggesting these exact wordings etc, these were just to hopefully get the idea across. If you have suggestions for better UX/wordings please comment and let us know!

Don't we have stats about usage?

Unfortunately not, we haven't implemented #146 yet and the survey I created only has a few responses and the results are almost perfectly split!

Open questions

1. Should we make the include everything the default going forward?
2. How do we make the UX not suck?
   1. How do we make the difference between the "include" and "exclude" views obvious?

[GentlemanHal]

I spoke to the awesome UX person at work and some more ideas came up.

1. Ask the tracking question while initially adding
2. Add tracking questions / options to the settings tab rather than the projects tab
3. Track new by default actually solves all problems
4. Periodic refresh of trays
5. Show removed projects on the monitor view

Points 1 and 2

The reason for points 1 and 2 is that this is likely to be an option that isn't changed frequently (if at all) once set. So asking the user when they are adding a tray lets us keep the project view simple and we can allow them to change it via the settings.

The same idea was raised for #63, adding the option to the settings rather than the project view.

Point 3

When adding a tray we now select all projects returned by default, if we also added the "track newly added projects" in settings and set that to true by default as well, then this effectively works as a "all projects" tracking option! So even though this is more complicated technically, it might be the best option to actually implement first.

Point 4

Calling refresh periodically in the background (every hour or day) would make it less likely that the user would see old projects on the Tracking page. This would potentially make it easier for users not familiar with Nevergreen to still update it successfully.

Point 5

This would make it more obvious that Nevergreen needs to be updated, especially when projects are renamed. This should be fairly easy to implement; If the server is asked to include a project that isn't returned in the CCTray XML feed it could return it with a new prognosis of removed or such like.

[GentlemanHal]

I've raised issue #276 to cover point 4 above.

I'm not going to raise issues to cover any of the other points (asking question when adding and showing removed projects) as I think the implementation of #63 should be good enough to satisfy most users. We can always raise new issues later if we are still seeing problems.

I'm also going to close this issue now as I think its served its purpose.