Exclude courses which have the download button disabled from mirror drives

[umar8hassan]

What are the relevant tickets?

closes https://github.com/mitodl/hq/issues/4473

Description (What does it do?)

This excludes the sites in offline mass build which has hide_download button set for them.

How can this be tested?

1. Switch main branch in ocw-studio
2. Create a site locally at localhost:8043/sites or use existing one
3. In metadata form for site, set hide_download to True
4. Trigger a mass_build: docker-compose exec web ./manage.py mass_publish <live/draft> --filter <site-name> and wait for the build to complete successfully.
5. Open minio console at localhost:9001 and check course data in ocw-content-offline-<live/draft>-local/courses/<site-name> for your site.
6. Verify that data has been updated recently.
7. Switch to umar/4473-exclude-courses-which-have-the-download-button-disabled-from-mirror-drives in ocw-studio
8. Repeat steps 2 - 6 and data should not be updated for offline version sites.

[ibrahimjaved12]

Can you please update the testing steps pertaining to local testing with mass build.

[ibrahimjaved12]

I tested the changes using upsert_mass_build_pipeline --offline and I can confirm that the courses are excluded from offline mass build that have download button disabled.

I also confirmed that they are excluded only from offline mass build, not the online one.
Looks good to me.

[pdpinch]

Where can I find a description about how this works? We have a goal of keeping studio indifferent to the project's data schema, and I feel like we're mixing things up here.

For example, if we had a project without a "show download" field, will this api still work?

[umar8hassan]

Where can I find a description about how this works? We have a goal of keeping studio indifferent to the project's data schema, and I feel like we're mixing things up here.

For example, if we had a project without a "show download" field, will this api still work?

Yes, it should be alright. It'd return an empty list instead of raising an error if the hide_download field does not exist in metadata property.

[gumaerc]

Where can I find a description about how this works? We have a goal of keeping studio indifferent to the project's data schema, and I feel like we're mixing things up here.

For example, if we had a project without a "show download" field, will this api still work?

@pdpinch I did suggest this solution, but yea now that you mention it this does go against our philosophy of not adding any application code to ocw-studio that makes decisions based on metadata properties. However, I don't really see a good way to avoid it in this case. The "hide download" option is a course level setting defined in the metadata, and the mass build is a deployment detail. For the online site, we simply don't render the download button in the theme if hide_download is true. With the offline build, there isn't really any code we could add to the Hugo code in the theme to not render the entire site if the flag is set. When you run hugo, a site is going to be built regardless of any settings so a decision has to be made somewhere outside of that to not run hugo to generate the offline site at all. I think in this case the best we can do is document the hide_download setting in the ocw-studio readme and detail exactly what it does within the context of the mass build.

[pdpinch]

Could we modify the code in the mass build script instead of the code in Studio? For example, exiting the build early if the download condition isn't met? Or, another option, could be build something else in this case, instead of the site? Maybe a page with a link?

[gumaerc]

Could we modify the code in the mass build script instead of the code in Studio? For example, exiting the build early if the download condition isn't met? Or, another option, could be build something else in this case, instead of the site? Maybe a page with a link?

The mass build pipeline is generated by ocw-studio. We could jump through the hoops required to parse out and read this setting, then choose not to execute the build within the pipeline, but I don't see how that would be worth it. At that point, we're still "crossing content configuration and deployment details" because the pipeline would be looking for hide_download.

If we do nothing with the ocw-studio code at all and tried to handle this in ocw-hugo-themes by somehow getting Hugo to render a page with a link in the offline theme, I don't think that would get us where we want either. You could potentially override the homepage and render a different template if hide_download is true, but you would still end up with the rest of the files that Hugo would generate and the site would still show up in the index. The existing solution here prevents the former. Since the site is never included in the pipeline, it's never built in the first place and the files won't be present in the mirror drive.

@umar8hassan The only thing I might add to this from a code review perspective is that we will also need to do this in update_websites_in_root_website. When ocw-studio publishes the "root website" (ocw-www) it publishes content along with that of type website that defines every website known to ocw-studio that is not the root website - https://github.mit.edu/mitocwcontent/ocw-www/tree/main/content/websites. This content is used to build the Fuse offline search index, and the site not being present in the mass build pipeline will not keep it out of here. We may want to just switch this function over to get the websites that should be included by calling get_publishable_sites. It also currently only updates or creates these records; it isn't set up to remove them.