Biweekly Technical Meetings

[jonmjoyce]

We are starting biweekly technical meetings for the Next-Gen DMAC project to give us a space for technical discussions with our collaborators. The meetings will be held at 3pm EST. Message me with your email address if you'd like me to send you a meeting invite.

Microsoft Teams meeting

Join on your computer, mobile app or room device

Click here to join the meeting

Meeting ID: 275 961 541 142

Passcode: VZfqyK

Download Teams | Join on the web

Or call in (audio only)

+1 213-357-2812,,1805532# United States, Los Angeles

Phone Conference ID: 180 553 2#

Find a local number | Reset PIN

Learn More | Meeting option

[jonmjoyce]

cc: @rsignell-usgs, @abkfenris, @emfdavid

[abkfenris]

That looks like it works for me.

[mwengren]

At the moment, I have a conflict every other two weeks at that time, including the first meeting on the 25th.

If we could move it an hour earlier or later on Wed that would help for me, but if not it's not a big issue. I'll try to join as I can.

[emfdavid]

I will attend as I can. I have a recurring conflict during this time.

[jonmjoyce]

Reminder that we're having our first open technical discussion today at 2pm ET.
We can adjust the day/time as needed for future meetings if this doesn't work for most people.

Hope to see you there!

[mwengren]

I have a double conflict this week, but I should be able join the next one. Either 1 PM ET or 3 PM ET regular times would be better for me with my current schedule (or pretty much any other time on Wednesdays).

[emfdavid]

Upvote for 3pm EST in future.

[rsignell-usgs]

Downvote for 3pm EST today, but okay for the future

[emfdavid]

My conflict got cancelled... see you shortly.

[emfdavid]

@rsignell-usgs the two links from fsspec/kerchunk#209 are the items for follow for the parquet kerchunk improvement?

[jonmjoyce]

We're shifting the meetings back an hour to 3pm ET now. The next meeting will be February 8th at 3pm ET. We had a great technical discussion the first meeting with a lot of knowledge sharing on kerchunk and different implementation strategies. Thank you to those who contributed, and looking forward to next time!

[mwengren]

@jonmjoyce Do you keep notes from the technical meetings? Sorry I couldn't make it today. I saw the calendar invite, if there are notes please add a link there.

Short turnaround question I have for the group (anyone that is who's following this repo and wants to weigh in):

What do you see as the most relevant activity we could be doing or tool we could deploy from the NextGen DMAC work so far if we were able to fund it operationally? For example:

• kerchunk all IOOS ocean forecast model output into virtual FMRC or other re-chunking schemes and serve via S3 open access
• build out all of the XPublish* tools listed here: https://asascience-open.github.io/nextgen-dmac/analysis/data-access.html
• create an IOOS observations data lake for ocean forecast model development/forcing
• ?

Where is our current DMAC system failing us or will be failing us soon unless we do things differently and better?

Looking for any input by this Friday (1/27). Thanks!

[mpiannucci]

In my personal opinion the first option is the most important. It is the foundational piece of functionality that is currently locked away behind the THREDDS workflow. Just focusing on this one area would give users the ability to work with these FMRCs and time series data however they would want or need to, directly from the cloud versus being forced to use opendap via thredds which has to be maintained and administered.

[jonmjoyce]

@mwengren I'm going to start a living document for the meeting notes. This first meeting we discussed some of our personal goals / desired outcomes from a technical perspective. The group shared what they've been working on in detail, and a lot of it centers around kerchunk and improving data access.

To answer your question, I'll second Matt's response with a little more context. If IOOS provided data in a cloud-optimized format (like kerchunk) then that would greatly improve accessibility to the data for customers wanting to access it using the cloud without downloading data. This is also a necessary first step to support the XPublish tools; they won't work any better than existing tools if the data isn't cloud-optimized. Jupyter notebook users would be able to point to the kerchunked data directly and perform their analysis more effectively.

The second part we need to consider then is how we kerchunk all the IOOS forecast model output. @emfdavid shared some of the challenges that Camus has faced trying to kerchunk a single model, the HRRR. Data delays, variable mismatches, computational slowdowns, etc. all come into play when operationalizing something like this. To manage some of those challenges we'd want to implement a robust execution workflow that's managing all the kerchunking, taking notes from the Data Ingest aspect of this project. Finally you need infrastructure to run all this which optimally would be Kubernetes.

Thanks for the question and feel free to reach out to discuss more details.

[abkfenris]

At least for me, I think it would help to separate out what are the declarative desired results of NetGen DMAC work vs the imperative. That is 'what do DMAC processes provide to users' vs 'a way to provide those outputs'.

I think this is useful as this is where some projects have stumbled in the past by our (developers) willingness to rush towards playing with or building new tools and loosing sight of the bigger picture. Then we have a tendency to try to rewrite the bigger picture around what we've built.

Thinking this way can also help the results of the work being too closely tied to a single infrastructure, which may not be transferable across RAs due to needing to interact and migrate from existing infrastructure. Then any reference implementation can be build with the eye towards using modular systems that allow individual pieces to be adapted as RA's evolve their infrastructure.

It can also help define what parts may be better off as a single national system funded by the IOOS office for the RAs to use, vs what things are the RAs responsibility.

Of course many of these concerns change if we're going full CIOOS-style collaborative shared national infrastructure.

[mwengren]

Thanks to everyone for weighing in so far.

@abkfenris I believe we are taking that approach in this project, or something like it - recognizing that DMAC is intentionally not a monolithic system and there needs to be flexibility both for practicality of implementation and also due to varying requirements, funding structures, etc across the regions.

I guess what I'm wondering is is what could we do as an early proof-of-concept that wouldn't force any changes on any RAs, but can add value to the DMAC system as a whole (assuming we could obtain funding to do it above what's already funded here)? What are added capabilities on a National scale that don't require any major changes to existing RA data systems?

I see the Kerchunk/FMRC work as being pretty non-disruptive and feasible (apart from challenges mentioned by @jonmjoyce above).

Another idea might be a federated ERDDAP instance for IOOS' in situ obs that aggregates each of the RA ERDDAP real-time data streams in a single service, and could serve as a pan-Regional portal for use data discover/data access for ocean model forcing (this is something that has been on IOOS' list for awhile actually but we never really pursued it). Bonus if there were some way to make the federated ERDDAP highly-performant and with some sort of caching functionality/layer to reduce load on the federated ERDDAPs (maybe a pipe dream, I know).

This is really just a brainstorming exercise, and I don't think I need full details about anything, so what you've all contributed already is great. I'm mostly looking to understand if we have some early ideas to build on down the road to know what to add to a request that came to me.

[emfdavid]

I have a conflict again this week. I will read notes after.
Camus is looking for public dataset like HRRR, but with longer time horizon (greater than 48 hours).
I can't find any other full historical forecast archives like HRRR with longer horizon on AWS/GCP. I don't see the North American Mesoscale model output on AWS/GCloud and GFS is only the current week. Any ideas?

[mpiannucci]

GFS on AWS has data back to 2021: https://noaa-gfs-bdp-pds.s3.amazonaws.com/index.html

Only GCP is limited to the last week for GFS.

[emfdavid]

I am pulled into another customer meeting this week.
Will look for notes.
Making progress on additional data sources - NAM & GFS.
Considering GOES 18 as well.
Plan to use kerchunk for all...

[jonmjoyce]

I've started a public meeting notes document here: https://docs.google.com/document/d/1oA70lZ-S7h0bDtrAbcRGJ-ggRKhY3YuIEBYIszAp6Ps/edit#