Airflow HTTPD Proxy development and configuration

[LucaCinquini]

No description provided.

[LucaCinquini]

Dependency: CS setup HTTPD proxy service on each target venue

Must proxy both the Airflow UI and the Airflow API (which are at the same base URL).
It will require consulting the Airflow documentation since Airflow assumes all URLs are served by a root URL,.

Instructions/documentation from CS:

• https://app.gitbook.com/o/xZRqGQeQXJ0RP4VMj7Lq/s/UMIRhLdbRQTvMWop8Il9/developer-docs/common-services/docs/developers-guide/httpd-server-deployment
• https://github.com/unity-sds/unity-proxy

[mike-gangl]

To integrate with the API Gateway, we must create a new resource in the venue api gateway:

• The venue API Gateway should have the naming convention of unity-{project}-{venue}-rest-api-gateway.
• a resource at the root of the API gateway must be created titled ogc (discussion, should this be "processing" ?)
• after that resource is created, we want to create another , proxy resources as a child of the ogc/path. It should essentially be ogc/{proxy+} .
• The proxy resource should point to the ogc load balancer (preferably over HTTPS), and should include the proxy+ content in the request to the OGC ALB:

[jpl-btlunsfo]

Had to poke at it a little bit, as Airflow unfortunately decided to use absolute pathing on its Location headers.

(stored in the /unity/unity/dev/cs/management/proxy/configurations/015-sps-airflow ssm param at the moment)

<Location "/sps/">
    ProxyPassReverse "/"
</Location>
<LocationMatch "^/sps/(.*)$">
    ProxyPassMatch "http://k8s-airflow-airflowi-8261b72237-194646091.us-west-2.elb.amazonaws.com:5000/$1"
    ProxyPreserveHost On
    FallbackResource /management/index.html
    AddOutputFilterByType INFLATE;SUBSTITUTE;DEFLATE text/html
    Substitute "s|\"/(?!sps/)([^\"]+)|\"/sps/$1|q"
</LocationMatch>

The Substitute call, above seems to work fine so far (no issues visible in the UI), but it'll only match a URL in quotes, which in my opinion could miss a edge case or two, but I haven't stumbled on one.

This doesn't handle the OGC endpoint all though- but I can work on that in a different PR.

[LucaCinquini]

In this case, what is the URL that is proxied to http://k8s-airflow-airflowi-8261b72237-194646091.us-west-2.elb.amazonaws.com:5000/$1?

[jpl-btlunsfo]

with the above configuration, the /sps/ path on the unity-proxy endpoint (currently, the unity-venue-dev "main" instance http://unity-dev-httpd-alb-285256043.us-west-2.elb.amazonaws.com:8080/sps/)

But that /sps/ path isn't set in stone, just picked during testing.

[LucaCinquini]

Note: the following links don't seem to be working:
o Clicking on the Airflow logo on the upper left
o Submitting a job from the UI (the. job gets submitted, but there is a problem displaying the confirmation page)

[jpl-btlunsfo]

Hmmm. That first link is a charming link to /

<a class="navbar-brand" rel="home" href="/" data-original-title="" title="">

I might be able to fiddle the regex to catch that. Done!

The second issue is an odd redirect to /home, which seems to double dip in picking up /sps prefixes (becomes /sps/sps/home). I'll take a look at that.

[jpl-btlunsfo]

Should be fixed by #170