[RFC] Discussion PR for HTTP/2 support (DO NOT MERGE).

[Lukasa]

What these changes does?

This is an example of the most basic HTTP/2 support in aiohttp. This is nowhere near close to production ready: instead, it's intended to spur a longer discussion about how best to add HTTP/2 support to aiohttp. This code does the following:

• Allows spinning up a HTTP/2-only aiohttp web server that cannot serve static files.

This code does not support any of the following:

• HTTP/2 flow control
• HTTP/2 stream priority
• HTTP/2 server push
• Serving static files
• 100 Continue
• Correct error pages (404 pages still serve using HTTP/1.1, which breaks things)
• Anything remotely tricky
• Basically, it's not RFC 7540 compliant, or even really all that close

I opened this because I believe that if aiohttp wants to support HTTP/2 it requires a very substantial change, and I did not want to work on that in isolation without discussing it with the aiohttp community. However, it's always tough to discuss such a change without having some concrete code to look at, so this provides that code.

How to test your changes?

The following test file is a variation on the example from the documentation, and can be tested using the Python hyper library.

This is the server file:

from aiohttp import web, web2, web_reqrep2


async def handle(request):
    name = request.match_info.get('name', "Anonymous")
    text = "Hello, " + name
    return web_reqrep2.Http2Response(body=text.encode('utf-8'))


app = web.Application(handler_factory=web2.Http2RequestHandlerFactory)
app.router.add_route('GET', '/{name}', handle)

web.run_app(app)

To test it, install the Python hyper library (pip install hyper) and then run hyper --h2 GET http://localhost:8080/name.

Related issue number

#863, #320

Discussion

When trying to implement HTTP/2 support in aiohttp, I discovered that aiohttp's architecture and API make it remarkably tricky to plumb HTTP/2 support through in a coherent and sensible way. The biggest problems here are:

1. Lack of encapsulation. A large swathe of the aiohttp code-base knows quite a lot about the wire-format of HTTP/1.1, rather than confining that knowledge to the ServerHttpProtocol and *Parser classes. For example, web_urldispatcher contains a _defaultExpectHandler that writes directly to the transport rather than passing that write through the ServerHttpProtocol or equivalent object. More generally, HttpMessage and friends all write directly to the transport, which means that to make HTTP/2 work we need to propagate quite a lot of state through aiohttp to get it from the ServerHttp2Protocol to those objects.
2. Subclassing internally. The aggressive use of subclassing within aiohttp makes it quite difficult to locate all the places that need changing. Ideally, all objects in aiohttp would direct their writes to the transport via the ServerHttpProtocol, but because there are so many different objects that have access to the backing transport (and also the reader/writer objects) it is very difficult to enforce that constraint. Additionally, for HTTP/2 they'd all need a correlator (their stream ID), which makes a further substantial change to the codebase. That wouldn't be too bad, except...
3. Subclassing as an API. aiohttp quite widely uses subclassing as an API: for example, ServerHttpProtocol and friends all expect to be subclassed. Because of this, there is likely quite a lot of third-party code that subclasses aiohttp's classes and reaches inside of them. That makes it very difficult to do a substantial refactor to the code. If aiohttp was unwilling to break that code, they'd need either to do a big breaking semver-major change (which will annoy a lot of people) or write an entirely parallel stack that is capable of doing HTTP/1.1 and HTTP/2 transparently and then ask people to migrate to that. Either seems like quite a lot of work.

Basically, from where I'm sat I think that adding HTTP/2 support transparently into aiohttp requires a very substantial change to the library, and I'm unwilling to do that without discussing with the community and maintainers first. Is there any interest in having such a substantial change made to aiohttp, or do the maintainers and community believe that HTTP/2 support is not worth it?

[asvetlov]

@Lukasa let's discuss it during US PyCon in a few days.

[Lukasa]

Sounds good to me.

[jchampio]

Okay, so as we near the end of the PyCon sprints, I've done far more reading than writing code. I was able to splice @Lukasa's contribution directly into the core in a dev branch (see above commit da7d903) so that we have a Response object that is capable of both HTTP/1.x and HTTP/2 communication. As expected, the result is an ugly mess. My hope is that the ugly parts will begin to show us where to untangle concerns.

To capture some of the side conversations we had in the sprint:

• It sounds @asvetlov is interested in splitting up the internal classes as necessary, with the general compatibility strategy of forwarding deprecated methods to their new equivalents. I'm sure we'll run into a pain point eventually, but we'll see when we get there.
• As @Lukasa alludes to above, server-side code does not appear to clearly separate the connection/transport from either the protocol being used on that connection or the responses being sent. This is one of the first places I intend to focus next.
• aiohttp supports sendfile() for plaintext static file serving. Doing the same for HTTP/2 would require explicit support in hyper-h2; see Add support for sendfile() based DATA frames. python-hyper/h2#236. It is probably less generally useful than its equivalent in HTTP/1.x, since no modern browser currently uses h2c, but it may still be useful for very specific use cases.

[fafhrd91]

Closing as no activity for almost a year. In any case client http2 support is more interesting from my prespective

[lock]

This thread has been automatically locked since there has not been
any recent activity after it was closed. Please open a new issue for
related bugs.

If you feel like there's important points made in this discussion,
please include those exceprts into that new issue.