Network overview for large networks fails with 500 error

[JackWilb]

On a large network, the AQL command that would be executed takes up a lot of RAM and the command stops instead of going over the memory limit. This leads to silent failure for the user. There is an option to stream results at the link below, which might be useful.

https://docs.python-arango.com/en/main/specs.html#arango.aql.AQL.execute

We'd want to pass stream in this line (if it's a good solution): https://github.com/multinet-app/multinet-api/blob/main/multinet/api/views/common.py#L37

[jjnesbitt]

Are you sure the problem is due to too much ram usage? How did you find that out? Also, I noticed this section of documentation from that parameter:

You should use this parameter only for short-running queries or without exclusive locks. Note: parameters cache, count and full_count do not work for streaming queries

We're currently using this function in full contrast of this statement: This would be for long running queries, and we currently do make use of full_count. I'm not against making use of this parameter, if it will solve more problems for us than it creates, but we should probably investigate it more.

[JackWilb]

I'm not sure that's the right option to use. I was thinking that the lock might only exist for queries that are mutating the data, not just reading. This would take a few minutes more research

Here's the stack trace that shows it will use too much memory:

multinet-api-django-1    |            ERROR    Internal Server Error:                            log.py:224
multinet-api-django-1    |                     /api/workspaces/updb/networks/updb/nodes/                   
multinet-api-django-1    |                     Traceback (most recent call last):                          
multinet-api-django-1    |                       File "/usr/local/lib/python3.8/site-packages/dj           
multinet-api-django-1    |                     ango/core/handlers/exception.py", line 47, in               
multinet-api-django-1    |                     inner                                                       
multinet-api-django-1    |                         response = get_response(request)                        
multinet-api-django-1    |                       File "/usr/local/lib/python3.8/site-packages/dj           
multinet-api-django-1    |                     ango/core/handlers/base.py", line 181, in                   
multinet-api-django-1    |                     _get_response                                               
multinet-api-django-1    |                         response = wrapped_callback(request,                    
multinet-api-django-1    |                     *callback_args, **callback_kwargs)                          
multinet-api-django-1    |                       File "/usr/local/lib/python3.8/site-packages/dj           
multinet-api-django-1    |                     ango/views/decorators/csrf.py", line 54, in                 
multinet-api-django-1    |                     wrapped_view                                                
multinet-api-django-1    |                         return view_func(*args, **kwargs)                       
multinet-api-django-1    |                       File "/usr/local/lib/python3.8/site-packages/re           
multinet-api-django-1    |                     st_framework/viewsets.py", line 125, in view                
multinet-api-django-1    |                         return self.dispatch(request, *args,                    
multinet-api-django-1    |                     **kwargs)                                                   
multinet-api-django-1    |                       File "/usr/local/lib/python3.8/site-packages/re           
multinet-api-django-1    |                     st_framework/views.py", line 509, in dispatch               
multinet-api-django-1    |                         response = self.handle_exception(exc)                   
multinet-api-django-1    |                       File "/usr/local/lib/python3.8/site-packages/re           
multinet-api-django-1    |                     st_framework/views.py", line 469, in                        
multinet-api-django-1    |                     handle_exception                                            
multinet-api-django-1    |                         self.raise_uncaught_exception(exc)                      
multinet-api-django-1    |                       File "/usr/local/lib/python3.8/site-packages/re           
multinet-api-django-1    |                     st_framework/views.py", line 480, in                        
multinet-api-django-1    |                     raise_uncaught_exception                                    
multinet-api-django-1    |                         raise exc                                               
multinet-api-django-1    |                       File "/usr/local/lib/python3.8/site-packages/re           
multinet-api-django-1    |                     st_framework/views.py", line 506, in dispatch               
multinet-api-django-1    |                         response = handler(request, *args, **kwargs)            
multinet-api-django-1    |                       File "/opt/django-project/multinet/api/auth/dec           
multinet-api-django-1    |                     orators.py", line 53, in wrapper                            
multinet-api-django-1    |                         return func(*args, **kwargs)                            
multinet-api-django-1    |                       File "/opt/django-project/multinet/api/views/ne           
multinet-api-django-1    |                     twork.py", line 139, in nodes                               
multinet-api-django-1    |                         paginated_query =                                       
multinet-api-django-1    |                     pagination.paginate_queryset(query, request)                
multinet-api-django-1    |                       File "/opt/django-project/multinet/api/views/co           
multinet-api-django-1    |                     mmon.py", line 37, in paginate_queryset                     
multinet-api-django-1    |                         cur: Cursor =                                           
multinet-api-django-1    |                     paginated_query.execute(full_count=True)                    
multinet-api-django-1    |                       File "/opt/django-project/multinet/api/utils/ar           
multinet-api-django-1    |                     ango.py", line 147, in execute                              
multinet-api-django-1    |                         return                                                  
multinet-api-django-1    |                     self.db.aql.execute(query=self.query_str,                   
multinet-api-django-1    |                     bind_vars=self.bind_vars, **kwargs)                         
multinet-api-django-1    |                       File "/usr/local/lib/python3.8/site-packages/ar           
multinet-api-django-1    |                     ango/aql.py", line 403, in execute                          
multinet-api-django-1    |                         return self._execute(request,                           
multinet-api-django-1    |                     response_handler)                                           
multinet-api-django-1    |                       File "/usr/local/lib/python3.8/site-packages/ar           
multinet-api-django-1    |                     ango/api.py", line 74, in _execute                          
multinet-api-django-1    |                         return self._executor.execute(request,                  
multinet-api-django-1    |                     response_handler)                                           
multinet-api-django-1    |                       File "/usr/local/lib/python3.8/site-packages/ar           
multinet-api-django-1    |                     ango/executor.py", line 65, in execute                      
multinet-api-django-1    |                         return response_handler(resp)                           
multinet-api-django-1    |                       File "/usr/local/lib/python3.8/site-packages/ar           
multinet-api-django-1    |                     ango/aql.py", line 400, in response_handler                 
multinet-api-django-1    |                         raise AQLQueryExecuteError(resp, request)               
multinet-api-django-1    |                     arango.exceptions.AQLQueryExecuteError: [HTTP               
multinet-api-django-1    |                     500][ERR 32] AQL: query would use more memory               
multinet-api-django-1    |                     than allowed (while executing)