Solution for paginating the Discover page (#496) #525

phantomjinx · 2024-09-10T18:14:51Z

hawtio-discover-paging-compressed.mp4

phantomjinx · 2024-09-10T18:15:25Z

Will do some videos to show exactly how it works ... to follow.

tadayosi

It's a bit tough to review such big changes effectively. Could you outline what's the core idea of the solution and where is the most important part of the changes?

tadayosi · 2024-09-11T01:07:07Z

Specifically, what's your solution to #496, how do you think the issue is solved with it?

phantomjinx · 2024-09-11T08:09:29Z

It's a bit tough to review such big changes effectively. Could you outline what's the core idea of the solution and where is the most important part of the changes?

Yeah, no problem. As I say, I'll do a video and talk through the changes

phantomjinx · 2024-09-25T13:26:24Z

Specifically, what's your solution to #496, how do you think the issue is solved with it?

Changes

Paging of pods (previous/next)
Skipping of pods (first/last/page)
Filter Namespace (filters out namespaces down to k8s level)
Filter Pods (filters out pods down to k8s level)
Sort (down to k8s level)
Preference for controlling number of pods displayed in each page
Preference for controlling the jolokia probe interval for checking on pod connectivity

Specifics

When a namespace is scanned, all the pods that contain a jolokia connection are indexed by name;
Using a given namespace limit (default 3), the first 'page' of pods are fetched and watcher connections established. Therefore, for each namespace containing jolokia pods, only the first '3' pods are ever being watched at any 1 time;
When the user click 'next' or 'prev', the watchers for the current set of pods are destroyed and a new set is created for the 'next' set of 3. Therefore, regardless of the project size the UI only displays a set limited number;
A preference is provided, should the user wish to increase the display from 3;
Both filter and sort have been refactored to perform those functions down at the k8s level rather than just the UI level. This makes sense since the filter then applies to all pods in the namespace rather than just those displayed in the current page of the namespace in the UI.

tadayosi

GJ

packages/kubernetes-api/src/client/client-instance.ts

packages/kubernetes-api/src/client/namespace-client.ts

tadayosi · 2024-09-26T05:03:51Z

packages/kubernetes-api/src/client/namespace-client.ts

maybe we can consider adding unit tests for newly introduced classes like this, if it's not so complex to do so.

tadayosi · 2024-09-26T05:06:48Z

packages/kubernetes-api/src/client/support.ts

these are a good place to add some more tests.

packages/kubernetes-api/src/client/ws-handler.ts

packages/kubernetes-api/src/kubernetes-service.ts

packages/kubernetes-api/src/paging-metadata-impl.ts

packages/management-api/src/index.ts

tadayosi · 2024-09-26T05:32:51Z

packages/online-shell/src/discover/discover-project.ts

another class maybe you can add tests.

…tio#496) * In order to paginate results, kubernetes uses 'limit' to limit the request and then returns a reference as a 'continue' property. By including the latter in the next query then next set of results can be obtained (see https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.30/#list-pod-v1-core) * globals.ts * Modifies KubeObjectList to support the optional properties of limit and continue defined in the KLimitMetadata type * Adds a PagingMetadata class for keeping track of paging references and the current index of the query, ie. <prev-ref> <index> <next-ref> * PagingMetadata support incremting and decrementing the current index in order to enable the moving to a previous or next set of results. * Kubernetes-service.ts * Refactored to support adding metadata limit to the watching of pods * Synchronises the pod watching code for both Namespace and Cluster config into a single function (initNamespaceClient). * Adds the namespace limit (nsLimit) to the pod options to be attached to the uri * If there is a continue reference for the current project/namespace then it too is added to the options for appending to the path uri * Watching the pods returns a result metadata as well as the pods which contains the podsRemaining property and a continue reference * The podRemaining property must be added to the 'current' paging metadata whilst the continueRef must be added to the 'next' paging metadata * Holds pods by project rather than just pods so that the project name can be used as an index reference for the pods * Holds metadata for each project so that the project name can be used as an index reference * Adds functions 'findNextPods' and 'findPrevPods' allowing clients to request the relevant alternative set of pods from the given namespace * collection.ts * Creates a metadata object for holding the limit and continue reference * Passes the metadata object through to the WsHandler responsible for doing the actual uri fetch * ws-handler.ts * Removes the support for websockets since they require the watch=true uri parameter which is incompatible with limit and continue. * Polling becomes the default mechanism instead for determine if there are any changes * OnMessage receives the response from the cluster and will also update the metadata object with the 'remaining' and 'continue' properties * object-poller.ts * Now responsible for fetching the cluster results, also communicates the metadata as part of the onMessage event back up to the WSHandler

* Current paging algorithm buggy and unnecessarily complex. Using limit, we are trying to save on network data load but this is not the issue with discover tab slowdown. Rather the slowdown is the number of pods being displayed and handled in the UI. * Better methodology: 1) Watch namespace and collect names of pods that are jolokia and add to a set 2) Take a page/chunk of pod names and watch them individually 3) Return only those pods watched for display in the UI 4) Prev/Next will move on the page/chunk in the pod name list and re-init the pod watchers accordingly * Removes the PagingMetadata * Fetches all the pods in a namespace using namespace watcher and adds names to list * Takes chunk from list determined by 'current' page and 'limit' size and starts watching each pod using fieldSelector. Provides only those pods as the collection of pods that k8Service displays for the namespace * Restores the wss handler and connections since no longer using limit and continue references. * No longer need to refresh continue tokens since not used * Changes name of Collection interface to Watched since it also now handles single entities being returned from k8 fieldSelector name API calls * NamespaceClients no longer need to be destroyed and recreated in order to work with pod changes. The pod watchers inside it are renewed if prev/next is called but otherwise those pod watchers already initialised are retained. Only those no longer in the 'page' are removed * Ref key from ClientFactor uses options name rather than continue ref * PagingMetadata interface refactored to Paging and applied to K8 Service and each namespace client

* Places the API Properties in a tab to give all tabs more space * Adds Prev and Next buttons to pods tab to choose the next set of pods to display in the namespace * Puts projects inside an Accordian to separate out the pods logically * KubernetesPods -> KubernetesProjectPods * KubernetesClient removed as folded into Kubernetes component

* Allows the namespace limit of the kubernetes service to be updated and resets all namespace clients to reconnect with the new limit value * Adds number input component to Kubernetes test app

* Allows user to extend/reduce the polling interval for updating of the pods

…awtio#496) * Discover.tsx * Move loading-panel into its own component DiscoverLoadingPanel * Puts the pods into tabs based on the projects/namespaces * New DiscoverProjectContent for each project * DiscoverProjectContent * Lists the limited selection of pods from management-service * When next button is clicked, the signal is sent to the management-service to fetch the 'next' selection of pods for the given namespace * DiscoverToolbar * Sorts the tab order since they are the namespaces * Sorts the pods in each namespace tab * Filters applied in line with patternfly guidelines * Filters of same attribute applied as OR * Filters of different attributes applied as AND * Filters are first concatenated together before being applied * HeaderMenuDropDown * Reimplemented with flyout sub-menus for the pods separated by namespace * context * Replacement of groupds / pods with projects * Addition of refreshing flag to indicate when pods are being refreshed awaiting a management update from prev/next buttons * discover-service split with new discover-project class that encapsulates the discover-projects and pods received from management

* Allows users to change the number of pods displayed in each Discover tab and the polling interval to determine heartbeat to jolokia container ports

…o#496) * The prev/next buttons were created when it was unknown how many pods would be returned for each namespace. Since we now have this information a patternfly pagination component can be implemented instead, complete with pod total, first and last functions. * namespace-client.ts * Returns total number of pods in namespace as separate parameter in callback * Adds first / last / page functions to accompany next and previous * management-service.ts * Adds Paging interface functions as pass-throughs to k8 service * Discover.ts * Adds total pods to tab titles * DiscoverProjectContent.ts * Replaces buttons with pagination component

* namespace-client.ts * Change name of 2nd callback parameter to fullPodCount to make it clear this is the total jolokia pods after filtering but before page slicing * Adds filtering and sorting getters/setters to affect the pods returned by the pod watchers * If any change occurs from filtering or sorting then the callback should be called in order to notify the UI * filter.ts * Refactors the TypeFilter to be a single class containing both namespace and pod name values * kubernetes-service.ts * Add/remove the pod projects to/from the service depending on if there are pods returned after filtering * management-service.ts * Add/remove the mgmt pod projects to/from the service depending on if there are pods returned after filtering * Discover context * Use only 1 filter rather than an array * Breaks out the empty state into its own component for use in other parts of the Discover page * DiscoverProjectContent.ts * Needs refresh Effect to reset the pagination index if it is greater than the project count returned * DiscoverToolbar.ts * When the buttons are clicked call the management service functions for filtering and sorting so that all pod projects down to the k8 service are effected. * discover-service.ts * Only responsible for grouping pods rather as filtering is re-assigned to other services

* client-instance.ts * Removes _ from parameter var * globals.ts * paging-metadata-impl.ts * Removes metadata: KubeSeachMetadata on account of no longer used * namespace-client.ts * Sort the jolokiaPods when using getter if a sort order has been specified * ws-handler.ts * Commas rather than + in log strings * Return type for getters * index.ts * Only export specified objects * kubernetes-service.ts * k8 -> k8s * managed-project.ts * Uses correct log from globals

* Since pods are fetched from an Object hashed by their uids, it is not certain that the pods will be correctly sorted when the sort button is clicked. Need to explicitly sort them. * Discover.tsx * context.ts * Adds a podOrder state object to context so that the DiscoverToolbar can update it and have it used when groupPods() is called * DiscoverToolbar.tsx * Set the pod order before telling the mgmt service to sort * discover-project.ts * When refreshing have it take into account the pod order that has been requested

phantomjinx requested review from tadayosi, grgrzybek and mmelko September 10, 2024 18:14

tadayosi linked an issue Sep 11, 2024 that may be closed by this pull request

Paginate the discover panel to only display a required number of pods at a time #496

Open

tadayosi reviewed Sep 11, 2024

View reviewed changes

phantomjinx force-pushed the watch-pod-lists branch 3 times, most recently from 4ec8d23 to b93524f Compare September 25, 2024 13:11

phantomjinx force-pushed the watch-pod-lists branch from b93524f to 566b526 Compare September 25, 2024 13:29

tadayosi approved these changes Sep 26, 2024

View reviewed changes

phantomjinx added 14 commits September 30, 2024 18:48

chore: Adds profiling of memory to Kubernetes test app

a896f2d

feat: Exposes the namespace limit for pod display (hawtio#496)

0d735b4

* Allows the namespace limit of the kubernetes service to be updated and resets all namespace clients to reconnect with the new limit value * Adds number input component to Kubernetes test app

feat: Exposes the jolokia polling interval (hawtio#496)

5e2892b

* Allows user to extend/reduce the polling interval for updating of the pods

Updates the management API and management test app

6bd873c

Exposes Discover Preferences (hawtio#496)

7f2cecd

* Allows users to change the number of pods displayed in each Discover tab and the polling interval to determine heartbeat to jolokia container ports

WIP

238fd65

phantomjinx force-pushed the watch-pod-lists branch from e797a7d to 238fd65 Compare September 30, 2024 17:55

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Solution for paginating the Discover page (#496) #525

Solution for paginating the Discover page (#496) #525

phantomjinx commented Sep 10, 2024 •

edited

Loading

phantomjinx commented Sep 10, 2024

tadayosi left a comment

tadayosi commented Sep 11, 2024

phantomjinx commented Sep 11, 2024

phantomjinx commented Sep 25, 2024

tadayosi left a comment

tadayosi Sep 26, 2024

tadayosi Sep 26, 2024

tadayosi Sep 26, 2024

Solution for paginating the Discover page (#496) #525

Are you sure you want to change the base?

Solution for paginating the Discover page (#496) #525

Conversation

phantomjinx commented Sep 10, 2024 • edited Loading

phantomjinx commented Sep 10, 2024

tadayosi left a comment

Choose a reason for hiding this comment

tadayosi commented Sep 11, 2024

phantomjinx commented Sep 11, 2024

phantomjinx commented Sep 25, 2024

Changes

Specifics

tadayosi left a comment

Choose a reason for hiding this comment

tadayosi Sep 26, 2024

Choose a reason for hiding this comment

tadayosi Sep 26, 2024

Choose a reason for hiding this comment

tadayosi Sep 26, 2024

Choose a reason for hiding this comment

phantomjinx commented Sep 10, 2024 •

edited

Loading