WIP - Added spec tendrl_performance_enhacements.adoc #174

shtripat · 2017-07-19T06:56:00Z

tendrl-bug-id: #172
Signed-off-by: Shubhendu shtripat@redhat.com

r0h4n · 2017-07-21T13:43:51Z

specs/tendrl_performance_enhacements.adoc

+
+* Annotate the tendrl flows in different definitions files of tendrl modules to
+define the tagged queue name where these jobs would be written
+


No impact on tendrl code due to changes to BaseObject.save and load ?

Haven't tried yet. I feel job creations which are done explicitly from code might get impacted.
I think you had some design in mind. You can add details as well.

r0h4n · 2017-07-21T13:45:43Z

specs/tendrl_performance_enhacements.adoc

+* Document the details of load incurred on storage nodes due to tendrl
+components within justified limits (so that storage admin can plan the resource
+requirements accordingly)
+


Add item:

For gluster and ceph sds sync, convert the sync thread to a job and start the job at integration service startup and then triggered periodically.

after the above change, each sds sync object (eg: volume) should be synced in a separate greenlet.

Ok. Will add the required details

r0h4n · 2017-07-21T13:46:46Z

specs/tendrl_performance_enhacements.adoc

+
+* REST layer to write the jobs in tagged queues based on definitions
+
+* Enhancements for tuning the response time for various GET endpoints


The changes to BaseObject ("data" attr) will have an API impact, please document it

Yes. Will add a dependency on API layer.

shtripat · 2017-07-24T09:27:24Z

@anivargi kindly review the API dependencies and section specific to it.

anivargi · 2017-07-24T17:36:22Z

specs/tendrl_performance_enhacements.adoc

+
+* Provide guidelines on setting up a clustered etcd for tendrl
+
+* Tuning of REST endpoints for better performance and predictable response time


What does this exactly involve?

I remember while initial discussions we talked about tuning GetNodeList kind of APIs which were taking lot of time to load details. You can comment more if such things to be done.

anivargi · 2017-07-24T17:36:56Z

specs/tendrl_performance_enhacements.adoc

+with defined tags would pick jobs from their specific tagged job queues for
+processing
+
+* Enhance REST layer to create job in tagged job queues based on flow annotation


We are already adding tags for jobs.

We are adding tags. We would need changes to submit the jobs to tagged queues instead of putting to /queue always. That's the change being talked about here.

@anivargi , @shtripat , we need this details in the docs. what are the chnages required on API side. Import flows needs to taken care of immediately and what else? I guess queue(to submit the job) related information to be read from the definition files

tendrl-spec: Tendrl/specifications#174 Signed-off-by: Shubhendu <shtripat@redhat.com>

shtripat · 2017-07-26T18:14:12Z

@r0h4n I have pushed the code changes PRs for commons/node-agent for this.
You would need to push your changes which you have done for performance enhancements under this spec (specifically for tagged job queues and others)

tendrl-spec: Tendrl/specifications#174 Signed-off-by: Shubhendu <shtripat@redhat.com>

shtripat · 2017-07-26T18:37:58Z

@nnDarshan I have the added the alternate mechanism which you suggested for making different sync calls at different intervals. Plz have a look.

@r0h4n plz check and comment.

nthomas-redhat · 2017-07-27T06:16:20Z

specs/tendrl_performance_enhacements.adoc

+
+== Proposed change
+
+* Annotate flows in tendrl definition files with tagged queue names (to which


Can you provide the details here? What are the different tagged queues and the flows attached to it? Also provide sample definition file with this change.

I think the queue changes needs to be a different spec.

@r0h4n may be you can start another spec linked to this as you have already thought through a lot around this.

I agree, should be a different spec.

nthomas-redhat · 2017-07-27T06:29:17Z

specs/tendrl_performance_enhacements.adoc

+* Annotate flows in tendrl definition files with tagged queue names (to which
+these flows would write the job to)
+
+* Introduce a tagged job queue mechanism in `tendrl-commons` module. Services


Provide specifics/psuedocode about the proposed change

nthomas-redhat · 2017-07-27T06:41:48Z

specs/tendrl_performance_enhacements.adoc

+with defined tags would pick jobs from their specific tagged job queues for
+processing
+
+* Enhance REST layer to create job in tagged job queues based on flow annotation


@anivargi , @shtripat , we need this details in the docs. what are the chnages required on API side. Import flows needs to taken care of immediately and what else? I guess queue(to submit the job) related information to be read from the definition files

nthomas-redhat · 2017-07-27T06:42:32Z

specs/tendrl_performance_enhacements.adoc

+* Enhance writing/reading to/from etcd to consider whole object details as
+single JSON. While writing we need to get the json representation of the object
+and write as single field under etcd. While reading, it should be read as single
+value and whole object should be weaved back from JSON.


Provide the psuedocode for this change

Isn't this logic very trivial in nature? Anyway will add sample.

nthomas-redhat · 2017-07-27T06:44:28Z

specs/tendrl_performance_enhacements.adoc

+and write as single field under etcd. While reading, it should be read as single
+value and whole object should be weaved back from JSON.
+
+* Fine tune REST endpoints for better and faster response times


do we have a list of APIs undergo change? what changes are required. I think this should be covered in a different spec

@shtripat it all depends on the etcd structure to what extent the endpoints be optimized, if etcd can maintain a index of how the objects are updated then we can do some intelligent caching on the API level. Also having dynamic (Memory) and static(os version) attributes separated would also help in improving the response time.
We already got a improvement in nodes list by removing the monitoring data from the response.

@r0h4n thoughts?

nthomas-redhat · 2017-07-27T06:45:57Z

specs/tendrl_performance_enhacements.adoc

+
+* Document the details of load incurred on storage nodes due to tendrl
+components within justified limits (so that storage admin can plan the resource
+requirements accordingly)


Lines 60-66 - where this documention available? Are we spec - ing this out here or in a different spec?

As part this spec only I have listed so we don't forget this. All we need is some wiki docs written for these.
If you want can create individual issues against the Documentation repo

nthomas-redhat · 2017-07-27T06:47:30Z

specs/tendrl_performance_enhacements.adoc

+
+* In {gluster/ceph}_integration, change the sds sync as a job and start this job
+while startup of these integration services. Once started, these jobs should be
+triggered periodically.


Psuedocode required

This was suggested recently. Also portions of this spec are to be implemented/already available from @r0h4n. I would request @r0h4n also to add flesh to this spec.

nthomas-redhat · 2017-07-27T06:47:37Z

specs/tendrl_performance_enhacements.adoc

+triggered periodically.
+
+* Change any explicit raw reads in tendrl components to use load() and then the
+required field from object.


Psuedocode required

This is for sure very trivial and I strongly feel there is no need for pseudo code.
All we would be doing is instead os NS._int.client.read we would be using NS.{mod ns}.objects.{Class}.load().

Do you feel we need a pseudo code for these kind of trivial things in spec?

nthomas-redhat · 2017-07-27T06:50:19Z

specs/tendrl_performance_enhacements.adoc

+
+* Tuning of REST endpoints for better performance and predictable response time
+
+* Tuning of different components of tendrl for better memory utilizations


What are changes proposed to address this?

@anivargi you are the best person to comment here.

Not sure what this has to do with only the API. Wouldn't this need to be done for all the backend components as well?

Ok. based on this discussion I would remove this line which is very specific to API layer. All backend changes anyway are listed in the spec here.

tendrl-bug-id: Tendrl#172 Signed-off-by: Shubhendu <shtripat@redhat.com>

brainfunked

Best to separate out the changes that impact the API into a separate spec issue and remove them from here. Correspondingly, I think this spec issue and the spec itself could be renamed to something more specific.

The new spec could be part of the Gluster Monitoring milestone @anivargi @nthomas-redhat @r0h4n

brainfunked · 2017-08-02T17:01:02Z

specs/tendrl_performance_enhacements.adoc

+
+* Tuning of REST endpoints for better performance and predictable response time
+
+* Tuning of different components of tendrl for better memory utilizations


Not sure what this has to do with only the API. Wouldn't this need to be done for all the backend components as well?

brainfunked · 2017-08-02T17:01:23Z

specs/tendrl_performance_enhacements.adoc

+
+== Proposed change
+
+* Annotate flows in tendrl definition files with tagged queue names (to which


I agree, should be a different spec.

shtripat · 2017-08-03T04:46:08Z

Not sure what this has to do with only the API. Wouldn't this need to be done for all the backend
components as well?

This is point under use cases and not specific to API only.

mkudlej · 2017-08-03T07:30:06Z

specs/tendrl_performance_enhacements.adoc

+
+== Testing:
+
+* Verify that load incurred on storage nodes due to tendrl components is within


What does mean defined limits in this context? I mean what is goal here? I think this should be defined here and if it cannot be achieved with specified HW configuration then result should be new limits with reasons why limits have been changed.

mkudlej · 2017-08-03T07:30:19Z

specs/tendrl_performance_enhacements.adoc

+* Verify that load incurred on storage nodes due to tendrl components is within
+the defined limits
+
+* Verify the REST endpoints for their response time and it should be within the


Similar question than above.

mkudlej · 2017-08-03T07:32:47Z

specs/tendrl_performance_enhacements.adoc

+
+* Provide guidelines on standard hardware requirements for tendrl server node
+
+* Provide guidelines on setting up a clustered etcd for tendrl


Are Tendrl components ready for clustered Etcd? Because right now there is only one IP/FQDN for accessing Etcd in configuration.

tendrl-spec: Tendrl/specifications#174 tendrl-bug-id: Tendrl#657 Signed-off-by: Shubhendu <shtripat@redhat.com>

mbukatov · 2017-08-22T17:17:58Z

specs/tendrl_performance_enhacements.adoc

+
+== Documentation impact:
+
+* Document for clustered setup of etcd


Do we really consider clustered etcd setup for the milestone #3? I'm asking because such change will have a huge impact on setup and configuration in general.

After discussion with dev team, this was moved out into 3rd milestone.

mbukatov · 2017-08-22T17:25:41Z

specs/tendrl_performance_enhacements.adoc

+
+* Document for hardware requirements for tendrl server
+
+* Document for load details on storage nodes due to tendrl components


What does this mean exactly? Does it refer to expected performance impact of tendrl components on storage nodes?

tendrl-spec: Tendrl/specifications#174 tendrl-bug-id: Tendrl#657 Signed-off-by: Shubhendu <shtripat@redhat.com>

r0h4n suggested changes Jul 21, 2017

View reviewed changes

shtripat force-pushed the spec/tendrl-performance-enhancements branch from bfbc96c to 64b8f81 Compare July 24, 2017 08:21

anivargi suggested changes Jul 24, 2017

View reviewed changes

shtripat pushed a commit to shtripat/node_agent that referenced this pull request Jul 26, 2017

Changes for performance improvement

6933975

tendrl-spec: Tendrl/specifications#174 Signed-off-by: Shubhendu <shtripat@redhat.com>

shtripat mentioned this pull request Jul 26, 2017

Changes for performance improvement Tendrl/node-agent#545

Closed

shtripat pushed a commit to shtripat/bridge_common that referenced this pull request Jul 26, 2017

Added logic to save and read whole object as single json

d297a31

tendrl-spec: Tendrl/specifications#174 Signed-off-by: Shubhendu <shtripat@redhat.com>

shtripat force-pushed the spec/tendrl-performance-enhancements branch from 64b8f81 to f595eca Compare July 26, 2017 18:36

nthomas-redhat requested changes Jul 27, 2017

View reviewed changes

shtripat force-pushed the spec/tendrl-performance-enhancements branch from f595eca to 92e6632 Compare July 27, 2017 08:13

WIP - Added spec tendrl_performance_enhacements.adoc

c850377

tendrl-bug-id: Tendrl#172 Signed-off-by: Shubhendu <shtripat@redhat.com>

shtripat force-pushed the spec/tendrl-performance-enhancements branch from 92e6632 to c850377 Compare July 27, 2017 08:27

brainfunked suggested changes Aug 2, 2017

View reviewed changes

mkudlej reviewed Aug 3, 2017

View reviewed changes

shtripat pushed a commit to shtripat/bridge_common that referenced this pull request Aug 8, 2017

Added logic to save and read whole object as single json

7846405

tendrl-spec: Tendrl/specifications#174 tendrl-bug-id: Tendrl#657 Signed-off-by: Shubhendu <shtripat@redhat.com>

mbukatov mentioned this pull request Aug 22, 2017

Update tendrl-ansible for Tendrl Milestone #3 (2nd milestone) Tendrl/tendrl-ansible#23

Closed

7 tasks

mbukatov reviewed Aug 22, 2017

View reviewed changes

mbukatov mentioned this pull request Aug 22, 2017

Performance enhancements for tendrl components #172

Open

mbukatov reviewed Aug 22, 2017

View reviewed changes

mbukatov mentioned this pull request Aug 23, 2017

Update tendrl-ansible for Tendrl Milestone #1 (3rd milestone) Tendrl/tendrl-ansible#25

Closed

6 tasks

shtripat pushed a commit to shtripat/bridge_common that referenced this pull request Aug 29, 2017

Added logic to save and read whole object as single json

51e8736

tendrl-spec: Tendrl/specifications#174 tendrl-bug-id: Tendrl#657 Signed-off-by: Shubhendu <shtripat@redhat.com>

shtripat pushed a commit to shtripat/bridge_common that referenced this pull request Aug 30, 2017

Added logic to save and read whole object as single json

7584bf5

tendrl-spec: Tendrl/specifications#174 tendrl-bug-id: Tendrl#657 Signed-off-by: Shubhendu <shtripat@redhat.com>

r0h4n closed this Jan 29, 2018


		* Annotate the tendrl flows in different definitions files of tendrl modules to
		define the tagged queue name where these jobs would be written


		* REST layer to write the jobs in tagged queues based on definitions

		* Enhancements for tuning the response time for various GET endpoints


		* Provide guidelines on setting up a clustered etcd for tendrl

		* Tuning of REST endpoints for better performance and predictable response time


		== Proposed change

		* Annotate flows in tendrl definition files with tagged queue names (to which


		* Tuning of REST endpoints for better performance and predictable response time

		* Tuning of different components of tendrl for better memory utilizations


		== Testing:

		* Verify that load incurred on storage nodes due to tendrl components is within


		* Provide guidelines on standard hardware requirements for tendrl server node

		* Provide guidelines on setting up a clustered etcd for tendrl


		== Documentation impact:

		* Document for clustered setup of etcd


		* Document for hardware requirements for tendrl server

		* Document for load details on storage nodes due to tendrl components

WIP - Added spec tendrl_performance_enhacements.adoc #174

WIP - Added spec tendrl_performance_enhacements.adoc #174

Conversation

shtripat commented Jul 19, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shtripat commented Jul 24, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shtripat commented Jul 26, 2017

shtripat commented Jul 26, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shtripat Jul 27, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shtripat Jul 27, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

brainfunked left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shtripat commented Aug 3, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mbukatov Aug 22, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shtripat Jul 27, 2017 •

edited

Loading

shtripat Jul 27, 2017 •

edited

Loading

mbukatov Aug 22, 2017 •

edited

Loading