Skip to content
This repository has been archived by the owner on Aug 30, 2024. It is now read-only.

maintenance mode 3.3 spec

Jump to bottom
Steve Jones edited this page Sep 6, 2017 · 9 revisions

Table of Contents

Maintenance mode

Overview

Maitenance mode operations should allow a cloud administrator to perform maitenance of node controllers with minimal intereference with the cloud workload. The general goal is to steer virtual machines away from the node to be maintained and then to re-integrate the node after maitenance has been performed. Two complementary approaches are supported:

  • Passive approach involves setting the node into maintenance mode, which means no new VMs will be scheduled on it, and waiting for the natural churn of instances to free up the node.
  • Active approach involves migrating VMs away from the node to be maintained using live migration features of the hypervisor.
We refer to the combination of these approaches as evacuation: the node is first placed into maintenance mode and then is cleared of all instances if sufficient resources for migrations are available at the time of the request. While evacuation is the only approach that is scheduled to be included in Eucalyptus 3.3, more fine-grained approaches to maitenance may be available in 3.3 or a later release:
  • Cloud admin may have the option of migrating only specific VMs
  • Cloud admin may have the option of migration to only specific destinations
  • Cloud admin may have the option of not only specifying, but reserving destination hosts
If resources necessary for migration are not available at the time of the migration or evacuation request, the request will fail. Currently, no mechanism is being developed for re-trying the migration until adequate resources are available. Such mechanism could be built with tools external to Eucalyptus by relying on the interface describe here.

Implementation overview

Two types operations are exposed through the CLC interface for the purposes of node maitenance:

  • Enable / Disable Node pair of operations request a specific node to enter or leave maintenance mode (i.e., enter or leave state in which no new instances will be scheduled on the node).
  • Migrate Instances operation requests migration of one or all instances from a source node to one or more destinations.
Components at all three levels (cloud, cluster, node) have methods corresponding to these operations, however their names, and the set of input or output parameters vary:
Maintenance-mode operations at different levels
Conceptual CLC API CC API NC/VB API
Enable / Disable Node
  • host name / IP
 EnableService
DisableService
  • ServiceId
  • Type
  • Host
  • Partition
 EnableService
DisableService
  • ServiceId
  • Type
  • Host
  • Partition
 ModifyNode
  • stateName
Migrate Instances
  • SourceHost
  • (InstanceId)
  • (Destinations)
 MigrateInstances
  • sourceHost
  • instanceId
  • destinationHost.n
  • allowHosts
 ccMigrateInstances
  • sourceHost
  • instanceId
  • destinationHost.n
  • allowHosts
 ccMigrateInstances
  • instance.n
  • action
  • credentials

Both types of requests are "synchronous" in that an upstream component waits for the corresponding downstream component to respond to the request before it replies. In case of the disable/enable request, the CLC waits for the CC to reply, which waits for the NC to persist the change of state to disk before replying. While CLC is waiting for the downstream components, it is "gating" requests to the cluster: most requests to the cluster, such as running a new instance or attaching a volume to an instance, are queued up.

Similarly, CLC "gates" requests while the Migrate Instance request is pending. Handler for MigrateInstance() on the CC creates the migration plan (i.e., computes the optimal schedule for migration based on the restrictions from the user, if any) and sends it to the source NC for storage. (The primary purpose of the MigrateInstances() request to the source NC is to save the migration plan on disk.) Once NC has acknowledged the migration plan, CC takes into account the reduced resource usage, as does the CLC, prior to unblocking any pending requests.

Instance migration proper is driven by CC in response to state changes on its NCs:

  • When a source NC reports itself as ready for outgoing migration, the CC issues the PREPARE request to the destination.
  • When a destination NC reports itself as ready for incoming migration, the CC issues the COMMIT request to the source, which effectively initiates the migration.
Failures during this process are handled locally when possible and are also advertised upstream when necessary. For instance, failure of the destination to prepare prompts CC to send the ROLLBACK request to the source (so no migration-related state will be reported henceforth).

Command-line tools

to be filled in by Garrett

Component APIs

Each level of the system (cloud level, cluster level, and node level) implements operations related to maitenance.

NOTATION: For Eucalyptus 3.3 minimal viable product, only parameters marked with ♣ must be implemented.

CLC an CC operations

Both CLC and CC implement the same operations with the same parameters:

MigrateInstances

Requests migration of either a specific instance or all instances on a specific NC or ESX(i) host. Optionally, the request may restrict the set of hosts that can be considered as potential migration destinations either by specifying a white list or a black list.

Request Parameters
Name Description Required
sourceHost♣ Name of the NC/ESX host from which all instances are to be migrated (mutually exclusive with instanceId).
  • Type: String
  • Default: None
Yes (XOR instanceId)
instanceId ID of the instance to migrate (mutually exclusive with sourceHost).
  • Type: String
  • Default: None
Yes (XOR sourceHost)
destinationHost.n Set of hosts that are either allowed or disallowed as targets of migration(s), depending on allowHosts. If the set is empty, then all hosts are potential targets for migration, with CC making the choice based on its standard scheduling algorithm.
  • Type: String
  • Default: Empty
No
allowHosts If true, the hosts in destinationHost set are the only allowed destinations for migration. (In the trivial case of a single host, there is no scheduling to perform.) If false, that set contains hosts explicitly disallowed as destinations. The parameter has no meaning if the set above is empty.
  • Type: Boolean
  • Default: None
No

Successful response from CLC indicates that:

  • the migration request is valid and the participants are present (instance is still running or host is still functioning)
  • a migration plan has been computed and recorded on the source NC or in the VB database
  • there were sufficient resources to perform the migration and they have been reserved (resource availability will reflect the additional resources necessary during migration)
Response Elements (from CLC)
Name Description Required
return♣ Request accepted if false and error if true
  • Type: Boolean
  • Default: None
Yes

Successful response from the CC will also include resource availability that resulted from the migrate request.

Response Elements (from CC)
Name Description Required
return♣ Request accepted if false and error if true
  • Type: Boolean
  • Default: None
Yes
resources Resource matrix (max and available for each VM type) after resources needed for migration are taken into account.
  • Type: ccResourceType (see eucalyptus_cc.wsdl)
  • Default: None
Yes

EnableService

DisableService

Requests to disable or enable a node controller are used for placing it into and out of maitenance mode. When in maitenance mode, the node controller:

  • is running, but is only responding to a subset of requests (most importantly, RunInstance requests are not accepted)
  • is monitoring virtual machines currently running on it
  • is not going to be the target of a migration unless it is specifically listed in the white list of a MigrateInstances
Maitenance mode may be used separately or in combination with a MigrateInstances request(s). For instance, evacuation of a node for the purposes of migration can be implemented as a combination of placing the node into maintenance mode, followed by the request to migrate all instances away from it.
Request Parameters
Name Description Required
ServiceId♣ what's this?
  • Type: String
  • Default: None
Y
Type♣ Must be "NODE" when used with NC or VB.
  • Type: String
  • Default: None
Y
Host♣ Name of the NC/ESX host that is to be enabled or disabled.
  • Type: String
  • Default: None
Y
Partition♣ Partition associated with the cluster in which NC/ESX host is located.
  • Type: String
  • Default: None
Y

Successful response to DisableService indicates that the node controller has been placed into maitenance mode. It will remain there until the administrator issues EnableService request for the host.

Response Elements (from CLC)
Name Description Required
return♣ Request accepted if false and error if true
  • Type: Boolean
  • Default: None
Yes

Successful response from the CC will also include resource availability that resulted from the request to enable or disable.

Response Elements (from CC)
Name Description Required
return♣ Request accepted if false and error if true
  • Type: Boolean
  • Default: None
Yes
resources Resource matrix (max and available for each VM type) after enabling or disabling of the node has been accounted for.
  • Type: ccResourceType (see eucalyptus_cc.wsdl)
  • Default: None
Yes

NC/VB operations

NC/VB impelements two operations related to maitenance mode:

ncMigrateInstances

Migration of an instance from one hypervisor host to another is accomplished through a sequence of ncMigrateInstances requests issued to the source and destination NCs with different action parameters. The sequence is as follows:

  1. Migrate request to the source with action set to prepare and a set of all instances to be migrated as part of the user's request. The intention of this request is, at the very least, to permanently record the migration plan so that it can be implemented even if higher-level components restart or failover and lose state. Additionally, source NC may perform checks or any other activities that necessarily precede preparations on the destination. Once ready to send an instance, the source NC sets its migration state to ready.
  2. Migrate request to each destination with action set to prepare and a set of instances destined for that NC. Using instance metadata and source node credetials included in the request, the NC performs everything necessary to receive an incoming migration: sets up the necessary directory structure, network devices, block devices, authorizes the anticipated incoming migration. Once cready to receive an instance, the destination NC sets its migration state to ready.
  3. Migrate request to the source with action set to commit and the instance to migrate. This triggers the actual migration.
In addition to driving the successful sequence, CC will detect error conditions and will request cleanup (rollback) from source or destination NCs by using action set to rollback. Roughly, if destination fails to prepare, cancellation (rollback) is sent to the source, while failure to migrate by the source leads to a cancellation sent to the destination.
Request Parameters
Name Description Required
instance.n♣ Set of one or more instances to migrate. This data structure contains all instance metadata, as reported in ncDescribeInstances, augmented with the three parameters specific to migration (source, destination, and state), listed below.
  • Type: instanceType (see eucalyptus_nc.wsdl)
  • Default: None
Yes
action♣ Specific migration-related action to perform.
  • Type: String
  • Default: None
  • Valid values: prepare, commit, rollback
Yes
credentials♣ String encoding the credentials necessary for the source to initiate a migration to the destination. This can be, for example, the public key of the source host.
  • Type: String
  • Default: Empty
Yes

Successful response indicates that the request is valid. Additionally, a response to prepare request to source indicates that the migration plan has been persisted to disk. Success of the actual operation can be determined asynchronously, based on the migration state reported in ncDescribeInstances:

  • preparing -> ready
  • migrating -> cleaning or none
Response Elements
Name Description Required
return♣ Request accepted if false and error if true
  • Type: Boolean
  • Default: None
Yes

ncModifyNode

Requests to disable or enable a node controller are both implemented on the NC/VB as a change of state requested with ncModifyNode operation.

Request Parameters
Name Description Required
stateName♣ The name of the state to set the Node Controller into.
  • Type: String
  • Default: None
  • Valid values: enabled, disabled
Y

Successful response to ncModifyNode indicates that the node controller has changed state and the latest state has been persisted to disk. NC/VB maintains its state across NC/VB service restarts and host reboots.

Response Elements
Name Description Required
return♣ Request accepted if false and error if true
  • Type: Boolean
  • Default: None
Yes

Migration states

Migration status is reflected in ncDescribeInstance and ccDescribeInstances responses. Within instance metadata the fields related to migration are:

Name Type Description
migrationSource♣ String Host name of the migration source NC/ESX host.
migrationDestination♣ String Host name of the migration destination NC/ESX host.
migrationState♣ String State of the migration on the NC that is responding or state as reported by the CC. The set of allowed state names is enumerated below.

The set of migration states reported by the CC to the CLC is smaller than the set of states reported by the NC/VB to the CC.

Migration States
Name Component Description
none NC/VB & CC No current migration-related activities on the host
preparing NC/VB & CC Source or destination is preparing for migration
ready NC/VB Source or destination is ready (to send or receive) a VM
migrating NC/VB & CC Source or destination is in the process of migration
cleaning NC/VB Source of the migration is cleaning up after the migration

NC-local States

The following diagram depicts the state machine for migration as it exists local to a single NC which participates in the migration. There are two participants (1) source and (2) destination whose states are indicated as appropriate.

CC States

From the perspective of the CC a migration's state is contributed by two NCs and combine to define a single state machine which determines the progress of the migration. The following diagram has this state machine.

Node to Node Communication and Authentication

  • NCs should mutually authenticate each other before performing a migration
  • credentials should
    • be unique to a migration
    • uniquely identify an NC
    • provide limited access to an NC
    • have a limited lifetime
    • be valid only for the duration of a migration
  • inter-NCs communications should be encrypted

Misc

  • terminate/run mode for instance store is out; no loss of in memory state
  • admin command for quarantine through CLC
  • RHEV & VMware seem to be the low hanging fruit functionally
  • KVM, Xen reliability are a major question
  • is there a way to pause the vm
  • vm maintenance mode
  • compromised
  • what are the benefits?
  • controlling workload schedule
  • slow down the system
  • live migration with attached volumes
tag:rls-3.3 tag:maintenance-mode
Clone this wiki locally