TMR @HOME 2024

Robot actions guide

By analyzing and cleaning the datasets generated for the fine-tuning, the actions required by the robot were defined as follows:

Action	Complement	Description	Navigation	Manipulation	Vision	HRI
go	place	Navigate to a place	✅	➖	➖	➖
find	person or object or object category	Find something	➖	✅	✅	➖
identify	characteristic + (person or object)	Determinate the object or person with the given characteristic and track it	➖	➖	✅	➖
count	characteristic + (person or object)	Determinate how many objects or persons posses the given characteristic	➖	✅	✅	➖
approach	person	Move near someone	✅	➖	✅	➖
interact	request	Process the request and communicate an interactive response. Doesn't expect follow-up	➖	➖	➖	✅
ask	request	Ask the required or expected information from the user	➖	➖	➖	✅
follow	person	Follow a person after announcing it	✅	➖	➖	✅
pick	object	Pick an object standing in a surface (table, container, etc.)	➖	✅	✅	➖
place	object	Place an object standing in a surface (table, container, etc.)	➖	✅	✅	➖
open	entrance	Open a door or a container (refrigerator, drawer)	➖	✅	➖	➖
close	entrance	Close a door or a container (refrigerator, drawer)	➖	✅	➖	➖
grab	object	Grab an object from the person in front	➖	✅	✅	➖
give	object	Give an object to the person in front	➖	✅	✅	➖
remember	something	Remember previous information from the previous action/location	➖	➖	➖	✅

List of possible characteristics:

• Clothing color (e.g. wearing blue t-shirt)
• Person's pose (standing, sitting, laying)
• Arm gestures (waving, pointing left, raising right hand, etc.)
• Object size (smallest, biggest)
• Object's weight (lightest)

This list is still tentative and in current development.