Experiment Checklists

[jkbhagatio]

This is divided into three main sections ("Pre-Experiment Checklist", "Experiment workflow start-up checklist", "In-Experiment Troubleshooting Tips") each with subsections. I think from previous discussion we agreed that all this information should go into the readme of this repo?

Please look this over and add any additions / corrections. In particular we still need to fill out the "Experiment workflow start-up checklist".

Each checklist item roughly follows this structure:

• Checklist Item Name
  • Details (optional):
  • Fix (optional):

Pre-Experiment Checklist

General arena and system checks

• Arena cleaned
  • Details: Remove tiles and walls; clean honeycomb grid; clean tiles both sides; clean walls both sides; replace tiles and walls (with clips) - no gaps between tiles
• Pose model trained: model has high accuracy for given arena
• Pose model path updated in experiment desktop shortcut
• Desktop shortcuts are up-to-date: SystemStartup, LightServer, Robocopy, <exp_workflow>
  • check with Goncalo and Andre
• Latest commit on experiment branch tagged on GitHub; repo root (aeon_experiments/) is clean
  • Details:
    • Navigate to the 'aeon_experiments' directory: cd /c/ProjectAeon/aeon_experiments
    • Checkout the experiment branch: e.g. git checkout social
      • Note: Bonsai and its dependencies may get reinstalled at this step, may take a few minutes.
    • Run git pull
    • Then run git log --oneline -n 1 and compare the commit returned here with the commit of the latest release of the experiment name branch here: https://github.com/SainsburyWellcomeCentre/aeon_experiments/tags. These commits should match. Additionally, run git status in the terminal: if there are any files found, delete them.
  • Fix: If, after pulling, the latest local commit does not match the commit of the latest experiment branch release on GitHub, then create a new tag on GitHub under the experiment branch name: https://github.com/SainsburyWellcomeCentre/aeon_experiments/releases/new. When creating the new tag, make sure to click "Generate Release Notes." Subsequently, git pull again locally on the acquisition PC.

Device checks

• Ceiling light luminance checked
  • Details: For every luminance level that may be used in the experiment, check the lux in the center of the arena using a luxmeter and the <lighttest_lux2lux> workflow, and check the intended_lux : actual_lux pair values match for each setting
  • Fix: If the lux setting is off from the lux reading by >10%, for any luminance level that may be used in the experiment, run the <lighttest_bit2lux> workflow and update the corresponding calibration file on ceph/aeon/aeon/config
• Infrared lights checked
  • Details: Ensure all infrared LED strips are turned and there is X IR Lux at the center of the arena
  • Fix 1: Check connection between strips and to power supply
  • Fix 2: Replace LED strip(s)
• Patch Wheels checked
  • Details: Check wheel start torque with blue tack and weights, if not equal within 10% tolerance...
  • Fix 1: unscrew and re-screw four screws that hold encoder box in place
  • Fix 2: fully disassemble and reassemble
• Patch Feeders checked
  • Details: Fill feeders with specified pellet count; Place pellet collector under each feeder on arena floor; Test "Reset" and "Deliver" GUI buttons (each need to be clicked twice for each patch, and "Reset" value needs to be Reset to a non-zero pellet count); pre-load the exposed pellet pockets of each wheel with pellets.
  • Fix 1: Restart workflow
  • Fix 2: Clean hopper using compressed air and any small thin tool to remove debris; align pin of feeder disk to hopper (being careful not to overtighten)
• RFIDs checked
  • Details: Test all RFIDs with the rfid_test workflow by manually moving a test chip along all locations: ensure there is no audible beep, but that rfid value appears in GUI
  • Fix 1: Check cabling and restart workflow
  • Fix 2: Replace antennas and replace PCBs
• Cameras checked
  • Details: Ensure all cameras are being recognized, triggered, and frames acquired with <cameratest> workflow and saved video files. Ensure all cameras are appropriately zoomed and focused: top view cam should see entire arena including corridors with a few pixels to spare on edges; patch cams should be able to see the full mouse body (including full tail, for ID purposes) while it is foraging.
  • Fix 1: Restart the workflow
  • Fix 2: Close the workflow, powercycle the cameras, check the cameras are all found by the PC via Spinview, restart the workflow
  • Fix 3: Ensure field of view is aligned with the centre of each patch
• Nest checked
  • Details: Add bedding, water, and enrichment (brick, balls); Check positioning and tare scale, if readings are unstable...
  • Fix: Level the scale

Experiment workflow start-up checklist

Pre experiment workflow launch

• If "new" epoch: All .tmp files (within workflow folder: e.g. /c/ProjectAeon/aeon_experiments/workflows/<exp_name>) deleted
• Storage space on D drive checked: delete contents of D/Tests/ folder
• SystemStartup and LightServer workflows checked
  • Details: both should be running (see if terminal windows are open with their names), if not, launch them from desktop shortcuts
  • Note: Ensure that there is a folder D:/ProjectAeon, if not, create this folder (this is the main data folder that robocopy looks at to copy over to ceph)
• Launch experiment workflow

Post experiment workflow launch

• Environment config yml file loaded -> Check the C:\ProjectAeon\aeon_experiments\workflows\social\Environmentsdirectory for theEnvironmentRandom.ymlandEnvironmentRandomHard.yml` - if there, click the "Reload Environment" button in the workflow GUI
• Cameras started (Shift + F1)
• Harp clock sync accuracy checked
  • Details: Check timestamps of saved files in D/ProjectAeon/<exp_name>/<acq_epoch>/
  • Fix: Restart PC
• Maintenance started; "test" subject added; Maintenance stopped
• "Deliver" button tested, for each patch
• "Reset" button tested, for each patch
• Pellet delivery via wheel spinning to threshold tested, for each patch
• Visible wheel pockets filled with pellets, for each patch
• Maintenance started; "test" subject removed
• Subjects weighed and physically placed in the nest in the arena (verify ids with RFID wand)
• Subjects added in GUI with weights; Maintenance stopped
• Comment added to pyrat for each subject specifying experiment name
• Add to Dario's spreadsheet (https://www.dropbox.com/scl/fi/lndzbtkmyywdyj3zrohd8/Experiment0.1_Dario_new.xlsx?dl=0&rlkey=651bg36nppevlp4dowrtwpff5)

In-Experiment Troubleshooting Tips

• General issue or strange occurrence
  - Check all cables are plugged in correctly, especially to and from expander boards
  • Details: Whenever any issue or strange occurrence, create an annotation in the experiment GUI. If any intervention (virtual or physical) is required, start maintenance, and then take appropriate steps to solve the issue. Even for physical interventions, avoid removing subject(s) unless absolutely necessary. If an issue happens during our evening (after light cycle goes light), we should just try and fix it before the next morning (before light cycle goes dark)
• PC restarted check
  • Details: After a PC restart, check all devices are online, esp. cameras and scale
  • Fix: If not all devices online, manually powercycle cameras and scale
• Tile wall-holder breaks
  • Details: Rarely, one or both of the wall-holders on the bottom of the tiles may break off.
  • Fix: Replace with a replacement tile: these can be found on the b2 rack with wall-holders already chloroformed on
• Ceiling light issues
  • Details: If ceiling lights are not obeying the intended light-cycle...
  • Fix 1: Check that the experiment workflow is reading from the appropriate <lightcycle>.config file.
  • Fix 2: Restart LightServer workflow
  • Fix 3: Restart experiment workflow
• Infrared light issues
  • Details: If a bulb goes out...
  • Fix: Manually check that the LED strips are still connected and the power supply is on and connected
• Camera issues
  • Details: If the main top overhead or any of the patch cameras stop working during the experiment...
  • Fix 1: Restart the workflow
  • Fix 2: Close the workflow, powercycle the cameras, check the cameras are all found by the PC via Spinview, restart the workflow
• Patch Feeder issues
  • Details: If the pellet beambreak or delivery signals are repeated consecutively, it could be that pellets are not actually dropping onto the wheel, multiple pellets are being delivered at a time, or the beambreak is acting extremely sensitively...
  • Fix 1: Try clicking the "Reset" Patch GUI button - you will need to specify a number of non-zero pellets to "reset to", and this button may need to be pressed twice. Look for proper pellet delivery and a delivery signal followed by a beambreak signal.
  • Fix 2: Manually check the patch: check the feeder and wheel for clogging; use a tool or compressed air to unclog; check and if necessary screw the pin of the disk of the feeder to properly align flush to the hopper (but be careful not to overtighten)
• Patch Wheel issues
  • Details: Wheel appears stuck when the animal is trying to move it
  • Fix: Manually unstick the wheel with applied force
• Nest issues
  • Details: Nest overhead camera view may become occluded by bedding and crap...
  • Fix: Manually readjust bedding and crap in nest
• Pose model issues:
  • Details: If the top camera starts lagging, or the model starts performing poorly or not at all...
  • Fix 1: Increase the frame-skip count for model inference (e.g. increase to every 3rd frame from every 2nd frame)
  • Fix 2: Check and if required update the pose model path
  • Fix 3: Restart the workflow / PC>)
• Robocopy issues:
  • Details: Robocopy stops copying files: we get an alert or notice in the workflow terminal or notice on ceph that files are not being copied over-
  • Fix: Launch Robocopy from desktop shortcut, NOT SystemStartup (we don't want to run harp sync again)
• On hard workflow / PC crashes:
  • Fix: Relaunch workflow following each step in "Experiment Workflow start-up checklist"
• BeyondTrust issues
  • Client console becomes unresponsive: "could not locate existing support session" error message appears when trying to connect
  • Fix: Reinstall client on acquisition computer in-person:
    • https://portal.swc.ucl.ac.uk/login/downloads -> "Jump Clients" Tab ->
    • "Aeon Group" in "Jump Group" field, in "Name" field (e.g. AEON4), in "Comments" field, all other fields can be left blank
    • Click "Create"
    • Download the installer, and install it.