Skip to content
This repository has been archived by the owner on Aug 30, 2022. It is now read-only.

Rag Autograder Documentation

Jump to bottom
Sam Joseph edited this page Jul 6, 2017 · 2 revisions

#Using the Autograder: RAG provides autograding as a service. Set up the grader payload correctly for your assignment, and our RAG instance will handle grading student submissions for you. RAG provides a flexible variety of grading engines, allows you to set arbitrary late penalties, gives students feedback on what tests they failed, and allows you to create your own assignments and can use.

The grader payload

    <problem>
  <coderesponse queuename="cs169-rev-development">   <!-- queue name should match the name of queue you asked edX to create in step 1 above -->
    <filesubmission points="100"/>                   <!-- points should be consistent with 'score:' property of JSON postback in step 2 above -->
    <codeparam>
      <grader_payload>
      {"assignment_name": "assignment3",
      "assignment_spec_uri": "git@github.com:saasbook/hw-bdd-cucumber-ci.git",
      "autograder_type": "HW3Grader",
	  "due_dates": {"2100-01-02": 1.0, "2100-01-03": 0.75, "2100-01-04": 0.50, "2100-01-05": 0.25}      }
      </grader_payload> <!-- arbitrary string passed to autograder; could be a JSON object; we recommend
                                                           using it to identify the assignment and/or problem being submitted -->
    </codeparam>
  </coderesponse>
</problem>

This is an example autograder payload used for CS169x, for HW3. To set up an assignment to use a grader payload, you will need to create a submission of this format using the advanced XML editor in edX Studio. The XML layout should conform to this format. The following sections will cover how to configure the grader_payload to work for your course.

Assignment name:

This is a name that identifies the name of the assignment.

#Setting up/ Modifying Autograder Operation:

  • Currently autograder is deployed to Amazon EC2 on the saasbook account
  • The instance is Aaron edXMOOCGrader Summer 15 Dev. Queue = cs169-rev-development, which uses the ssh key RAG-Summer-15.pem. Armando Fox, Chick Markley, Sam Joseph, Aaron Zhang have this key. aaron.zhang@berkeley.edu
  • There are two instances of the autograder that serve two edX XQueues each running in a detached tmux window.
    • The first one is in the tmux session autograder which serves the queue cs169-rev-development and is meant for development
    • The second one is the tmux session rag-production-console which serves the queue cs169-rev-production and is meant for production grading.
  • The main point of entry to rag is the run_autograder.rb file, which requires necessary files and starts an autograder according to the config file.
  • The autograder meant to be crash resilient but some badly formatted submissions will cause it to crash. * Therefore, we run it in a bash loop so that it will restart on failure. Each time a crash happens logs are flushed to disk in logs/ for debugging.
  • Command to run autograder: while true; do bundle exec ruby run_autograder.rb /home/ubuntu/rag_production/config/config.yml;

Set up EC2 Instance:

  • To set up EC2 instance, boot an Ubuntu AMI. We use ubuntu/images/ebs/ubuntu-trusty-14.04-amd64-server-20140416.1 - ami-018c9568
  • Copy this script https://github.com/saasbook/rag/blob/master/ubuntu-install.sh to ~/, and then run it to set up dependencies
  • For solutions in private repos you will need an ssh key so the autograder can pull them. This can be placed in ~/.ssh/id_rsa.

Start/Restart the Autograder:

  • If you need to restart the autograder for whatever reason, go into to the tmux session, kill the current process, and start it again. You can then detach it again to allow it to run in the background :
ubuntu@ip-10-81-170-195:~$ tmux list-sessions
3: 1 windows (created Fri Oct  2 17:57:10 2015) [178x43]
4: 1 windows (created Fri Oct  2 18:00:50 2015) [178x43]
autograder: 1 windows (created Tue Sep 22 02:55:24 2015) [178x43]
rag-production-console: 1 windows (created Fri Oct  2 17:54:59 2015) [143x39]
$ tmux attach -t rag-production-console 
$ ^C # kill current autograder, if running
$ while true; do bundle exec ruby run_autograder.rb /home/ubuntu/rag_production/config/config.yml; done # run the autograder in a bash loop

Autograder Logs

Logs will only be saved to disk if the value log_value: true is set in the config file, otherwise log contents are printed to the console (STDOUT stream). Under the top level directory ~/rag/, the folder logs/ will store all logs from the autograder. Logs are timestamped with system time of when the autograder was started. Logging can be set with severity level, 0 for debug level logging , 1 for error level logging. Levels above that won't produce any logs.

Autograder Rubrics

Solution rubrics are stored in the rubrics/ folder. The grader payload has a field for the URI of where the solutions can be found, which will accept .git repositories and direct download links. Solution rubrics are for all graders with the exception of the HW4 Grader .rb RSpec file, which the autograder will use to score student files. Rubrics will stay cached with the autograder for a day. For documentation on how to write your own rubric see the wiki page for rubrics.

Autograder Submissions

All student submissions are stored in the submissions/ folder. Submissions are stored in the following way: <one_way_edX_student_hash>/<assignment_name>/<assignment_time>/. The contents of the actual submission will be stored inside that folder. If the submission is a zip file, the file directory is unzipped first before being placed into the assignment_time/ folder.

Autograder Queue Configuration: Autograder Troubleshooting: HW4:

Clone this wiki locally