Grading Scheme for RoboCup
Posted by kyokko on 2009/10/31
Team Grade: 40%
Performance of soccer team – 10%
- Fails to beat original team – 0/10
- Wins occasionally – 4/10
- Consistently wins by narrow margin – 8/10
- Consistently wins by large margin – 10/10
Report – 10%
Written report explaining everything that was accomplished, remaining problems, TODO for people who might be working with our code. Mark should reflect completeness, justifications and clarity of writing.
Documentation – 10%
Comments in code and design document.
Screencast Demo – 10%
Demo on YouTube or similar service showing a client running.
Individual Grades: 60%
- Participation – 10%
- Testing – 10%
- Completion of Tasks (Code Sprint, Individual Tasks) – 40%
Participation – 10%
- Writing requirements: blog posts, minutes, etc.
- Meeting attendance, communication skills, etc.
- Peer evaluations
Each team member had some tasks assigned. For this project it was hard to find completely non-overlapping problems for each member to work on, also tasks got reassigned occasionally for various reasons. Our individual grading schemes are taking that into account.
Testing/Performance – 10%
Note: RoboCup is a real-time running application, so for most of the functions it is impossible to create Unit tests. We use Debug UI and Human Controlled player to evaluate performance of the client.
Testing for people working on incorporating Machine Learning (ML) algorithms for high-level decision-making is a standard procedure which involves creating a test dataset and using it to assess how well the trained model generalizes to new data.
Tasks from code sprint weekend – 10%
- Ioana/Chani – Debug UI
- Patrik/Yulia – Human Controlled Player ver.1
- Alex – Java Monitor
Individual tasks – 30%
These are projects each person has been working on since Code Sprint. Since we all focused on different tasks, individual grading schemes reflect the amount of effort put into particular area.
- Basic Actions (-improvements to basic actions: kicking, dashing, turning, etc) – 15%
- Complex Actions (developing multi-tick actions, that are based on the basic actions and allow for more complex behaviour : dashing to position, following the ball, finding the ball when it hasn’t been seen for a while, etc) – 15%
- Sight Improvements (integration of better vision algorithm into the complex actions) – 15%
- ML (using Decision Trees (DT) or Neural Networks (NN) for better decision-making)- 15%
- ML (experiments with various parameters for DTs and bugfixes to parser) – 15%
- Actions (debugging and improving basic actions and multi-tick actions, in particular better passing) – 15%
- ML testing and experiments (creating dataset for evaluation of ML algorithm’s performance, various experiments with different training data and parameters) – 15%
- ML infrastructure (parser for server logs, Human Client ver. 2, script to feed data to ML algorithm) – 15%
- Basic actions (debugging and improving algorithms for basic actions : kicking, turning, seeing, etc) – 15%
- Complex actions (multi-tick actions, that are composed of basic actions and enable more complex behaviour: interception, dribbling to a position, chasing the ball, etc) – 15%