Autograding - How to assign a custom score based on the number of passes for autograding within a test for the auto grading reporter

[hongbyungwoo]

Hi everyone,

I have a question regarding the Autograding in GitHub. I would like to know how I could assign a custom value to the test score in the Test runner summary at using Autograding. For a single test, we could set a maximum score and the test score will be given zero or all depending on the failure or success. Even though there are several failures within a single test, I would like to assign a custom score value to the Test Score in the Test runner summary table. One can test a single function with varying input using DDP packages, but there is only two choices, zero or the maximum score, for the test.

It would be appreciated if anyone could give advices on this matter!

[maddie0501]

Hey GitHub Classroom’s Autograding currently supports only binary scoring per test either full points or zero, based on pass/fail. It doesn’t natively support partial scoring within a single test (e.g., assigning custom scores based on how many sub-cases pass).

You can try to split the test into multiple smaller tests, each with its own score. This gives more granular control.

Use a custom script to calculate partial scores, then print results in a format for manual review—though these won’t reflect in the official summary table.

[hongbyungwoo]

Thank you for your kind response. It would be greatly appreciated if you could advice how to deal with this problem.

Let us assume that the assignment is given to build a classifier and the evaluation is performed based on the accuracy.

For example, if the output accuracy is 85.5, then how can we build a test code to assign 85.5 as the test score in the autograding summary table and the csv file to download.

It does not seem to be reasonable to write for example 100 test codes for a varying ranges of thresholds and check for each threshold value to add one point.

Thank you for your help in advance!

[maddie0501]

You're right — writing 100 separate tests for each threshold is not practical. Instead, calculate the accuracy in a single test script, then output the score to a file (like score.txt). While GitHub Autograding can't show that score directly in the summary, you can download the file for manual grading or aggregate scores later.

[hongbyungwoo]

Thank you for your kind advice. As far as I know, any saved files by the test codes within a submission repository will be deleted at the termination of all the autograding procedures. Thus, the saved score to a file (like score.txt) will be deleted at the termination of all the autograding procedures.

I am wondering if there is any way to load the saved file (e.g. score.txt) and apply the stored point value and the maximum point value to the test score in the summary table and the css file in the classroom.yml.