Organizer_Leaderboard

Leaderboard nuts and bolts

The Codalab leader board offers many features. It can be configured to display results on multiple datasets, show execution time, show rank on all scores, etc. etc. This page will help you out...

1. Leaderboard format.

In the YAML file, you can specify the leaderboard format. This is best explained via an example from the AutoML challenge:

leaderboard:
    leaderboards:
        Results: &RESULTS
            label: RESULTS                      # This will be the title of the table
            rank: 1
    columns:
        set1_score:                             # This corresponds to 1st value provided by the scoring program
            leaderboard: *RESULTS
            label: Set 1                        # This is the name of the column in the table
            numeric_format: 4                   # This is the number of decimals
            rank: 2                             # This is the number of the column (column 2)
        set2_score:                             # This corresponds to 2nd value provided by the scoring program
            leaderboard: *RESULTS
            label: Set 2
            numeric_format: 4
            rank: 3
        set3_score:                             # This corresponds to 3rd value provided by the scoring program
            leaderboard: *RESULTS
            label: Set 3
            numeric_format: 4
            rank: 4
        set4_score:                             # This corresponds to 4th value provided by the scoring program
            leaderboard: *RESULTS
            label: Set 4
            numeric_format: 4
            rank: 5
        set5_score:                             # This corresponds to 5th value provided by the scoring program
            leaderboard: *RESULTS
            label: Set 5
            numeric_format: 4
            rank: 6
        ave_score:                              # We will show the average of several scores
            leaderboard: *RESULTS
            label: <Rank>                       # This is the column name in the table
            numeric_format: 4
            rank: 1                             # This will be the 1st column
            computed:                           # These are the scores that will be averaged
                operation: Avg
                fields: set1_score, set2_score, set3_score, set4_score, set5_score
        Duration:                               # We will also display duration (in seconds)
            leaderboard: *RESULTS
            label: Duration 
            numeric_format: 2
            rank: 7

The leaderboard will display the results of the file score.txt, which is returned by the scoring program, and has the following format:

set1_score: 0.813519760093
set2_score: 0.745929379262
set3_score: 0.523620210384
set4_score: 0.793769274565
set5_score: 0.692495854273
Duration: 12

The leaderboard does not show all the decimals, it will limit itself to the number of decimals specified in the numeric_format.

User_DetailedResultsPage