Procedure

The ranking of tools is determined shortly after the live competition. Tools with incorrect results (either observed during the live competition due to a YES/NO conflict, or communicated afterwards by the tool authors to the SC) are excluded from the ranking (and the affected answers are marked as ERRONEOUS). A few weeks after each live competition there is a full run of tools on all eligible problems in the COPS database. Authors of tools with incorrect results have the possibility to submit a corrected version for the full run. The tools versions for the full run will be incorporated into CoCoWeb.