Bot Problems about org HOT 4 OPEN

Bot Races are difficult to judge

We have reached a point where it's almost impossible to manually judge each report fairly. Judges have only 24 hours to assess 20 reports, some of which exceed 100k lines each (!).

Recommended Solution

• Reduce report length by dropping some categories (e.g. NC and R)
• Reduce the number of available spots as 20 is too many

Bot Races abuse "farming" issues

Currently, each variant of an issue can be "farmed" to get more points. Some concrete examples include:

• Missing NatSpec @author
• Missing NatSpec @title
• Missing NatSpec ...

This floods the report with a lot of issues, it's unfair as this behavior gets more points, and increases judging efforts

Recommended Solution

Issues that share the same root cause, and are split, should be penalized instead of awarded.

NC/R issues do not provide value to the sponsor

We need a sponsor survey to prove this, but I don't think these issues are valuable to sponsors. Nevertheless, most NC/R issues are subjective (e.g., tests should always have 100% coverage), yet they still contribute points. These issues are very easy to add, so bot racers are more inclined to include them instead of focusing on security issues.

Recommended Solution

• NC/R categories should no longer be considered valid issues
  or
• NC/R categories should get a fraction of L issues point-wise (e.g. 1/10 or less)

from org.

Recommended Solution For Different Severity Definitions

I think bot racer should not be able to edit reports. There should be a c4 server that our bots are running and everyones report should be automatically generated & published without any changes by racer.

from org.

Advance Knowledge of Bot Race Judge

Because of their role at Code4rena, some, but not all, bot racers know in advance who will be the judge for a bot race. Because there is a great deal of variance in how bot races are judged, and because recognizable judging patterns have arisen over time, the knowledge of who will be the judge in advance of the race represents an unfair advantage. A bot racer that knows in advance who the judge will be can tailor their report to receive a better grade by that judge. Bot racers that do not know who will judge the race are left at a disadvantage to the racers that do. (This issue was raised in #147.)

Recommended Solution

To level the playing field, one of the following options should be implemented:

1. No bot racer should be allowed to know in advance who will judge a race.
2. The judge for a race is known in advance to all bot racers at the same time. No bot racer would know before any other.

from org.

Bot Race Judging is Inconsistent

As there are no standardized guidelines for bot race judging, each bot race judge makes up and follows their own rules for each race. In addition, a given judge may change their judging process from one race to the next. There is not always full transparency in how bot reports are evaluated.

Recommended Solution

Bot race judging standards, designed to maximize the value of a bot race report to the sponsor, should be developed and implemented. The standards should accommodate a mostly automated judging process, allowing for human review that can more easily, effectively, and fairly be done in the short allotted time frame for judging.

from org.