giantpizzahead / judgelite Goto Github PK

Is your feature request related to a problem? Please describe.
Diff is a pretty good checker for most types of problems, but there are times when it sort of falls short. For example, any problem requiring an output of floating point numbers makes it hard to use diff. The ability to use custom checkers would be really helpful for these problems.

Describe the solution you'd like
Problem creators should be able to create a custom checker to check the output of the user's submitted program. This checker would be given the submission's output as its input, along with the test case's input and output files, in order to check if it is correct. This could probably be done by placing the input, output, and answer files in the same directory as the checker, then running the checker. It would also make sense to reuse the isolate code that compiles and runs a program to do this.

Additional context
This is not an easy issue to tackle, as it would require restructuring a good portion of the submission judge code. However, if you're feeling up for the challenge, then by all means, go ahead! (We will probably implement this sooner or later as well.)

Is your feature request related to a problem? Please describe.
Right now, the documentation for creating a problem is contained in the sample_problem_info directory. The 'documentation' isn't really documentation at all; in fact, it's pretty much just a set of tips.

Describe the solution you'd like
A more complete documentation of how to create problems would be helpful. It would probably be placed in the "Creating Problems" Wiki page.

Additional context
I'll probably write this documentation at some point in the future.

Self reminder: Update wiki / club website when this is added

Describe the bug
Disk quota is not enabled for isolate, only fsize is. That means that you could bypass the max file size limit by creating tons of small files, overloading the system.

To Reproduce

1. Write code that creates a whole bunch of small files
2. Submit it to JudgeLite
3. Make JudgeLite hang (if the target system doesn't have enough storage space)

Expected behavior
JudgeLite should not hang (ideally, disk quota should be enabled).

Additional context
This isn't too important of an issue to fix, since no one would do this by accident. Plus, who would even think to try this?
However, it's important enough to leave this note here, in case someone does know how to fix this. I don't remember why we disabled disk quota, but there's probably a reason. Oh well.

Describe the bug
Microsoft Azure seems to do some sort of throttling on VMs that have been inactive for a while. So, JudgeLite reports an incorrect memory used value (~30 mb too high) for the first submission after a period of inactivity.

To Reproduce

1. Go to HHS Programing Club's website
2. Submit something and look at the memory usage
3. Submit again and look at the lower memory usage.

Expected behavior
The memory used value should be accurate.

Additional context
This is probably Azure's fault. However, it could be countered by running a 'dummy' submission before the actual one after a period of inactivity.

Is your feature request related to a problem? Please describe.
Currently, JudgeLite is very easily DOS-able: Just submit your code a ton of times, and it will hang! Unfortunately, that's not really a good thing. The big issue is that a normal user may accidentally do this, by clicking the submit button too many times.

Describe the solution you'd like
JudgeLite should keep track of the usernames associated with submissions. When a new submission comes in, JudgeLite should only accept the new submission if it's been at least some number of seconds since that user last submitted code.

Describe alternatives you've considered
Another way to fix this could be to only allow new submissions once the old one has been completed.

Additional context
This could probably be implemented with the Redis database's EXPIRE command. If anyone wants to tackle this one, go ahead!

Is your feature request related to a problem? Please describe.
The webhook Wiki page needs to be filled in.

Describe the solution you'd like
There's already a pretty comprehensive description of what the webhook does in env_vars.py. It just needs a bit of flavor text / maybe an example, and the Wiki page would be complete!

Is your feature request related to a problem? Please describe.
Currently, there is a run_judgelite.sh script that pretty much automates the entire process of setting up JudgeLite on Linux. Sadly, I am a complete Powershell noob, so I have no idea how to translate that script to Windows.

Describe the solution you'd like
The run_judgelite.sh script should be translated into Powershell.

Additional context
Anyone with experience in Powershell (or willing to learn)... PLEASE HELP 😢