modal doesn't support fixing CPU so we can't actually do code execution there. we can do everything else (download test data, run tests simultaneously, grading, etc) there though.

For USACO problems that have file I/O, we should both pipe the input into standard input and have an input file. If an output file is detected, use that. Otherwise, use standard output.

One user reported seeing this error. I think it's possible that a long-lasting lambda function might run out of space. Either way, probably not very high priority

never mind, I'm also seeing this error, no clue why

Remaining Tasks

• Write tests
• Figure out how to handle large input files (robo cow herd is 10MB which exceeds lambda limit)

Goal

Add support for submitting and grading code for entire problems (ex. USACO problems), to be used by things like USACO Guide Groups.

Rough guidelines

Execute Lambda

https://github.com/cpinitiative/online-judge/tree/serverless/execute

• Truncate large standard inputs / outputs (aws lambda has a 6mb limit; see below)
• Return a hash of the output. Make sure to normalize whitespace (ie we don't want to return WA if the user is off by whitespace only).

Submit Lambda

https://github.com/cpinitiative/online-judge/tree/serverless/submit

• Distinguish between three types of requests: Create a new problem submission (POST), get a problem submission status (GET), and create an execution request for a single test case like ide.usaco.guide (POST).
• When a new problem submission is created, create a new DynamoDB row and return the row ID. Then, use one execution lambda request to compile the code. Fetch all the test cases from S3 somehow. For every test case, call another execution lambda. Then, compare the execution lambda output hash with the expected output hash to determine whether the answer is correct or not. Also check for memory usage. Update the dynamodb row as needed
• For get problem submission status requests, just fetch data from DynamoDB and return it

Things to keep in mind

• AWS Lambda has a 6mb output size limit, and some USACO problems exceed that, so we'll need to use hashing to check whether an output is correct or not.
• DynamoDB is very expensive and charges by size. We should avoid storing large text like input / output / etc; if we ever need to do this, we should use S3.
• However, we do want to store source code, compilation errors, and small test case outputs for USACO Guide groups (to help users debug their code on small cases more easily). We should put a limit on how big the source code can be (64kb is what Codeforces uses), and use gzip to compress any text we store.

https://ide.usaco.guide/Mpmo86nPK_P2chy166K?lang=cpp

Steps to reproduce:

1. Write some C++ code that asserts false
2. A core dump file is created in /tmp and is never deleted
3. eventually /tmp runs out of space and the lambda fails all future requests

especially bits/stdc++.h so we can speed up compile times

Statistics like https://newjudge0.usaco.guide/statistics

Right now for code executions, if status is compile error, the compilationMessage isn't set but stderr is. We should set compilationMessage even for compile errors for consistency

CSES Subordinates:

AC Code

#include "bits/stdc++.h"
 
int dfs(int node, std::vector<std::vector<int>>& a, std::vector<int>& ans) {
    int temp = 0;
    for(auto& i : a[node]) temp += dfs(i, a, ans);
    return ans[node] = 1 + temp;
}
 
signed main() {
    std::ios::sync_with_stdio(false); std::cin.tie(nullptr); std::cout.tie(nullptr);
    int n; std::cin >> n;
    std::vector<std::vector<int>> a(n + 1);
    std::vector<int> ans(n + 1, 0);
    for(int i = 2, x; i <= n; ++i) {
        std::cin >> x;
        a[x].push_back(i);
    }
    dfs(1, a, ans);
 
    for(int i = 1; i <= n; ++i) std::cout << ans[i] - 1 << " ";
}

gives RTE on our grader. I think this is because our stack size is too small. Fix with ulimit -s unlimited: https://stackoverflow.com/questions/14471564/what-does-ulimit-s-unlimited-do

Some test cases can have multiple outputs such as https://cses.fi/problemset/task/1640. Currently, I believe that the judge doesn't validate dynamic outputs looking at the source code:

Also, submitting to it on one of my USACO Guide Groups gives me partial WA while CSES gives me all AC.

Goal

To create a low-cost, reliable, fast, and consistent online judge that supports C++, Java, and Python.

See serverless branch for more information

https://github.com/cpinitiative/online-judge#api

Use https://github.com/cpinitiative/online-judge/blob/submit/submit/src/types.ts as a starting point

At least for POST /execute, you can probably base it off of the unit tests.