This project contains the files that can be used for Stress Testing the C++ solution, i.e., to find a test case in which the main solution fails.
This tester is divided into three types, for testing three different category of problems:
- Normal tester: To compare the main solution with a correct brute-force solution.
- Custom tester: To validate a solution with more than one correct output.
- Interactive tester: To test the interactive problems.
Note:
-
To directly execute the script from the terminal, add the file
stress-test.shto the PATH variable, or move it to/usr/bin/directory, by executing the following command from the root of this directory:sudo cp stress-test.sh /usr/bin/
-
To directly run the stress-tester from an IDE (e.g. Sublime Text), modify the build file of the IDE by adding the commands to execute this script. For Sublime Text, the following lines shall be added to the build file: (Example of C++ Build File)
{ "variants": [ { "name": "Stress Tester", "shell": false, "cmd": ["bash", "-c", "gnome-terminal -- bash -c 'stress-test.sh \"${file}\" normal 500 2; read'"] }, { "name": "Custom Tester", "shell": false, "cmd": ["bash", "-c", "gnome-terminal -- bash -c 'stress-test.sh \"${file}\" custom 500 2; read'"] }, { "name": "Interactive Tester", "shell": false, "cmd": ["bash", "-c", "gnome-terminal -- bash -c 'stress-test.sh \"${file}\" interactive 500 2; read'"] }, ] }
./stress-test.sh <file.cpp> ...[optional_parameters]...
All the cpp files should be present in the same directory
Optional Parameters:
These parameters must follow the order given:
<testing_type> Must be either 'normal', 'custom' or 'interactive'.
normal: Testing the normal problems
custom: Testing the solutions which require custom tester
interactive: Testing the interactive problems
[DEFAULT]: normal
<no_of_testcases> Total number of testcases to check.
[DEFAULT]: 500
<timeout> Timeout of each execution (given in seconds)
[DEFAULT]: 2 (2 seconds)
Examples:
./stress-test.sh main.cpp./stress-test.sh main.cpp custom 50./stress-test.sh main.cpp interactive 100 1
Make sure that the file stress.sh is executable. If it isn't, then run sudo chmod +x stress-test.sh to make it executable.
The description is messed up, feel free to contribute and improve it :)
For all the below examples, the main solution has the filename: main.cpp.
The extra files required for testing are named as 'main file' + '__' + '[type].cpp', as described below. Note that __ denotes two underscores.
Usage: ./stress-test.sh main.cpp ...[optional parameters]...
The following files are required:
main.cpp
- The main solution which the user wants to validate.
main__Good.cpp
- The correct, brute force solution, for comparing the final output.
main__Generator.cpp
- The file to generate random test cases.
- Output the test case to the standard output stream (
stdout) usingcout.
The following files are required:
main.cpp
- The main solution which the user wants to validate.
main__Checker.cpp
- The checker to validate a solution with more than one correct output.
- Take input from the standard input stream (
stdin) usingcinin the following order:- The first input is the testcase produced by the generator:
main__Generator.cpp. - The second input is the answer produced by the main solution file:
main.cpp.
- The first input is the testcase produced by the generator:
- Give verdict, such as
Expected [...] found [...]to the standard error stream (stderr) usingcerr. - Finally,
return 0in case of SUCCESS, orreturn 1in case of FAILURE.
main__Generator.cpp
- The file to generate random test cases.
- Output the test case to the standard output stream using
cout.
The following files are required:
main.cpp
- The main solution which the user wants to validate.
main__Generator.cpp
- This file acts as a generator as well as an interactor (to generate random test case and to interact with the main solution).
- Use
cinandcoutfor interaction (cin- take user's query,cout- give back response). For printing anything to the console, or for giving verdict, usecerr. - First, generate a random test case - e.g. the number to guess, and store it in some container.
- Then loop until the number of queries exceed the allowed query, or the user guesses the correct answer.
- Now, inside the loop, first take the output produced by the main solution from the standard input stream using
cin. - Then, give back the response to the standard output stream using
cout. - After every interaction, check:
- Whether the number of questions asked is within limits.
- Whether the response of the user is correct or not.
- In case of any error, output it using
cerrand simultaneouslyreturn 1. - In case the answer is correct, give appropriate verdict using
cerrandreturn 0.
For debugging purposes,
- Output every user's query to the standard error stream using
cerr, (i.e., for everycin, also output the values tocerr). - Print the correct answer at the end using
cerrbefore returning 0 or 1.
In case of interactor, make sure to flush the output by using endl, and not \n.
main.cpp
#include <bits/stdc++.h>
using namespace std;
int main() {
int n;
cin >> n;
int answer;
// Write the main solution here
cout << answer << endl;
}main__Good.cpp
#include <bits/stdc++.h>
using namespace std;
int main() {
int n;
cin >> n;
int answer;
// Write the brute-force correct solution here
cout << answer << endl;
}main__Generator.cpp
#include <bits/stdc++.h>
using namespace std;
mt19937 rng(chrono::steady_clock::now().time_since_epoch().count());
int rand(int l, int r){
uniform_int_distribution<int> uid(l, r);
return uid(rng);
}
int main() {
// Generate random test case
int testcase = rand(1, 100);
// Output the random test case
cout << testcase << endl;
}Getting the current testcase in the Generator (to generate fixed test cases, if required):
#include <bits/stdc++.h>
using namespace std;
// Add the argc and argv in the main function to accept the arguments
int main(int argc, char **argv){
// The testcase is passed as the first argument to the generator (argv[1])
// Testcases starts from 0 upto (TOTAL_TESTCASE - 1)
// Convert the argument to an integer
int test = atoi(argv[1]);
// Generate a fixed testcase according to the testcase number
cout << test << " " << test * 2 << endl;
}main__Checker.cpp
#include <bits/stdc++.h>
using namespace std;
int main() {
// Take the testcase as input
int testcase;
cin >> testcase;
// Take the answer produced by main solution as input
int answer;
cin >> answer;
// Check whether the answer produced satisfies the test case
int expected_answer;
bool is_correct = false;
// ...
if (is_correct) {
// Output the verdict to stderr
cerr << endl;
cerr << "Expected " << expected_answer << ", found " << answer << endl;
return 1; // return 1 in case of FAILURE
}
return 0; // return 0 in case of SUCCESS
}main__Interactor.cpp
#include <bits/stdc++.h>
using namespace std;
mt19937 rng(chrono::steady_clock::now().time_since_epoch().count());
int rand(int l, int r){
uniform_int_distribution<int> uid(l, r);
return uid(rng);
}
// ALWAYS USE `endl` FOR FLUSHING THE OUTPUT, NOT `\n`
int main() {
// Generate random test case
int answer = rand(1,100);
int asked_questions = 0;
while (true) {
char user_response_type;
int response;
// Take the user's response as input
cin >> user_response_type >> response;
// Outputting this to stderr for debugging purposes
cerr << user_response_type << " " << response << endl;
asked_questions++;
if (asked_questions > 100) {
cerr << "Too many questions" << endl; // Outputting every verdict to stderr
return 1; // return 1 in case of FAILURE
}
if (user_response_type == '?') {
// Give proper response to stdout
// FOR INTERACTING WITH USER, GIVE THE RESPONSE USING `cout`, FOR GIVING VERDICT, USE `cerr`
int interactor_response;
// ...
cout << interactor_response << endl;
cerr << interactor_response << endl; // Outputting this to stderr for debugging purposes
} else if (user_response_type == '!') {
if (response == answer) {
cerr << "Correct solution" << endl;
break;
} else {
cerr << "Incorrect solution." << endl;
return 1; // return 1 in case of FAILURE
}
}
}
// Output the correct answer and return 0 (SUCCESS)
cerr << "Correct Answer: " << answer << endl;
return 0;
}sample/: Folder contains some sample interactors.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent