Some code cleaning, some std::cout <<, turn that into logging and add a .clang-format file for linting etc

Lots of bits of the tourniquet API currently take at least three parameters: a module_name, line, and col uniquely locating some source code feature. These should be abstracted behind a Location or similar dataclass, allowing us to replace three parameters with one.

Even more aggressively: We could have a Locator class, which can be derived from and supports concretize to produce a concrete Location.

This is wrong; it should intelligently default to C for inputs that end with .c, and C++ for inputs that end with the common C++ suffixes.

We should build binary wheels for macOS and Linux and distribute them via PyPI.

The dynamic linkage to clang might pose a problem, although auditwheel should be able to vendor the appropriate clang libraries. We'll see.

We should use SQLAlchemy to abstract the process of table creation, as well as replace the raw queries currently used to access the DB.

We should make sure that Tourniquet builds and functions correctly with Clang and LLVM 10.

Some other expressions we could consider:

• Function(): Like Variable(). The constructor could take additional concretization constraints, like an optional return type or parameter types.

• Call(Function(), Variable(), ..., Variable()) and/or Call(Lit("..."), Variable(), ..., Variable()).

• Asmt(Lhs, Expression()) for lhs = <expression>;. Lhs can be either a preexisting variable or a fresh variable; we should think about ways to express the latter.

Once #34 is merged, we should remove it from mypy's ignores and add a proper stub file for its functions.

In particular, after #25, those interfaces are:

def extract_ast(filename: os.PathLike, is_cxx: bool) -> Dict[str, Any]:
    ...

def transform(filename: os.PathLike, is_cxx: bool, replacement: str, start_line: int, start_col: int, end_line: int, end_col: int) -> bool:
    ...

When running a new file create a new key from module_name --> filename

We should have custom exceptions instead of ValueError, where appropriate.

Auto patch should be better now with raw stmt information stored into the DB. This is because when we are doing search, as long as the types match what is in the fix patterns it's just guess and check.

In MATE we only had access to variable level source info, now in the DB we have matched nodes with their source level strings so we can just pull from the DB to have much better templates.

Port over the C++ code actually responsible for patching, maybe use stringstreams instead of using the actual rewriter this time.

Export extractor only as tourniquet.extractor rather than having extractor and tourniquet be distinct python modules

Don't always build the Python extension in debug mode.

We need to talk, generally, about tourniquet's approach to building source files and testing patches.

As of #37, that approach looks like the following:

• Patching: Patches are applied to the original source file (not a copy) within a context manager, and then rolled back when the context manager ends its scope.
• Testing: Programs are assumed to be single-file, standalone executables with no special flags. They're passed directly into clang; builds that error are skipped. The auto-patcher assumes that the target program can be tested using inputs on stdin and exits with with an exit code on various different error conditions.

By and large, I think this approach to patching is fine for now: we'll probably want to not patch the original source file further down the line, but hiding it inside a context manager keeps things clean for the time being.

On the other hand, the approach to testing needs to be rethought and redesigned. In particular, we should think about the following:

• How are we going to compile source files that belong to a larger (multi-file) program? Are we going to recompile the entire program each time, or attempt to recompile only the particular translation unit with the help of compilation records from blight? How are we going to ensure that we faithfully re-link the modified translation unit?

• How are we going to test patched programs? We should probably implement a set of methodologies, with "throw stuff at stdin and see how it exits" being one of them. We should probably come up with an abstract "test" interface that each methodology refines; we can then make the process of selecting the appropriate methodology for a program something that requires human feedback.

We should use a conventional src structure, with src/tourniquet/__init__.py to provide the proper package/module structure instead of tourniquet/tourniquet.py and tourniquet/__init__.py to fix the former up.

Write tests to confirm that the proper tables are made and that the proper info is in the tables

For different fix patterns, have a view mode as well to see what a non concretized version looks like

Exporting the AST info gives us some good information to reason about, but there is still a lot of information left in the clang hierarchy. It will be hard to match syntactic patterns without using the clang API.

I think we should still support some type of syntactic matcher expressions like before, but instead of making ASTMatchers bindings we can just have another C++ extension that takes a string and constructs a matcher expression. The code for this should already be in clang-query, so we can take that.

This is still better than what we had before, because

1. You can actually specify fix patterns etc from the user facing component
2. You are not limited by syntactic patterns, they are just a suggestion now.
3. Before the match API call would also collect source info etc about your match, we don't have to do that now
4. We don't have to send patch-lang type information back and forth, its nice.

Check out what we could do if we had syntactic patterns (str --> match --> True/False) + regular python
I imagine Node.matches actually invokes the C++ extension

This is an example of matching a syntactic pattern (arithmetic operations) and trying to relate it a math poi
that says "yeah thats something we should check out". The replacement pattern says lets replace that node with
some arbitrary BinaryOperator. This means during search, we will try searching for other potential operators.

new_t = template(
     Node.matches(binaryOperator(anyOf(
                             hasOperatorName("<<"),
                             hasOperatorName("+"),
                             hasOperatorName("-"),
                             hasOperatorName("*"),
                             hasOperatorName("/"),
                            ))) and cpg.run_math_poi(Node) == True, 
     pattern(
        Node, 
        BinaryOperator()
      )
    )
  

Right now, subclasses of Expression and Statement hardcode their abstract syntax features in the view and concretize methods. To avoid duplication and to make addition of new subclasses easier, these should be lifted to a _template (or similar) class attribute that contains a Handlebar (or other) template.

For example:

class IfStmt(Statement):
    _template = """
    if ({{ cond }}) {
      {{ expr }}
    }
    """

view and concretize could then use _template as appropriate.

We should do almost exactly what blight does: https://github.com/trailofbits/blight/blob/master/.github/workflows/docs-deploy.yml