Giter VIP home page Giter VIP logo

nixt's Introduction

Nixt

Nixt is a unit-testing tool for Nix.

Installation

Imperatively

Using nix-env:

$ nix-env -if https://github.com/nix-community/nixt/archive/master.tar.gz

Using nix profile:

$ nix profile install github:nix-community/nixt

Declaratively

Add Nixt as a flake to your configuration:

{
  inputs.nixt = {
    url = "path:/home/ldlework/src/nixt";
    inputs.nixpkgs.follows = "nixpkgs";
  };
}

Then add the package to your packages:

{
  environment.systemPackages = [
    inputs.nixt.defaultPackage.x86_64-linux
  ];
}

Help

nixt

  Test-runner for nixlang.

Options

  -p, --path string   Path to the test suite
  -w, --watch         Watch for changes at path
  -v, --verbose       Show additional test info
  -l, --list          List, but don't run, tests
  -d, --debug         Show nixt-developent relevant info
  -h, --help          Prints this usage guide

Running Tests

If it does not find a registry, the nixt CLI discovers and runs tests located at -p/--path:

$ nixt ./nix/

Found 14 cases in 8 suites over 3 files.

  ✗ 2 cases failed.

┏ /home/ldlework/src/nixt/cli/nix/get-testset.test.nix
┃   mkSuites
┃     ✗ always fails

┏ /home/ldlework/src/nixt/cli/nix/utils.test.nix
┃   broken test
┃     ✗ undefined variable

  ⚠ Couldn't import 1 files:
    - /home/ldlework/src/nixt/cli/nix/invalid.test.nix
      Import error: called with unexpected argument 'nixt'
      Did you forgot to add the 'nixt' argument to your test expression?

Adding in the -v/--verbose flag will show passing cases and additional information on failed cases:

$ nixt ./nix/ -v

Found 14 cases in 8 suites over 3 files.

  ✗ 2 cases failed.

┏ /home/ldlework/src/nixt/cli/nix/get-testset.test.nix
┃   mkSuite
┃     ✓ creates correct structure
┃   mkSuites
┃     ✗ always fails
┗     ✓ creates correct structure

┏ /home/ldlework/src/nixt/cli/nix/utils.test.nix
┃   broken test
┃     ✗ undefined variable
┃       error: undefined variable 'baz'
┃       at /home/ldlework/src/nixt/cli/nix/utils.test.nix:12:30:
┃       11|     "broken test" = {
┃       12|       "undefined variable" = baz;
┃       |                              ^
┃       13|     };
┃       (use '--show-trace' to show detailed location information)
┃   dirFiles
┃     ✓ empty list for non-existent path
┃     ✓ non-empty list for existing path
┃   findNixFiles
┃     ✓ empty list for non-existent path
┃     ✓ non-empty list for existing path
┃   getDir
┃     ✓ empty list for non-existent path
┃     ✓ non-empty list for existing path
┃   isNix
┃     ✓ false for non-nix files
┃     ✓ true for nix files
┃   isTestSuite
┃     ✓ false for non-test suites
┗     ✓ true for test suites

  ⚠ Couldn't import 1 files:
    - /home/ldlework/src/nixt/cli/nix/invalid.test.nix
      Import error: called with unexpected argument 'nixt'
      Did you forgot to add the 'nixt' argument to your test expression?

Two -v -v verbose flags implies --show-trace.

Listing Tests

To list discovered tests without actually evaluating their cases use the --l/-list flag:

$ nixt ./nix/ -l

Found 14 cases in 8 suites over 3 files.

  ⚠ Couldn't import 1 files:
    - /home/ldlework/src/nixt/cli/nix/invalid.test.nix
      Import error: called with unexpected argument 'nixt'
      Did you forgot to add the 'nixt' argument to your test expression?

Or with the -v/--verbose flag:

$ nixt ./nix/ -l -v

Found 14 cases in 8 suites over 3 files.

┏ /home/ldlework/src/nixt/cli/nix/get-testset.test.nix
┃   mkSuite
┃     - creates correct structure
┃   mkSuites
┃     - always fails
┗     - creates correct structure

┏ /home/ldlework/src/nixt/cli/nix/utils.test.nix
┃   broken test
┃     - undefined variable
┃   dirFiles
┃     - empty list for non-existent path
┃     - non-empty list for existing path
┃   findNixFiles
┃     - empty list for non-existent path
┃     - non-empty list for existing path
┃   getDir
┃     - empty list for non-existent path
┃     - non-empty list for existing path
┃   isNix
┃     - false for non-nix files
┃     - true for nix files
┃   isTestSuite
┃     - false for non-test suites
┗     - true for test suites

  ⚠ Couldn't import 1 files:
    - /home/ldlework/src/nixt/cli/nix/invalid.test.nix
      Import error: called with unexpected argument 'nixt'
      Did you forgot to add the 'nixt' argument to your test expression?

Writing Tests

Nixt tests are written in blocks. Users may use flakes or standalone testing.

With standalone testing, a block is put in its own file which:

  • Contains a function taking attrset args nixt and pkgs
  • Evaluates to a Block

Each block is composed of one or more suites; Each suite is composed of one or more cases. Each case should be an expression or list of expressions that evaluate to booleans.

For those curious:

Block = struct "Block" {
  path = path;
  suites = list TestSuite;
};
TestSuite = struct "TestSuite" {
  name = string;
  cases = list TestCase;
};
TestCase = struct "TestCase" {
  name = string;
  expressions = list bool;
};

Library Functions

grow

Args:

  • attrset containing
    • blocks: list of Block
    • settings: Optional attrset of settings

Builds the nixt registry for cli consumption. Only relevant to flakes.

{
  inputs = {
    nixt.url = "github:nix-community/nixt";
  };

  outputs = {
    nixt,
    ...
  } @ inputs:
  {
    __nixt = nixt.lib.grow {
      blocks = [
        nixt.lib.block' ./flake.nix {
          "nixt"."passes this test" = true;
          "nixt"."fails this test" = false;
        }
      ];
    };
  };
}

block

Args:

  • path: path to the current file
  • suites: list of TestSuites

Creates a Block from a path and list of TestSuite.

{
  nixt,
  pkgs ? import <nixpkgs> {}
}: let
  inherit (nixt) block describe';
in
  block ./block.spec.nix [
    (describe' "nixt" {
      "passes this test" = true;
      "fails this test" = false;
    })
  ]

block’

Args:

  • path: path to the current file
  • suites: attrset of suites

Creates a Block from a path and attrset.

{
  nixt,
  pkgs ? import <nixpkgs> {}
}:
nixt.block ./block.spec.nix {
  "nixt"."passes this test" = true;
  "nixt"."fails this test" = false;
}

describe

Args:

  • name: string
  • cases: list of TestCase

Creates a TestSuite from a string and list of TestCase

{
  nixt,
  pkgs ? import <nixpkgs> {}
}: let
  inherit (nixt) block describe it;
in
  block ./block.spec.nix [
    (describe "nixt" [
      (it "passes this test" true)
      (it "fails this test" false)
    ])
  ]

describe’

Args:

  • name: string
  • cases: attrset of cases

Creates a TestSuite from a string and attrset

{
  nixt,
  pkgs ? import <nixpkgs> {}
}: let
  inherit (nixt) block describe';
in
  block ./block.spec.nix [
    (describe' "nixt" {
      "passes this test" = true;
      "fails this test" = false;
    })
  ]

it

Args:

  • name: string
  • expressions: bool or list of bool

Creates a TestCase from a string and bool or list of bool

{
  nixt,
  pkgs ? import <nixpkgs> {}
}: let
  inherit (nixt) block describe it;
in
  block ./block.spec.nix [
    (describe "nixt" [
      (it "passes this test" true)
      (it "fails this test" false)
    ])
  ]

inject

Args:

  • path: path to a test file

Provides arguments to compliant files. For standalone support and cli use.

nixt's People

Contributors

dustinlacewell avatar github-actions[bot] avatar lord-valen avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

nixt's Issues

Integration into nixpkgs

Are there any plans or reasons not to include this in nixpkgs?

I think we lack something like this as go-to solution that just comes with nixpkgs.

New code (non-flake import), infinite recursion

Attempting to use the non-flake import on the typescript branch fails.

building '/nix/store/q13rlagpl9d65wgrjccpydf7dg0f6v8m-nixt-0.1.0.drv'...
building
Hint: used config file '/nix/store/6f0inr5ikfha63lxmcqn53wq3cj8gh0q-x86_64-unknown-linux-gnu-nim-wrapper-1.6.4/etc/nim/nim.cfg' [Conf]
Hint: used config file '/nix/store/6f0inr5ikfha63lxmcqn53wq3cj8gh0q-x86_64-unknown-linux-gnu-nim-wrapper-1.6.4/etc/nim/config.nims' [Conf]
...................................................................................................................
/nix/store/f5bnxczqawd26z1446diw20frarsffjz-nim/nix.nim:9:15 Warning: inherit from a more precise exception type like ValueError, IOError or OSError. If these don't suit, inherit from CatchableError or Defect. [InheritFromException]
/nix/store/f5bnxczqawd26z1446diw20frarsffjz-nim/nix.nim:27:3 Error: expression expected, but found 'keyword let'
error: builder for '/nix/store/q13rlagpl9d65wgrjccpydf7dg0f6v8m-nixt-0.1.0.drv' failed with exit code 1
error: 1 dependencies of derivation '/nix/store/n2krbilz3lkj5fnq1rmwhbk043pgcfxb-system-path.drv' failed to build
error: 1 dependencies of derivation '/nix/store/zxdxp2b93b7vp1igp7vdq3zwrskhrmxj-nixos-system-mini-one-22.05pre360796.73ad5f9e147.drv' failed to build

Add continous run mode

feature request - built in option/integration with watchman/examples docs
poor mans implementation

environment.interactiveShellInit = ''
      function _nixt()
      {
        nixt $@ $(realpath "''${NIXOS_CONFIG%/*}")
      } 
      alias nixt=_nixt

      function nixty()
      {
        while true
        do
          _nixt > /tmp/nixt-out
          clear
          cat /tmp/nixt-out
          sleep 3
        done
      } 
      alias nixt=_nixt
  '';

should return non zero code when test fail

I just discovered this unit test tool

I'm trying it with this dummy program

{ nixt }:

nixt.mkSuites {
  "Hello" = {
    "world" = hello == "Hello world";
  };
}
nixt tests && echo "everything is good" || echo "something is bad"
Found 0 cases in 0 suites over 1 files.

  ✓ 0 cases failed.

  ⚠ Couldn't import 1 files:
    - /home/pinage404/Project/nix-sandboxes/nix/tests/hello.test.nix
      Command failed: nix eval --json --impure  --expr 'import /nix/store/08lsh2ydkxfp5hsx40vs9rd9bf86vjfa-nixt-0.2.2/lib/node_modules/nixt/nix/get-testspec.nix { path = "/home/pinage404/Project/nix-sandboxes/nix/tests/hello.test.nix"; }'
      error: undefined variable 'hello'
      
             at /home/pinage404/Project/nix-sandboxes/nix/tests/hello.test.nix:5:15:
      
                  4|   "Hello" = {
                  5|     "world" = hello == "Hello world";
                   |               ^
                  6|   };
      

everything is good

everything is good is outputed, so nixt returns a 0 code

Maybe it's because there is a compilation error

So, i also tested with a valid program that fails

{ nixt }:

let
  hello = "Hello anything but world";
in

nixt.mkSuites {
  "Hello" = {
    "world" = hello == "Hello world";
  };
}
nixt tests && echo "everything is good" || echo "something is bad"
Found 1 cases in 1 suites over 1 files.

  ✗ 1 cases failed.

┏ /home/pinage404/Project/nix-sandboxes/nix/tests/hello.test.nix
┃   Hello
┗     ✗ world


everything is good

But there is still the same issue

tes examples seem broken

after cloning the repo and entering shell via direnv:

> nixt examples/valid.spec.nix 
Path provided: standalone mode

Found 0 cases in 0 suites over 1 files.


  ⚠ Couldn't import 1 files:
    - /home/adrian/code/nixt/examples/valid.spec.nix     

Documentation

Looking for suggestions on how to handle documentation for Nixt.

Investigate inconsistency between `nix build` and `nix develop`

Currently nix build works as intended, it runs the preinstall script without issue and outputs nixt in result/bin. Nix develop fails to build. It appears that tsc is not running with the correct working directory, causing tsc to print its help screen instead of transpiling. If the shell node2nix provides is meant for dev use then npm install probably shouldn't be running at all. If it's meant to be run, npm install should return the same result as it does when building. I may have to roll my own devShell.

> [email protected] preinstall /nix/store/9230zmgs46fpc1ggw68d0485ax44cikz-node>
> npm run build


> [email protected] build /nix/store/yw2xs0sw9bpyi3gm92w1qachw3wxa0s9-node-dependencies-nixt-0.2.1/nixt
> tsc

Version 4.7.4
tsc: The TypeScript Compiler - Version 4.7.4

COMMON COMMANDS

  tsc
  Compiles the current project (tsconfig.json in the working directory.)

  tsc app.ts util.ts
  Ignoring tsconfig.json, compiles the specified files with default compiler options.

  tsc -b
  Build a composite project in the working directory.

  tsc --init
  Creates a tsconfig.json with the recommended settings in the working directory.

  tsc -p ./path/to/tsconfig.json
  Compiles the TypeScript project located at the specified path.

  tsc --help --all
  An expanded version of this information, showing all possible compiler options

  tsc --noEmit
  tsc --target esnext
  Compiles the current project, with additional settings.

COMMAND LINE FLAGS

--help, -h
Print this message.

--watch, -w
Watch input files.

--all
Show all compiler options.

--version, -v
Print the compiler's version.

--init
Initializes a TypeScript project and creates a tsconfig.json file.

--project, -p
Compile the project given the path to its configuration file, or to a folder with a 'tsconfig.json'.

--build, -b
Build one or more projects and their dependencies, if out of date

--showConfig
Print the final configuration instead of building.

COMMON COMPILER OPTIONS

--pretty
Enable color and formatting in TypeScript's output to make compiler errors easier to read.
type: boolean
default: true

--target, -t
Set the JavaScript language version for emitted JavaScript and include compatible library declarations.
one of: es3, es5, es6/es2015, es2016, es2017, es2018, es2019, es2020, es2021, es2022, esnext
default: es3

--module, -m
Specify what module code is generated.
one of: none, commonjs, amd, umd, system, es6/es2015, es2020, es2022, esnext, node16, nodenext
default: undefined

--lib
Specify a set of bundled library declaration files that describe the target runtime environment.
one or more: es5, es6/es2015, es7/es2016, es2017, es2018, es2019, es2020, es2021, es2022, esnext, dom, dom.iterable, webworker, webworker.importscripts, webworker.iterable, scripthost, es2015.core, es2015.collection, es2015.generator, es2015.iterable, es2015.promise, es2015.proxy, es2015.reflect, es2015.symbol, es2015.symbol.wellknown, es2016.array.include, es2017.object, es2017.sharedmemory, es2017.string, es2017.intl, es2017.typedarrays, es2018.asyncgenerator, es2018.asynciterable/esnext.asynciterable, es2018.intl, es2018.promise, es2018.regexp, es2019.array, es2019.object, es2019.string, es2019.symbol/esnext.symbol, es2020.bigint/esnext.bigint, es2020.date, es2020.promise, es2020.sharedmemory, es2020.string, es2020.symbol.wellknown, es2020.intl, es2020.number, es2021.promise/esnext.promise, es2021.string, es2021.weakref/esnext.weakref, es2021.intl, es2022.array/esnext.array, es2022.error, es2022.intl, es2022.object, es2022.string/esnext.string, esnext.intl
default: undefined

--allowJs
Allow JavaScript files to be a part of your program. Use the 'checkJS' option to get errors from these files.
type: boolean
default: false

--checkJs
Enable error reporting in type-checked JavaScript files.
type: boolean
default: false

--jsx
Specify what JSX code is generated.
one of: preserve, react, react-native, react-jsx, react-jsxdev
default: undefined

--declaration, -d
Generate .d.ts files from TypeScript and JavaScript files in your project.
type: boolean
default: `false`, unless `composite` is set

--declarationMap
Create sourcemaps for d.ts files.
type: boolean
default: false

--emitDeclarationOnly
Only output d.ts files and not JavaScript files.
type: boolean
default: false

--sourceMap
Create source map files for emitted JavaScript files.
type: boolean
default: false

--outFile
Specify a file that bundles all outputs into one JavaScript file. If 'declaration' is true, also designates a file that bundles all .d.ts output.

--outDir
Specify an output folder for all emitted files.

--removeComments
Disable emitting comments.
type: boolean
default: false

--noEmit
Disable emitting files from a compilation.
type: boolean
default: false

--strict
Enable all strict type-checking options.
type: boolean
default: false

--types
Specify type package names to be included without being referenced in a source file.

--esModuleInterop
Emit additional JavaScript to ease support for importing CommonJS modules. This enables 'allowSyntheticDefaultImports' for type compatibility.
type: boolean
default: false

You can learn about all of the compiler options at https://aka.ms/tsc

npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] build: `tsc`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the [email protected] build script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /build/.npm/_logs/2022-07-19T23_32_19_956Z-debug.log
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] preinstall: `npm run build`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the [email protected] preinstall script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /build/.npm/_logs/2022-07-19T23_32_20_182Z-debug.log

error: builder for '/nix/store/aly55j3zr5s2vhzqj2yc535rywf1hb5v-node-dependencies-nixt-0.2.1.drv' failed with exit code 1;
       last 10 log lines:
       > npm ERR! errno 1
       > npm ERR! [email protected] preinstall: `npm run build`
       > npm ERR! Exit status 1
       > npm ERR!
       > npm ERR! Failed at the [email protected] preinstall script.
       > npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
       > 
       > npm ERR! A complete log of this run can be found in:
       > npm ERR!     /build/.npm/_logs/2022-07-19T23_32_20_182Z-debug.log

The NixService.nixPath function needs to calculate a path one further up the hierarchy

Running the following:

nix run github:nix-community/nixt -- --debug ./config.test.nix

Gives an error:

[
  'path = "[REDACTED]/config.test.nix";'
]
Import error!
  Command failed: nix eval --json --impure  --expr 'import /nix/store/6b5kfla0zjgcl2i22bpzwjdf01y6gjzy-nixt-0.3.1/lib/node_modules/nixt/dist/nix/get-testspec.nix { path = "[REDACTED]/config.test.nix"; }'
error: getting status of '/nix/store/6b5kfla0zjgcl2i22bpzwjdf01y6gjzy-nixt-0.3.1/lib/node_modules/nixt/dist/nix/get-testspec.nix': No such file or directory

Running tests!
Rendering!

Found 0 cases in 0 suites over 1 files.

Transformer for checks

It would be quite nice if nixt could make derivations from test blocks (which purposefully fail builds if the test fails). Such a function should probably take a list of blocks, as grow does, and build a derivation from each block (possibly each suite). The obvious use case is for the flake checks output.

README.md contian invalid code examples

for example:

image

I guess it should be:

{nixt, pkgs ? import <nixpkgs> {}}:

nixt.block ./block.spec.nix [
  (describe' "nixt" [
    ["passes this test" true]
    ["fails this test" false]
  ])
]

__nixt registry schema

This is one of the things which opens up interoperability with https://github.com/divnix/std.
We should output a registry, __nixt, which contains the data nixt needs to run tests. Right now I'm considering a schema like so:

graph TD;
	A[__nixt];
	B[testSpec];
	BA[...testFiles];
	BAA[...testSuites];
	BAAA[...testCases];
	C[options];
	D[__schema];
    A-->B;
    B-->BA-->BAA-->BAAA;
    A-->C;
	A-->D;
Loading

https://github.com/colinhacks/zod may be useful here.

Doesn't find any tests

I wanted to give nixt a try but i can't seem to get it to work. I quickly uploaded a small reproducer here: https://github.com/gilligan/nixt-issue

$ nix-shell
$ nixt ./tests/
Found 0 test suites

🎉 0 tests PASSED 🎉

It's not clear to me what the problem is here. I played around with the functions in nix/utils.nix and it does find the file and isTestSuite does say it is a test so maybe the problem is in the nim code? I don't know. Maybe you can have a look?

Would love to try out nixt.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.