workiva / dart_dev Goto Github PK

There should at least be an option to run tests in checked mode. As tests are currently being run in a separate process using the pub run test executable, it would require adding the -c/--checked flag to the pub run command. If the test task was updated to run tests programmatically, it would probably just inherit the checked flag from the pub run command used to run dart_dev.

Problem

Currently we are using processes to run the executables for the following tasks:

• dart formatter
• tests
• coverage
• docs

Solution

If possible, we should switch these to using the APIs directly for several reasons:

• executing the task programmatically is much faster than starting a process to do so
• typed interaction and results allows us to provide more useful return types for the APIs and more useful output for the CLIs
• consumers would no longer need to also have the above packages installed as immediate dependencies because we would no longer be using the executables

Resources

• dart formatter programmatic usage

I want to be able to run a subset of the tests:

$ pub run dart_dev test test/my_module/

$ pub run dart_dev test test/my_module/test_module.dart

The --functional flag should cause the functional test suite (if configured by the project) to run.

• coverage
• dart_style
• test

pub serve example --port=9000

• dartfmt line length option #14
• Tests #15
• Licenses #17
• Update installation instructions to point to actual version (or get rid of it in favor or auto generated version on Pub) #33
• Changelog #12
• Prep pubspec.yaml #27
• Travis CI setup #29
• Codecov.io setup #30

By default, the format task should use the project's configuration for files to format. But, we should also allow running the formatter on a specific file or set of files. This can be done by allowing additional args to the format CLI that override the default configuration.

Investigate the possibility of adding watchers to tasks that devs might want to run continually (analyze, format, tests).

The dartanalyzer follows imports, which means it's only necessary to analyze the top-level entry points since all other files (that are imported) will be reachable and will be analyzed.

On this same note, the "entryPoints" config option may not be very intuitive since it accepts directories or files and expands directories to look for all top-level files. We may want to leave the file expansion to the shell and only accept a list of files.

Use Case

Consider a project where..

• Unit tests can be run without any dependencies.
• Integration tests require an HTTP server to be running.

Currently, the Test task supports this use case by allowing a project to start the integration server in the before section and kill it in the after section. But, it doesn't do this dynamically based on whether or not the --integration flag was actually included. So when only running unit tests, the integration server would be run unnecessarily, slowing down what should be an extremely fast unit test run.

Possible Solutions

Add fields to the TestConfig class that would allow specific before/after dependencies for each type of test suite. This creates quite a bit of bloat on that config object, however.

Support the passing of selected options/flags to the before/after functions so that they can be dynamically altered at runtime. This would allow the integration server to only start up if the --integration flag is set. This seems like a better approach, but I'm unsure of how to best pass that configuration into the functions because it will require a static function interface.

Given that a LICENSE file is present, copy that file to every source file in the repository that does not already include it.

Options

directories: which directories to traverse and add license headers within.
fileTypes: file types to check (.dart, .html, .css)

Need to add an await here: https://github.com/Workiva/dart_dev/blob/master/lib/src/tasks/coverage/cli.dart#L98

The coverage package has a utility where they use sockets to find an open port. We should use that when selecting a port for observing VM tests for coverage so that parallelized coverage collection doesn't step on itself.

• Description
• Authors
• Homepage
• Documentation

Make sure to run a pub publish --dry-run

We are currently depending on ^0.2.0 of dart_style, which is still very new. There are a lot of projects that will not have upgraded to that version, which can produce a version range discrepancy. We should widen this range since the our usage is compatible with older versions.

Currently the logs will display an error noting when a test suite fails, but the coverage is simply ignored for that suite and the overall coverage task still succeeds. This produces a coverage report with lower-than-expected coverage.

Each task should have two static methods:

static Future<TaskResult> run(...);
static Task start(...);

Additionally, each task's done Future should resolve with the TaskResult instance. The CLIs should return this same TaskResult.

Calling .run() is simpler and eventually returns the end result that consumers want. However, calling .start() gives the consumer the opportunity to stream the output to somewhere like stdout for live feedback.

This will allow our integration tests to call into the CLIs directly to programmatically execute the tests instead of spawning new processes to do that. The returned result can then be used easily by the test to verify information about the task without having to parse the stdout output.

When we merge the individual coverage collections, we skip the first item and use it as a starting point. Since we delete the files in the loop after we read them, the first one doesn't get deleted.

Dependent on #22.

Analyze task summary should include:

• entry points analyzed
• errors
• warnings
• hints
• overall result (pass/fail)

Currently the tests are split up between unit and integration tests. Eventually, there will be a functional test suite as well. We may add the ability to dynamically define test suites to better organize tests.

With this in mind, it could be helpful to have a -l flag that lists out the test suites that are available.

Additionally, we may want to opportunistically inform the CLI user of the available test suites.. for example:

• User runs ddev test, which runs the unit test suite
• Ddev notices that the project has defined an integration test suite as well
• Print out a line before starting the test run that there are also integration tests available and to use the --integration flag to run them

Guys love the idea of ddev... Hopefully by endless grinder task can go away..

Anyhow run into this problem

ddev coverage

::: Collecting coverage

    Running test suite test/cxs_store_test.dart
    dart --observe=56484 test/cxs_store_test.dart

        Observatory listening on http://127.0.0.1:56484
        00:00 +0: A group of tests First Test
        00:00 +1: All tests passed!

    Collecting coverage for test/cxs_store_test.dart
    pub run coverage:collect_coverage --port=56484 -o coverage/collection/test/cxs_store_test.dart.json
    Formatting coverage
    pub run coverage:format_coverage -l --package-root=packages -i coverage/coverage.json -o coverage/coverage.lcov --verbose --report-on=lib/


        Provided package root "packages" is not directory.

        Usage: dart format_coverage.dart [OPTION...]

        -s, --sdk-root        path to the SDK root
        -p, --package-root    path to the package root
        -i, --in              input(s): may be file or directory
        -o, --out             output: may be file or stdout
                              (defaults to "stdout")

            --report-on       which directories or files to report coverage on
        -j, --workers         number of workers
                              (defaults to "1")

        -r, --pretty-print    convert coverage data to pretty print format
        -l, --lcov            convert coverage data to lcov format
        -v, --verbose         verbose output
        -h, --help            show this help
    Coverage formatting failed. Could not generate coverage/coverage.lcov
Unhandled exception:
Uncaught Error: Exception: Coverage formatting failed. Could not generate coverage/coverage.lcov
Stack Trace:
#0      CoverageTask._format.<_format_async_body> (package:dart_dev/src/tasks/coverage/api.dart:280:7)
#1      _RootZone.runUnary (dart:async/zone.dart:1166)
#2      _Future._propagateToListeners.handleValueCallback (dart:async/future_impl.dart:494)
#3      _Future._propagateToListeners (dart:async/future_impl.dart:577)
#4      _Future._completeWithValue (dart:async/future_impl.dart:368)
#5      _Future._asyncComplete.<anonymous closure> (dart:async/future_impl.dart:422)
#6      _microtaskLoop (dart:async/schedule_microtask.dart:43)
#7      _microtaskLoopEntry (dart:async/schedule_microtask.dart:52)
#8      _runPendingImmediateCallback (dart:isolate-patch/isolate_patch.dart:96)
#9      _RawReceivePortImpl._handleMessage (dart:isolate-patch/isolate_patch.dart:149)

#0      _rootHandleUncaughtError.<anonymous closure> (dart:async/zone.dart:895)
#1      _microtaskLoop (dart:async/schedule_microtask.dart:43)
#2      _microtaskLoopEntry (dart:async/schedule_microtask.dart:52)
#3      _runPendingImmediateCallback (dart:isolate-patch/isolate_patch.dart:96)
#4      _RawReceivePortImpl._handleMessage (dart:isolate-patch/isolate_patch.dart:149)

Dart Version

Dart VM version: 1.12.0-dev.5.9 (Mon Aug 24 03:56:54 2015) on "macos_x64"

Description

I can make this happen consistently, if you use stagehand to set a package-simple project. You will certain hit the problem.

Thanks in advance and keep the work... Just need a way to run dart_doc now.

Similar to the --check flag on the format task.
Could be used in CI builds.

A more generic version of the "examples" task that essentially proxies pub serve.

Tasks like analyze and format currently only target the lib/ directory by default. This could be made more intelligent by checking for the existence of other standard directories (bin/, example/, tool/) and including them only if they exist.

Several tasks in dart_dev take a list of patterns and expand it to find a list of files. We should consider making a utility for this if we can get enough reuse across the project.

Feature Request

As a developer I should be able to execute code generation commands with dart dev. I should be able to define a "library search path" in the configuration that defines all the code paths to apply a source_gen Generator class. I should be able to specify a list of of generator classes that will be applied to the given search path items. When I run pub run dart_dev generate the tool should use my configuration for to determine the list of generators and the search paths to generate code.

Currently, running coverage and generating the HTML report without having lcov installed will produce an exception with a nasty stack trace. We should check for existence of genhtml command before running it and print a helpful error message if it's not installed.

Need a null check on dependencies and devDependencies when parsing the pubspec.yaml.

• Setup reporting during CI build
• Add badge to README

Command

test

Flags

• --unit runs unit tests (defaults to on, negatable)
• --integration runs integration tests (defaults to off)
• --functional runs functional tests (defaults to off)

Default configuration

• Runs test runner from test package (pub run test)
• --integration, --functional have no effect

Configuration

TBD

Support code coverage collection.

• Setup CI
• Add badge to README

The dart_dev executable throws an exception when it encounters a dart2js warning when serving a non-dartium client. This exception causes the dart_dev process to exit leaving the child pub serve process running.

    [example] GET ....
    [Info from Dart2JS]:
    Compiling <omitted_path>/main.dart...
Unhandled exception:
Uncaught Error: [Dart2JS on <omitted_path>/main.dart]:
#0      _rootHandleUncaughtError.<anonymous closure> (dart:async/zone.dart:895)
#1      _microtaskLoop (dart:async/schedule_microtask.dart:43)
#2      _microtaskLoopEntry (dart:async/schedule_microtask.dart:52)
#3      _runPendingImmediateCallback (dart:isolate-patch/isolate_patch.dart:96)
#4      _RawReceivePortImpl._handleMessage (dart:isolate-patch/isolate_patch.dart:149)

This seems to be an issue when a child process of dart_dev emits output to stderr.

Steps to Reproduce

• run pub run dart_dev examples with an example that includes code that throws a warning or hint in the dart2js compiler.
• serve a dartium client to confirm and confirm no error
• serve a non-dartium(chrome) client and confirm it throws an exception

The dartformatter removes trailing whitespace from comments, which means after we apply the license text to all source files, the formatter will need to be run. If the copy license task trimmed each line, the generated code should already be "clean".

There's a good opportunity to get some exposure for this tooling via presenting it on Dart Live.

https://pub.dartlang.org/packages/dart_dev

Issues

• GitHub tables are not standard markdown - use HTML tables instead
• Blockquotes end to early..
  • at the beginning rationale
  • under "CLI Usage"
  • under "Programmatic Usage"
• Headers under "Getting Started" are too small
• "Project Configuration" code block is missing most of the code?

This will depend on #22

Currently the integration tests are being run as separate processes, but this prevents coverage collection from recording the impact of these tests. We need to execute those tests programmatically to fix this.