romhut / criterion Goto Github PK

The poller interval is not clearing when data.status.code == '1' or '0'

What are your thoughts on creating a Dockerfile for Criterion?

So a user could simply pull the Dockerfile, Docker would then install all dependencies for Criterion, run composer to pull in Criterion, and expose the application on a port automatically.

For those who want it of course, another way to get up and running quickly

The images are caching in the readme. According to this post, its because they are not https. which could be an issue. I have added the no cache headers.

travis-ci/travis-ci#263

Over the next few days we shall be converting to PSR-2.

First of all, nice job with this project.

However, i would love to see if this project could also support different database engines (RiakDB, Couch or even SQL-based engine?). We could use something like Doctrine perhaps?

Abstracting away the model from hardcoded Mongo-specific syntax and support driver-based db will immenselly helpful for others that want to have this app but already depend on other db engine than Mongo.

Thanks.

I'm really not a fan of "test" as the keyspace for running things in the file. "script" like what travis has makes much more sense. I think you got "test" from phpci, which is a bad name. CI is more than just testing, it's integration so you may be running build scripts, compilation or even setting up keys etc.

Also, "pass" and "fail" should be optional. Looks messy having them in there but empty.

The UI should have network hooks after the tests have failed or passed. In this screen there could also be options like notifications (send email when failed) and where to send them.

The simplest hook could be "when passed hit the url". The url would then do whatever it needs, similar to how git hooks work when commits are pushed.

A more complex iteration could be "push to remote". If the tests pass, push this commit onto another remote automatically. A lot of people already use git based deployment so this would be ideal.

Should we bother doing this? Since its not going to be running at a large scale just yet it may not be an issue. @marcqualie @outrunthewolf

Allow local folders to skip the "copying" process for large projects.

Add test status images. This can be as simple as redirecting to /test/image/123 to pass.jpg or fail.jpg. Or it can be more complex if we want it to be.

The UI should have options to add ENV variables to projects. This can be for things like databases and configuration.

I was going to try and add this tonight but didn't have time. Would be pretty awesome to setup when Github hooks are fully automated like what travis does.

https://github.com/blog/1227-commit-status-api

Convert everything and continue to follow PSR-2 standard.

We need to add emails notifications.

I kinda want to build "callbacks" in general so we can make anything happen after a failed/success deploy. And you just add it to your list in your .criterion.yml file.

For example:

pass:
    - sh deploy.sh
    - email_notifications
fail:
    - email_notifications

Allow duplicate projects (same source). This may break github hooks, but people generally won't have duplicate projects for github projects. If this does become an issue then we will have to pass the project ID or something into the github hook.

We recently added authentication but it's enabled by default. Some people might simply just want to firewall it to certain IPs or have it completely open on a certain port.

We need to be able to specify the SSH keys for a specific clone to use.

I think the only real way of doing this is the "ssh-agent" but its a bit annoying to work with.

When adding environmental variables (#32) we should add the option to run mutiple php versions. Criterion should be responsible for installing these php binaries onto the machine by downloading their source code and compiling.

Then inside .criterion.yml we can allow them to do version: 5.4 etc like travis does. This is especially useful for testing libraries and making sure you don't accidentally add 5.5 only features when coding and testing against your own builds of php.

Build a basic auth system using http basic auth.

I have been thinking about this for a while now. The way Criterion is built, it will allow any tests to be run from any language, but should we mainly build for PHP?

I know this will probably be a controversial question amongst people, but it's just a question at this stage.

By "mainly build for PHP" i mean that so most of our features work awesomely with PHP and integrate in nicely. Such as #38

We need to checkout the specified branch after copying a local folder.

app/Helper/Repo.php 36

This would be an awesome addition. We could display one of the badges for how much code coverage a project has, which could be calculated by extending phpunit's code coverage system!

I'm not sure if we should be using /src for the code. That's normally reserved for libraries along with /lib. Standalone applications tend to use /app. This would be pretty easy to change within composer.json as wouldn't affect any code (unless you've put in absolute paths to things).

Also, since there is only ever one config file would it be better to have that config.json file in the root of the repo? I don't think we need a whole directory for configs since we don't do different environments right now and probably won't ever need to due to the nature of the app.

Travis has a cool feature to be able to skip a build by adding [ci skip] to the commit message. This is especially useful for documentation updates. We should implement this to save some server overhead when updating READMEs for example.

http://about.travis-ci.org/docs/user/how-to-skip-a-build/

Add a Github Service:
https://github.com/github/github-services

Need to fix the exit codes from CLI tool to allow git hooks to work properly.

Add an installer to the console:

• Check dependencies
• Create first user account
• Generate config.json
  • Url
  • Email
• chmod +x bin/git
• Offer to add "hook" user
• Offer to add sample projects (Criterion, MongoMinify, Hoard)

When using a token, we need to check that it belongs to an admin. This is because for instance a Github hook can create a project, but only admins should be able to create them.

Have a look through the README file and make sure everything is sorted. We should also add licensing/author data etc before open sourcing.

Things to look out for would be like "script" instead of "test" now in the examples, and making sure all the links point to the right place.

Add the ability to use local repos:

• Check that it's a local folder in Repo::cloneCommand()
• Issue "cp" command in Repo::cloneCommand()
• Run usual process from then on (excluding clone)

Add the ability to push to a remote on deployment. Refs #33

We need to add support for supplying a name to a project. This would be very good for when having different setups of the same source.

When running tests I get this in the web ui.

$ rm -rf /tests/51e17c684c3e4c5cf48ba5cd/51e17cf44c3e4c5cf4286add
There is no output for this command.

I think this should be hidden because it's no use in the UI. Is there a way to mark certain commands as "internal" that aren't part of the tests themselves?

The status image should really say "passing" instead of "passed". Makes more sense for a status and that's how all over CI tools display it.

Allow the branch to be picked in project config settings. It will use this one unless a test is specifically passed in (from hooks).

There have been multiple requests for server side configuration which will override any settings inside criterion.yml. This will help when you need to run subsets of commands or different tests/checks to what is available within the project repository.

I suggest this configuration be in the same format, a yml editior in the web interface which will then be decoded and stored in the database with the project as "config" key and json data.

We need a one line installer such as:

curl -s https://raw.github.com/romhut/criterion/master/installer | sh

This would:

• Download latest tarball
• Extract
• Download, Install and Run Composer
• Run bin/cli install

PHPunit has a cool feature called "test suites". It would be cool if would integrate this into Criterion somehow. Would be very easy to detect if there are suites configured and run them.

The UI would have to be updated to be able to run different commands based on the suite selected, and the suite running interface itself would need to be custom as PHPunit doesn't support it very well natively.

This kind of functionality would save me from creating multiple projects with the same codebase all of the time.

The builds shouldn't simply fail if no criterion.yml file does not exist. Insyead it should fail gracefully and try to "figure out" the system. 90% of the time, you won't need to include a file in the repo.

By default all php system's should run "phpunit" as the only command and this will work. If a composer.json exists in the repo root simply run "composer install" before running "phpunit". Will save a lot of hassle adding files everywhere as most repos wont' need a lot of the extra functionality, and if they do.. simply add the file to overwrite the defaults.

Move the Mongo connection strings to the file. Specify in installer.

The issue we're having with the installer could be solved by having a simple web installer. Prompting for input during an automated install process is bad practice and will cause a lot of issues, stopping it being scripted which is bad for what I want to extend onto.

I've thought up a new process. The install process simple builds the system and it's dependencies, making it workable. The configuration then comes later. It's down the user to either run

bin/configure --db=mongo://127.0.0.1 --url=http://criterion.example.com

or go to the web interface and configure through there. Each option they do not pass to the cli tool they will be prompted for. If people want to fully automate the install they can simply build their own script which uses our installer then calls configure with all of their options set.

Convert the helpers to none static so we can use __construct to set \Criterion\Application()