Test Harness about kubicorn HOT 6 CLOSED

Hi,
Sorry that I didn't notice this one before. I think using travis CI might be the best place for us to do integration tests and unit tests. The following questions are identifying short term goals and long term jobs.

On unit tests, my $0.02 is to build a simple unit test framework as soon as possible (in one sprint) and require any other coming new PR writing the tests. Unit tests for existing features will be long term jobs. The CI tests will mostly around running some building from src tests as well as some real use case (say that deploy a very small scale cluster and apply some kubicorn commands there)

Thoughts?

from kubicorn.

I would call deploying a cluster an integration test..

But I completely agree.. we just need to set up the testing infra.. that is more on my plate than anything else as i need to set up a kubicorn account in the clouds so we can keep $$ accounted for correctly..

Do you have testing experience/passion/ideas? Would you like to start working on a suite knowing that it will one day be automated on each PR?

from kubicorn.

Hi,
I do agree that deploying a cluster is the best way to test as it is the practice. But it is just too expensive. What I had in mind are: 1) dev should handle that the code is fully tested (as if someone is working on the aws part, make sure it is deployed and tested) 2) we have CI test that will check every PR and make sure it itself is not breaking, and merging it into the master will not break, and 3) a unit test that check pieces. Plus the fact that the code reviewer should try to deploy the patch if they have the chance.

Do you think it is reasonable? In this case all of the effort can be done via travis ci (I think). Of cause we can try to get some Azure or AWS resources to run some nightly build just to make sure (I believe it saves a lot too).

from kubicorn.

So I started working on a testing harness and a new infrastructure testing library called charlie.. links and code coming soon!

I am still playing with the idea of deploying a cluster on ever PR - it does seem expensive, but I really want this tool to be taken seriously and respected for it's stability - so it might be what we need. I am fine with footing the bill (within reason) and I am sure we can come up with clever ways to scrape together a hundred bucks or so every month to offset our cloud bills.

But a nightly build might be a nice alternative.. thinking thinking thinking... hrmm

from kubicorn.

I see your point.
I think running real deployment tests for every PR is orthogonal to travis CI -- you still need something to run unit test and some mock deploy test while developing. Right?
(I will keep an eye on charlie, see you there)

from kubicorn.

Closing this as we have make test and make ci in #158

from kubicorn.