assimilation / assimilation-official Goto Github PK

The nanoprobe is "universal", but still there need to be packages to install it with. So, we need at least a single RPM package and a single Debian package. Maybe more if they don't always work out. These need to be published in a a public repo.

See: https://trello.com/c/XRhZqwWu

• Events: One event should be generated for each new failure and each new success. Currently these events are ephemeral, but eventually will be persistent.
• Searchable. A relationship or node should exist for each failure, and maybe for each success. The general idea is that one should be able to easily search for all systems that failed tests, a total count of failures should be available, and it should be possible to collect the list of machines with any failure, or those which fail any given rule.

Here are some sample outputs:

Aug  3 18:35:39 servidor <80><93>þk<8a>^?[23135]: DEBUG: _netio_setsockbufsize.278: trying to set input buffer size to 10485760
Aug  3 18:35:39 servidor <80><93>þk<8a>^?[23135]: DEBUG: _netio_setsockbufsize.278: trying to set output buffer size to 1048576

The process ID (23235) is the parent process for the CMA:

assimil+ 23135 23132  0 18:35 pts/1    00:00:05 python ./cma.py --debug 2 --erasedb --foreground

This is very consistent and happens every time we start the CMA.
There are corresponding log messages from the CMA itself, and they look just fine:

Aug  3 18:35:55 servidor cma INFO: Socket input buffer size:  20971518
Aug  3 18:35:55 servidor cma INFO: Socket output buffer size: 2097150

We have a failed build here:
https://travis-ci.org/assimilation/assimilation-official/builds/101161253

It appears to be caused by the fact it can't find the neoauth program.

Neoauth gets installed when you install the py2neo python package. I see it got installed, but I don't know why this command isn't found. Perhaps a PATH problem? On my machine, it's installed under /usr/local/bin. Not sure if that's universal or not...

Here's where this shows up (16 times) in the build output:
CalledProcessError: Command '['neoauth', 'neo4j', 'neo4j', ')(UJZUBbLXRkez&P']' returned non-zero exit status 1

This is the cause of this failure.

I presume that the command isn't found. You does not have to be root for this to work. You just have to know the default password ("neo4j").

If you're not root, then you'll get a different failure in the same module ;-) -- but not this one.

The failures cause crashes in the python/C interface

Sadly, when this kind of thing happens, it's ugly to figure out. Really ugly. Discovered by @dpquigl. The good news (so to speak) is that it happens under Docker as well. This eliminates any kernel involvement. Libraries and new python code are possible causes.

https://docs.travis-ci.com/user/deployment/bintray

d9b4504
i added the .travis.yml here and enabled travis to listen to pushes to the repo. once we iron out all the errors, i can merge this to master.

the builds can be seen here: https://travis-ci.org/assimilation/assimilation-official/

and this is the latest build log (with errors): https://travis-ci.org/assimilation/assimilation-official/builds/80584496

ignore the twitter message i sent @Alan-R , thats old now

Automatically extract nanoprobe binary from nanoprobe docker image as part of docker/nanoprobe/dockit script

In other words, add that step to the tail end of that dockit script.

http://linux-ha.org/source-doc/assimilation/html/index.html

i can prob find and fix this but if someone else fixes it first, i wont stop you :)

Not sure how to do this, but Coverity implies it can be done. https://scan.coverity.com/users/sign_in. Can also import the repository somehow. I have a script which does it as well, but on-demand (manually).

The symptoms point to sort giving different results on different platforms - without any errors being indicated. https://travis-ci.org/assimilation/assimilation-official/builds/81379935

The only thing I know of that could possibly cause that is the locale. Sort command man page suggests setting LC_ALL=C. So I'm trying that.

So, you don't need to run it in a separate step.

I don't remember why you put the netconfig discovery in the stream. If you're going to run them, I'd discovery commands before the build step. That way if things mess up, we'll have that information available.

From my perspective, you might run the os discovery. It's probably more interesting to know if something goes wrong.

Here's the message:
meson.build:107: WARNING: The variable(s) 'BPINSTALL', 'DISCOVERYINSTALL', 'LICENSE_LONG', 'LICENSE_SHORT', 'MONRULEINSTALL', 'PROJVERS', 'QUERYINSTALL' in the input file 'include/projectcommon.h.in' are not present in the given configuration data.

At least DISCOVERYINSTALL and probably *INSTALL are all full-blown errors. This has to be fixed.

Not sure how to make the build fail because of this?

logic based on getting the tarball from mercurial
http://hg.linux-ha.org/|experimental/assimilation/archive

but i dont know what the github counterparts of experimental/assimilation/archive are

Not clear why. It failed for @dpquigl. Presumably it will fail for me too - but only on a real Fedora 22 kernel ;-)

https://travis-ci.org/assimilation/assimilation-official/jobs/638934844#L19162

Details are here: https://trello.com/c/ZQO848ju

For a variety of platforms - specifically CentOS, Ubuntu/Debian, SUSE, others as needed...

But start with CentOS and Ubuntu (or Debian)

For more info about FPM:

• https://github.com/jordansissel/fpm
• https://fpm.readthedocs.io/en/latest/

Need to fix that. I'm surprised that @ejratl got her tests to run given that. But her fix was obviously solid.

I've_ forked ctypesgen to here https://github.com/assimilation/ctypesgen, since upstream seems to be abandoned. I tried adding the patches as prescribed by ctypesgen/ctypesgen#61 and ctypesgen/ctypesgen#62 and I'm still hitting the same errors indicated by the issues. @Alan-R can you give it a shot at fixing ctypesgen? (make changes directly to assimilation/ctypesgen)

I've also tried other forks like olsonse/ctypesgen but they don't help either.

travis output showing ctypesgen problem: https://www.travis-ci.org/assimilation/assimilation-official/jobs/490881626#L1640

ill create a pull request for this when ready

This involves rewriting the cmake libraries.
I'd also like to build our own copy of libsodium - compiled from source. It's easy to compile & test.

The clientlib, serverlib, and libsodium need to be included in the Docker image, so you can't stop building libraries - it's just that you need to fully bind everything for the nanoprobe.

broken
http://linux-ha.org/source-doc/assimilation/html/md__r_e_a_d_m_e.html

When things break, one wants to answer the question "What changed?". If we make events persistent, then we can answer that question.

We can also (eventually) answer questions about how long it took to resolve an issue, how long servers were down, and related things. This provides the data to make these things possible.

In other words we want a docker image for the CMA that includes:

• the CMA python code and dependencies
• the client and server libraries (the exact ones built for the nanoprobe)
• llbsodium (the exact version we built from source for the nanoprobe)
• glib
  etc as needed

Note that the CMA will need to communicate with a docker image running Neo4j. There is already a Neo4j package that I'm using for testing. It is necessary for some of the Neo4j files to be accessible from this image (mainly to initialize the Neo4j password).

A relationship or node should exist for each failure, and maybe for each success. The general idea is that one should be able to easily search for all systems that failed tests, a total count of failures should be available, and it should be possible to collect the list of machines with any failure, or those which fail any given rule.

This is a subset of #19 - which has been closed and this item split off.