Comments (5)
If the 1Hz tick and clocks are working one should also see "Stop FLYWHEEL" event from CFE_TIME approximately 2 seconds after that operational message.
But overall I would advocate a more holistic approach to sanity checking the build. For instance the latter event should be checked by subscribing to the event stream and checking for event CFE_TIME_FLY_OFF_EID
. Checking the stdout for specific messages is neither reliable nor sustainable for a longer term solution.
from cfs.
Yeah, definitely debated internally what level to go to. Any positive check is better than none, but I agree a specific message on stdout is pretty fragile.
Certainly encourage ideas and thought about really what's useful/appropriate for this Travis CI check.
We lack documentation covering the bigger picture/goals of more sophisticated test setups like dockers emulated platforms, etc but from that perspective/context (and eventually automating build verification) maybe a non-empty file for this test is good enough (and less dependent on text changing)? I'm trying to catch the case where we get a false pass because nothing happened. I'd think it'd be sufficient to just check that something happened (and no warn/fail/err).
from cfs.
It would indeed be nice to automatically check the behavior of the executable, in addition to running unit tests.
In this context, I have implemented a proof of concept consisting of a fully dockerized end-to-end automated test. In substance, this solution builds and runs the cFS (currently: on CentOS 7, Ubuntu 18.04, or Ubuntu 20.04). Simultaneously, a pseudo-GSW—in fact: a program running the test—tries to enable the telemetry of the former and, if successful, asserts that an expected event (e.g. Stop FLYWHEEL
) has been emitted by the FSW. Such an assertion is performed by inspecting the datagrams sent by the FSW through the network connection, not via stdout
or stderr
.
When the test passes, the testing tool exits with code 0. When one modifies the cFS code to simulate an error (e.g. make the FSW not emit a Stop FLYWHEEL
), the error is indeed detected by the tool, and a non-zero exit code is returned. This is shown by the following screen captures.
(Note: the green and red borders are for illustrative purposes only).
In short, this prototype checks the cFS execution by simulating GSW/FSW interactions, where the GSW is, in fact, a test runner. And because the tool is fully dockerized, it would be conceivable to directly integrate it into the CI process in the future.
Obviously, this is not a full-fledged testing solution at the moment, but I think that it could help to illustrate the possibilities offered by this approach as well as concretely show its limitations and the technical challenges that it introduces.
In this context, I am wondering if it would be reasonable for me to open a pull request that would submit this tool as well as explain the underlying methodology in more detail.
from cfs.
@lethuillierg Excellent ideas, and appreciate the input! We have discussed a docker approach internally and possibly also testing for multiple platforms/OS's utilizing QEMU, eventually building up to a continuous verification system across multiple architectures. Tagging @jphickey @astrogeco and @acudmore if they have more inputs. I encourage opening a pull request!
from cfs.
Thank you @skliper. I will soon open a pull request for this purpose.
from cfs.
Related Issues (20)
- Create unit test for vxworks_sysmon HOT 1
- Large object pushed on stack by function UT_CallTaskPipe()
- TOC links not working in CONTRIBUTING.md
- Renaming the project name to the name of the containing folder causes make SIMULATION=native prep to fail HOT 8
- Consider analyzing conditionals for mc/dc coverage HOT 2
- Remove format check on PR title
- Shared workflow scripts need updates for node.js version
- 79 - coverage-es-ALL (SEGFAULT) HOT 2
- Update CodeQL to v3
- Rename unit-test-coverage.yml
- Update platform defined values to be more 'flight like'/work with BVT tests
- How hardware components are integrated with cFS applications? HOT 6
- Training guide apps and current cFS version compatibility. HOT 1
- cFS releases HOT 3
- cFS start guide
- cFS Documentation and Guides workflow fail
- python3 GroundSystem.py depends on PyQt5 and PyQt5 won't install in linux 20.04.06 HOT 1
- documentation about the pieces HOT 1
- pyqt5 woes HOT 1
- draco does not build HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from cfs.