project-mjolnir / sindri Goto Github PK
View Code? Open in Web Editor NEWSever-side code for the Mjolnir system to store, serve/display, and monitor data from remote sensors.
Home Page: https://hamma.dev
License: MIT License
Sever-side code for the Mjolnir system to store, serve/display, and monitor data from remote sensors.
Home Page: https://hamma.dev
License: MIT License
If something goes sideways with brokkr
or the disk is full, the log file sindru
reads can be empty. Sindri
fails to build in these cases, with the ultimate error coming from Pandas: pandas.errors.EmptyDataError: No columns to parse from file
We need to sidestep empty files, in a similar manner as the fix for #1 I imagine.
Edge case. Low priority.
When one of the log files has a bad line, Sindri will bork and the site no longer updates. This can happen when Brokkr (the logger) fails in the middle of file write. The CSV-ness is lost.
Close GitHub milestone and ensure all issues are resolved/moved
In a bash-like shell locally and on the Pi/server, set up the variables needed (otherwise, replace them manually)
# Full version number to release, e.g. '1.2.3b4'
VERSION='0.3.0'
# Git branch to make the release from, e.g. 'main', 'master' or '1.x'
MAIN_BRANCH='master'
# Prepare-release branch name
PREPARE_RELEASE_BRANCH='multi-sensor-dashboard'
# Release branch name, if a new major/minor release, e.g. '1.x' or '0.4.x'
RELEASE_BRANCH='0.3.x'
# Set mode to 'client' (if on a Pi) or 'server' (if on the central VPS)
MODE='MODE_NAME'
or
VERSION='0.3.0'; MAIN_BRANCH='master'; PREPARE_RELEASE_BRANCH='multi-sensor-dashboard'; RELEASE_BRANCH='0.3.x'; MODE='MODE_NAME'
Update local repo
git reset --hard && git switch $MAIN_BRANCH && git pull upstream $MAIN_BRANCH
Clean local repo
git clean -xdi
Perform a quick local smoke-test of the final development version
python -b -X dev -m pip install -e .
python -bb -X dev -W error -m sindri --version
python -bb -X dev -W error -m sindri serve-website --mode test
# Check that output website displays and behaves as expected, with no JS errors
Create a new branch for the release
git switch -c $PREPARE_RELEASE_BRANCH
Ensure docs and metadata are up to date and commit any changes
Update CHANGELOG.md
with the latest changes
Update __version__
in __init__.py
(set release version, remove .dev0
)
nano src/sindri/__init__.py
Create release commit
git commit -m "Release Sindri version $VERSION"
Push the prepare-release branch to your fork
git push -u origin $PREPARE_RELEASE_BRANCH
Open a pull request for the branch
On the Pi/server, pull the prepare-release branch
git reset --hard && git fetch --all && git switch $PREPARE_RELEASE_BRANCH
Clean repository of old artifacts
git clean -xdi
Create and activate a fresh virtual environment for testing
python -m venv test-env && source test-env/bin/activate
Install/update the packaging stack
python -m pip install --upgrade pip
pip install --upgrade build setuptools wheel
Build the distribution packages
python -bb -X dev -W error -m build
Install from built wheel
python -b -X dev -m pip install dist/sindri-$VERSION-py3-none-any.whl
Check environment
pip check; python -I -bb -X dev -W error -m sindri --version
Test package
python -I -bb -X dev -W error -m sindri serve-website --mode $MODE
# Check that output website displays and behaves as expected, with no JS errors
On the Pi/server(s), activate the production virtual environment
Disable and stop Sindri service
sudo systemctl disable sindri-hamma-$MODE && sudo systemctl stop sindri-hamma-$MODE
Reinstall the package from the built wheel
python -b -X dev -m pip install dist/sindri-$VERSION-py3-none-any.whl
Reinstall service
sudo /$PATH_TO_VENV/bin/python -I -bb -X dev -W error -m sindri install-service --account $ACCOUNT
Restart services
sudo systemctl restart sindri-hamma-$MODE
Verify still running, no errors and functioning correctly after 60 seconds
systemctl status sindri-hamma-$MODE
Activate the appropriate venv/conda environment
Clean local repo
git clean -xdi
Update the packaging stack
python -m pip install --upgrade pip
pip install --upgrade --upgrade-strategy eager build setuptools twine wheel
Build source distribution and wheel
python -bb -X dev -W error -m build
Check distribution archives
twine check --strict dist/*
Check installation
python -b -X dev -m pip install dist/sindri-$VERSION-py3-none-any.whl
Check environment
pip check; python -I -bb -X dev -W error -m sindri --version
Check functionality
python -I -bb -X dev -W error -m sindri serve-website --mode test
# Check that output website displays and behaves as expected, with no JS errors
Upload distribution packages to PyPI
twine upload dist/*
Create release tag
git tag -a v$VERSION -m "Sindri version $VERSION"
Merge the prepare-release branch to $MAIN_BRANCH
, or the pull request
git switch $MAIN_BRANCH && git merge $PREPARE_RELEASE_BRANCH
If new major or minor version, create release branch and push
git switch -c $RELEASE_BRANCH && git push -u origin $RELEASE_BRANCH && git push upstream $RELEASE_BRANCH && git switch $MAIN_BRANCH
Update __version__
in __init__.py
(increment to next, add .dev0
)
nano src/sindri/__init__.py
Create a back to work
commit with the next anticipated version on the branch
git commit -m "Begin development of version $VERSION"
Reinstall the development version locally in editable mode
python -b -X dev -m pip install -e .
Push new release commits and tags to $MAIN_BRANCH
git push upstream $MAIN_BRANCH --follow-tags
Create a GitHub release from the tag with the changelog contents
Open a GitHub milestone as needed for the next release
On the Pi/server, install the released version from PyPI
pip install sindri
Or, pull the release branch, checkout the tag and editable-install
git fetch --all && git switch $RELEASE_BRANCH && git checkout v$VERSION && pip install -e .
Restart the service and verify working
sudo systemctl restart sindri-hamma-$MODE && sleep 10 && systemctl status sindri-hamma-$MODE
On the Pi/server, remove the clean test virtual environment
rm -rfd test-env
Delete the prepare-release branch locally and on the Pi/server
git branch -d $PREPARE_RELEASE_BRANCH
Delete the prepare-release branch on the remote
git push -d origin $PREPARE_RELEASE_BRANCH
Update the color scheme of a few of the variables display on the website.
Some of these is probably specific to our use case, so at some point these values may need to be broken out to a config file, etc.
Typical use has given us an idea of what the default layout should. More important blocks should go higher up, logged values that have little variation don't need to be plotted, etc.
Right now, the location on the server of where the webpages are pushed is in src/website/mjolnir-website.lektorproject
, and this is in version control. This value needs to be different for each sensor, so we need to find a way to have a way to specify the location differently for each sensor site.
Depending on the logging frequency and the number log files, it can take a long time (~10 minutes) for Sindri to generate the website.
When we upgrade to Pandas 1.3+, we get a deprecation warning on error_bad_lines
. This was used in this fix.
Looks like we now need to use on_bad_lines='skip
instead.
(Pandas 1.3+ is happening on current builds, so this should be tested and fixed soon.)
We have a new server, so let's use it! (There are changes to make in other packages as well....)
The links don't work when the site is not deployed to the root of the server, as they assume that. This needs to be fixed in the Python that generates the Javascript, either by injecting the path as included in a config file, or using a relative link (ideal, but we'll need to know the current relative location that the generated code is getting deployed to, which will in turn either need to be done by the Python generation code, or injected by jinja.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.