sindri's Issues

Build failure on empty file

If something goes sideways with brokkr or the disk is full, the log file sindru reads can be empty. Sindri fails to build in these cases, with the ultimate error coming from Pandas: pandas.errors.EmptyDataError: No columns to parse from file

We need to sidestep empty files, in a similar manner as the fix for #1 I imagine.

Edge case. Low priority.

Sindri Borks on File Read

When one of the log files has a bad line, Sindri will bork and the site no longer updates. This can happen when Brokkr (the logger) fails in the middle of file write. The CSV-ness is lost.

Release Sindri 0.3.0

Release Checklist

Prepare

Close GitHub milestone and ensure all issues are resolved/moved

In a bash-like shell locally and on the Pi/server, set up the variables needed (otherwise, replace them manually)

# Full version number to release, e.g. '1.2.3b4'
VERSION='0.3.0'
# Git branch to make the release from, e.g. 'main', 'master' or '1.x'
MAIN_BRANCH='master'
# Prepare-release branch name
PREPARE_RELEASE_BRANCH='multi-sensor-dashboard'
# Release branch name, if a new major/minor release, e.g. '1.x' or '0.4.x'
RELEASE_BRANCH='0.3.x'
# Set mode to 'client' (if on a Pi) or 'server' (if on the central VPS)
MODE='MODE_NAME'

VERSION='0.3.0'; MAIN_BRANCH='master'; PREPARE_RELEASE_BRANCH='multi-sensor-dashboard'; RELEASE_BRANCH='0.3.x'; MODE='MODE_NAME'

Update local repo

git reset --hard && git switch $MAIN_BRANCH && git pull upstream $MAIN_BRANCH

Clean local repo
```
git clean -xdi
```

Perform a quick local smoke-test of the final development version

python -b -X dev -m pip install -e .
python -bb -X dev -W error -m sindri --version
python -bb -X dev -W error -m sindri serve-website --mode test
# Check that output website displays and behaves as expected, with no JS errors

Commit

Test

On the Pi/server, pull the prepare-release branch

git reset --hard && git fetch --all && git switch $PREPARE_RELEASE_BRANCH

Clean repository of old artifacts
```
git clean -xdi
```
Create and activate a fresh virtual environment for testing
```
python -m venv test-env && source test-env/bin/activate
```

Install/update the packaging stack

python -m pip install --upgrade pip
pip install --upgrade build setuptools wheel

Build the distribution packages
```
python -bb -X dev -W error -m build
```

Install from built wheel

python -b -X dev -m pip install dist/sindri-$VERSION-py3-none-any.whl

Check environment

pip check; python -I -bb -X dev -W error -m sindri --version

Test package

python -I -bb -X dev -W error -m sindri serve-website --mode $MODE
# Check that output website displays and behaves as expected, with no JS errors

Stage

On the Pi/server(s), activate the production virtual environment

Disable and stop Sindri service

sudo systemctl disable sindri-hamma-$MODE && sudo systemctl stop sindri-hamma-$MODE

Reinstall the package from the built wheel

python -b -X dev -m pip install dist/sindri-$VERSION-py3-none-any.whl

Reinstall service

sudo /$PATH_TO_VENV/bin/python -I -bb -X dev -W error -m sindri install-service --account $ACCOUNT

Restart services

sudo systemctl restart sindri-hamma-$MODE

Verify still running, no errors and functioning correctly after 60 seconds
```
systemctl status sindri-hamma-$MODE
```

Build

Activate the appropriate venv/conda environment
Clean local repo
```
git clean -xdi
```

Update the packaging stack

python -m pip install --upgrade pip
pip install --upgrade --upgrade-strategy eager build setuptools twine wheel

Build source distribution and wheel
```
python -bb -X dev -W error -m build
```

Check

Check distribution archives
```
twine check --strict dist/*
```

Check installation

python -b -X dev -m pip install dist/sindri-$VERSION-py3-none-any.whl

Check environment

pip check; python -I -bb -X dev -W error -m sindri --version

Check functionality

python -I -bb -X dev -W error -m sindri serve-website --mode test
# Check that output website displays and behaves as expected, with no JS errors

Release

Upload distribution packages to PyPI
```
twine upload dist/*
```

Create release tag

git tag -a v$VERSION -m "Sindri version $VERSION"

Merge the prepare-release branch to $MAIN_BRANCH, or the pull request
```
git switch $MAIN_BRANCH && git merge $PREPARE_RELEASE_BRANCH
```

If new major or minor version, create release branch and push

git switch -c $RELEASE_BRANCH && git push -u origin $RELEASE_BRANCH && git push upstream $RELEASE_BRANCH && git switch $MAIN_BRANCH

Finalize

Update __version__ in __init__.py (increment to next, add .dev0)
```
nano src/sindri/__init__.py
```
Create a back to work commit with the next anticipated version on the branch
```
git commit -m "Begin development of version $VERSION"
```
Reinstall the development version locally in editable mode
```
python -b -X dev -m pip install -e .
```
Push new release commits and tags to $MAIN_BRANCH
```
git push upstream $MAIN_BRANCH --follow-tags
```
Create a GitHub release from the tag with the changelog contents
Open a GitHub milestone as needed for the next release

Deploy

On the Pi/server, install the released version from PyPI
```
pip install sindri
```
Or, pull the release branch, checkout the tag and editable-install
```
git fetch --all && git switch $RELEASE_BRANCH && git checkout v$VERSION && pip install -e .
```

Restart the service and verify working

sudo systemctl restart sindri-hamma-$MODE && sleep 10 && systemctl status sindri-hamma-$MODE

Cleanup

On the Pi/server, remove the clean test virtual environment
```
rm -rfd test-env
```
Delete the prepare-release branch locally and on the Pi/server
```
git branch -d $PREPARE_RELEASE_BRANCH
```
Delete the prepare-release branch on the remote
```
git push -d origin $PREPARE_RELEASE_BRANCH
```

General color coding improvements of website

Update the color scheme of a few of the variables display on the website.

Some of these is probably specific to our use case, so at some point these values may need to be broken out to a config file, etc.

Change default layout of website

Typical use has given us an idea of what the default layout should. More important blocks should go higher up, logged values that have little variation don't need to be plotted, etc.

Location on server of webpage push

Right now, the location on the server of where the webpages are pushed is in src/website/mjolnir-website.lektorproject, and this is in version control. This value needs to be different for each sensor, so we need to find a way to have a way to specify the location differently for each sensor site.

Long time to generate web pages

Depending on the logging frequency and the number log files, it can take a long time (~10 minutes) for Sindri to generate the website.

Deprecation Warning with Pandas

When we upgrade to Pandas 1.3+, we get a deprecation warning on error_bad_lines. This was used in this fix.

Looks like we now need to use on_bad_lines='skip instead.

(Pandas 1.3+ is happening on current builds, so this should be tested and fixed soon.)

Website push should point to new server

We have a new server, so let's use it! (There are changes to make in other packages as well....)

Fix links in archive table when site is not deployed to the root of the server

The links don't work when the site is not deployed to the root of the server, as they assume that. This needs to be fixed in the Python that generates the Javascript, either by injecting the path as included in a config file, or using a relative link (ideal, but we'll need to know the current relative location that the generated code is getting deployed to, which will in turn either need to be done by the Python generation code, or injected by jinja.

project-mjolnir / sindri Goto Github PK

sindri's People

Contributors

Stargazers

Watchers

Forkers