For convenience, it would be nice to have some high-level tools, e.g to handle the tasks outlined in the repo example.

I would like to know the correct steps for removing corrupted or unwanted versions from the repository. Here's what I assume should be done:

1. Modify targets.json metadata:

   • Remove the entries related to the specific version from the signed.targets object. This includes removing the .tar.gz and .patch references that correspond to the version we want to remove.

2. Remove files from the repository:

   • Delete the actual .tar.gz and .patch files from the repository's targets directory.

Could someone please confirm if my understanding is correct? Are there any additional steps or precautions I should be aware of when performing this task?

Is there any chance that this will support github releases, S3 buckets and release channels like pyupdater. Also, I know this sounds like an odd question but what are the chances this repo gets archived like pyupdater I would hate to have to find a new way to update python apps haha. Thanks for your time in advance

Description

If an update server requires authentication, it may be necessary to provide credentials using HTTP headers.

However, notsotuf.client.Client does not support custom HTTP headers.

Background

The notsotuf.client.Client class is a subclass of tuf.ngclient.Updater, which does not support custom headers out-of-the-box.

Two solutions are suggested in tuf issue 1011:

• provide a .netrc file for requests to use (workaround)
• provide a custom Fetcher that handles the headers

Proposed solution

Implement a custom Fetcher that supports custom HTTP headers.
This class should support the FetcherInterface.
Ideally it would be an minimal extension of the tuf.ngclient._internal.requests_fetcher.RequestsFetcher class (except for the fact that this is clearly marked _internal...)

Description

The client currently has a single Client.update() method that checks for available updates and asks for confirmation via the command line before downloading and applying the update:

https://github.com/dennisvang/notsotuf/blob/1321f81b499c36c743287de3c6e1106bdb01344c/src/notsotuf/client.py#L84

This works for simple applications, but a separate update-check method would provide more flexibility.

Current:

notsotuf_client = Client(...)
notsotuf_client.update(...)

Proposed:

notsotuf_client = Client(...)
if notsotuf_client.check_for_updates(...):
    notsotuf_client.update(...)

Our repo code includes an import from setuptools, but this causes a failure on the client side if the app is bundled using PyInstaller.

https://github.com/dennisvang/notsotuf/blob/499ad1e60c4fdbc174d8cad04f8544ce74efc749/src/notsotuf/repo/__init__.py#L7

This is due to PyInstaller issue 6911.

The import succeeds on the repo side, and is not required on the client side, so we should be able to work around this problem.

As mentioned in the README, after creating and publishing the initial version of notsotuf, I found out that there once existed another package called Not-So-TUF. Moreover it seems this was actually the original name for PyUpdater.

Although this is a coincidence, and the not-so-tuf name seems to have been abandoned, it still doesn't feel right to continue using it.

Therefore I propose to rename the github repo and corresponding package from notsotuf to tufup. Short for TUF-updater, or tuffen-up or whatever you like. It is a bit shorter, which is convenient.

I decided against tuf-up because I don't like hyphens in package names and want repository name and package name to match, to prevent ambiguity.

As there don't seem to be any users yet, now would be a good time to make this change.

@NicklasTegner Sorry for the inconvenience.

python-tuf #2047 is a blocking issue for us, so we need to release a workaround, as long as the issue has not been fixed on the python-tuf side.

PR awaiting review.

Current __version__ is 2022.0. Although there is nothing wrong with using the year as major version, it feels inconvenient for a package. Moreover, from a semantic versioning point of view, we should use major version zero during initial development, as "Anything MAY change at any time."

As it's still early days, now would be the time to switch to major version zero, and forget all about this.

The recently released python-tuf v2.1.0 introduces some breaking changes (for us).

This may be related to our use of the private Updater._trusted_set. Need to investigate further.

As a temporary workaround, I've pinned python-tuf to v2.0.*.

UPDATE:
pyton-tuf v3.0.0 has now been released, so we should adapt to that.

In the tufup.client.Client API, what does the refresh_required flag do?

I'm to the point where the app opens, downloads the new update and alerts the user and then just closes. Trying to think through the best way to present this scenario to the user. I guess auto opening the app again after update isn't a thing is it?

Is the update process with tufup compatible with an exe app created in PyQT6?

Background

The .tufup-repo-config file, that is created when initializing a repository, includes absolute paths for the keys_dir and repo_dir.

Although these can easily be modified, when necessary, the file should be portable out-of-the-box.

Proposal

It would be more convenient to use relative paths in the .tufup-repo-config file. That is, relative to the config file location.

Description

The str type is not enforced in TargetMeta.__str__:

https://github.com/dennisvang/notsotuf/blob/f1766fd429f1b57d4ef3a8b9e42401a96ca3118b/src/notsotuf/common.py#L40-L41

This causes a TypeError if TargetMeta.target_path_str is not a str.

How to reproduce

Run this:

import pathlib
from notsotuf.common import TargetMeta

str(TargetMeta(target_path=pathlib.Path('something')))

Expected behavior

• TargetMeta.__str__ should always return a str, regardless of the type of target_path_str
• target_path_str should always be a str
• the TargetMeta argument target_path should accept both str and pathlib.Path

Describe the bug
Creating the patch file can take x4 the time it takes to generate a new version archive, while the file is marginally smaller.

I am releasing the updates on a raspberry pi 4 which is not a powerful computer but the target device is also a raspberry pi so I am restricted to that device.

Generating a new update archive takes around 10 minutes while the patch can sometimes take over an hour, the difference in the file size is only around 20-30mb. This behavior is consistent across multiple project that we are developing and use tufup.

To Reproduce
Steps to reproduce the behavior:

1. Create a project on Raspberry Pi
2. Pack the application with PyInstaller using directory option
3. Generate update and patch files
4. See error

Expected behavior
An option to skip the generation of patch files while creating a new release.

Screenshots

System info (please complete the following information):

• OS: Raspbian
• Python version [e.g.3.10.6]: 3.9

Additional context
python = "3.9.*"
numpy = "1.23.1"
sounddevice = "0.4.4"
tufup = "0.4.8"
pandas = "1.4.3"
pytz = "2022.1"
"RPi.GPIO" = "0.7.1"
spidev = "3.5"
setuptools = "61.0.0"
kivy = {git = "https://github.com/kivy/kivy.git", rev = "2.1.0"}
kivymd = "^1.1.1"

Background

The Client provides a way to specify a custom install function, using the install argument for donwload_and_apply_update, for example:

...
client.download_and_apply_update(.., install=my_custom_install_function, ...)
...

However, in many cases, on Windows, the only thing that really needs to be modified is the content of the batch file that is written by the default install function. This batch file content is defined in WIN_MOVE_FILES_BAT.

This can be done using a custom install function, but that's quite cumbersome, as becomes clear e.g. from this discussion.

An alternative would be to monkey-patch the WIN_MOVE_FILES_BAT variable, but that is quite hacky.

Proposal

For this reason it would be convenient if users could specify a custom batch file template, which would be used instead of WIN_MOVE_FILES_BAT.

In addition, it would be nice to allow custom template variables, in addition to the ones that are defined in WIN_MOVE_FILES_BAT.

I already have some users using an exe compiled with pyupdater, how can I push out an update that switches them over to this library ?

Currently we have a minimal pyproject.toml, minimal setup.py (for editable installs), and a setup.cfg containing the actual project metadata.

It looks like pyproject.toml is no longer experimental, and can replace setup.cfg when using modern versions of pip and setuptools.

To prevent confusion, it would be nice to move the project metadata into pyproject.toml and remove setup.py and setup.cfg.

Some detailed documentation would be nice.

Hello,

My executable has a digital signature applied to it for my company. When tufup applies a new patch, the new executable is no longer signed. What options do we have to keep the application with a digital signature?

The current implementation of the windows install script uses robocopy with the /purge option.

This removes everything from the destination directory that is not in the current update archive.
However, the user may not want to purge everything.

Therefore, it would be convenient to allow the user to specify additional robocopy options for the windows install script, e.g. /xf to exclude files from purge.

Feature request:

One thing I've learned from the logging is why the update always takes 30 extra seconds.

From the log file:

2022/09/22 20:11:20 ERROR 32 (0x00000020) Copying File C:\Users\aaronb\AppData\Local\Temp\tufup\AMP.exe
The process cannot access the file because it is being used by another process.

Waiting 30 seconds... Retrying...
	    Older     		  62.8 m	AMP.exe
  1.5%
  3.1%
  4.7%

It seems like every time it starts the patch process it detects the app is still running, waits 30 seconds and then is able to patch.

Would it be possible to get a variable to set the timeout option so I can make it a little quicker than 30 seconds?

I'm using the progress hook to send updates to a GUI progress bar to provide feedback to users as the update is being downloaded.

client.download_and_apply_update(progress_hook=progress_hook)

I then have the progress_hook for processing the data.

def progress_hook(bytes_downloaded, bytes_expected):
          progress_percent = bytes_downloaded / bytes_expected * 100
          self.onProgress(progress_percent)
          if progress_percent >= 100:
              print("Update has finished")
              self.thread_finished()  
              buttonUpdate = QMessageBox.information(self,'Update Success!', "AMP was updated successfully. Please start the app again.", QMessageBox.StandardButton.Ok, QMessageBox.StandardButton.Ok)
              if buttonUpdate == QMessageBox.StandardButton.Ok:
                  sys.exit()

Instead of being able to update, this function just locks up and never gets called and receives the hook data

Any thoughts? The update downloads fine. It just doesn't call the hook.

When installing a new update to an executable, the tufup script that runs in the cmd line window deletes ALL files in the same folder as your original executable.

My users will commonly keep my .exe on their desktop or in their downloads folder. In either of those locations, users open the app, a new update is detected, the update is downloaded and applied but then EVERYTHING in the same folder as the original executable is deleted and the only thing remains is the new properly updated executable.

The patch process works perfectly but all the files being deleted in the same folder is a CRITICAL error.

Describe the bug
I have generated a Version 1 of a python application buddled with pyinstaller. This package contains images, libraries, the .exe and my .py files that have been converted to .pyd (binaries). One of those .pyd files states the version of the file. If only change the version of that .pyd file without running pyinstaller again to generate the second version of the bundle with tufup I get a file difference of 200MB, which is crazy if you take into account that the whole package is 340MB. The last modification date for these files are the same except the .pyd file file that states the version. Using the bsdiff4.file_diff() method between these two version produces the same result. I can provide both of these files If needed.

To Reproduce
Steps to reproduce the behavior:

1. Run cython and pyinstaller with .spec required to make the bundle.
2. Copy all files except 1 folder containing some images
3. Modify version.py file and re-run cython for that file to generate version.pyd and copy over the 1 folder mentioned above from the source (folder has not changed and copied with shutil.cptree() so the modification dates are the same)
4. When I run the repo.add_bundle(new_bundle_dir=bundle_dir) method I get the filesize mentioned above for the patch

Expected behavior
A patch size that is less than 10 MB. On a previous run, I regenerated just the .exe (running pyinstaller and just copying the .exe and deleting everything else while I follow the steps mentioned above.) The .exe filesize is 17mb while the generated patch was 35MB for that run.

System info (please complete the following information):

• OS: [Window 11]
• Python version 3.9
• Pyinstaller version 5.9.0
• Tufup version 0.4.9
• bsdiff4 version 1.2.3

Hi.
After reading the readme and thinking, I would highly advice to make MacOS support a requirement for version 1.0.
Very few people (including myself) would sadly not be able to use this library, just because it doesn't support MacOS.

I am doing some tests with tufup on an application and when tufup installs the update from the local server, it works just fine. Only, it installs in a subfolder within the installation directory. So the application can never start because it never actually get overwritten by the new one.

AppData
|---Local
    |---Scheduling        (Installation Folder)
        |---Scheduling    (Where tufup installs the updates)

Steps to reproduce the behavior:
Following exact steps from tufup-example, but with my own application.

1. Create app with version 0.1
2. Bundle with tufup
3. Change version to 0.2 & recreate
4. Bundle with tufup
5. Copy extracted files from Scheduling-0.1.tar.gz to C;/Users/newing/AppData/Local/Scheduling/
6. Use http.server host from repository folder
7. Run application, it stops, cmd opens for a few seconds (installing)
8. All files from the update are in C:/Users/newing/AppData/Local/Scheduling/Scheduling/

Log file from after update:
install.log

Code from settings.py file in application:
(Almost an exact match to the example)

import toml, pathlib, os, sys
from file import getFilePath

FROZEN = getattr(sys, "frozen", False)

APP_NAME = "Scheduling"
APP_VERSION = "0.2"

# ?:/Program Files
# ?:/ProgramData
WIN_PER_MACHINE_PROGRAMS_DIR = pathlib.Path(os.getenv("ProgramFiles"))
WIN_PER_MACHINE_DATA_DIR = pathlib.Path(os.getenv("PROGRAMDATA"))

# ?:/Users/user/AppData/Local
# ?:/Users/user/AppData/Local/Programs
WIN_PER_USER_PROGRAMS_DIR = pathlib.Path(os.getenv("LOCALAPPDATA"))
WIN_PER_USER_DATA_DIR = WIN_PER_USER_PROGRAMS_DIR / "Programs"

# sys._MEIPASS if FROZEN, else 'updater' folder 
MODULE_DIR = pathlib.Path(__file__).resolve().parent

DEV_DIR = MODULE_DIR.parent.parent / "releases"

PROGRAMS_DIR = WIN_PER_USER_PROGRAMS_DIR if FROZEN else DEV_DIR
DATA_DIR = WIN_PER_USER_DATA_DIR if FROZEN else DEV_DIR
INSTALL_DIR = PROGRAMS_DIR / APP_NAME
UPDATE_CACHE_DIR = DATA_DIR / APP_NAME / "update_cache"
METADATA_DIR = UPDATE_CACHE_DIR / "metadata"
TARGET_DIR = UPDATE_CACHE_DIR / "targets"

toml_file = toml.load(getFilePath("config/appconfig.toml", False))
METADATA_BASE_URL = toml_file["updates"]["metadata_url"]
TARGET_BASE_URL = toml_file["updates"]["target_url"]

if FROZEN:
    TRUSTED_ROOT_SRC = MODULE_DIR.parent / "root.json"
else:
    TRUSTED_ROOT_SRC = MODULE_DIR.parent.parent / "repo/deploy/metadata/root.json"
TRUSTED_ROOT_DST = METADATA_DIR / "root.json"

Subfolder with the updated app file:

System Info:

• OS: Windows 10
• Python version: 3.8.6
• Pip Version: 22.3
• Tufup Version: 0.4.4

If I missed something important just let me know.

Background

The tufup client extracts new update archives to a temporary directory. By default, on Windows, this is a per-user temp dir: AppData\Local\Temp\tufup

Tufup then uses a batch script to move the extracted files to the installation directory. The temporary dir is then assumed to be empty, and is left in place for future use.

Note that Windows does not automatically clear content of its default temporary directories.

Problem

The problem with this approach is that any files that are left-over in the tufup temporary dir (for whatever reason), will get moved along with newly extracted archives. This can happen e.g. when files are not moved because they already exist in the install dir.

This can lead to stale files cluttering the new install directory, as mentioned here. This is not a problem per se, but can be confusing, and may lead to unexpected issues in the future.

How to reproduce

Run an update with the tufup-example. After the update, some files that have not changed between app versions have not been moved, so they remain in the AppData\Local\Temp\tufup.

These files will be moved along with future update files, even if the app content has changed completely.

Proposed solution

The best solution would be to properly clean up the tufup temp dir after moving the files.

However, users may override the install process, so we cannot always be sure the tufup temp dir has been cleaned up.

Therefore, we should also make sure the tufup temp dir is empty before extracting a new archive.

Description

Currently dependencies are specified both in setup.cfg and in requirements.txt.

This allows us to pin specific package versions via requirements.txt, but it can also lead to problems if the two files are out of sync, as can be seen here.

Proposed solution

A single source of truth for dependencies is much more convenient.

We should specify minimal versions for dependencies in setup.cfg only (see e.g. 2, 3), and use the . trick in requirements.txt (see e.g. here).

We can also get rid of the requirements-dev.txt file, and just do the following in requirements.txt:

# install dev dependencies
build
twine
# install dependencies from setup.cfg (or pyproject.toml) in editable mode
-e .

Update installation as admin is currently not supported out-of-the-box.

The user needs to provide a custom install function if they require elevated rights.

However, this can be tricky, so it would be more convenient for users if admin install is supported out-of-the-box.

To keep things simple, the initial implementation was aimed at a single key per top-level role.

However, TUF supports multiple keys per role, with signature thresholds.

Thus, if we want to improve TUF conformance, we should also support multiple keys.

This mostly has to do with the implementation of our key_map:

https://github.com/dennisvang/notsotuf/blob/77fce835be5a320131c74b1fc866b7d1e5deed98/src/notsotuf/repo.py#L113

As mentioned by @NicklasTegner here, support for progress hooks would be useful.

A progress hook, in this context, is a callable that will be called after each downloaded chunk, in order to provide feedback to the user about download progress.

Unfortunately python-tuf does not support progress hooks out-of-the-box, as discussed in 1057 and 1213, but they can be implemented using a custom Fetcher.

As we already use a custom Fetcher for authentication, adding support for progress hooks should not be a big problem.

Our AuthRequestsFetcher._get_session override can handle basic <scheme>://<server> urls, e.g. https://example.com, but it breaks down if we add a url path, e.g. https://example.com/some/path : session.auth remains None

Here's the culprit:

https://github.com/dennisvang/notsotuf/blob/a2a98dc1e8e44b6324d7d5830e3009446703ff60/src/notsotuf/client.py#L295

We need to remove the path from the url key.

A workaround would be to use the complete url as key in the session_auth dict.

The recently released python-tuf v2.1.0 introduces an experimental repository package.

(UPDATE: still experimental in python-tuf v3.0.0)
(UPDATE: still experimental in python-tuf v3.1.0)

This is not part of python-tuf's stable API yet, but, as soon as it does become stable, we should replace our custom repository tools by those from python-tuf (as far as possible).

As mentioned in #9, we should use standard tools from python-tuf as much as possible.

When using tufup on Linux, everything works until the installation step of a patch. Is there any reason why it has not been implemented? If there are no limitations, I would try to implement it myself

python-tuf version 2.0 has been released, so we need to upgrade

I propose making a series of tests that goes through an actual PyInstaller and notsotuf updater workflow (minus the PyInstaller building).

These tests would run after the current ones if possible, so we know that everything should work.
So something like the following:

• Initialize repository
• Commit first version of zipped app
• Make patch
• Upload to repo (in this case just sign and move files)
• Run initial app and check if it updates.

This would insure that implementations on all platform we test for will continue to work, regardless of changes internally.

Currently we can only add targets to the tuf repository.

However, it should also be possible to remove targets.

To protect integrity, we should only be able to remove the latest target.

Basically a rollback action.

Feature Request:
Would you please consider adding a new CLI argument to 'targets' to allow it to simply copy the package file directly (and update targets.json) without needing to gzip or make patches? We're using PEX to package our app instead of Pyinstaller so we don't need to gzip or create patches. We simply need to copy the pex from from eg. /temp/dist to /repository/targets and register it's hash/file size in target.json. This will make tufup more versatile to cater to other packaging solutions.

Is this still supported? Pyupdater is no longer being maintained.

Hello, is there a way to see a summary of new things added in the updated versions? As I'm pretty new with this framework I'd like to be able to see a broad summary of the updates so I know about potential impacts to my workflow.

This is going to fall more into a procedural question than a problem with tufup, but I believe this could be helpful to others as well. I'm experiencing problems with my .exe failing with "timestamp.json is expired" when it begins a .check_for_udpates().

I've followed the feedback from The Update Framework's Frequently Asked Questions where it recommends having timestamp.json expiring after a day.

The Timestamp and Snapshot metadata should normally have a short expiration (1 day), whereas the Root and Targets metadata should expire less often (1 year). A good rule of thumb is the more often the metadata changes, the sooner it should expire.

So, my timestamp is reporting it's expired as expected with it being set for a day. It's keeping my executable from now running. I am completely missing what is supposed to be resigning the timestamp.json after one day?

I have my metadata files hosted in an S3 bucket that the app is checking for updates. What is the process to keep the file signed and updated and the executable not failing?

Currently, remove_path expects a pathlib.Path as argument.

https://github.com/dennisvang/notsotuf/blob/3fc5d84c59540a746d23e47a979abfbaf433e93a/src/notsotuf/utils/__init__.py#L8

This is a bit fragile and inconvenient, so it may be better to accept str arguments as well.

Describe the bug
If a patch is registered in the repo, and it is smaller than the full bundle, the client will try to download the patch.
However, if the patch file is missing, whereas the full bundle is available, the client does not fall back to the full bundle.

Instead, we get a tuf.api.exceptions.DownloadHTTPError: 400 Client Error: Bad Request for url: .../my-app-1.2.3.patch

To Reproduce
Steps to reproduce the behavior:

1. create a new app bundle
2. add the bundle to the tufup repo, with patch
3. upload the metatdata and full app bundle to your update server, but do not upload the patch
4. try to update an existing client

this results in a status 400 client error due to missing patch file

Expected behavior

probably a fallback to the existing full bundle

Description

Keys.__init__ and Keys.create currently assume that key names match role names:

https://github.com/dennisvang/notsotuf/blob/424004b3fca6edbdb9724d78b7cd3b05ccfd984b/src/notsotuf/repo.py#L137

https://github.com/dennisvang/notsotuf/blob/424004b3fca6edbdb9724d78b7cd3b05ccfd984b/src/notsotuf/repo.py#L162

However, it would be much more convenient if the user could map role names to key names.

A mapping could be supplied to Keys.__init__ as a key_map argument.

In addition, Keys.encrypted should contain key names instead of role names (the key is encrypted, not the role).

On Windows we don't use tar.gz, usually .zip how can you use .zip for targets with your package?

An issue to track discussion regarding the disign and implementation of notsotufs CLI.

"To get started, create a local example repository by running the

repo_workflow_example.py script."

I run repo_workflow_example and this error appeared