I recently found your project while struggling to sanely implement aptly with weekly snapshots for different environments. However, while trying to install it on ubuntu (xenial), I came to realize that the repo is actually empty. I am saddened greatly.

As in title, following the "Documentation" link will lead to an address, where authentication is needed and probably isn't meant for the public.

I do not think this is intentional..

Rotated snapshots are not needed and could be cleaned up. Repositories (e.g. Ubuntu) include all packages so a cleanup of snapshots wouldn't really help as the newest snapshot would still include all packages. Hence the aptly filter (only latest package) would need to be implemented.

Once this is implemented and snapshots are also cleaned up a aptly db cleanup would remove packages which are not references anymore.

Issue in aptly to be completed: adfinis-forks/aptly#28

Tasks to be implemented in pyaptly:

• Using new filter option in mirror filter
• Delete snapshots which are not current

The link to the documentation in the readme leads to a gitlab-pages 404 page.

Is this repo still maintained by the adfinis sygroup? (old issues & PRs, last commit 2017)
And if you do not want to maintain it, could you release it into the hands of someone who would like to maintain it, please?

When many snapshots (e.g. 1200) are present in aptly read_snapshot_map takes long time to process the data.

It would be nice to have a command line parameter to redirect the (debug) output of pyaptly / aptly into a separate file. Example:

$ pyaptly --debug --log output.log -c trusty.yml snapshot update

What do you think?

The snapshot update command fails when having publish endpoints, which get published directly (without snapshot), configured:

$ pyaptly -c trusty.yml snapshot update
Traceback (most recent call last):
  File "/usr/local/bin/pyaptly", line 1794, in <module>
    main()
  File "/usr/local/bin/pyaptly", line 775, in main
    args.func(cfg, args)
  File "/usr/local/bin/pyaptly", line 1284, in snapshot
    for cmd in cmd_snapshot(cfg, snapshot_name, snapshot_config)
  File "/usr/local/bin/pyaptly", line 1534, in cmd_snapshot_update
    if is_publish_affected(publish_name, publish_conf_entry)
  File "/usr/local/bin/pyaptly", line 1519, in is_publish_affected
    for snap in publish['snapshots']:
KeyError: 'snapshots'

Example publish endpoint where the repo is published directly:

  icaclient:
    -
      distribution: "stable"
      origin: "icaclient"
      architectures: ["amd64", "i386"]
      components: "main"
      repo: "icaclient"
      automatic-update: true

Hello,

I need to setup some repositories, but for some reason I need to publish by example docker mirrors, like this:

aptly publish -gpg-key="<GPG_KEY_IDENTIFIER>" snapshot docker-bookworm-<date> docker/bookworm
aptly publish -gpg-key="<GPG_KEY_IDENTIFIER>" snapshot docker-bullseye-<date> docker/bullseye

There is no documentation for the project or to be more precise since docs.adfinis-sygroup.ch stopped, probably.
Can you port documentation on Github ?

Or please, can you help me to configure the local publish endpoint ?

[publish]
# Bookworm
[[publish.bookworm-main]]
gpg-key = "<GPG_KEY>"
skip-contents = true
automatic-update = true
components = "main"
distribution = "bookworm"
[[publish.bookworm.snapshots]]
name = "bookworm-main-%T"
timestamp = "current"
archive-on-update = "archived-bookworm-main-%T"

[[publish.bookworm-updates]]
gpg-key = "<GPG_KEY>"
automatic-update = true
components = "main"
distribution = "bookworm-updates"
[[publish.bookworm-updates.snapshots]]
name = "bookworm-updates-%T"
timestamp = "current"
archive-on-update = "archived-bookworm-updates-%T"

[[publish.bookworm-security]]
gpg-key = "<GPG_KEY>"
automatic-update = true
components = "main"
distribution = "bookworm-security"
[[publish.bookworm-security.snapshots]]
name = "bookworm-security-%T"
timestamp = "current"
archive-on-update = "archived-bookworm-security-%T"

[[publish.docker-bookworm]]
gpg-key = "<GPG_KEY>"
automatic-update = true
components = "stable"
distribution = "bookworm"
[[publish.docker-bookworm.snapshots]]
name = "docker-bookworm-%T"
timestamp = "current"
archive-on-update = "archived-docker-bookworm-%T"

[[publish.php-bookworm]]
gpg-key = "<GPG_KEY>"
automatic-update = true
components = "main"
distribution = "bookworm"
[[publish.php-bookworm.snapshots]]
name = "php-bookworm-%T"
timestamp = "current"
archive-on-update = "archived-php-bookworm-%T"

[[publish.postgres-bookworm]]
gpg-key = "<GPG_KEY>"
automatic-update = true
components = "main"
distribution = "bookworm-pgdg"
[[publish.postgres-bookworm.snapshots]]
name = "postgres-bookworm-%T"
timestamp = "current"
archive-on-update = "archived-postgres-bookworm-%T"

# Bullseye
[[publish.bullseye-main]]
gpg-key = "<GPG_KEY>"
skip-contents = true
automatic-update = true
components = "main"
distribution = "bullseye"
[[publish.bullseye.snapshots]]
name = "bullseye-main-%T"
timestamp = "current"
archive-on-update = "archived-bullseye-main-%T"

[[publish.bullseye-updates]]
gpg-key = "<GPG_KEY>"
automatic-update = true
components = "main"
distribution = "bullseye-updates"
[[publish.bullseye-updates.snapshots]]
name = "bullseye-updates-%T"
timestamp = "current"
archive-on-update = "archived-bullseye-updates-%T"

[[publish.bullseye-security]]
gpg-key = "<GPG_KEY>"
automatic-update = true
components = "main"
distribution = "bullseye-security"
[[publish.bullseye-security.snapshots]]
name = "bullseye-security-%T"
timestamp = "current"
archive-on-update = "archived-bullseye-security-%T"

[[publish.docker-bullseye]]
gpg-key = "<GPG_KEY>"
automatic-update = true
components = "stable"
distribution = "bullseye"
[[publish.docker-bullseye.snapshots]]
name = "docker-bullseye-%T"
timestamp = "current"
archive-on-update = "archived-docker-bullseye-%T"

[[publish.php-bullseye]]
gpg-key = "<GPG_KEY>"
automatic-update = true
components = "main"
distribution = "bullseye"
[[publish.php-bullseye.snapshots]]
name = "php-bullseye-%T"
timestamp = "current"
archive-on-update = "archived-php-bullseye-%T"

[[publish.postgres-bullseye]]
gpg-key = "<GPG_KEY>"
automatic-update = true
components = "main"
distribution = "bullseye-pgdg"
[[publish.postgres-bullseye.snapshots]]
name = "postgres-bullseye-%T"
timestamp = "current"
archive-on-update = "archived-postgres-bullseye-%T"

If the configuration file is "wrong", it leads to seemingly fatal crashes, like the following when the configuration doesn't have any snapshots defined:

# pyaptly publish config.toml create
ERROR: snapshot
               Wrote traceback to: /tmp/tmpwpp159mr

For better UX, it should explain what is missing/wrong... I assume that's something which exists in various places.

Currently when setting up the aptly lastest/next-stable/stable system we cannot bootstrap all these publish point at once.

When calling the following commands:

release=bionic
  pyaptly -c /etc/pyaptly/${release}_rlc.yml repo create
  echo "Creating mirror..."
  pyaptly -c /etc/pyaptly/${release}_rlc.yml mirror create
  echo "Updating mirror..."
  pyaptly -c /etc/pyaptly/${release}_rlc.yml mirror update
  echo "Working on: ${release}"
  echo "Creating snapshot..."
  pyaptly -c /etc/pyaptly/${release}_rlc.yml snapshot create
  pyaptly -c /etc/pyaptly/${release}_rlc.yml publish create
  echo "Creating stable snapshot..."
  pyaptly -c /etc/pyaptly/${release}_rlc.yml publish create ubuntu/stable

The configuration looks like

...
      distribution: "bionic"
      origin: "Ubuntu"
      architectures: ["amd64", "i386", "source"]
      components: ["main", "restricted", "universe", "multiverse"]
      snapshots:
        - name: "bionic-main_foo-keyring-stable-%T"
          timestamp: "previous"
          archive-on-update: "archived-bionic-main_foo-keyring-stable-%T"
        - name: "bionic-restricted-stable-%T"
          timestamp: "previous"
          archive-on-update: "archived-bionic-restricted-stable-%T"
        - name: "bionic-universe-stable-%T"
          timestamp: "previous"
          archive-on-update: "archived-bionic-universe-stable-%T"
        - name: "bionic-multiverse-stable-%T"
          timestamp: "previous"
          archive-on-update: "archived-bionic-multiverse-stable-%T"
      automatic-update: false
...

Creating snapshot...
Creating stable snapshot...
ERROR: unable to publish: snapshot with name bionic-main_foo-keyring-stable-20180825T0000Z not found
Traceback (most recent call last):
  File "/usr/local/bin/pyaptly", line 1842, in <module>
    main()
  File "/usr/local/bin/pyaptly", line 803, in main
    args.func(cfg, args)
  File "/usr/local/bin/pyaptly", line 1289, in publish
    cmd.execute()
  File "/usr/local/bin/pyaptly", line 267, in execute
    self._finished = subprocess.check_call(self.cmd)
  File "/usr/lib/python2.7/subprocess.py", line 190, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['aptly', 'publish', '-origin=Ubuntu', '-component=main,restricted,universe,multiverse', '-distribution=bionic', '-architectures=amd64,i386,source', 'snapshot', 'bionic-main_foo-keyring-stable-20180825T0000Z', 'bionic-restricted-stable-20180825T0000Z', 'bionic-universe-stable-20180825T0000Z', 'bionic-multiverse-stable-20180825T0000Z', 'ubuntu/stable']' returned non-zero exit status 1

• Generally there is not enough docs
• Reenable generated dev-docs
• Here are some ideas how to extend the docs in this merged PR

In my configuration I have two publish endpoints, one with a snapshot and timestamp: current and another one with timestamp: previous, but after an update they both publish the same snapshot.

mirror:
  trusty-main:
    archive: "http://ch.archive.ubuntu.com/ubuntu/"
snapshot:
  trusty-main-%T:
    mirror: "trusty-main"
    timestamp: {"time": "00:00"}
publish:
  ubuntu/latest:
    - distribution: "trusty"
      snapshots:
        - {"name": "trusty-main-%T", "timestamp": "current"}
  ubuntu/stable:
    - distribution: "trusty"
      snapshots:
        - {"name": "trusty-main-%T", "timestamp": "previous"}

Expected result:

  * trusty/latest [amd64] publishes {main: [trusty-main-20160613T0000Z]: Snapshot from mirror [trusty-main]: http://ch.archive.ubuntu.com/ubuntu/ trusty}
  * trusty/stable [amd64] publishes {main: [trusty-main-20160612T0000Z]: Snapshot from mirror [trusty-main]: http://ch.archive.ubuntu.com/ubuntu/ trusty}

Actual result:

  * trusty/latest [amd64] publishes {main: [trusty-main-20160613T0000Z]: Snapshot from mirror [trusty-main]: http://ch.archive.ubuntu.com/ubuntu/ trusty}
  * trusty/stable [amd64] publishes {main: [trusty-main-20160613T0000Z]: Snapshot from mirror [trusty-main]: http://ch.archive.ubuntu.com/ubuntu/ trusty}

Mirror creation/updating is still done when --pretend is used. It's because the mirror code currently doesn't use the Command feature but runs commands directly. I'm not sure if the solution is to unify the code and also use a Command for the mirroring (though there might be implications, not sure). Or if we should just propagate the config option to mirrors and check if pretend mode is activated.

@ganwell I don't know the pytest suite very well, but it already seems like we're generating tests. Is it possible to generate test cases that run with --pretend twice and make sure the output is exactly the same both times?

And since we're already at it, I'd propose renaming pretend to dryrun, it's much more popular.

Currently modifications in the yaml file are ignored. If possible pyaptly should updated the aptly setup once modifications are done.

Maybe add a special cmdline flag to trigger an update?

E.g.: Remove the i386 architecture from a mirror should also remove it in the aptly mirror (if requested).

Currently such changes have to be done manually;

aptly mirror edit -architectures="amd64" google-chrome

Thanks @erickeller for reporting the issue.

Aptly offers the possibility to pass the gpg key to use for signing the publish as an option -gpg-key.
This should be added as a possible configuration in the pyaptly yaml file.

I switched to toml, because I had an issue with the nestedness of the config and because I wanted to get rid of C-extensions in pyaptly. I did not find an acceptable pure python yaml parser.

While the converter from yaml to toml did an ok job for the configs we use in our tests. It made a complete mess on the existing productions configs. Currently I think, there is a nice config that is correct toml, but the converter is just messing it up.

What I am going to do:

• Try to fix the converter

Questions:

• Is manual cleanup acceptable if we can't fix the converter?
• Do we really need to get rid of C-extensions?
• Should we just support toml and yaml?
• Forget toml completely?
• Should the nestedness be fixed in the data-structure?

The new -max-tries= parameter for mirror update cannot be set through the pyaptly config at the moment:

https://www.aptly.info/doc/aptly/mirror/update/

This needs at least aptly version 1.0.0 (https://github.com/smira/aptly/releases/tag/v1.0.0)!