Comments (11)
akulakhan
commented on August 28, 2024
I also need this to be added. Not having a way to combine "all" and "amd64" architecture packages breaks specific bootstrap installer that I rely on, specifically the diskimage-builder ubuntu-minimal stuff here: https://docs.openstack.org/diskimage-builder/latest/elements/ubuntu-minimal/README.html
from pulp_deb.
quba42
commented on August 28, 2024
@akulakhan Can you explain what exact version of Ubuntu software and what exact commands do not work as expected?
This is important to us, since it would be the first confirmed example of something that does not work.
In the meantime, you could probably still work around the problem by using the "verbatim publisher", which publishes the repo that you synced exactly as it was (including all metadata and signatures).
from pulp_deb.
akulakhan
commented on August 28, 2024
@akulakhan Can you explain what exact version of Ubuntu software and what exact commands do not work as expected? This is important to us, since it would be the first confirmed example of something that does not work.
In the meantime, you could probably still work around the problem by using the "verbatim publisher", which publishes the repo that you synced exactly as it was (including all metadata and signatures).
Provided a log for the failure. The issue occurs during the 08-debootstrap step which seems to be solely reliant on the DIB_DISTRIBUTION_MIRROR provided, which can only be a single repo, and doesn't seem to use the normal apt-get workflow (which does work).
2022-04-12 15:07:27.432 | ++ default_ubuntu_mirror=http://archive.ubuntu.com/ubuntu
2022-04-12 15:07:27.432 | ++ export DIB_DISTRIBUTION_MIRROR=https://<pulp 3 server hostname>:8443/pulp/deb/upstream/ubuntu-bionic/2021-05-18-000000/
2022-04-12 15:07:27.432 | ++ DIB_DISTRIBUTION_MIRROR=https://<pulp 3 server hostname>:8443/pulp/deb/upstream/ubuntu-bionic/2021-05-18-000000/
2022-04-12 15:07:27.432 | dib-run-parts Sourcing environment file /tmp/dib_build.AF3pu4pz/hooks/root.d/../environment.d/10-ubuntu-minimal.bash
2022-04-12 15:07:27.434 | + source /tmp/dib_build.AF3pu4pz/hooks/root.d/../environment.d/10-ubuntu-minimal.bash
2022-04-12 15:07:27.434 | ++ export 'DIB_DEBIAN_COMPONENTS_WS=main universe'
2022-04-12 15:07:27.434 | ++ DIB_DEBIAN_COMPONENTS_WS='main universe'
2022-04-12 15:07:27.434 | ++ DIB_APT_SOURCES_CONF_DEFAULT='default:deb https://<pulp 3 server hostname>:8443/pulp/deb/upstream/ubuntu-bionic/2021-05-18-000000/ bionic main universe'
2022-04-12 15:07:27.434 | ++ DIB_UBUNTU_MIRROR_DISTS=
2022-04-12 15:07:27.434 | ++ export 'DIB_APT_SOURCES_CONF=default:deb [trusted=yes arch=amd64] https://<pulp 3 server hostname>:8443/pulp/deb/upstream/ubuntu-bionic/2021-05-18-000000/ bionic main restricted universe
2022-04-12 15:07:27.434 | updates:deb [trusted=yes arch=amd64] https://<pulp 3 server hostname>:8443/pulp/deb/upstream/ubuntu-bionic/2021-05-18-000000/ bionic-updates main restricted universe
2022-04-12 15:07:27.434 | security:deb [trusted=yes arch=amd64] https://<pulp 3 server hostname>:8443/pulp/deb/upstream/ubuntu-bionic/2021-05-18-000000/ bionic-security main restricted universe
2022-04-12 15:07:27.434 | updates:deb [trusted=yes arch=amd64] https://<pulp 3 server hostname>:8443/pulp/deb/upstream/ubuntu-bionic-updates/2021-05-18-000000/ bionic-updates main restricted universe
2022-04-12 15:07:27.434 | security:deb [trusted=yes arch=amd64] https://<pulp 3 server hostname>:8443/pulp/deb/upstream/ubuntu-bionic-security/2021-05-18-000000/ bionic-security main restricted universe'
2022-04-12 15:07:27.434 | ++ DIB_APT_SOURCES_CONF='default:deb [trusted=yes arch=amd64] https://<pulp 3 server hostname>:8443/pulp/deb/upstream/ubuntu-bionic/2021-05-18-000000/ bionic main restricted universe
2022-04-12 15:07:27.434 | updates:deb [trusted=yes arch=amd64] https://<pulp 3 server hostname>:8443/pulp/deb/upstream/ubuntu-bionic/2021-05-18-000000/ bionic-updates main restricted universe
2022-04-12 15:07:27.434 | security:deb [trusted=yes arch=amd64] https://<pulp 3 server hostname>:8443/pulp/deb/upstream/ubuntu-bionic/2021-05-18-000000/ bionic-security main restricted universe
2022-04-12 15:07:27.434 | updates:deb [trusted=yes arch=amd64] https://<pulp 3 server hostname>:8443/pulp/deb/upstream/ubuntu-bionic-updates/2021-05-18-000000/ bionic-updates main restricted universe
2022-04-12 15:07:27.434 | security:deb [trusted=yes arch=amd64] https://<pulp 3 server hostname>:8443/pulp/deb/upstream/ubuntu-bionic-security/2021-05-18-000000/ bionic-security main restricted universe'
2022-04-12 15:07:27.434 | dib-run-parts Sourcing environment file /tmp/dib_build.AF3pu4pz/hooks/root.d/../environment.d/11-ubuntu-init-system.bash
2022-04-12 15:07:27.436 | + source /tmp/dib_build.AF3pu4pz/hooks/root.d/../environment.d/11-ubuntu-init-system.bash
2022-04-12 15:07:27.436 | ++ [[ bionic == \t\r\u\s\t\y ]]
2022-04-12 15:07:27.436 | ++ export DIB_INIT_SYSTEM=systemd
2022-04-12 15:07:27.436 | ++ DIB_INIT_SYSTEM=systemd
2022-04-12 15:07:27.436 | dib-run-parts Sourcing environment file /tmp/dib_build.AF3pu4pz/hooks/root.d/../environment.d/11-ubuntu-kernel.bash
2022-04-12 15:07:27.438 | + source /tmp/dib_build.AF3pu4pz/hooks/root.d/../environment.d/11-ubuntu-kernel.bash
2022-04-12 15:07:27.438 | ++ export DIB_UBUNTU_KERNEL=linux-image-generic
2022-04-12 15:07:27.438 | ++ DIB_UBUNTU_KERNEL=linux-image-generic
2022-04-12 15:07:27.438 | dib-run-parts Sourcing environment file /tmp/dib_build.AF3pu4pz/hooks/root.d/../environment.d/12-ubuntu-repo-dists.bash
2022-04-12 15:07:27.440 | + source /tmp/dib_build.AF3pu4pz/hooks/root.d/../environment.d/12-ubuntu-repo-dists.bash
2022-04-12 15:07:27.440 | ++ export DIB_UBUNTU_MIRROR_DISTS=updates,security,backports
2022-04-12 15:07:27.440 | ++ DIB_UBUNTU_MIRROR_DISTS=updates,security,backports
2022-04-12 15:07:27.440 | dib-run-parts Sourcing environment file /tmp/dib_build.AF3pu4pz/hooks/root.d/../environment.d/14-manifests
2022-04-12 15:07:27.442 | + source /tmp/dib_build.AF3pu4pz/hooks/root.d/../environment.d/14-manifests
2022-04-12 15:07:27.442 | ++ export DIB_MANIFEST_IMAGE_DIR=/etc/dib-manifests
2022-04-12 15:07:27.442 | ++ DIB_MANIFEST_IMAGE_DIR=/etc/dib-manifests
2022-04-12 15:07:27.442 | ++ export DIB_MANIFEST_SAVE_DIR=/outputdir/tiny.d/
2022-04-12 15:07:27.442 | ++ DIB_MANIFEST_SAVE_DIR=/outputdir/tiny.d/
2022-04-12 15:07:27.442 | dib-run-parts Sourcing environment file /tmp/dib_build.AF3pu4pz/hooks/root.d/../environment.d/20-network-interface-names
2022-04-12 15:07:27.443 | + source /tmp/dib_build.AF3pu4pz/hooks/root.d/../environment.d/20-network-interface-names
2022-04-12 15:07:27.443 | ++ export 'DIB_NETWORK_INTERFACE_NAMES=eth0 eth1'
2022-04-12 15:07:27.443 | ++ DIB_NETWORK_INTERFACE_NAMES='eth0 eth1'
2022-04-12 15:07:27.443 | dib-run-parts Sourcing environment file /tmp/dib_build.AF3pu4pz/hooks/root.d/../environment.d/50-devuser
2022-04-12 15:07:27.445 | + source /tmp/dib_build.AF3pu4pz/hooks/root.d/../environment.d/50-devuser
2022-04-12 15:07:27.445 | ++ export DIB_DEV_USER_USERNAME=dev
2022-04-12 15:07:27.445 | ++ DIB_DEV_USER_USERNAME=dev
2022-04-12 15:07:27.445 | ++ export DIB_DEV_USER_SHELL=
2022-04-12 15:07:27.445 | ++ DIB_DEV_USER_SHELL=
2022-04-12 15:07:27.445 | ++ export DIB_DEV_USER_PWDLESS_SUDO=yes
2022-04-12 15:07:27.445 | ++ DIB_DEV_USER_PWDLESS_SUDO=yes
2022-04-12 15:07:27.445 | ++ export DIB_DEV_USER_AUTHORIZED_KEYS=
2022-04-12 15:07:27.445 | ++ DIB_DEV_USER_AUTHORIZED_KEYS=
2022-04-12 15:07:27.445 | ++ export DIB_DEV_USER_PASSWORD=<a password>
2022-04-12 15:07:27.445 | ++ DIB_DEV_USER_PASSWORD=<a password>
2022-04-12 15:07:27.445 | dib-run-parts Sourcing environment file /tmp/dib_build.AF3pu4pz/hooks/root.d/../environment.d/99-dib-init-system
2022-04-12 15:07:27.447 | + source /tmp/dib_build.AF3pu4pz/hooks/root.d/../environment.d/99-dib-init-system
2022-04-12 15:07:27.447 | ++ '[' -z systemd ']'
2022-04-12 15:07:27.447 | ++ case $DIB_INIT_SYSTEM in
2022-04-12 15:07:27.447 | dib-run-parts Running /tmp/dib_build.AF3pu4pz/hooks/root.d/08-debootstrap
2022-04-12 15:07:27.478 | I: Retrieving InRelease
2022-04-12 15:07:27.503 | I: Checking Release signature
2022-04-12 15:07:27.506 | I: Valid Release signature (key id 4E0FDBC2E1296EAC6C7EC7E93DA3CBF27F36FB4F)
2022-04-12 15:07:27.536 | I: Retrieving Packages
2022-04-12 15:07:27.562 | I: Validating Packages
2022-04-12 15:07:27.612 | I: Retrieving Packages
2022-04-12 15:07:27.674 | I: Validating Packages
2022-04-12 15:07:28.650 | I: Resolving dependencies of required packages...
2022-04-12 15:07:34.565 | I: Resolving dependencies of base packages...
2022-04-12 15:07:39.072 | I: Found additional required dependencies: e2fslibs fdisk libaudit1 libblkid1 libbz2-1.0 libc6 libcap-ng0 libcom-err2 libdb5.3 libdbus-1-3 libdebconfclient0 libexpat1 libext2fs2 libfdisk1 libgcrypt20 libgpg-error0 liblz4-1 libmount1 libncursesw5 libpcre3 libsemanage1 libsmartcols1 libss2 libstdc++6 libsystemd0 libtinfo5 libudev1 libuuid1 libzstd1
2022-04-12 15:07:39.072 | I: Found additional base dependencies: gpgv libapt-pkg5.0 libffi6 libgmp10 libgnutls30 libhogweed4 libidn2-0 libnettle6 libp11-kit0 libseccomp2 libtasn1-6 libunistring2
2022-04-12 15:07:39.079 | I: Checking component main on https://<pulp 3 server hostname>:8443/pulp/deb/upstream/ubuntu-bionic/2021-05-18-000000...
2022-04-12 15:07:39.178 | I: Checking component universe on https://<pulp 3 server hostname>:8443/pulp/deb/upstream/ubuntu-bionic/2021-05-18-000000...
2022-04-12 15:07:39.796 | E: Couldn't find these debs: ca-certificates gnupg2
2022-04-12 15:07:40.025 | INFO diskimage_builder.block_device.blockdevice [-] State already cleaned - no way to do anything here
The build step did not produce a kernel and initramfs. Check builds/dib-output.log
Makefile:104: recipe for target 'builds/tiny.initramfs' failed
make: *** [builds/tiny.initramfs] Error 1
Note that I have tried using the verbatim publisher, however this doesn't work for me since it seems to have various issues. Not the least of which is that it doesn't support translation files. In the workflows I must support, I do not have control over every client to disable fetching of these files, and therefore the builds fail.
In the meantime I am simply pointing these builds to use the pulp 2 instance which I will maintain until pulp 3 supports this.
Also, I have monitored the nginx logs on the pulp3 server to confirm that this process is only grabbing the amd64 package index, which does not contain the missing packages since they are "all" architecture. (gnupg2, for example)
from pulp_deb.
akulakhan
commented on August 28, 2024
@quba42 do you have any idea when this might be fixed or if it's on the roadmap at all? I am presently having to maintain both a pulp 2 and pulp 3 server as a result of this issue, and it would be nice to have an end of life estimation for the former.
from pulp_deb.
akulakhan
commented on August 28, 2024
Anyone have any update on whether this is on the roadmap?
from pulp_deb.
quba42
commented on August 28, 2024
Having discussed this issue internally, I am afraid it has not currently made it onto our roadmap, since none of our use cases are currently hitting this as a problem and we don't have the time to work on it. If any external contributors do want to have a go at it, I can give some pointers (I don't think this would be that hard to implement).
from pulp_deb.
akulakhan
commented on August 28, 2024
I can look into implementing it, any pointers would be helpful.
from pulp_deb.
quba42
commented on August 28, 2024
@akulakhan I guess the place for specifying the option, perhaps named no_support_for_architecture_all, would be after this line: https://github.com/pulp/pulp_deb/blob/main/pulp_deb/app/models/publication.py#L34
Most of the code for actually implementing the feature would have to go in https://github.com/pulp/pulp_deb/blob/main/pulp_deb/app/tasks/publishing.py In particular, I believe something like the following would need to be added here:
if no_support_for_architecture_all and package.architecture=='all':
for architecture in architectures:
package_serializer.to822(self.component).dump(
self.package_index_files[architecture][0]
)
self.package_index_files[architecture][0].write(b"\n")
But that is just a rough draft off the top of my head and some of those variables are currently not available in that function. The basic Idea is: "if we have an architecture all package it needs to be added to every package index, and not just the all index".
You do have one other option that does not involve coding a new pulp_deb feature: You could try using the verbatim publisher instead of the APT publisher, to just mirror the upstream repo you are trying to use exactly.
from pulp_deb.
akulakhan
commented on August 28, 2024
Thanks for the info. I have tried the verbatim publisher but it does not work in this case since the systems consuming it will fail if the language packages aren't available. These are static/released images that must be supported without changes, so there is no option to make changes on the client side to work around this, unfortunately.
I will take a crack at implementing this feature at some point soon.
from pulp_deb.
quba42
commented on August 28, 2024
since the systems consuming it will fail if the language packages aren't available
I am afraid I have more bad news here: pulp_deb currently ignores all language files during the sync (if it synced them, then the verbatim publisher would publish them). It looks like pulp_deb won't be usable for your use case without some significant further development. We do have the long term goal of making pulp_deb sync absolutely every type of file that can exist in an upstream repo, and then mirroring those files exactly as they were via the verbatim publisher. However, due to resource constraints, this is more a long term goal than an actual roadmap at this point.
from pulp_deb.
quba42
commented on August 28, 2024
Actually I may have been overly pessimistic with the language files. It is plausible that with verbatim publisher the upstream Release files will advertise language files, that are actually missing, prompting the clients to complain. When using the APT publisher, the language files will also be missing, but they also won't be advertised as present in the Release file, so the clients might be fine with that. In that case all you would need is the "No-Support-for-Architecture-all: Packages" mode described by this issue. The only way to be 100% certain though would be to implement this issue and then test.
from pulp_deb.
Related Issues (20)
- Serializers will reject the repository parameter when creating artifactless content HOT 1
- Add retrieve function to all relevant serializers
- Convert pulp_deb docs to new stack HOT 3
- Suppress the warnings about python3-apt not being installed
- Enable full skip mode when using optimize with mirror mode
- Use by-hash files when syncing content
- Retrieve functionality breaks semver, requires task monitoring HOT 6
- Start using pulpcore 3.49 with python>=3.9
- Publishing a source package that belongs to two dists raises IntegrityError HOT 2
- Allow source files to publish at by-hash paths
- Support duplicate content for source debs HOT 1
- Support by-hash files for verbatim publications
- Investigate performance of php repository
- Make plugin compatible with with pulpcore 3.55+
- Include performance tests for the nightly CI
- pulp_deb serves unpublished content HOT 4
- New user question: How to rollback a debian verbatim distribution ? (3.43.0 all in one container) HOT 4
- Setting APT_BY_HASH doesn't enable the apt-by-hash feature
- An additional character "1:" is being added after the (_)underscore with the filename of Ubuntu (deb) package in Pulp3 repository. HOT 1
- Cannot create repository version since there are newly added packages with the same name, version, and architecture - Docker CE HOT 5
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pulp_deb.