Once a week or so, I run pkg_comp targeting my NetBSD server. SANDBOX_TYPE=netbsd-release, no X sets.

As of this pkgsrc/mk commit, my builds became incomplete, missing 100+ of the usual packages. (pkg_comp reported no errors, so I only noticed the problem after attempting to update my system with the new package set.) The log showed pbulk starting at [113/496] instead of the expected [1/496].

Based on the above commit, I tried adding an explicit X11_TYPE=modular in extra.mk.conf. Indeed, this restored me to complete builds.

Possibly a second causal factor here is that freefonts (one of the packages in my list) grew dependencies on encodings and mkfontscale since my last full build two weeks ago, perhaps forcing the X11 issue. Guessing this isn't particular to freefonts but more generally to do with font handling in pkgsrc.

In the condition where pbulk skips trying to build some packages, would it be possible for pkg_comp to error out -- ideally right away?

Is there a helpful-but-not-too-clever way that pkg_comp could auto-generate the needed X11_TYPE setting? Or should users perhaps be required to explicitly set it?

On macOS Sierra 10.12.4, pkg_comp can't download the distfiles for packages via HTTPS. See sandboxctl issue #2 for why this is. This is problematic since a number of packages provide their distfiles via HTTPS only (e.g., devel/protobuf and devel/googletest). A workaround is to download the distfiles by hand outside of the sandbox and place them in /var/pkg_comp/distfiles, but that's obviously a pain and far from automated.

When you run

# pkg_comp -c <config> bootstrap

and you forgot to create the sandbox beforehand, it will give you a very misleading error message about a missing real path on mount (the error message is of course correct, but only a follow up of the actual root cause, the missing sandbox).
It would be great if there would be a sandbox-detection routine to either create the sandbox on the fly if it doesn't exist on calling bootstrap, or to at least give the user a proper error message like "please run pkg_comp -c sandbox-create, first".

After setting up pkg_comp-cron on a new host, I ran a full bulk build. It claimed success, and a visual scan of the output showed that all packages were built.

But when I went to install the packages, 5 of them (out of about 65) weren't built.

So I tried one of them (git-docs) with pkg_comp auto -f devel/git-docs and it ran through the process, claimed that every package built successfully, and claimed overall success. But when I looked closely, the git-docs wasn't even listed as an attempted package.

Digging deeper, I couldn't find any build logs to explain what had happened. (Per issue #3.) I did try manually creating the sandbox (so it would be left behind) and running another build, and I still couldn't find any logs in the sandbox. I was able to find pbulk's reports, and report.txt did show that there were errors, but it didn't show which packages actually had the errors. (This seems like a bug in pbulk, but I'm not sure.) I finally discovered that report.html had more details than report.txt had, and the underlying problem was that python wasn't building because it wanted X11 and I didn't have it installed. So apparently the python package was "masked" and that made dependent builds be automatically skipped.

I don't know if this is fixable via pbulk config; hopefully it is. But even if it is, I think pkg_comp still needs a way to verify if all the requested packages were actually built. It seems like pbulk can't be trusted to report success or failure reliably (at least not when building a specific list of packages), so some sort of post-build check is needed.

I have made earlier attempts at such a script; here is one and here is another that might be helpful.

What I did: following the recent howto, updated my 1.x config to 2.0 and ran my usual weekly pkg_comp build.

What I expected: roughly the same set of binary packages as last week's pkgtools/pkg_comp1-powered build, including lang/php56 and a handful of extensions.

What happened instead: the same binary packages plus lang/php70 and extensions built against it and lang/php71 and extensions built against it.

Why it got in my way: my little script to pkg_add everything failed due to conflicts between the various PHP versions.

What fixed it for me: adding NO_MULTI_PKG= #defined to extra.mk.conf so that pbulk only builds the PHP I asked for.

What I'm suggesting for pkg_comp: set this variable by default, and probably don't even worry about needing to allow it to be overridden by the user. As far as I can tell, the default behavior makes sense for typical pbulk usage but is never desirable in pkg_comp usage.

I can't bootstrap on macOS High Sierra. I do the following:

$ sudo pkg_comp sandbox-create
$ sudo pkg_comp bootstrap

And the bootstrap fails like this:

===> Bootstrapping bmake
===> running: /bin/sh /pkg_comp/work/pkg/bootstrap/bin/install-sh -d -o root -g
wheel /pkg_comp/work/pkg/bootstrap/bmake
===> running: (cd /pkg_comp/work/pkg/bootstrap/bmake && /bin/sh configure  --pre
fix=/pkg_comp/work/pkg/bootstrap --with-default-sys-path=/pkg_comp/work/pkg/boot
strap/share/mk --with-machine-arch=x86_64 )
checking for gcc... cc -isystem /usr/include
checking for C compiler default output file name...
configure: error: in `/pkg_comp/work/pkg/bootstrap/bmake':
configure: error: C compiler cannot create executables
See `config.log' for more details.
===> exited with status 77
aborted.

It seems that the C compiler does not work in the sandbox. I filed sandboxctl issue #7 for this.

Per my original comments:

It's still not clear to me how to see the output logs. It looks like only the cron job will log to /var/pkg_comp/log, but that's not clear from the blog post. ("...you can find the logs for all builds...") I also am still not clear on how to see the detailed logs from the individual packages, both during the build, and after. I found them once, but I don't remember where. (I know that's probably more of a pbulk issue, but it would help to explain it here.)

I think there's a 2x2 matrix of cases:

1a. During cron build
1b. After cron build
2a. During manual (auto -f) build
2b. After manual build

Ideally it would always just be "here's where the active and/or residual logs are, from outside the sandbox" but maybe it's not that easy.

I'd find it very useful if pkg_comp, still using sandboxctl and requiring root to run, could be configured to create an unprivileged user inside the sandbox and do an --unprivileged build as that user.

I'd use this in a VM configured to match the OS of a hosting provider where I have shell access but can't compile much of anything (thanks to process limits). I'd build binary packages in the VM, taking care that the sandbox $HOME matches my hosting provider $HOME, and then upload and install them.

The macOS installer comes with the program /usr/local/bin/bindfs, but not the shared library /usr/local/lib/libosxfuse.2.dylib that bindfs needs.

I'm guessing this is just an oversight in the packing list for the package.

Per http://julio.meroh.net/2017/02/pkg_comp-2.0-tutorial-netbsd.html#comment-3187176460, it's not currently possible to bootstrap pkg_comp without pkgsrc or pkgconfig. It would be nice if it could be done.

# unchanged from usual docs
pkg_add pkgin pkg_comp-cron
echo 'file:///var/pkg_comp/packages/All' >>/etc/pkgin/repositories.conf
pkgin update

# simple new way to install all the requested packages
pkgin import /var/pkg_comp/list.txt

(pkgin import seems to obey the grep -v '^#' rule, too.)

And I haven't had a chance to test it yet, but maybe if repositories.conf had the local path as the only source, pkgin would even complain about missing packages (i.e. before trying to install them)!

https://github.com/truist/settings/blob/master/bin/util/pkg_comp_util.sh is a set of utility functions, including a "hygiene" function, to help catch errors/issues with the build builds. Maybe some of it will be useful to you.

https://github.com/truist/settings/blob/master/bin/server/pkg_comp_wrapper is my new interface to day-to-day usage of pkg_comp. It ensures I always start with a clean (but pre-created) sandbox, and then actually confirms that all the desired packages were built, and then leaves the sandbox behind (so I can inspect the errors) if they weren't. I'm also going to try running it (with no args) instead of pkg_comp4cron in my weekly cron job.

If that works, I think I'm going to fall back to bare pkg_comp instead of pkg_comp-cron. pkg_comp-cron was great for helping me get set up, but now that I've essentially merged the two packages (via shared configuration and a manual cron job), it seems extraneous.

All of this is just in the spirit of sharing ideas and "real-user" feedback; I don't have any specific critique or suggestion for pkg_comp. So feel free to close this issue; this was just the best way I knew to draw your attention to these things.

My pkg_comp build system has been unable to build the erlang package since Erlang 24.

Erlang 24 introduced a JIT-compiler and uses that compiler during it's build process. The JIT-compiler implementation on NetBSD uses shm_open in virtmem.cpp. For erlang to build it needs a working shm_open setup, namely a correctly mounted /var/shm mount.

By default the sandbox prepared by pkg_comp does not have /var/shm mounted, hence the build fails.

My work around at the moment is the following hook inside sandbox.conf:

pkg_comp_post_mount_hook() {
        mount -t tmpfs -o rw -o -m1777 -o -sram%10 tmpfs /var/pkg_comp/sandbox/var/shm
        true
}

Is that the best way to mount /var/shm inside the sandbox? Could an option be added to pkg_comp as an alternative to the custom hook above?

NOTE: I have also submitted this as pkg/57158 becuase it may affect other packages & build systems.

I ran the pkg_comp auto command on macOS Sierra 10.12.5:

# pkg_comp auto

and the pbulk build failed on the very first package (cwrappers):

Resolving...
Building...
Initialisation complete.
[1/417] Starting build of  cwrappers-20170611
[1/417] Failed to build    cwrappers-20170611
Building pkg_summary...
Building SHA512...
Build reports...
Sending report mail...
Uploading report...
Uploading packages...
pkg_comp: I: Destroying sandbox

The pbulk log shows that it built the package successfully, but the problem was in installing it; the following appears in bulklog/cwrappers-20170611/package.log:

=> Creating binary package /pkg_comp/work/pkg/pkgtools/cwrappers/work/.packages/cwrappers-20170611.tgz
pkg_add: A different version of cwrappers-20170611 is already installed: cwrappers-20170112
pkg_add: 1 package addition failed

Building individual packages via pkg_comp seems to end up overwriting the pkg_summary file(s) with just the packages that were built during that session, even if lots of other packages were already present in the package directory.

So if you e.g. just want to add a package to your system, and you build it via e.g. pkg_comp auto -f some/package, and follow that with pkgin update, you end up with a very short list of available packages.

devel/libidn2 2.0.3 is packaged from a .tar.lz distfile.

As a one-off build directly from pkgsrc, it extracts fine. I've also been watching pkgsrc-bulk@ and not seeing any reported failures. That's why I suspect the following must be a bug in pkg_comp:

With my usual weekly build, I get the following in configure.log:

=> Bootstrap dependency digest>=20010302: found digest-20160304
=> Checksum SHA1 OK for libidn2-2.0.3.tar.lz
=> Checksum RMD160 OK for libidn2-2.0.3.tar.lz
=> Checksum SHA512 OK for libidn2-2.0.3.tar.lz
===> Installing dependencies for libidn2-2.0.3
=> Tool dependency libtool-base>=2.4.2nb9: found libtool-base-2.4.6
=> Tool dependency lzip>=1.14: found lzip-1.19
=> Tool dependency pkgconf-[0-9]*: found pkgconf-1.3.5
=> Tool dependency checkperms>=1.1: found checkperms-1.11nb1
=> Build dependency cwrappers>=20150314: found cwrappers-20170611
=> Full dependency libunistring>=0.9.4: found libunistring-0.9.7
===> Overriding tools for libidn2-2.0.3
===> Extracting for libidn2-2.0.3
lzip: not found
tar: End of archive volume 1 reached
tar: Sorry, unable to determine archive format.
*** Error code 1

Stop.
bmake: stopped in /pkg_comp/pkgsrc/devel/libidn2
*** Error code 1

Stop.
bmake: stopped in /pkg_comp/pkgsrc/devel/libidn2

I can reproduce just as well with a simple pkg_comp auto libidn2 (though when I do it this way, the build logs disappear before I can save them off!).

The first version of pkg_comp 2.0 forced the use of a bootstrapped pkg_install for simplicity but this gets in the way of using binary packages on platforms where pkg_install is native (NetBSD). It also makes it impossible to mix pkg_comp-built packages with the official binary packages.

pkg_comp should be able to use the native pkg_install binaries for the final packages so that using those packages does not require having to use a different path to the package tools nor having to configure the host's pkg_path.conf file.

To consider: are there any implications on the host's pkg_install.conf file, for example, to properly configure e.g. PKG_DBDIR to match whatever was done in the pkg_comp configuration?

I'm wanting to do my NetBSD 8.x package builds with clang. Newer NetBSD has clang in the base system, but 8.x doesn't, so I've built it from pkgsrc on the host.

What's a good way to get clang pkg_add'd in the sandbox and have it get used to build the bootstrap and all my packages?

On macOS Sierra 10.12.4, executing the bootstrap subcommand produces messages about mDNSResponder:

# pkg_comp bootstrap
pkg_comp: I: Bootstrapping pkg tools
pkg_comp: I: Setting up bootstrap in /opt/pkg using binary kit
/System/Library/LaunchDaemons/com.apple.mDNSResponder.plist: Operation not permitted while System Integrity Protection is engaged
/private/var/pkg_comp/sandbox/private/tmp/com.apple.mDNSResponder.plist: service already loaded
/private/var/pkg_comp/sandbox/private/tmp/com.apple.mDNSResponder.plist: Operation not permitted while System Integrity Protection is engaged
/System/Library/LaunchDaemons/com.apple.mDNSResponder.plist: service already loaded
pkg_comp: I: Setting up /pkg_comp/pkg.mk.conf
pkg_comp: I: Setting up /opt/pkg/etc/pkg_install.conf
pkg_comp: I: Bootstrapping pbulk tools
pkg_comp: I: Setting up bootstrap in /pkg_comp/pbulk using binary kit
/System/Library/LaunchDaemons/com.apple.mDNSResponder.plist: Operation not permitted while System Integrity Protection is engaged
/private/var/pkg_comp/sandbox/private/tmp/com.apple.mDNSResponder.plist: service already loaded
/private/var/pkg_comp/sandbox/private/tmp/com.apple.mDNSResponder.plist: Operation not permitted while System Integrity Protection is engaged
/System/Library/LaunchDaemons/com.apple.mDNSResponder.plist: service already loaded
pkg_comp: I: Setting up /pkg_comp/pbulk.mk.conf
pkg_comp: I: Setting up /pkg_comp/pbulk/etc/pkg_install.conf
/System/Library/LaunchDaemons/com.apple.mDNSResponder.plist: Operation not permitted while System Integrity Protection is engaged
/private/var/pkg_comp/sandbox/private/tmp/com.apple.mDNSResponder.plist: service already loaded
===> Binary install for pbulk-0.65
=> Installing pbulk-0.65 from /pkg_comp/packages/pbulk/All
pkg_add: Warning: package `pbulk-0.65' was built for a platform:
pkg_add: Darwin/x86_64 16.4.0 (pkg) vs. Darwin/x86_64 16.5.0 (this host)
pkg_add: Warning: package `rsync-3.1.2' was built for a platform:
pkg_add: Darwin/x86_64 16.4.0 (pkg) vs. Darwin/x86_64 16.5.0 (this host)
mkdir: /etc/rc.d: Read-only file system
===========================================================================
The following files should be created for rsync-3.1.2:

	/etc/rc.d/rsyncd (m=0755)
	    [/pkg_comp/pbulk/share/examples/rc.d/rsyncd]

===========================================================================
pkg_add: Warning: package `pbulk-base-0.53' was built for a platform:
pkg_add: Darwin/x86_64 16.4.0 (pkg) vs. Darwin/x86_64 16.5.0 (this host)
pkg_add: Warning: package `digest-20160304' was built for a platform:
pkg_add: Darwin/x86_64 16.4.0 (pkg) vs. Darwin/x86_64 16.5.0 (this host)
pbulk-0.65: copying /pkg_comp/pbulk/share/examples/pbulk/pbulk.conf to /pkg_comp/pbulk/etc/pbulk.conf
pbulk-0.65 successfully installed.
/private/var/pkg_comp/sandbox/private/tmp/com.apple.mDNSResponder.plist: Operation not permitted while System Integrity Protection is engaged
/System/Library/LaunchDaemons/com.apple.mDNSResponder.plist: service already loaded

Similarly for the build subcommand:

# pkg_comp build devel/protobuf
pkg_comp: I: Adding packages to be built to (existent) pbulk.list
pkg_comp: I: Starting pbulk build in the sandbox
/System/Library/LaunchDaemons/com.apple.mDNSResponder.plist: Operation not permitted while System Integrity Protection is engaged
/private/var/pkg_comp/sandbox/private/tmp/com.apple.mDNSResponder.plist: service already loaded
Scanning...
[...]
Building pkg_summary...
Building SHA512...
Build reports...
Sending report mail...
Uploading report...
Uploading packages...
/private/var/pkg_comp/sandbox/private/tmp/com.apple.mDNSResponder.plist: Operation not permitted while System Integrity Protection is engaged
/System/Library/LaunchDaemons/com.apple.mDNSResponder.plist: service already loaded

Similarly for the sandbox-shell subcommand:

# pkg_comp sandbox-shell
/System/Library/LaunchDaemons/com.apple.mDNSResponder.plist: Operation not permitted while System Integrity Protection is engaged
/private/var/pkg_comp/sandbox/private/tmp/com.apple.mDNSResponder.plist: service already loaded
sh-3.2#

Twice now I've uninstalled pkg_comp-cron and reinstalled it, and later realized that it has re-added the @daily cron task. I change it to @Weekly, so I end up with it configured twice.

It's not really surprising that it does that, but it would be nice if it would e.g. check if anything in the crontab already had pkg_comp4cron in it, and just skipped adding it again.

As per https://mail-index.netbsd.org/tech-repository/2017/06/10/msg000637.html, the old jsonn repos are going away.

Per my comment on your blog post:

Then, because of that symlink (which I had to recreate, but I didn't realize that until after I'd gone a ways down the path without it), I ended up with an /etc/mk.comp (from bootstrap) that has an inside-the-sandbox path in it. That feels a little weird, but I haven't figured out a better option so I'm just living with it.

To rephrase: the mk.conf that is extracted from the bootstrap archive has .sinclude "/pkg_comp/pkg.mk.conf" in it. That path only makes sense from inside the sandbox, but that file ends up in a place where it has an effect outside the sandbox. It's confusing, and worst-case someone could end up with a matching path and unintended consequences.

On macOS Sierra 10.12.3, I followed the Easy pkgsrc on macOS with pkg_comp 2.0 tutorial (using pkg_comp-2.0-20170223-macos.pkg) and then ran an auto build (as root):

# pkg_comp auto

(Just in case it matters, I actually ran two shorter auto builds before this where I listed a few packages on the command line after the auto subcommand just to test that it works.)

It attempted to build 485 packages (some of which succeeded and some of which failed), and seemed to run OK and exited with an exit status of 0. However, when I went to review the build logs, I found that there were none:

% ls -alF /var/pkg_comp/log/
total 0
drwxr-xr-x  2 root  wheel   68 Feb 23 16:05 ./
drwxr-xr-x  7 root  wheel  238 Mar 22 04:59 ../

I rebuild packages a lot sometimes and I don't want to wear down my disk more than I need to, so I always specify the work directory to point to a ram drive (also speeds up builds significantly). On my NetBSD systems I have a slightly modified pkg_comp which introduces a REAL_WORKDIR for this purpose.

It would be neat if pkg_comp 2.0 could do this out-of-the-box.

As raised in my first comment to your blog post:

I was pretty confused about how the various mk.conf and extra.mk.conf files would come together. (And the bootstrap /usr/pkg/etc/mk.conf would have blown away my /etc/mk.conf if I hadn't already deleted /usr/pkg before extracting it [because of my symlink].) I don't think the post nor the docs really explain it well. I think it would help for you to just say "nearly everything you usually put in /etc/mk.conf should go in extra.mk.conf, except A, B, C".