Comments (7)
jmmv
commented on August 22, 2024
I'm keeping the duplicate sandboxctl issue open for now as I don't know where the fix is going to happen.
For now, I think that adding the following to extra.mk.conf should workaround the problem:
USE_BUILTIN.curl=no
Let me know if it does.
from pkg_comp.
jlmuir
commented on August 22, 2024
Unfortunately, that doesn't work. I don't think www/curl has a built-in capability (i.e., there's no www/curl/builtin.mk).
I tried the following in extra.mk.conf:
TOOLS_PLATFORM.curl=
That caused bmake fetch to install pkgsrc's curl, but it said it couldn't resolve the host for each URL it tried to download the distfile from:
# bmake fetch
=> Bootstrap dependency curl-[0-9]*: NOT found
=> Verifying bin-install for ../../www/curl
===> Binary install for curl-[0-9]*
=> Installing curl-[0-9]* from /pkg_comp/packages/pkg/All
[...]
curl-7.53.1nb1 successfully installed.
=> Returning to build of googletest-1.8.0
=> Bootstrap dependency digest>=20010302: found digest-20160304
===> Skipping vulnerability checks.
=> Fetching googletest-1.8.0.tar.gz
=> Total size: 1281617 bytes
curl: (6) Couldn't resolve host 'github.com'
fetch: Unable to fetch expected file googletest-1.8.0.tar.gz
curl: (6) Couldn't resolve host 'cdn.NetBSD.org'
fetch: Unable to fetch expected file googletest-1.8.0.tar.gz
curl: (6) Couldn't resolve host 'ftp6.NetBSD.org'
fetch: Unable to fetch expected file googletest-1.8.0.tar.gz
curl: (6) Couldn't resolve host 'ftp.fr.NetBSD.org'
fetch: Unable to fetch expected file googletest-1.8.0.tar.gz
curl: (6) Couldn't resolve host 'ftp.fr.NetBSD.org'
fetch: Unable to fetch expected file googletest-1.8.0.tar.gz
curl: (6) Couldn't resolve host 'ftp.NetBSD.org'
fetch: Unable to fetch expected file googletest-1.8.0.tar.gz
curl: (6) Couldn't resolve host 'ftp.NetBSD.org'
fetch: Unable to fetch expected file googletest-1.8.0.tar.gz
*** Error code 1
Stop.
bmake: stopped in /pkg_comp/pkgsrc/devel/googletest
However, I'm still having issue #15 where DNS resolution with curl does not seem to work in the sandbox, so I'm thinking that is hiding this HTTPS issue. For example, native and pkgsrc curl are giving a different error message than that reported in sandboxctl issue #2; the error is now about being unable to resolve the host as opposed to an SSL certificate problem, so I think now it's not even getting to the SSL handshake:
# /usr/bin/curl -IsS https://github.com/ | head -n 1
curl: (6) Could not resolve host: github.com
# /opt/pkg/bin/curl -IsS https://github.com/ | head -n 1
curl: (6) Couldn't resolve host 'github.com'
I don't know when it started happening, but issue #15 seems like it might need to be the first thing to sort out.
from pkg_comp.
jlmuir
commented on August 22, 2024
OK, now that issue #15 is fixed (Thanks for that!), I tried again, and the native curl still doesn't work in the pkg_comp sandbox:
# /usr/bin/curl -IsS https://github.com/ | head -n 1
curl: (35) SSL certificate problem: Couldn't understand the server certificate format
For the case of downloading the devel/googletest distfile, it emits the same curl error message as above, but the distfile does get downloaded, so I assume pkgsrc is falling back to another fetch tool or a different download location that is not HTTPS:
# cd devel/googletest
# bmake fetch
=> Bootstrap dependency digest>=20010302: NOT found
=> Verifying bin-install for ../../pkgtools/digest
===> Binary install for digest>=20010302
=> Installing digest>=20010302 from /pkg_comp/packages/pkg/All
pkg_add: Warning: package `digest-20160304' was built for a platform:
pkg_add: Darwin/x86_64 16.4.0 (pkg) vs. Darwin/x86_64 16.5.0 (this host)
digest-20160304 successfully installed.
=> Returning to build of googletest-1.8.0
===> Skipping vulnerability checks.
WARNING: No /opt/pkg/libdata/pkgdb/pkg-vulnerabilities file found.
WARNING: To fix run: `/opt/pkg/sbin/pkg_admin -K /opt/pkg/libdata/pkgdb fetch-pkg-vulnerabilities'.
=> Fetching googletest-1.8.0.tar.gz
=> Total size: 1281617 bytes
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
curl: (35) SSL certificate problem: Couldn't understand the server certificate format
fetch: Unable to fetch expected file googletest-1.8.0.tar.gz
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 1251k 100 1251k 0 0 1181k 0 0:00:01 0:00:01 --:--:-- 1181k
Then I tried forcing the use of the pkgsrc curl instead of the native curl by adding the following to extra.mk.conf:
TOOLS_PLATFORM.curl=
This made the bmake fetch download the distfile without an error message:
# bmake fetch
=> Bootstrap dependency curl-[0-9]*: NOT found
=> Verifying bin-install for ../../www/curl
===> Binary install for curl-[0-9]*
=> Installing curl-[0-9]* from /pkg_comp/packages/pkg/All
pkg_add: Warning: package `libidn-1.33' was built for a platform:
pkg_add: Darwin/x86_64 16.4.0 (pkg) vs. Darwin/x86_64 16.5.0 (this host)
pkg_add: Warning: package `gettext-lib-0.19.8.1' was built for a platform:
pkg_add: Darwin/x86_64 16.4.0 (pkg) vs. Darwin/x86_64 16.5.0 (this host)
gettext-lib-0.19.8.1: copying /opt/pkg/share/examples/gettext/locale.alias to /opt/pkg/share/locale/locale.alias
libidn-1.33: registering info file /opt/pkg/info/libidn-components.png
libidn-1.33: registering info file /opt/pkg/info/libidn.info
pkg_add: Warning: package `perl-5.24.1' was built for a platform:
pkg_add: Darwin/x86_64 16.4.0 (pkg) vs. Darwin/x86_64 16.5.0 (this host)
openssl-1.0.2k: copying /opt/pkg/share/examples/openssl/openssl.cnf to /opt/pkg/etc/openssl/openssl.cnf
curl-7.53.1nb1 successfully installed.
=> Returning to build of googletest-1.8.0
=> Bootstrap dependency digest>=20010302: found digest-20160304
===> Skipping vulnerability checks.
WARNING: No /opt/pkg/libdata/pkgdb/pkg-vulnerabilities file found.
WARNING: To fix run: `/opt/pkg/sbin/pkg_admin -K /opt/pkg/libdata/pkgdb fetch-pkg-vulnerabilities'.
=> Fetching googletest-1.8.0.tar.gz
=> Total size: 1281617 bytes
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 132 0 132 0 0 952 0 --:--:-- --:--:-- --:--:-- 992
100 1251k 100 1251k 0 0 1711k 0 --:--:-- --:--:-- --:--:-- 1711k
However, the pkgsrc curl still fails the https://github.com/ test, but with a different error message:
# /opt/pkg/bin/curl -IsS https://github.com/ | head -n 1
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.haxx.se/docs/sslcerts.html
curl performs SSL certificate verification by default, using a "bundle"
of Certificate Authority (CA) public keys (CA certs). If the default
bundle file isn't adequate, you can specify an alternate file
using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
the bundle, the certificate verification probably failed due to a
problem with the certificate (it might be expired, or the name might
not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
the -k (or --insecure) option.
So, I tried installing the security/mozilla-rootcerts package and installing the certs (all still in the sandbox):
# pkg_add mozilla-rootcerts
# cd /opt/pkg/etc/openssl/certs
# mozilla-rootcerts extract
# mozilla-rootcerts rehash
And now, the pkgsrc curl passes the https://github.com/ test:
# /opt/pkg/bin/curl -IsS https://github.com/ | head -n 1
HTTP/1.1 200 OK
I tried the native curl one more time with the --capath option to try to make it use the same certificate directory that the pkgsrc curl is using, but that didn't work:
# /usr/bin/curl --capath /opt/pkg/etc/openssl/certs -IsS https://github.com/ | head -n 1
curl: (35) SSL certificate problem: Couldn't understand the server certificate format
So, I still don't know why the native curl doesn't work. But if the decision is made to punt on that and just use the pkgsrc curl, then just setting TOOLS_PLATFORM.curl= in extra.mk.conf seems to be insufficient to solve the problem: the root certs from the pkgsrc mozilla-rootcerts package are needed as shown above (i.e., the mozilla-rootcerts package needs to be installed, and the certs need to be extracted and hashed).
from pkg_comp.
jmmv
commented on August 22, 2024
I dug into this a bit last week, seeing if maybe some obvious missing path in the sandbox was behind these failures, but couldn't find anything to explain why curl fails. I plan to research this further because, if curl is failing in this manner, other things will fail too: the sandbox is not working appropriately.
I'd rather not set up curl from pkg_comp to workaround this issue because it's complex as you discovered (need the certificates).
If we cannot fix the sandbox to make curl run... then I'd rather investigate making USE_BUILTIN.curl=no work properly in pkgsrc or similar so that pkg_comp can trivially tell pkgsrc to avoid curl without requiring special build logic.
from pkg_comp.
jlmuir
commented on August 22, 2024
In the sandbox, I can't seem to get the native curl's --cacert nor --capath options to work. So, I must be using them incorrectly, the file or directory they point at is not in the required format, or they're being ignored.
I don't know how far you got with researching why the native curl isn't working, but I'm finding things that make we wonder if the problem is that the native curl uses the System or login keychain for getting the certificates, and that won't work in a sandbox (?).
The things I found are as follows.
The curl(1) man page on macOS Sierra says the following about the --cacert option:
(iOS and macOS only) If curl is built against Secure Transport, then this option is supported for backward compatibility with other SSL engines, but it should not be set. If the option is not set, then curl will use the certificates in the system and user Keychain to verify the peer, which is the preferred method of verifying the peer's certificate chain.
A curl mailing list email from 2013 says (among other things):
In Mavericks, Apple changed from curl 7.24.0 to 7.30.0, and in the process, they switched the TLS/SSL engine used by their curl, from OpenSSL to their own Secure Transport engine.
and
Unless Apple changed curl's code behind my back, the --cacert option no longer works. This is because the new engine reads its certificates from a security database (the Keychain) instead of from certificate files.
A Phusion blog post says:
curl on macOS uses the macOS Security framework internally to load the client cert.
from pkg_comp.
jmmv
commented on August 22, 2024
Yeah, that (curl using the native keychains) was the theory I was chasing as well -- and attempted to mount various Keychain-related directories in the sandbox to no avail.
One thing I wanted to try was to play with security and see if that worked at all. Maybe you'd give it a try? I can't look until tomorrow at the earliest.
from pkg_comp.
jmmv
commented on August 22, 2024
Found the problem playing with security list-keychains. This is a sandbox-creation issue, so closing in favor of jmmv/sandboxctl#2.
from pkg_comp.
Related Issues (20)
- `pkg_comp auto some/package` overwrites pkg_summary HOT 5
- Some bits of code you might find interesting HOT 2
- No build logs on macOS in /var/pkg_comp/log/ HOT 1
- On macOS, DNS resolution in sandbox does not work HOT 3
- Bootstrap error message on non-existent sandbox
- Allow use of native pkg_install
- "pkg_comp auto" fails: pkg_add: A different version of ... is already installed HOT 4
- Specify work directory
- Default conf mentions deprecated repos HOT 1
- can't extract distfiles with EXTRACT_SUFX=.tar.lz HOT 9
- Can't bootstrap on macOS High Sierra because C compiler does not work HOT 1
- request: really simple UNPRIVILEGED support HOT 2
- use alternate compiler in sandbox HOT 1
- seriously incomplete builds with recent pkgsrc
- Best way to mount /var/shm in the sandbox?
- pkg_comp should override pbulk's multi-package default HOT 6
- Failed/missing packages are not reported HOT 4
- Reinstalling pkg_comp-cron adds semi-duplicate entry to crontab HOT 2
- Suggested improvements for first-install steps HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pkg_comp.