Discussed in #83

Describe the bug
git HEAD is 47e1c47
When configuring in a build directory (build) outside of the source directory make check fails:

doug@XU-22:/a/appl/pkcs11-provider-git/build$ make check
Making check in src
make[1]: Entering directory '/a/appl/pkcs11-provider-git/build/src'
make[1]: Leaving directory '/a/appl/pkcs11-provider-git/build/src'
Making check in tests
make[1]: Entering directory '/a/appl/pkcs11-provider-git/build/tests'
make  tsession tgenkey ttls tdigests \
  helpers.sh setup-softhsm.sh setup-softokn.sh softhsm-proxy.sh test-wrapper tbasic teccsha2 thkdf toaepsha2 trsapss
make[2]: Entering directory '/a/appl/pkcs11-provider-git/build/tests'
make[2]: 'tsession' is up to date.
make[2]: 'tgenkey' is up to date.
make[2]: 'ttls' is up to date.
make[2]: 'tdigests' is up to date.
make[2]: Nothing to be done for '../../pkcs11-provider/tests/helpers.sh'.
make[2]: Nothing to be done for '../../pkcs11-provider/tests/setup-softhsm.sh'.
make[2]: Nothing to be done for '../../pkcs11-provider/tests/setup-softokn.sh'.
make[2]: Nothing to be done for '../../pkcs11-provider/tests/softhsm-proxy.sh'.
make[2]: Nothing to be done for '../../pkcs11-provider/tests/test-wrapper'.
make[2]: Nothing to be done for '../../pkcs11-provider/tests/tbasic'.
make[2]: Nothing to be done for '../../pkcs11-provider/tests/teccsha2'.
make[2]: Nothing to be done for '../../pkcs11-provider/tests/thkdf'.
make[2]: Nothing to be done for '../../pkcs11-provider/tests/toaepsha2'.
make[2]: Nothing to be done for '../../pkcs11-provider/tests/trsapss'.
make[2]: Leaving directory '/a/appl/pkcs11-provider-git/build/tests'
make  check-TESTS
make[2]: Entering directory '/a/appl/pkcs11-provider-git/build/tests'
make[3]: Entering directory '/a/appl/pkcs11-provider-git/build/tests'
SKIP: basic-softokn
SKIP: basic-softhsm-proxy
SKIP: eccsha2-softokn
SKIP: oaepsha2-softokn
SKIP: hkdf-softokn
SKIP: rsapss-softokn
SKIP: digests-softokn
SKIP: digests-softhsm-proxy
SKIP: genkey-softokn
SKIP: genkey-softhsm
SKIP: session-softokn
SKIP: session-softhsm-proxy
SKIP: tls-softokn
SKIP: tls-softhsm-proxy
============================================================================
Testsuite summary for pkcs11-provider 0.1
============================================================================
# TOTAL: 14
# PASS:  0
# SKIP:  14
# XFAIL: 0
# FAIL:  0
# XPASS: 0
# ERROR: 0

To Reproduce

Steps to reproduce the behavior:

1. cd to git source (in my case: /a/appl/pkcs11-provider-git/pkcs11-provider)
2. autoreconf -fi
3. cd ..
4. mkdir build
5. cd build
6. ../pkcs11-provider/configure
7. make
8. make check

Expected behavior
make check works when building in the source but not when built outside the source

Operating environment (please complete the following information):

• OS: XUbuntu
• Version 22.10

Additional context

Adding set -xv to testwrapperfailure to sourcetestvars` shows each test fails with 77 like this one:

TEST_PARAMS=(${1//-/ })
+ TEST_PARAMS=(${1//-/ })

TEST_NAME=$(basename ${TEST_PARAMS[0]})
++ basename ../../pkcs11
+ TEST_NAME=pkcs11
TEST_PATH=$(dirname ${TEST_PARAMS[0]})
++ dirname ../../pkcs11
+ TEST_PATH=../..
TOKEN_DRIVER=${TEST_PARAMS[1]}
+ TOKEN_DRIVER=provider/tests/basic

if [ -f "./tmp.${TOKEN_DRIVER}/testvars" ];  then
    source ./tmp.${TOKEN_DRIVER}/testvars
else
    exit 77 # token not configured, skip
fi
+ '[' -f ./tmp.provider/tests/basic/testvars ']'
+ exit 77
SKIP basic-softokn (exit status: 77)

Describe the bug
All tests fail, except for tls_softtokn.

It appears that the tests omit to load this provider?

To Reproduce
Steps to reproduce the behavior:

1. Clone, apply the patch to make compilation possible, configure, build.
2. Do make check
3. See error (8 tests skipped, 9 tests failed, 1 test passed).

test-suite.log
tls-softhsm-proxy.log
tls-softokn.log
readkeys-softhsm-proxy.log
readkeys-softokn.log
session-softhsm-proxy.log
session-softokn.log
genkey-softhsm.log
genkey-softokn.log
digests-softhsm-proxy.log
digests-softokn.log
rsapss-softokn.log
hkdf-softokn.log
oaepsha2-softokn.log
eccsha2-softokn.log
certs-softhsm-proxy.log
certs-softokn.log
basic-softhsm-proxy.log
basic-softokn.log
setup-softhsm.log
setup-softokn.log

Expected behavior
All tests passed.

Operating environment (please complete the following information):

• OS: MacOS Ventura
• Version 13.1

Token and application used (please complete the following information):

• Device: [e.g. ACME HSM Pro 1.7]
• PKCS11 Driver version: master

Additional context
Using OpenSSL master (3.2.0dev).

Here's the patch to compile pkcs11-provider - without it (at least on MacOS) it fails to get ssize_t type and load definition for snprintf() function.

diff --git a/src/provider.h b/src/provider.h
index 07b66c8..23a9e35 100644
--- a/src/provider.h
+++ b/src/provider.h
@@ -10,6 +10,7 @@
 #include <stdbool.h>
 
 #include "pkcs11.h"
+#include <sys/types.h>
 #include <openssl/core_dispatch.h>
 #include <openssl/core_object.h>
 #include <openssl/types.h>
diff --git a/src/session.c b/src/session.c
index 57e4169..0e1339e 100644
--- a/src/session.c
+++ b/src/session.c
@@ -1,8 +1,8 @@
 /* Copyright (C) 2022 Simo Sorce <[email protected]>
    SPDX-License-Identifier: Apache-2.0 */
 
-#include "provider.h"
 #include <string.h>
+#include "provider.h"
 
 /* Slot stuff */
 struct p11prov_slot {
diff --git a/src/util.c b/src/util.c
index ed996cf..0aad107 100644
--- a/src/util.c
+++ b/src/util.c
@@ -1,12 +1,13 @@
 /* Copyright (C) 2022 Simo Sorce <[email protected]>
    SPDX-License-Identifier: Apache-2.0 */
 
-#include "provider.h"
 #include <string.h>
 #include <time.h>
-#include "platform/endian.h"
+#include <sys/types.h>
 #include <openssl/bn.h>
 #include <openssl/x509.h>
+#include "provider.h"
+#include "platform/endian.h"
 
 #define FA_RETURN_VAL(x, _a, _b) \
     do { \

Describe the bug
The SoftHSM supports only CKM_ECDSA mechanism, which is raw EC signature. The pkcs11-provider fails when we try to create hashed signature using any digest+signature fails with the following logs:

[interface.gen.c:291] p11prov_CopyObject(): Calling C_CopyObject
[objects.c:168] cache_key(): Key 17:25 cached as 27:27
[signature.c:1630] p11prov_ecdsa_set_ctx_params(): ecdsa set ctx params (ctx=0x2164b50, params=(nil))
[signature.c:1454] p11prov_ecdsa_digest_sign_update(): ecdsa digest sign update (ctx=0x2164b50, data=0x451f00, datalen=65)
[interface.gen.c:153] p11prov_GetMechanismInfo(): Calling C_GetMechanismInfo
[interface.gen.c:157] p11prov_GetMechanismInfo(): Error: 0x00000070; Error returned by C_GetMechanismInfo
C_GetMechanismInfo for CKM_ECDSA_SHA512(4166) failed 112

The provider should fall back to the CKM_ECDSA if the CKM_ECDSA_SHA* mechanism is not available by doing the digest inside OpenSSL. This might be problematic with the certifications, but it should improve the imteroperability with mode pkcs11 providers or smart cards.

To Reproduce
Steps to reproduce the behavior:

1. Go to https://gitlab.com/libssh/libssh-mirror/-/merge_requests/334
2. See the latest failed pipeline results

Expected behavior
The signatures using ECDSA keys work with CKM_ECDSA mechanism only.

Operating environment (please complete the following information):

• OS: Fedora
• Version 36

Token and application used (please complete the following information):

• Device: SoftHSM
• PKCS11 Driver version: 2.6.1
• Application: libssh
• Version: master

Additional context
This should also allow us enabling the ecdsa signature tests with softhsm pipeline. The following change:

diff --git a/tests/Makefile.am b/tests/Makefile.am
index c3c58aa..87fae4d 100644
--- a/tests/Makefile.am
+++ b/tests/Makefile.am
@@ -46,7 +46,7 @@ dist_check_SCRIPTS = \
 test_LIST = \
 	basic-softokn basic-softhsm-proxy \
 	certs-softokn certs-softhsm-proxy \
-	eccsha2-softokn \
+	eccsha2-softokn eccsha2-softhsm-proxy \
 	oaepsha2-softokn \
 	hkdf-softokn \
 	rsapss-softokn \

Now gives the same error here:

Error signing raw input data                                                     
Public Key operation error                                                       
804B32944F7F0000:error:40800070:pkcs11:p11prov_GetMechanismInfo:An invalid mechanism was specified to the cryptographic operation:interface.gen.c:157:Error returned by C_GetMechanismInfo
804B32944F7F0000:error:40800001:pkcs11:p11prov_sig_operate_init:reason(1):signature.c:605:Failed to open session on slot 17

Describe the bug
When parsing parts of the PKCS#11 URI, the function parse_attr() allocates just the len + 1 bytes for the members (for example uri->object, but when these are compared against the token data, the hardcoded length from the PKCS#11 specification (32B) is used in p11prov_uri_match_token(). This leads to the buffer overruns (probably not reported because the openssl's allocs are used) and failures to match the keys.

To Reproduce
Steps to reproduce the behavior:

1. Try to select key by URI
2. See the matching failed

Expected behavior
Matching works

Operating environment (please complete the following information):

• OS: Fedora
• Version 36

Token and application used (please complete the following information):

• Device: SoftHSM, p11-kit
• PKCS11 Driver version: latest
• Application libssh
• Version master

Additional context
The parse_attr either needs to allocate the full length of the buffers it is checked against in p11prov_uri_match_token() or the p11prov_uri_match_token() needs to be adjusted to not compare behind the end of the buffer, ignoring trailing whitespace.

Describe the bug
Doing RSA PKCS1.5 signature with pre-calculated hash with hash information is calculated wrong. A DigestInfo struct should be included in the data to be signed.

To Reproduce
Patch to add for the test suite:

diff --git a/tests/tbasic b/tests/tbasic
index 7153cb3..07015e9 100755
--- a/tests/tbasic
+++ b/tests/tbasic
@@ -54,6 +54,19 @@ pkeyutl -verify -inkey "${PUBURI}"
                 -in ${TMPPDIR}/sha256.bin
                 -sigfile ${TMPPDIR}/sha256-sig.bin'
 
+title PARA "Sign and Verify with provided Hash and RSA with DigestInfo struct"
+ossl 'dgst -sha256 -binary -out ${TMPPDIR}/sha256.bin ${SEEDFILE}'
+ossl '
+pkeyutl -sign -inkey "${PRIURI}" -pkeyopt digest:sha256
+              -in ${TMPPDIR}/sha256.bin
+              -out ${TMPPDIR}/sha256-sig.bin'
+
+ossl '
+pkeyutl -verify -inkey "${PUBURI}" -pkeyopt digest:sha256
+                -pubin
+                -in ${TMPPDIR}/sha256.bin
+                -sigfile ${TMPPDIR}/sha256-sig.bin'
+
 title PARA "Sign and Verify with provided Hash and EC"
 ossl '
 pkeyutl -sign -inkey "${ECBASEURI}"

Steps to reproduce the behavior:

1. Apply above patch
2. make check

Expected behavior
Modified test suite passes.

Additional context
The OpenSSL core will properly set rsasig context, and result in log:

Set OSSL_SIGNATURE_PARAM_DIGEST to sha256

However, the sequence to call sign/verify will end up in a call to p11prov_sig_set_mechanism(digest_sign=false) and the digest gets completely ignored there.

It seems PKCS11 does not support signature with any of the CKM_SHA256_RSA_PKCS etc., so it is likely that the provider code should calculate and produce the DigestInfo structure with the hash info and use CKM_RSA_PKCS sign/verify mechanism.

Current strategy is to create PKCS11 sessions on the spot when needed. However is not ideal from interoperability and performance:

• Some modules may have hard limit on few or even one concurrent session. In this case the first thread gets the sessions, and other threads will fail with failure to create session.
• Other modules may support indefinite sessions, but have large CPU/time overhead on creation the session (e.g. session maps directly to a TCP connection). Thus using cache or pool of sessions could remove this overhead.
• Also, as further optimization, if logging in to session is needed to use the key/object, we can remove need for extra logins for consecutive use of object if session is kept in logged-in state.

Additional potential PKCS11 complication to account for is that all sessions share the logged-in user and state. Thus some operations might be possible if there are sessions using from logged-in user (e.g. some thread doing crypto as normal user, but another thread wanting to create/delete objects needing SO user).

This ticket is to discuss the strategy to manage sessions and concurrent access.

Two options to consider:

• Implement session pooling similar to libp11's pooling. Down side is the need for locking even in single threaded use cases.
• Using per-thread cached session in pthread "setspecific" area. These can have a destructor associated so sessions can be cleaned up on thread exit. Additionally C_CloseAllSessions can be used to flush them globally if needed.

Perhaps additional options brew up during discussion of the problem.

In some cases (HW accelerator) we want to insure all operations done with our keys need to stay on the token.
If we allow exporting keys though, openssl will try hard to switch to the default provider to perform operations.
By allowing config that prevents export of public keys we may force computations to stay on the token.

This can't be the default because on key generation you reallu need to be able to export a public key so that it can be handed over, so this is not a cut and dry thing.

Describe the feature
Provide feedback to user while generating keys by using OpenSSL clabback system when possible.

Expected behavior
When the token supports it provide user feedback on key generation.

Additional context
PKCS11 support the use of CK_CALLBACK_FUNCTION to allow feedback when the token knows long operations are expected.
This could be used to pipe this feedback all the way to OpenSSL callling application by setting appropriate provider callbacks.

Describe the bug
If the attribute requested is sensitive or otherwise and is "required" the function may return short w/o processing all data and with the wrong error code (CKR_HOST_MEMORY)

To Reproduce
N/A

Expected behavior
All available attributes should be processed and returned even on error
P11PROV_debug statements (or even P11PROV_raise for required ones) are emitted so that there is a log of which attributes were required/requested and were not returned

Operating environment (please complete the following information):
N/A

Token and application used (please complete the following information):
N/A

Additional context
N/A

Some infrastructure is already present but there are no keymgmt or signature operations exported.

The OASIS PKCS#11 header files that you have included are non-free.
Please consider replacing them with the drop-in replacements from the NSS library or pkcs11.h from OpenSC.

I would like to build and run the provider on a Windows system.
Is Windows compatibility planned in the near future?

Describe the feature
PKCS11 provider may provide the same algorithms with or without FIPS property. It depends on the underlying token certification status. Algorithm properties should be manageable via config.

Expected behavior

• Minimal requirement: if we have smth like "fips=1" in the configuration section, all the algorithms provided by provider should have "fips=yes" property.
• Real-life requirement: support of syntax like "fips-capable = alg1, alg2" allows specifying which algorithms provided by driver have the "fips=yes" property.

In setups with multiple slots/tokens, openssl key commands (pkey and pkeyutl) fails with the following error message:

OPENSSL_CONF=ossl_ock.conf openssl pkey -in [${key}](pkcs11:object=test_rsa1k:pub;type=public -pubin -pubout -out key.pub
Could not read key of Public Key from pkcs11:object=test_rsa1k:pub;type=public
802BE023C27F0000:error:40800005:pkcs11:p11prov_session_free:General Error:session.c:304:Potential double free in the calling code

The problem occured in a setup wiht multiple slots. Ma test setup (opencryptoki) has one slot with a sw-token and another slot without a a token (CKF_TOKEN_PRESENT not set). If I remove the second (empty) slot, the openssl command terminates correctly.

/usr/local/etc/opencryptoki/opencryptoki.conf

slot 0
{
stdll = libpkcs11_sw.so
tokversion = 3.12
}
slot 1
{
stdll = libep11.so
}

ossl_ock.conf

HOME = .

# Use this in order to automatically load providers.
openssl_conf = openssl_init

config_diagnostics = 1

[openssl_init]
providers = provider_sect

[provider_sect]
default = default_sect
pkcs11 = pkcs11_sect
base = base_sect

[base_sect]
activate = 1

[default_sect]
activate = 1

[pkcs11_sect]
module = /path/to/pkcs11_provider.so
pkcs11-module-path = libopencryptoki.so.0
pkcs11-module-token-pin = file:/path/to/pinfile.txt
activate = 1

It looks like the store_load() function iterates over all slots for finding keys. At the beginnig of each iteration, the session should be reset by calling p11prov_session_free() for the session from the previous iteration. This causes the error later, if the store is closed (p11prov_store_close) and all sessions of the context are free-ed.

Backtrace of the first p11prov_session_free():

Thread 1 "openssl" hit Breakpoint 1, p11prov_session_free (session=0x5555556b0ad0) at session.c:291
291         P11PROV_debug("Session Free %p", session);
(gdb) bt
#0  p11prov_session_free (session=0x5555556b0ad0) at session.c:291
#1  0x00007ffff7871c34 in store_load (ctx=0x555555660430, pw_cb=0x0, pw_cbarg=0x0) at store.c:245
#2  0x00007ffff7871e1d in p11prov_store_open (pctx=0x5555556596b0, uri=0x7fffffffe5d2 "pkcs11:object=test_rsa1k:prv;type=private")
    at store.c:311
#3  0x00007ffff7d32c69 in OSSL_STORE_open_ex () from /lib/x86_64-linux-gnu/libcrypto.so.3
#4  0x00005555555ef44f in ?? ()
#5  0x00005555555f00e3 in ?? ()
#6  0x00005555555f062b in ?? ()
#7  0x00005555555c1d5e in ?? ()
#8  0x00005555555ba151 in ?? ()
#9  0x0000555555596351 in ?? ()
#10 0x00007ffff78ae20a in __libc_start_call_main (main=main@entry=0x555555596190, argc=argc@entry=12, argv=argv@entry=0x7fffffffe2d8)
    at ../sysdeps/nptl/libc_start_call_main.h:58
#11 0x00007ffff78ae2bc in __libc_start_main_impl (main=0x555555596190, argc=12, argv=0x7fffffffe2d8, init=<optimized out>,
    fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffffffe2c8) at ../csu/libc-start.c:389
#12 0x0000555555596481 in ?? ()

Backtrace of the second p11prov_session_free():

Thread 1 "openssl" hit Breakpoint 1, p11prov_session_free (session=0x5555556b0ad0) at session.c:291
291         P11PROV_debug("Session Free %p", session);
(gdb) bt
#0  p11prov_session_free (session=0x5555556b0ad0) at session.c:291
#1  0x00007ffff787189b in p11prov_store_ctx_free (ctx=0x555555660430) at store.c:184
#2  0x00007ffff7872417 in p11prov_store_close (pctx=0x555555660430) at store.c:462
#3  0x00007ffff7d32886 in ?? () from /lib/x86_64-linux-gnu/libcrypto.so.3
#4  0x00007ffff7d3364e in OSSL_STORE_close () from /lib/x86_64-linux-gnu/libcrypto.so.3
#5  0x00005555555ef62f in ?? ()
#6  0x00005555555f00e3 in ?? ()
#7  0x00005555555f062b in ?? ()
#8  0x00005555555c1d5e in ?? ()
#9  0x00005555555ba151 in ?? ()
#10 0x0000555555596351 in ?? ()
#11 0x00007ffff78ae20a in __libc_start_call_main (main=main@entry=0x555555596190, argc=argc@entry=12, argv=argv@entry=0x7fffffffe2d8)
    at ../sysdeps/nptl/libc_start_call_main.h:58
#12 0x00007ffff78ae2bc in __libc_start_main_impl (main=0x555555596190, argc=12, argv=0x7fffffffe2d8, init=<optimized out>,
    fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffffffe2c8) at ../csu/libc-start.c:389
#13 0x0000555555596481 in ?? ()

Before digging deeper into the session handling: has someone else tested the pkcs11-provider successfully in a multi-slot or multi-token environment?

Describe the feature
On iterating slots, if we need a specific mechanism, we should check whether the slot supplies this mechanism.

In src/sessions.c we try to come up with some numbers to use to limit the amount of sessions we use at the same time and the number we can keep open as a cache.

This code really needs to be rethought, as these numbers are pulled out of thin air and the calculations are rough and inconsistent.

The OpenSSL API allows to enumerate multiple objects in a store.

This can turn useful if a token stores multiple certificates with their private keys and we want to enumerate the available certificate to find out if we have a private key to match an incoming request.
This is useful in environments where a single HSM is used to serve multiple services and therefore it stores multiple keys/certificate, for example on a terminating load balancer.

Implement support for enumerating all objects on the token instead of the single shot API currently enforced.

Describe the bug

It looks like OpenSSL adds RSA:rsaEncryption:1.2.840.113549.1.1.1 to its name table in ossl_namemap_add_names as umber 0 then when it tries to use it, the assert in evp_fetch.c:112 fails.
It is hard to debug as OpenSSL uses many inline macros. It is also not clear when the names are added as it looks like these are added when a provider is first used.

A pkcs11-spy shows the public key is read from card by OpenSC (which is using same version of OpenSSL as provider)

(gdb) run
Starting program: /opt/ossl-dev/bin/openssl pkey -in pkcs11:type=public\;id=%01\;pin-value=123456 -pubin -pubout -out /tmp/xxx.pub
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
../openssl/crypto/evp/evp_fetch.c:112: OpenSSL internal error: Assertion failed: name_id > 0 && name_id <= METHOD_ID_NAME_MAX

Program received signal SIGABRT, Aborted.
__pthread_kill_implementation (no_tid=0, signo=6, threadid=140737352845120) at ./nptl/pthread_kill.c:44
44	./nptl/pthread_kill.c: No such file or directory.
(gdb) where
#0  __pthread_kill_implementation (no_tid=0, signo=6, threadid=140737352845120) at ./nptl/pthread_kill.c:44
#1  __pthread_kill_internal (signo=6, threadid=140737352845120) at ./nptl/pthread_kill.c:78
#2  __GI___pthread_kill (threadid=140737352845120, signo=signo@entry=6) at ./nptl/pthread_kill.c:89
#3  0x00007ffff7442476 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#4  0x00007ffff74287f3 in __GI_abort () at ./stdlib/abort.c:79
#5  0x00007ffff7a73533 in OPENSSL_die (message=0x7ffff7cc5608 "Assertion failed: name_id > 0 && name_id <= METHOD_ID_NAME_MAX", 
    file=0x7ffff7cc55e0 "../openssl/crypto/evp/evp_fetch.c", line=112) at ../openssl/crypto/cryptlib.c:260
#6  0x00007ffff7a3dca9 in ossl_assert_int (expr=0, exprstr=0x7ffff7cc5608 "Assertion failed: name_id > 0 && name_id <= METHOD_ID_NAME_MAX", 
    file=0x7ffff7cc55e0 "../openssl/crypto/evp/evp_fetch.c", line=112) at ../openssl/include/internal/common.h:28
#7  0x00007ffff7a3de30 in evp_method_id (name_id=0, operation_id=10) at ../openssl/crypto/evp/evp_fetch.c:112
#8  0x00007ffff7a3e5a6 in inner_evp_generic_fetch (methdata=0x7fffffffd430, prov=0x555555690470, operation_id=10, 
    name=0x7ffff7ebadd8 "RSA:rsaEncryption:1.2.840.113549.1.1.1", properties=0x0, new_method=0x7ffff7a4b518 <keymgmt_from_algorithm>, 
    up_ref_method=0x7ffff7a4bc30 <EVP_KEYMGMT_up_ref>, free_method=0x7ffff7a4bc90 <EVP_KEYMGMT_free>) at ../openssl/crypto/evp/evp_fetch.c:323
#9  0x00007ffff7a3e7a6 in evp_generic_fetch (libctx=0x0, operation_id=10, name=0x7ffff7ebadd8 "RSA:rsaEncryption:1.2.840.113549.1.1.1", properties=0x0, 
    new_method=0x7ffff7a4b518 <keymgmt_from_algorithm>, up_ref_method=0x7ffff7a4bc30 <EVP_KEYMGMT_up_ref>, free_method=0x7ffff7a4bc90 <EVP_KEYMGMT_free>)
    at ../openssl/crypto/evp/evp_fetch.c:364
#10 0x00007ffff7a4bc2a in EVP_KEYMGMT_fetch (ctx=0x0, algorithm=0x7ffff7ebadd8 "RSA:rsaEncryption:1.2.840.113549.1.1.1", properties=0x0)
    at ../openssl/crypto/evp/keymgmt_meth.c:221
#11 0x00007ffff7bc57ca in try_key_ref (data=0x7fffffffd620, ctx=0x555555693a20, provider=0x555555690a20, libctx=0x0, propq=0x0)
    at ../openssl/crypto/store/store_result.c:200
#12 0x00007ffff7bc5e9d in try_key (data=0x7fffffffd620, v=0x7fffffffd7f0, ctx=0x555555693a20, provider=0x555555690a20, libctx=0x0, propq=0x0)
    at ../openssl/crypto/store/store_result.c:398
#13 0x00007ffff7bc54ed in ossl_store_handle_load_result (params=0x7fffffffd710, arg=0x7fffffffd7f0) at ../openssl/crypto/store/store_result.c:134
--Type <RET> for more, q to quit, c to continue without paging--
#14 0x00007ffff7eb2073 in p11prov_store_load (pctx=0x5555556a0540, object_cb=0x7ffff7bc5207 <ossl_store_handle_load_result>, object_cbarg=0x7fffffffd7f0, 
    pw_cb=0x7ffff7a832a1 <ossl_pw_passphrase_callback_dec>, pw_cbarg=0x555555693a68) at ../../src/src/store.c:493
#15 0x00007ffff7bc1fcd in OSSL_STORE_load (ctx=0x555555693a20) at ../openssl/crypto/store/store_lib.c:432
#16 0x000055555561085e in load_key_certs_crls (uri=0x7fffffffe1dd "pkcs11:type=public;id=%01;pin-value=123456", format=0, maybe_stdin=1, pass=0x0, 
    desc=0x5555556389c2 "Public Key", ppkey=0x0, ppubkey=0x7fffffffda08, pparams=0x0, pcert=0x0, pcerts=0x0, pcrl=0x0, pcrls=0x0)
    at ../openssl/apps/lib/apps.c:987
#17 0x000055555560f615 in load_pubkey (uri=0x7fffffffe1dd "pkcs11:type=public;id=%01;pin-value=123456", format=0, maybe_stdin=1, pass=0x0, e=0x0, 
    desc=0x5555556389c2 "Public Key") at ../openssl/apps/lib/apps.c:606
#18 0x00005555555d5d90 in pkey_main (argc=7, argv=0x7fffffffde00) at ../openssl/apps/pkey.c:216
#19 0x00005555555cd53c in do_cmd (prog=0x555555692460, argc=7, argv=0x7fffffffde00) at ../openssl/apps/openssl.c:418
#20 0x00005555555cd0c6 in main (argc=7, argv=0x7fffffffde00) at ../openssl/apps/openssl.c:298

To Reproduce
Run the test script below.

Expected behavior
A clear and concise description of what you expected to happen.

Operating environment (please complete the following information):

• OS: Ubuntu
• Version 22.04

Token and application used (please complete the following information):

• Device: NIST beta PIV card 2
• PKCS11 Driver version: opensc-pkcs11.so (master 0.23.0-rc1 plus)
• Application openssl pkey
• Version tag-3.0.5, tag-3.0.7 and master cloned from [email protected]:openssl/openssl.git

Additional context
test script:

#!/bin/sh
#set -xv

OSSL=/opt/ossl-dev
# debug  pkcs11-provider

export PKCS11_PROVIDER_DEBUG="file:/tmp/pp-debug"
export OPENSSL_CONF=$OSSL/ssl/openssl-pp.cnf

# Incase we are using pkcs11-spy.so in openssl-pp.cnf:
export PKCS11SPY=$OSSL/lib/opensc-pkcs11.so
export PKCS11SPY_OUTPUT=/tmp/pkcs11-spy.log

# may be in opensc.conf
#export OPENSC_DEBUG=3

gdb --args $OSSL/bin/openssl pkey \
     -in "pkcs11:type=public;id=%01;pin-value=123456" \
     -pubin -pubout -out /tmp/xxx.pub

diff of changes added to openssl.cnf

+++ openssl-pp.cnf	2022-11-06 06:53:00.983719361 -0600
@@ -13,7 +13,7 @@
 # defined.
 HOME			= .
 
-# Use this in order to automatically load providers.
+ # Use this in order to automatically load providers.
 openssl_conf = openssl_init
 
 # Comment out the next line to ignore configuration errors
@@ -60,6 +60,20 @@
 # included fipsmodule.cnf.
 # fips = fips_sect
 
+base = base_sect
+pkcs11 = pkcs11_sect
+
+[base_sect]
+activate = 1
+
+[pkcs11_sect]
+module = /opt/ossl-dev/lib/pkcs11_provider.so
+#pkcs11-module-path = /opt/ossl-dev/lib/opensc-pkcs11.so
+pkcs11-module-path = /opt/ossl-dev/lib/pkcs11-spy.so
+pkcs11-module-allow-export = 1
+activate = 1
+
+
 # If no providers are activated explicitly, the default one is activated implicitly.
 # See man 7 OSSL_PROVIDER-default for more details.
 #
@@ -69,7 +83,7 @@
 # OpenSSL may not work correctly which could lead to significant system
 # problems including inability to remotely access the system.
 [default_sect]
-# activate = 1
+ activate = 1

Describe the bug
openssl ca commad with yubikey tokens fails.

To Reproduce
Steps to reproduce the behavior:
mkdir demoCA
mkdir demoCA/newcerts demoCA/private
echo "01" > ./demoCA/serial
touch ./demoCA/index.txt

#Creating CA cert:
openssl req -x509 -new -nodes -key "pkcs11:type=private;id=%02" -days 1024 -out ./demoCA/cacert.pem -sha256

#Creating certificate from csr:
openssl ca -out newcert.pem -days 3650 -in tmp.csr -keyfile "pkcs11:type=private;id=%02" -md sha256

error:
CA certificate and CA private key do not match
4087EFF7627F0000:error:05800075:x509 certificate routines:X509_check_private_key:unknown key type:../crypto/x509/x509_cmp.c:411:

Expected behavior
newcert.pem file

Operating environment (please complete the following information):

• OS: Ubuntu
• Version 20.10

Token and application used (please complete the following information):

Device: Yubikey5
PKCS11 Driver version: [e.g. 1.3.4]
Application openssl
Version 3.0.5

Additional context

There is already some place holder code for CKO_CERTIFICATE / CKC_X_509 support. This is a feature place holder to implement supporting X.509 certificate objects on PKCS#11 module.

This is most probably not really needed, but since this is in initial state, it should be probably documented what version of C is going to be used. My vote goes to C11/C17. It's well supported to some extent by all still supported distros including RH7. I don't think there's a strong argument for C99 in 2022.

Once we have a version of OpenSSL that includes openssl/openssl#19648 or equivalent we should add tests that insure no regressions on CA cert manipulation.

Good examples are contained in issues #94 and #99

OpenSSL wants DER encoded ECDSASignature, but PKCS#11 uses RAW byte sequences.

See PR #121 for a test case.

Describe the bug
openssl ocsp commad with yubikey tokens fails

To Reproduce
Steps to reproduce the behavior:

1. create demoCA:
   $ mkdir -p demoCA/newcerts
   $ touch demoCA/index.txt
   $ echo 01 > demoCA/serial

create openssl.cnf with pkcs11 provider and add:

[ usr_cert ]
authorityInfoAccess = OCSP;URI:http://127.0.0.1:8080

[ v3_OCSP ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
extendedKeyUsage = OCSPSigning

$ export OPENSSL_CONF=openssl.cnf
$ openssl genrsa -out rootCA.key 1024
$ openssl req -new -x509 -days 3650 -key rootCA.key -out rootCA.crt
$ openssl genrsa -out certKey.key 1024
$ openssl req -new -x509 -days 3650 -key certKey.key -out certificate.crt
$ openssl x509 -x509toreq -in certificate.crt -out CSR.csr -signkey certKey.key
$ openssl ca -batch -startdate 150813080000Z -enddate 250813090000Z -keyfile rootCA.key -cert rootCA.crt -policy policy_anything -notext -out certificate.crt -infiles CSR.csr

Creating the OCSP server:

$ openssl req -new -key "pkcs11:type=private" -out ocspSigning.csr -sha256
$ openssl ca -keyfile rootCA.key -cert rootCA.crt -in ocspSigning.csr -out ocspSigning.crt

Running the OCSP server:

$ openssl ocsp -index demoCA/index.txt -port 8080 -rsigner ocspSigning.crt -rkey "pkcs11:type=private;id=%02" -CA rootCA.crt -text -out log.txt

Result:

ACCEPT 0.0.0.0:8080 PID=31356
ocsp: waiting for OCSP client connections...

Running client:

$ openssl ocsp -CAfile rootCA.crt -issuer rootCA.crt -cert certificate.crt -url http://127.0.0.1:8080 -resp_text -noverify

Server result:

ocsp: Received request, 1st line: POST / HTTP/1.0
Segmentation fault (core dumped)

Describe the feature
The open_ex interface provide more information to the open interface including a UI method that carries information for the the store.

Describe the feature
We should not try to generate new keys using a particular slot if it doesn't support the necessary mechanism.

Describe the bug
openssl req command fails.
No errors are displayed as a result of this command but no csr file is created and the return value from the openssl req command is 1.
openssl pkey and openssl rsautl commands are working well.

To Reproduce
Steps to reproduce the behavior:
$ openssl req -new -batch -key "pkcs11:type=private;id=%02" -out tmp.csr
Expected behavior
csr file and return valur 0

Operating environment (please complete the following information):

• OS: Ubuntu
• Version 22.10

Token and application used (please complete the following information):

• Device: Yubikey5
• PKCS11 Driver version: [e.g. 1.3.4]
• Application openssl
• Version 3.0.5

p11prov-debug.log

We have fallback code to load an associated public key from a private key object.
We need to extend this to also try to use a certificate object if a public key object is not available as some tokens store only certificate and private key and depend on software to retrieve the public key from the certificate.

Describe the bug
The creation of a SSL server context fails, if the pkcs11-prrovider is configured.

To Reproduce

1. build the attached c-example (gcc -o ssl-ctx-new ssl-ctx-new.c -l ssl -l crypto)
2. build the pkcs11-provider (make check)
3. execute the example with the openssl-configuration for nss (OPENSSL_CONF=tests/openssl.cnf PKCS11_PROVIDER_DEBUG="file:p11prov-debug.log" ./ssl-ctx-new)

The SSL_CTX_new() call returns NULL, if the pkcs11-provider is configured. If the line pkcs11 = pkcs11_sect in openssl-cnf is commented out, the creation of the context works as expected.

Expected behavior
The SSL_CTX_new() call should also work with the pkcs11-provider.

Operating environment (please complete the following information):

• OS: debian, Fedora
• Version: debian sid, fedora 36

Token and application used (please complete the following information):
NSS Soft-Token. Not key material is involved for the creation of the SSl context.

Additional context
Attachments: the c-example and the debug log.

Originally reported in #77 as I noticed this during the copr build:

To Reproduce
Steps to reproduce the behavior:

1. Run make check on s390x machine (with different endianness than x84_64)
2. Test are failing with error:

## Sign and Verify with provided Hash and RSA
openssl dgst -sha256 -binary -out ${TMPPDIR}/sha256.bin ${SEEDFILE}
openssl pkeyutl -sign -inkey "${PRIURI}" -in ${TMPPDIR}/sha256.bin -out ${TMPPDIR}/sha256-sig.bin
openssl pkeyutl -verify -inkey "${PUBURI}" -pubin -in ${TMPPDIR}/sha256.bin -sigfile ${TMPPDIR}/sha256-sig.bin
000003FFB0772720:error:0200008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:crypto/rsa/rsa_pk1.c:75:
000003FFB0772720:error:02000072:rsa routines:rsa_ossl_public_decrypt:padding check failed:crypto/rsa/rsa_ossl.c:599:
Signature Verification Failure
FAIL test.sh (exit status: 1)

Expected behavior
all checks pass

Operating environment (please complete the following information):

• OS: Fedora
• Version all

Token and application used (please complete the following information):

• Device: NSS swton
• PKCS11 Driver version: all

Additional context
The comment in #77 (comment) shows the potential source or problem, where the endianness is changed and I did not find this mentioned in the pkcs11 specifications, but did not get back to check it.

At least in some soft token a token key is really slow because it is stored encrypted on disk, so at each use it would have to be recovered with an expensive operation.

To speed up things for application that hold a private key structure we should copy the key to a session key object (this causes the soft toke to cache the private key) on the first operation that actually sues the private key and hold both handles on the key object.

This means the key objects will need to hold a session.

Currently, the debug messages are very rough and not much verbose. Getting hang of what is going on there is sometimes hard for the people who did not write the code.

From debugging other projects for years, I was very happy with the verbosity of debug messages, that contain at least a function name in them so it is easy to figure out where the message comes from. For example, now messages can look like this:

Get session on slot 18
Error: 224
Failed to get session to load keys

The way to figure out which error is the above is mostly about grepping for some strings that look like they should be fixed (sometimes already in other function or in other file) and then guessing where the code went through. The least would be the function name, similarly like this which should already give rough estimate where the message comes from (this is something like OpenSSH):

p11prov_get_session: Get session on slot 18
p11prov_session_open: Error: 224
store_fetch: Failed to get session to load keys

Even better would be to have there exact location in the file, as sometimes functions are not always in the file where I would expect them to be. Adding pid or thread ID might also come very helpful when this will be used in multi-thread/multi-process applications in the future (this is similar what we have in OpenSC now):

[1234][T1] session.c:714 p11prov_get_session: Get session on slot 18
[1234][T1] session.c:471 p11prov_session_open: Error: 224
[1234][T1] store.c:106 store_fetch: Failed to get session to load keys

Additionally, I found useful in the past log messages at the beginning of any non-trivial function that record the function was called. This makes easy to trace the program and see where it went off the path I would expect.

The last minor note would be interpretation of the CKR values or printing these values in hexadecimal, as that is easier to find in the header files. Having the following log would already give much more information than the decimal number in the previous cases:

Error: 0xD0 (TEMPLATE_INCOMPLETE) 

This is mostly request for comments if somebody sees some additional information that would be useful to get logged or if you see some drawbacks of logging some of the information before I will start rewriting this code.

Describe the bug
OpenSSL has two separate phases in dealing with storemgmt.
It expects to be able to open the store w/o requiring credentials, and then passed any callback (that cna provide passwords) at time of key loading.

When I changed the code to be able to list public objects w/o requiring a login I broke the fallback code that was going to retry reading the store in the load phase when a password is available.

To Reproduce
Steps to reproduce the behavior:
Remove the PIN from the condifuration file and execute anopenssl command that requires the PIN (anything that uses a private key).
No prompting will happen and the command fails to execute with an error about not being able to open the pkcs11 uri passed in.

Expected behavior
Prompting happens and the operation completes.

Operating environment (please complete the following information):
N/A

Token and application used (please complete the following information):
N/A

Additional context
Potentially always deferring the open operation could solve this.

Hey,

before, I start doing this (it will require some tweaking), we had great success with adding and using .clang-format to reformat files automatically. I can try to model the .clang-format according to the current code (or just use linux format). But before I do that, is this something you would be willing to accept? There are numerous plugins to editors to format on save and it makes the life much easier in the end.

Please add:
pkg-config
libtool

Thanks

If a private key has CKA_ALWAYS_AUTHENTICATE set CK_TRUE to do any crypto operations on it, a secondary C_Login(CKU_CONTEXT_SPECIFIC) is needed in connection with the crypto operation.

The provider-keymgmt API supports key generation. This ticket is feature holder to implement the EC/RSA key generation support.

Describe the bug

./configure fails to locate /opt/local/lib/nss/libsoftokn3.so.

To Reproduce
Steps to reproduce the behavior:

1. Go to 'pkcs11-provider'
2. Do autoreconf -fi
3. Do ./configure
4. See error

checking for /opt/local/lib/nss/libsoftokn3.so... no
.  .  .
configure: WARNING: Softoken library missing, tests will fail!

Expected behavior
Find /opt/local/lib/nss/libsoftokn3.dylib, because (of course) dynamic libraries on Mac have extension .dylib rather than .so.

Operating environment (please complete the following information):

• OS: MacOS Ventura
• Version 13.1
• Compiler: Clang from Xcode-14.2

Token and application used (currently irrelevant):

• Device: Yubikey 5
• PKCS11 Driver version: OpenSC master
• Application OpenSSL
• Version 3.2.0dev

Additional context
Add any other context about the problem here.

Most of PKCS#11 tokens support RNGs. We need to use these RNGs as a part of openssl RNG support (probably just seeding and/or separate RNG)

The RSA and ECDSA are the most common keys these days in the PKCS#11 world, but the EdDSA is becoming also more and more popular. There are HSMs supporting these keys for some time (yubiHSM) as well as some smart cards/tokens based on OpenPGP (NitroKey with the gnuk applet). It will also become more important when the FIPS 186-5 will get approved.

Describe the feature
On initialization, the slots without tokens should be avoided

Describe the bug
configure: WARNING: The p11-kit client not found. Can not run SoftHSM tests

To Reproduce
./configure

Expected behavior
A clear and concise description of what you expected to happen.

Operating environment (please complete the following information):

• OS: [Ununtu 22.10]

Additional context
$ softhsm2-util --version
2.6.1

The code that extract public key info can fallback to pull that information out of a certificate if there is no corresponding public key object in the token.
Unfortunately it is not easy to add a certificate w/o pkcs12-util also covertly extracting and adding a public key object to softokn.

The PKCS#11 WG has a page with common bugs and while reading one of the pkcs11 traces from pkcs11-provider, I noticed that it does this: Calling FindObjectsInit, then the GetAttributeValue and then moving on to other objects.

C_FindObjectsInit/Find/End can't be interrupted

Interrupting the sequence C_FindObjectsInit/C_FindObjects/C_FindObjectsEnd with a different call results in inconsistent/erratic results. This makes it difficult to safely perform any general processing of multiple objects rather than just using the find-objects capability as a basic "find me this one object".

https://wiki.oasis-open.org/pkcs11/CommonBugs

Last time I was writing something like this I think I was pointed to that page by somebody more experienced that this is wrong (even though the wording is not very clear). I changed the code to run the Find, store the handles and then read the objects attributes like this:

https://github.com/OpenSC/OpenSC/blob/master/src/tests/p11test/p11test_case_common.c#L800-L875

I could not find any similar wording in the PKCS#11 specification though and I believe any sane PKCS#11 module should handle this, but if there is somebody with more experience who could add more information, it would be appreciated

Having a pin on the pkcs11_uri is useful but also not great as those strings are shared in many places.
A configuration option to specify a pin (directly) and a pin file (so permissions can be locked down) should be provided.

On fork a lot of PKCS#11 object handles become irrelevant. We need to have a test like "login-fork-do stuff" to check whether all these handles are dealt with correctly.

Describe the bug
The PKCS#11 modules should be generally usable before login at least for listing of the keys. The NSS is taking this to the extreme, asking pin before any operation, which is OK, according to the standards, but painful for the users [1][2].
For this, the profile objects were invented in PKCS#11 3.0, but there are many PKCS#11 modules not supporting this yet a lot of software that is using OpenSSL and will be expecting this behavior [libssh].

To Reproduce
Steps to reproduce the behavior:

1. Remove the pkcs11-module-token-pin directive from openssl.cnf
2. Run search for a public key: pkey -in $PUBURI -pubin -pubout -out ${TSTOUT}.pub
3. This fails on cryptic message somewhere deep in creating sessions while attempting to login

Expected behavior
The applications should be allowed to list public objects without being prompted for a PIN and without requiring the pin in the configuration file.

Operating environment (please complete the following information):

• OS: Fedora
• Version 36

Token and application used (please complete the following information):

• Device: SoftHSM, NSS Softokn from testsuite
• PKCS11 Driver version: *
• Application: libssh, openssl
• Version any

Additional context
The pkcs11 provider should try to list the objects on the card even without login. The CKF_LOGIN_REQUIRED flag says only

True if there are some cryptographic functions that a user MUST be logged in to perform

so this does not mean that the user needs to be logged in to do ANY operation on the module.

An reproduced and ugly solution that solved the issue for me is available in the follwing branch (but it will probably need some more work to work in all corner cases
https://github.com/Jakuje/pkcs11-provider/tree/login (main...Jakuje:login)

This is both a RFE issue as well as an open question to other project contributors.

Although this attribute is no in c11 which is our minimum standard, it is supported by gcc and clang, and makes cleanup of variables a lot nicer as it avoids goto statements in most functions.

I'll leave this here for a little while and see if there is feedback.
If I do not get any major push back I might pick it up and implement later.

Describe the feature
Currently pkcs11 provider fails to login if a hardware token uses an onboard pinpad or some other out of band authentication method

Expected behavior
Authentication via out of band token authentication should work

Additional context
The token_login function unconditionally fails if a pin is not available w/o trying a C_Login operation. A C_login operation should be still tried if no pin is available but the token advertises
CKF_PROTECTED_AUTHENTICATION_PATH