SSH with openssl "mux digest failed" about cryptodev-linux HOT 5 OPEN

I also have this problem. When the cryptodev module is loaded, the remote host cannot log in to this machine via ssh. Is there any idea for this problem?

from cryptodev-linux.

You may need to set up the whole chain to investigate: openssl, cryptodev or linux and run some of the openssl tests. It looks as if something causes cryptodev to close midair. If you want to investigate you can start looking for what's causing that effect. It may be some changes in linux (what kernel is being used?) that are not accounted for in cryptodev. Or some defect on error path handling in cryptodev. Or some plain bug. Is this a regression apparent when moving to newer kernels? Did it work with kernels from 3-4 years ago? If it did, was it because there were some algorithms not available in the kernel?

You may need to refine the debugging depending on where you get with these steps.

from cryptodev-linux.

I use gdb to debug openssh, and the results are very different when the cryptodev module is loaded and not loaded:

1）Debug openssh when cryptodev is not loaded and the final result is as expected：

Starting program: /usr/sbin/sshd -d -p 22222
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib64/libthread_db.so.1".
debug1: sshd version OpenSSH_8.0, OpenSSL 1.1.1k FIPS 25 Mar 2021

Breakpoint 6, BN_priv_rand_range (r=0xaaaaaabad790, range=0xaaaaaabad7b0) at crypto/bn/bn_rand.c:184
184 return bnrand_range(PRIVATE, r, range);
(gdb) s
bnrand_range (flag=PRIVATE, r=0xaaaaaabad790, range=0xaaaaaabad7b0) at crypto/bn/bn_rand.c:118
118 if (range->neg || BN_is_zero(range)) {
(gdb) s
BN_is_zero (a=a@entry=0xaaaaaabad7b0) at crypto/bn/bn_lib.c:905
905 return a->top == 0;
(gdb) n
bnrand_range (flag=PRIVATE, r=0xaaaaaabad790, range=0xaaaaaabad7b0) at crypto/bn/bn_rand.c:113
113 static int bnrand_range(BNRAND_FLAG flag, BIGNUM *r, const BIGNUM range)
(gdb) s
bnrand_range (range=0xaaaaaabad7b0, r=0xaaaaaabad790, flag=PRIVATE) at crypto/bn/bn_rand.c:123
123 n = BN_num_bits(range); / n > 0 */
(gdb) n
127 if (n == 1)
(gdb) n
129 else if (!BN_is_bit_set(range, n - 2) && !BN_is_bit_set(range, n - 3)) {
(gdb) n
162 if (!bnrand(flag, r, n, BN_RAND_TOP_ANY, BN_RAND_BOTTOM_ANY))
(gdb) n
165 if (!--count) {
(gdb) n
170 while (BN_cmp(r, range) >= 0);

2）Debug openssh when the cryptodev module is loaded，final execution fails：
Unable to load host key "/etc/ssh/ssh_host_rsa_key": invalid format
Unable to load host key: /etc/ssh/ssh_host_rsa_key
accumulate_host_timing_secret: ssh_digest_start

Starting program: /usr/sbin/sshd -d -p 22222
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib64/libthread_db.so.1".

Breakpoint 1, main (ac=4, av=0xfffffffff3e8) at sshd.c:1514
1514 {
(gdb) c
Continuing.
debug1: sshd version OpenSSH_8.0, OpenSSL 1.1.1k FIPS 25 Mar 2021

Breakpoint 2, bnrand_range (flag=PRIVATE, r=0xaaaaaabaf190, range=0xaaaaaabaf1b0) at crypto/bn/bn_rand.c:118
118 if (range->neg || BN_is_zero(range)) {
(gdb) c
Continuing.

Breakpoint 2, bnrand_range (range=0xaaaaaabaf1b0, r=0xaaaaaabaf190, flag=PRIVATE) at crypto/bn/bn_rand.c:123
123 n = BN_num_bits(range); /* n > 0 */
(gdb) n
127 if (n == 1)
(gdb) n
129 else if (!BN_is_bit_set(range, n - 2) && !BN_is_bit_set(range, n - 3)) {
(gdb) n
162 if (!bnrand(flag, r, n, BN_RAND_TOP_ANY, BN_RAND_BOTTOM_ANY))
(gdb) n
165 if (!--count) {
(gdb) n
bnrand_range (flag=PRIVATE, r=, range=) at crypto/bn/bn_rand.c:120
120 return 0;
(gdb) bt
#0 bnrand_range (flag=PRIVATE, r=, range=) at crypto/bn/bn_rand.c:120
#1 0x0000fffff7c05624 in BN_BLINDING_create_param (b=b@entry=0x0, e=e@entry=0xaaaaaabaeb70, m=m@entry=0xaaaaaabaedc0,
ctx=ctx@entry=0xaaaaaaba9180, bn_mod_exp=0xfffff7c07598 <BN_mod_exp_mont>, m_ctx=) at crypto/bn/bn_blind.c:273
#2 0x0000fffff7ce28a4 in RSA_setup_blinding (rsa=rsa@entry=0xaaaaaaba8410, in_ctx=0x0) at crypto/rsa/rsa_crpt.c:177
#3 0x0000fffff7ce2a94 in RSA_blinding_on (rsa=0xaaaaaaba8410, ctx=ctx@entry=0x0) at crypto/rsa/rsa_crpt.c:99
#4 0x0000aaaaaaae44c0 in sshkey_private_deserialize (buf=buf@entry=0xaaaaaabac7b0, kp=kp@entry=0xffffffffebb0) at sshkey.c:3261
#5 0x0000aaaaaaae4bf0 in sshkey_parse_private2 (blob=blob@entry=0xaaaaaabac390, passphrase=passphrase@entry=0xaaaaaab205a0 "",
keyp=keyp@entry=0xffffffffed20, commentp=0x0, type=11) at sshkey.c:3790
#6 0x0000aaaaaaae5538 in sshkey_parse_private_fileblob_type (commentp=, keyp=0xffffffffed20, passphrase=0xaaaaaab205a0 "",
type=11, blob=0xaaaaaabac390) at sshkey.c:4165
#7 sshkey_parse_private_fileblob_type (blob=0xaaaaaabac390, type=11, passphrase=0xaaaaaab205a0 "", keyp=0xffffffffed20,
commentp=) at sshkey.c:4140
#8 0x0000aaaaaaae875c in sshkey_load_private (filename=0xaaaaaaba6630 "/etc/ssh/ssh_host_rsa_key", passphrase=0xaaaaaab205a0 "",
keyp=0xffffffffed20, commentp=0x0) at authfile.c:258
#9 0x0000aaaaaaaae43c in main (ac=, av=) at sshd.c:1825

I found that this last step was a weird jump （return 0） that led to the final failure
I don’t understand why loading the cryptodev module will cause such a weird jump in openssh ？
My kernel： 5.10.21-2.el8*.aarch64

from cryptodev-linux.

My gdb ：gdb-8.2-12.el8.aarch64

from cryptodev-linux.

openssh uses openssl's hardware encryption engine by default：

openssl_engine=no
AC_ARG_WITH([ssl-engine],
[ --with-ssl-engine Enable OpenSSL (hardware) ENGINE support ],
[
if test "x$withval" != "xno" ; then
if test "x$openssl" = "xno" ; then
AC_MSG_ERROR([cannot use --with-ssl-engine when OpenSSL disabled])
fi
openssl_engine=yes
fi
]
)

Remove --with-ssl-engine from configure options, sshd OK

from cryptodev-linux.