Thanks for answering my previous questions =)

I just wanted to share this idea and why I ended up choosing against the custom TLV format. Instead, I store certs in a generic KLV format, that I already implemented for other purposes in my code base.

As alluded to in #2, I'm not a fan of using a custom hard coded TLV format. Personally, It's a deal-breaker.

Why?

• hard coded type values are not future proof, and annoying to change
• one-off formats increase code size
• one-off formats increase code complexity
• it is not extensible by 3rd parties. I can't add a lot of my own fields.
• custom formats often die quickly, I want code to last. I don't want it to be deprecated.
• its more annoying to document, because the format changes as types are added
• types are not human readable

Instead, we really should:

• move the format to its own *.c an *.h file
• document it, so its not as proprietary
• use a keyLen,key,dataLen,data format, so we don't need hardcoded type values
• it should be generic and generally useful
• it should support versioning
• consider making it an ESP-IDF component

The generic KLV format I use looks like this:

Edit: also worth considering using DER format.

[EKLV]         (4 bytes) // magic value, "espKeyLengthValue"
[versionMajor] (1 bytes)
[versionMinor] (1 bytes)
[totalItemsLen] (8 bytes) // total items length in bytes

// item 1
[keyLenLen] (1 byte)    // keyLen size is configurable
[keyLen]    (X bytes)
[key]       (X bytes)
[padLen]     (2 bytes)   // for aligning 'data'
[pad]        (X bytes)
[dataLenLen] (1 byte)    // dataLen size is configurable
[dataLen]    (X bytes)
[data]       (X bytes)

// item 2
[keyLenLen] (1 byte) 
[keyLen]    (X bytes)
[key]       (X bytes)
[padLen]     (2 bytes) 
[pad]        (X bytes)
[dataLenLen] (1 byte) 
[dataLen]    (X bytes)
[data]       (X bytes)

...repeat...

The TLV format currently used here is a deal-breaker for me.

NVS seems like a good choice. It is already supported by ESP-IDF, and rather stable.

NVS is likely to still be supported in 5+ years, etc.

Why did you switch to TLV?

Thanks.

Generating on device brings a few advantages:

1. generally simpler
2. the host PC just needs to sign the public key with the CA
3. less reliance for your build system to generate keys in "just the right way"
4. private keys never leave the device

Can you recommend a way to generate the private/public keys (Y,M,Rb,MPrime variables )on device? Using mbedTLS for instance?

If not, Espressif should consider adding a library to ESP-IDF which does that.

BitString was renamed to BitStream in Python 3.7+. I renamed in the IDF source and it seems to work.

jonsmirl@ares:~/aosp/esp-matter/examples/lowpan$ configure_esp_secure_cert.py -p /dev/ttyACM0 --keep_ds_data_on_host --efuse_key_id 1 --ca-cert cacert.pem --device-cert client.crt --private-key client.key --target_chip esp32c3 --secure_cert_type cust_flash_tlv --configure_ds
Traceback (most recent call last):
File "/home/jonsmirl/esp/esp-idf/components/esptool_py/esptool/espefuse.py", line 15, in
import espressif.efuse.esp32 as esp32_efuse
File "/home/jonsmirl/esp/esp-idf/components/esptool_py/esptool/espressif/efuse/esp32/init.py", line 1, in
from . import operations
File "/home/jonsmirl/esp/esp-idf/components/esptool_py/esptool/espressif/efuse/esp32/operations.py", line 18, in
from . import fields
File "/home/jonsmirl/esp/esp-idf/components/esptool_py/esptool/espressif/efuse/esp32/fields.py", line 17, in
from .. import base_fields
File "/home/jonsmirl/esp/esp-idf/components/esptool_py/esptool/espressif/efuse/base_fields.py", line 14, in
from bitstring import BitArray, BitString, CreationError
ImportError: cannot import name 'BitString' from 'bitstring' (/home/jonsmirl/.espressif/python_env/idf4.4_py3.10_env/lib/python3.10/site-packages/bitstring.py)

Note: I realize this should probably go in the ESP-IDF repo. But just continuing the discussion from before.

@AdityaHPatwardhan. Please add functions to generate ESP-DS Peripheral Params on device.

This is more secure than generating private keys in Python, and copying to device. It should be an option that users have.

Most important (RSA -> DS Params):
esp_err_t esp_ds_mbedtls_generate_params(const mbedtls_rsa_context* rsa, esp_ds_p_data_t* params);

Super Convenient (generate new rsa key & burn fuses & output pubKeyPem):
esp_err_t esp_ds_mbedtls_generate_keypair_and_burn_fuses(char* pubKeyPemBuf, size_t bufLen);

I've already done some of the code, tested working.

// Generate RInv & MPrime from RSA private numbers
void esp_mbedtls_calculate_rinv_mprime(const mbedtls_rsa_context* rsa, mbedtls_mpi* rinv, uint32_t* mprime)
{
    // python equivalent code:
    //
    //    key_size = private_key.key_size # in bits
    //
    //    # calculate rinv == Rb
    //    rr = 1 << (key_size * 2)
    //    rinv = rr % pub_numbers.n # RSA r inverse operand
    //
    //    # calculate MPrime
    //    a = rsa._modinv(M, 1 << 32)
    //    mprime = (a * -1) & 0xFFFFFFFF # RSA M prime operand

    // rr = 1 << (key_size * 2) # in bits
    mbedtls_mpi rr;
    mbedtls_mpi_init(&rr);
    mbedtls_mpi_lset(&rr, 1);
    mbedtls_mpi_shift_l(&rr, sizeof(pd_4096_bit_t) * 8 * 2);

    // rinv = rr % rsa.N 
    mbedtls_mpi_mod_mpi(&rinv, &rr, &N);

    // ls32 = 1 << 32
    mbedtls_mpi ls32;
    mbedtls_mpi_init(&ls32);
    mbedtls_mpi_lset(&ls32, 1);
    mbedtls_mpi_shift_l(&ls32, 32);

    // a = rsa._modinv(N, 1 << 32)
    mbedtls_mpi a;
    mbedtls_mpi_init(&a);
    mbedtls_mpi_inv_mod(&a, &N, &ls32);

    // a32 = a 
    uint32_t a32 = 0;
    mbedtls_mpi_write_binary_le(&a, (uint8_t*) &a32, sizeof(uint32_t));

    // mprime
    *mprime = ((int32_t) a32 * -1) & 0xFFFFFFFF;

    //
    // Results
    //
    printMpi(&rinv, "RInv");
    printf("mprime: 0x%x\n", *mprime);
}

void printMpi(const mbedtls_mpi* mpi, const char* name)
{
    size_t olen = 0;
    uint32_t len = MBEDTLS_MPI_MAX_SIZE * 2 + 1;
    char* pp = (char*)calloc(1, len);
    mbedtls_mpi_write_string(mpi, 16, pp, len, &olen);
    printf("%s: %s\n", name, pp);
    free(pp);
}

Is it possible to use the stored private key to generate a CSR and store a newly signed certificate in the esp_secure_cert partition?

Device naming is different between espefuse and configure_esp_secure_cert for esp32s3.
espefuse uses: "esp32s3beta2"
configure_esp_secure_cert uses: "esp32s3"

configure_esp_secure_cert error message:
espefuse: error: argument --chip/-c: invalid choice: 'esp32s3' (choose from 'auto', 'esp32', 'esp32s2', 'esp32s3beta2', 'esp32c3')

I have followed this example to write certs and keys to the ESP32-S3-WROOM-2 DEV KIT-C N32R8: https://github.com/FreeRTOS/iot-reference-esp32c3/blob/main/GettingStartedGuide.md

My question is what is the magic byte and is it something I should be writing?

I get the following error when I try to read using the latest version of this library:

E (2935) esp_secure_cert_tlv: Expected magic byte is BA5EBA11, obtained magic byte = 9058D003
E (2945) esp_secure_cert_tlv: Could not find the tlv of type 1

    uint32_t len = 0;
    char *addr = NULL;
    esp_err_t esp_ret = ESP_FAIL;
    esp_ret = esp_secure_cert_get_device_cert(&addr, &len);
    if (esp_ret == ESP_OK) {
        ESP_LOGI(TAG, "Device Cert: \nLength: %d\n%s", strlen((char *)addr), (char *)addr);
    } else {
        ESP_LOGE(TAG, "Failed to obtain flash address of device cert");
    }

Partition Map

# Name,   Type, SubType, Offset,   Size, Flags
# Note: if you have increased the bootloader size, make sure to update the offsets to avoid overlap
# esp_secure_cert type custflash 0x3F
esp_secure_cert, 0x3F,    ,     0xD000,   0x6000,   encrypted
nvs,             data, nvs,     0x13000,  0x6000,   
otadata,         data, ota,     0x19000,  0x2000,   encrypted
phy_init,        data, phy,     0x1b000,  0x1000,   encrypted
factory,         app,  factory,        ,      5M,   encrypted
ota_0,           app,  ota_0,          ,      5M,   encrypted
ota_1,           app,  ota_1,          ,      5M,   encrypted
storage,         data, nvs,            , 0x10000,   encrypted
nvs_key,         data, nvs_keys,       , 0x1000,    encrypted