Accuracy mismatch between clear and FHE predictions in CIFAR-10 example about concrete-ml HOT 4 CLOSED

Hello @RomanBredehoft, thank you for your prompt response and also for answering my additional query. As for the FHEModelDev lines (l92-93), I have commented them out from my end (thank you for pointing this out though).

I am closing the issue from my end as I do not have any further questions (for now). I genuinely appreciate your help.

from concrete-ml.

Hello @bhuvneshchaturvedi2512 ,
Thanks for the report, we were unfortunately not able to reproduce what you seem to explain 😕 Could you therefore give us more information about your issue ? More precisely :

• are you using one of our latest commit ? if not, would it be possible to do so and try again ?
• could you provide the specific steps you've followed ?

For information, in the cifar_brevitas_training directory, I've ran python3 bnn_pynq_train.py --epochs 10 --data ./data --experiments ./experiments, updated evaluate_torch_cml.py with the new checkpoint path and then ran python3 evaluate_torch_cml.py, which gave me some expected results :

Torch accuracy top1: 0.6928401898734177
Concrete ML accuracy top1: 0.6919501582278481
Torch accuracy top5: 0.9740901898734177
Concrete ML accuracy top5: 0.9739912974683544

Please tell me if the issue you're describing is different from the above steps !

Thanks 😃

from concrete-ml.

Hello @RomanBredehoft, thank you for your response.

The current version is working fine and I am getting similar accuracy on multiple reruns of the script. I guess this was a problem with the previous version of the code which I tested in the first week of September. I see the code was updated a month ago, possibly in the third or fourth week of September.

On an additional note, the script evaluation_one_example_fhe.py (which is named evaluate_one_example_fhe.py in the repository but not in the command python3 evaluation_one_example_fhe.py) runs the inference in the encrypted setting which takes around 15 to 20 minutes on my machine. Can you please let me know if it is possible to run the inference in simulation mode? This will make it easy for me to test some of the stuff quickly.

I appreciate your help.

from concrete-ml.

Hello again @bhuvneshchaturvedi2512 ,
Great to know that it works well !

Regarding the evaluate_one_example_fhe.py file, thanks for pointing out the typo, I've created a small PR to fix it. As for running the script in simulation, I think the easiest solution for you would be to simply comment all lines starting at l144 and only print the simulated results ( print(f"Expected prediction: {expected_quantized_prediction}"), as done in l171 ! This is because simulation is already executed at l141 😄

Quick notes about it :

• don't forget to git lfs pull before if you haven't done so
• by default, this script only runs a single inference ! You can change that by running NUM_SAMPLES=100 python3 evaluate_one_example_fhe.py for 100 inferences for example
• you'll need to remove the FHEModelDev lines (l92-93) if you want to execute the script several times (I'll removed them as well as they are not used anymore anyway)

from concrete-ml.