I converted your onnx models to tf saved model and with the current tfjs converter to tfjs models and it works very well. here is my code (based a lot on yours):
https://github.com/Jonathhhan/ofEmscriptenExamples/blob/main/emscriptenCLIP2/clip.js
This is how I converted the models:
tensorflowjs_converter --input_format tf_saved_model clip-text-vit-32-tf clip-text-vit-f16-tfjs --quantize_float16
tensorflowjs_converter --input_format tf_saved_model clip-image-vit-32-tf clip-image-vit-uint8-tfjs --quantize_uint8

https://huggingface.co/Jona0123456789

Hi Joe :)

There is an issue with browsers that I hope you could help me with!

On Chrome, running locally, it failed to load vips.worker.js when initializing models but ran OK when deployed to GitHub page.

On Firefox, running locally, it loaded vips.worker.js successfully. So I suppose this is an issue with the browser. Firefox does not support File System Access API, so I can only use the reddit image dataset provided.

--- Details when running locally on Chrome ---

I ran into this error when initializing the models:

Failed to execute 'importScripts' on 'WorkerGlobalScope': The script at 'http://localhost:63342/clip-image-sorter/vips/vips.worker.js' failed to load.`.

However, the file vips.worker.js is indeed available:

Any help would be greatly appreciated! Thank you :)
Huyen