the example is unable to run on iphone 11 pro. about whisperkit HOT 8 OPEN

Here is the problem

[WhisperKit] Cache Length: 2 Input Token: 50359
[WhisperKit] Key Cache | Val Cache | Update Mask | Decoder Mask | Position
[WhisperKit] -0.125732 | 0.048828 | 0 | 0 | 0
[WhisperKit] 0.308350 | -0.556641 | 0 | 0 | 1
[WhisperKit] 0.000000 | 0.000000 | 1 | 0 | 2
[WhisperKit] 0.000000 | 0.000000 | 0 | -10000 | 3
[WhisperKit] Fallback https://github.com/argmaxinc/WhisperKit/pull/1.0 (logProbThreshold)
[WhisperKit] Decoding Temperature: 0.2

The loop is hitting its log prob threshold before end of text, which is likely part of the early stopping mechanism in the app. Normally it's not an issue that early but perhaps on iphone 11 it is.

One step to see if it solves the issue is to try editing line 929 in the example app ContentView.swift and add a param for logprob threshold like this:

        let options = DecodingOptions(
            verbose: false,
            task: task,
            language: languageCode,
            temperatureFallbackCount: 3, // limit fallbacks for realtime
            sampleLength: Int(sampleLength), // reduced sample length for realtime
            usePrefillPrompt: enablePromptPrefill,
            usePrefillCache: enableCachePrefill,
            skipSpecialTokens: !enableSpecialCharacters,
            withoutTimestamps: !enableTimestamps,
            clipTimestamps: seekClip,
            logProbThreshold: 0 //<-------- add this line
        )

Will attempt to reproduce this in the meantime, thanks for the report 👍

from whisperkit.

Thank you for the report! TestFlight feedback also showed that iPhone 11 and iPhone XS (the 2 oldest devices still supported by iOS17) are consistently having issues. My current suspicion is that this is due to the Neural Engine specifically. If you are able to rebuild the app after changing the compute units to GPU and test it on your 11 Pro, would be much appreciated :)

from whisperkit.

Looking a bit deeper into your debug logs, the model forward passes seem to be successful but model outputs are likely to be corrupt if you are getting <|endoftext|> right away.

from whisperkit.

Yes one thing standing out to me here is that the cache size is only 2. Can you post the full debug logs from the beginning of the loop all the way to the end? That way we can see what the tokens are coming through as - typically for base you would want to see

0: <|startoftranscription|>
1: <|en|> (or whatever language option set)
2: <|transcribe|> OR <|translate|>
3: <|0.00|> OR <|notimestamps|>
4..<n: (predicted transcript)
n: <|endoftext|>

from whisperkit.

I tried the base model with ios app whisperBoard, which was ok.
But the WhisperAX example was failed, here is the log message and error message:

doUnloadModel:options:qos:error:: model=_ANEModel: { modelURL=file:///var/mobile/Containers/Data/Application/F857AEFB-FBFC-4530-825D-81DA7772EC2D/Library/Caches/com.moody.whisperkit.WhisperAX/com.apple.e5rt.e5bundlecache/21C66/55FDBDC681E9BEC3377E76410A13B6A36D117901B2EACD6CCEC8EF6477BBCE68/F154AEEA75FF5C72ECAEBA3C899508F053C1576E8D7761AF31DBC258BF1ABFC3.bundle/H12.bundle/main/main_eir/ : sourceURL= (null) : key={"isegment":5,"inputs":{"encoder_output_embeds_eir":{"shape":[1500,1,1,512,1]}},"outputs":{"key_cast_fp16":{"shape":[1500,1,1,512,1]},"value_cast_fp16":{"shape":[1500,1,1,512,1]}}} : identifierSource=0 : cacheURLIdentifier=C231E55A5F6BD6E18E60C70EF622BACAF5215E7A45E150A265BFE09C379D6CA7_2393CF36BE99B057765D2CBDB5257F2EF24668A91C43BF3732D9D8E6D0053626 : string_id=0x70ac000000264af7 : program=_ANEProgramForEvaluation: { programHandle=4755813074256 : intermediateBufferHandle=4755813104416 : queueDepth=127 } : state=3 : programHandle=4755813074256 : intermediateBufferHandle=4755813104416 : queueDepth=127 : attr={
ANEFModelDescription = {
ANEFModelInput16KAlignmentArray = (
);
ANEFModelOutput16KAlignmentArray = (
);
ANEFModelProcedures = (
{
ANEFModelInputSymbolIndexArray = (
0
);
ANEFModelOutputSymbolIndexArray = (
0,
1
);
ANEFModelProcedureID = 0;
}
);
kANEFModelInputSymbolsArrayKey = (
"encoder_output_embeds_eir"
);
kANEFModelOutputSymbolsArrayKey = (
"key_cast_fp16@output",
"value_cast_fp16@output"
);
kANEFModelProcedureNameToIDMapKey = {
"net_5" = 0;
};
};
NetworkStatusList = (
{
LiveInputList = (
{
BatchStride = 1540096;
Batches = 1;
Channels = 512;
Depth = 1;
DepthStride = 1540096;
Height = 1;
Interleave = 1;
Name = "encoder_output_embeds_eir";
PlaneCount = 512;
PlaneStride = 3008;
RowStride = 3008;
Symbol = "encoder_output_embeds_eir";
Type = Float16;
Width = 1500;
}
);
LiveOutputList = (
{
BatchStride = 1540096;
Batches = 1;
Channels = 512;
Depth = 1;
DepthStride = 1540096;
Height = 1;
Interleave = 1;
Name = "key_cast_fp16@output";
PlaneCount = 512;
PlaneStride = 3008;
RowStride = 3008;
Symbol = "key_cast_fp16@output";
Type = Float16;
Width = 1500;
},
{
BatchStride = 1540096;
Batches = 1;
Channels = 512;
Depth = 1;
DepthStride = 1540096;
Height = 1;
Interleave = 1;
Name = "value_cast_fp16@output";
PlaneCount = 512;
PlaneStride = 3008;
RowStride = 3008;
Symbol = "value_cast_fp16@output";
Type = Float16;
Width = 1500;
}
);
Name = "net_5";
}
);
} : perfStatsMask=0} was not loaded by the client.
[WhisperKit] Loaded text decoder
[WhisperKit] Loading models from /var/mobile/Containers/Data/Application/F857AEFB-FBFC-4530-825D-81DA7772EC2D/Documents/huggingface/models/argmaxinc/whisperkit-coreml/openai_whisper-base with prewarmMode: false
[WhisperKit] Loading feature extractor
[WhisperKit] Loaded feature extractor
[WhisperKit] Loading audio encoder

[WhisperKit] Loaded audio encoder
[WhisperKit] Loading text decoder
Validation failure: Invalid input tensor channel 1 and format size 2 bytes, must be aligned on 64 bytes
Validation failure: Invalid input tensor channel 1 and format size 2 bytes, must be aligned on 64 bytes
Validation failure: Invalid input tensor channel 1 and format size 2 bytes, must be aligned on 64 bytes
Validation failure: Invalid input tensor channel 1 and format size 2 bytes, must be aligned on 64 bytes
Validation failure: Invalid input tensor channel 1 and format size 2 bytes, must be aligned on 64 bytes
Validation failure: Invalid input tensor channel 1 and format size 2 bytes, must be aligned on 64 bytes
[WhisperKit] Loaded text decoder
[WhisperKit] Loading tokenizer for base

[WhisperKit] Loaded tokenizer
[WhisperKit] Loaded models for whisper size: base

tcp_input [C2.1.1:3] flags=[R.] seq=67868, ack=3630046153, win=32950 state=LAST_ACK rcv_nxt=67868, snd_una=3630046129
[WhisperKit] Current audio size: 32000 samples, most recent buffer: 1600 samples, most recent energy: (0.14581555, 0.0028157167, 0.009438675, 1.1093871e-06)

[WhisperKit] Current audio size: 64000 samples, most recent buffer: 1600 samples, most recent energy: (0.07724396, 0.0031911524, 0.009436812, 1.4101643e-06)

[WhisperKit] Current audio size: 96000 samples, most recent buffer: 1600 samples, most recent energy: (0.48100042, 0.03933922, 0.13287143, 2.9166695e-05)

[WhisperKit] Current audio size: 128000 samples, most recent buffer: 1600 samples, most recent energy: (0.293728, 0.011141126, 0.041470844, 5.8486767e-06)

[WhisperKit] Current audio size: 160000 samples, most recent buffer: 1600 samples, most recent energy: (0.3946247, 0.022941826, 0.073072806, 3.928726e-06)

[WhisperKit] Current audio size: 192000 samples, most recent buffer: 1600 samples, most recent energy: (0.054507904, 0.0029965553, 0.0102628, 8.92207e-07)

[WhisperKit] Decoder init time: 0.006031990051269531
[WhisperKit] Prefill time: 0.00011301040649414062
[WhisperKit] Prefill prompt: ["<|startoftranscript|>", "<|en|>", "<|transcribe|>", "<|0.00|>"]
[WhisperKit] Decoding Seek: 0
[WhisperKit] Decoding 0.0s - 13.3s
[WhisperKit] Decoding with tempeartures [0.0, 0.2, 0.4, 0.5996]
[WhisperKit] Decoding Temperature: 0.0
[WhisperKit] Running main loop for a maximum of 224 iterations, starting at index 0
[WhisperKit] Forcing token 50258 at index 0 from initial prompt
[WhisperKit] --------------- DECODER INPUTS DEBUG ---------------
[WhisperKit] Cache Length: 0 Input Token: 50258
[WhisperKit] Key Cache | Val Cache | Update Mask | Decoder Mask | Position
[WhisperKit] 0.000000 | 0.000000 | 1 | 0 | 0
[WhisperKit] 0.000000 | 0.000000 | 0 | -10000 | 1
[WhisperKit] 0.000000 | 0.000000 | 0 | -10000 | 2
[WhisperKit] 0.000000 | 0.000000 | 0 | -10000 | 3
[WhisperKit] tokenIndex: 0, token: 50362, word: <|nocaptions|>
[WhisperKit] Forcing token 50259 at index 1 from initial prompt
[WhisperKit] --------------- DECODER INPUTS DEBUG ---------------
[WhisperKit] Cache Length: 1 Input Token: 50259
[WhisperKit] Key Cache | Val Cache | Update Mask | Decoder Mask | Position
[WhisperKit] -0.125732 | 0.048828 | 0 | 0 | 0
[WhisperKit] 0.000000 | 0.000000 | 1 | 0 | 1
[WhisperKit] 0.000000 | 0.000000 | 0 | -10000 | 2
[WhisperKit] 0.000000 | 0.000000 | 0 | -10000 | 3
[WhisperKit] tokenIndex: 1, token: 50359, word: <|transcribe|>
[WhisperKit] Forcing token 50359 at index 2 from initial prompt
[WhisperKit] --------------- DECODER INPUTS DEBUG ---------------
[WhisperKit] Cache Length: 2 Input Token: 50359
[WhisperKit] Key Cache | Val Cache | Update Mask | Decoder Mask | Position
[WhisperKit] -0.125732 | 0.048828 | 0 | 0 | 0
[WhisperKit] 0.308350 | -0.556641 | 0 | 0 | 1
[WhisperKit] 0.000000 | 0.000000 | 1 | 0 | 2
[WhisperKit] 0.000000 | 0.000000 | 0 | -10000 | 3
[WhisperKit] Fallback #1.0 (logProbThreshold)
[WhisperKit] Decoding Temperature: 0.2
[WhisperKit] Running main loop for a maximum of 224 iterations, starting at index 0
[WhisperKit] Forcing token 50258 at index 0 from initial prompt
[WhisperKit] --------------- DECODER INPUTS DEBUG ---------------
[WhisperKit] Cache Length: 0 Input Token: 50258
[WhisperKit] Key Cache | Val Cache | Update Mask | Decoder Mask | Position
[WhisperKit] -0.125732 | 0.048828 | 1 | 0 | 0
[WhisperKit] 0.308350 | -0.556641 | 0 | -10000 | 1
[WhisperKit] 0.000000 | 0.000000 | 0 | -10000 | 2
[WhisperKit] 0.000000 | 0.000000 | 0 | -10000 | 3
[WhisperKit] tokenIndex: 0, token: 50362, word: <|nocaptions|>
[WhisperKit] Forcing token 50259 at index 1 from initial prompt
[WhisperKit] --------------- DECODER INPUTS DEBUG ---------------
[WhisperKit] Cache Length: 1 Input Token: 50259
[WhisperKit] Key Cache | Val Cache | Update Mask | Decoder Mask | Position
[WhisperKit] -0.125732 | 0.048828 | 0 | 0 | 0
[WhisperKit] 0.308350 | -0.556641 | 1 | 0 | 1
[WhisperKit] 0.000000 | 0.000000 | 0 | -10000 | 2
[WhisperKit] 0.000000 | 0.000000 | 0 | -10000 | 3
[WhisperKit] tokenIndex: 1, token: 50359, word: <|transcribe|>
[WhisperKit] Forcing token 50359 at index 2 from initial prompt
[WhisperKit] --------------- DECODER INPUTS DEBUG ---------------
[WhisperKit] Cache Length: 2 Input Token: 50359
[WhisperKit] Key Cache | Val Cache | Update Mask | Decoder Mask | Position
[WhisperKit] -0.125732 | 0.048828 | 0 | 0 | 0
[WhisperKit] 0.308350 | -0.556641 | 0 | 0 | 1
[WhisperKit] 0.000000 | 0.000000 | 1 | 0 | 2
[WhisperKit] 0.000000 | 0.000000 | 0 | -10000 | 3
[WhisperKit] [0.00 --> 13.30] <|startoftranscript|><|en|><|transcribe|><|0.00|><|endoftext|>
[WhisperKit] ---- Transcription Timings ----
[WhisperKit] Audio Load: 0.00 ms / 1 runs ( 0.00 ms/run) 0.00%
[WhisperKit] Audio Processing: 0.60 ms / 1 runs ( 0.60 ms/run) 0.03%
[WhisperKit] Mels: 135.32 ms / 1 runs ( 135.32 ms/run) 6.90%
[WhisperKit] Encoding: 939.16 ms / 1 runs ( 939.16 ms/run) 47.92%
[WhisperKit] Matrices Init: 6.03 ms / 1 runs ( 6.03 ms/run) 0.31%
[WhisperKit] Prefill: 0.11 ms / 1 runs ( 0.11 ms/run) 0.01%
[WhisperKit] Decoding: 855.78 ms / 4 runs ( 213.94 ms/run) 43.66%
[WhisperKit] Non-inference: 17.11 ms / 4 runs ( 4.28 ms/run) 0.87%
[WhisperKit] - Sampling: 10.92 ms / 4 runs ( 2.73 ms/run) 0.56%
[WhisperKit] - Kv Caching: 4.92 ms / 4 runs ( 1.23 ms/run) 0.25%
[WhisperKit] - Windowing: 0.15 ms / 1 runs ( 0.15 ms/run) 0.01%
[WhisperKit] Fallbacks: 773.91 ms / 0 runs ( 0.00 ms/run) 39.48%
[WhisperKit] Decoding Full Loop: 1953.54 ms / 4 runs ( 488.39 ms/run) 99.67%
[WhisperKit] -------------------------------
[WhisperKit] Model Load Time: 3.55 seconds
[WhisperKit] Inference Duration: 1.96 seconds
[WhisperKit] - Decoding Loop: 1.95 seconds
[WhisperKit] Time to first token: 1.79 seconds
[WhisperKit] Total Tokens: 9
[WhisperKit] Tokens per Second: 2.05 tok/s
[WhisperKit] Real Time Factor: 0.15
[WhisperKit] Fallbacks: 0.0
[WhisperKit] [0.00 --> 13.30] <|startoftranscript|><|en|><|transcribe|><|0.00|><|endoftext|>

from whisperkit.

I am sorry, Still same error.

from whisperkit.

@woodymoo Checking in on this, it appears to be a device-specific bug. If anyone can reproduce this consistently on an iPhone 11 Pro - please post here with any other info.

from whisperkit.

Feel like this is related but why does the Tesflight version run faster than the Xcode one? Debug/Release build or further optimizations? Transcription appears with a 10 second delay with Xcode (same model size)

from whisperkit.