How to use flan-t5-small or all-MiniLM-L6-v2 in .net iOS?

Hi @mc

You can do this in a plain .NET for iOS app. The models run through ONNX Runtime, which handles both of them with the same code.

1. Add the package

dotnet add package Microsoft.ML.OnnxRuntime

This ships native binaries for iOS device and simulator, and includes the CoreML and XNNPACK execution providers. The CoreML provider needs iOS 13+ and must be registered explicitly on the session options — it isn't on by default.

2. Export the models to ONNX

optimum-cli export onnx --model sentence-transformers/all-MiniLM-L6-v2 ./minilm-onnx
optimum-cli export onnx --model google/flan-t5-small ./flant5-onnx

Add the .onnx files and the tokenizer files to your iOS project as BundleResource, then read them from NSBundle.MainBundle.BundlePath at runtime.

3. Tokenize in C#

Use Microsoft.ML.Tokenizers:

• all-MiniLM-L6-v2 → BertTokenizer (WordPiece, needs vocab.txt). Gives you input_ids, attention_mask, token_type_ids.
• flan-t5-small → SentencePiece tokenizer (spiece.model), not the Bert one.

4. Wire up your "ask questions about a txt file" flow

The two models do different jobs, so use both:

1. Split the text file into chunks (a few hundred tokens each).
2. Run each chunk through MiniLM → take the last hidden state → mean pooling over the attention mask → L2-normalize. That's your 384-dim embedding. Cache these.
3. Embed the question the same way, rank chunks by cosine similarity, keep the top 2–3.
4. Build a prompt like question: {q} context: {top chunks} and run it through flan-t5.

One thing to plan for: T5 is encoder-decoder, so it isn't a single session.Run(). You run encoder_model.onnx once, then loop decoder_model.onnx (feeding it the encoder hidden states plus the tokens generated so far) until EOS, then decode the ids back to text. Use the _with_past decoder variant and pass the KV cache back in, otherwise generation is slow.

Note:

• flan-t5-small is small. Answers to open-ended questions will be limited. If your questions are the "find the answer in the text" kind, an extractive QA model (MiniLM-based, SQuAD-tuned) gives better results at the same size.
• App size and memory: the models plus the runtime add up in the IPA, and iOS is strict about memory. Test on a real device, not just the simulator.
• If you hit a linker error like library 'onnxruntime.xcframework.zip' not found on iOS, that's a known packaging issue — see https://github.com/microsoft/onnxruntime/issues/22661 for versions and workarounds.

Hope this helps. If you found my response helpful or informative, I would greatly appreciate it if you could follow this guidance or provide feedback.

Thank you.