Optimize Sentence Encoding with LLMs
We need to be able to encode large corpora efficiently, need to perform some benchmarking and research around how to do that. Different models are likely to have different performance profiles.
One option: Look into feasibility of using multi-core CPU/ANE to parallelize sentence embeddings. This should be technically possible, but we need to think through how to do it optimally and benchmark the process.
Edited by Jim Wallace