Cohere says request shape affects embedding and reranking inference choices
Cohere says the choice between shared and dedicated inference for Embed and Rerank depends on work per request, traffic patterns, utilization and latency targets. Requests per minute alone can mislead: batch catalog indexing is throughput-oriented, while interactive search needs low latency.
Event / data period:
Published: · Updated:
Cohere’s illustrative calculation uses a batch of 100 texts at about 200 tokens each, or 20,000 tokens; under one continuously used setup, dedicated capacity reaches its modeled crossover at roughly 20 requests per minute. The post gives about four requests per minute for batches of longer documents and 29 for reranking 50 candidates.
These are not universal thresholds. Cohere’s calculation assumes one hourly billed NVIDIA A10 GPU, no negotiated discounts and traffic sustained around the clock. The company says the cost crossover changes with active hours, pricing, hardware and latency requirements.
Data
Cohere says request shape affects embedding and reranking inference choices
- · Occurred
Cohere says request shape affects embedding and reranking inference choices
Cohere says the choice between shared and dedicated inference for Embed and Rerank depends on work per request, traffic patterns, utilization and latency targets. Requests per minute alone can mislead: batch catalog indexing is throughput-oriented, while interactive search needs low latency.
Methodology
This event record summarizes the dated statement in the cited primary source. Product or test claims from the source are not presented as independently verified results.
Updated:
Subjects and places
Subject proposals
- Science and technologySuggested
- Preserved editorial category/topic; broad candidate onlycohere.com ↗