lm-route-proxy | overview

links:

main:

flowchart LR
    A[openrouter base-url] --> B[lm route proxy]
    B --> C{picks the best provider}
    C --> D[fastest]
    C --> E[cheapest]
    C --> F[most reliable]
    C --> G[cache aware]
    D & E & F & G --> H[ranks descending]
    H --> I[openrouter]

always route to the best openrouter provider available [price, cache, tps, latency] - smart routing decisions. lm_route_proxy is a database-free, asynchronous aiohttp/python api proxy designed to operate between openai-compatible clients and upstream routers such as openrouter.

during request processing, candidate providers are first subjected to hard filtering against required model capabilities, client restrictions, and operator privacy gates (prompt retention / training prohibitions). eligible providers are then evaluated using a sum-normalised composite score derived from latency %, throughput (tps) %, and request-sensitive effective pricing (input /output /cache-read /cache-write -costs).

specifics:

flowchart TD
    pool{{"Candidate Pool"}}

    phase1[/"Phase 1:
Hard Constraints"/]
    note1["Instantly drops candidates lacking capabilities,
missing metadata, or violating configured
privacy gates."]

    phase2[/"Phase 2:
Base Score"/]
    note2["Surviving providers receive a
sum-normalized score blending:
Latency (p50/p75), Throughput (TPS),
and Context-aware effective price."]

    out[/"1. top-ranked provider
2. next-best provider
3. third-best provider"/]

    degrade["Graceful Degradation
Missing provider metrics automatically
default to neutral medians to prevent
artificial ranking bias."]

    pool --> phase1
    phase1 -- "surviving candidates" --> phase2
    phase2 --> out

    phase1 -.- note1
    phase2 -.- note2
    degrade -. "applies to" .-> phase2

    classDef stage fill:#111,stroke:#fff,color:#fff
    classDef annotation fill:#fff,stroke:#fff,color:#000
    class pool,phase1,phase2,out stage
    class note1,note2,degrade annotation

to optimize cache continuity, an in-memory, count- and ttl-bounded RouteHistory logs normalized context prefixes. incoming requests matching prior context above an overlap threshold receive a non-linear stickiness score boost. the proxy maintains no local error retries or provider exclusion feedback loops; it appends the ranked candidate list into the outbound provider.only payload field and relies natively on openrouter to handle fallback execution within. successful routes to the provider that got hit are mapped for cache stickiness. when the cache ttl exceeds, sticky boost is dropped.