Prompt context window limits hinder migration of large prompts to self‑hosted LLMs
问题概述
Users moving large prompts (e.g., 35KB) from hosted models with million‑token windows to self‑hosted models hit strict context window caps (often ~65K tokens), causing crashes and loss of functionality. This forces them to split prompts and manage multiple sessions, adding complexity and reducing productivity.
痛点标签
- Context window size limits in self‑hosted models
- Need to split large prompts across multiple inference sessions
- Frustration with Ollama's suitability for large prompts
待验证问题
- Do you currently split large prompts manually or with scripts?
- What is the typical token length of the prompts you are trying to run?
- Which self‑hosted inference stack (e.g., llama.cpp, Ollama, vLLM) are you using?
证据
原话
Local models have a smaller context window, so your 35kB prompts that worked fine against a hosted 1 Million token window, crash out when you only have a 65K (!) token window locally.
原话
you need to cut your prompt up. If you believe LLMs work, have the LLM help you shape the overall plan, and then have multiple sessions run each step in the plan without being bloated with the context of previous successful steps.
事实
Most people and companies run their inference engine, regardless of the model, you run out of useful context that the model can accurately attend to around the 250k mark no matter how much they advertise their context size is.
免责声明：本卡片由 AI 从社区讨论中提炼，属于「痛点证据」，是需求分析与验证的起点，不等同于「已验证的市场需求」。AI 存在分析局限，关键决策前请打开上方原始讨论人工复核。