We built a sales proposal bot on gpt-35-turbo. The model took a week. Everything else took two months.
ChatGPT's model landed in Azure in March. Building a proposal-answering bot on it taught me that the LLM is the smallest component in an enterprise LLM system.
🔧 We built a sales proposal bot on gpt-35-turbo. The model took a week. Everything else took two months.
When the ChatGPT model (gpt-35-turbo) arrived in Azure OpenAI Service in March, my team started on something our sales colleagues had wanted for years: a bot that answers questions from our proposal and RFP history — what have we said before about X, which past responses cover Y — instead of someone spelunking through folders at midnight before a submission.
It works, people use it, and almost nothing that made it hard was the model. Notes from the trenches:
1. The real engineering was in Microsoft Graph, not the LLM. The knowledge lives in SharePoint and Teams, which means the Graph API — and that is where the weeks went. Application permissions versus delegated permissions is not a footnote; it is the security architecture. Build it lazily with app-level access and your bot will happily quote documents to people who were never allowed to open them. We used delegated access so every retrieval runs as the asking user, and the platform's own permissions do the trimming. Add throttling, pagination, token refresh and files saved in every format known to mankind — that was the project.
2. A 4K context window is a forcing function. You cannot paste proposals into 4,000 tokens; a single RFP response is ten times that. So the architecture became: chunk the documents, embed them, retrieve only the relevant pieces per question, and let the model answer from those pieces alone. What felt like a workaround for a small context window turned out to be the correct design anyway — it is what makes answers traceable.
3. "Answer only from the provided sources, and cite them" is the whole trust story. Unconstrained, the model produces beautiful, confident, occasionally fictional answers — lethal in a domain where someone will paste the output into a client document. Constrained to retrieved chunks with citations, wrong answers become checkable answers. Every response links to the source proposal. No citation, no answer.
4. The corpus was the surprise villain. Fifteen years of proposals contain contradictions — different pricing language, superseded offerings, three versions of the same case study. The bot did not create that problem; it exposed it, at query speed. We ended up curating which documents count as truth before quality was acceptable. Garbage in was always garbage out; LLMs just remove the delay.
5. Salespeople ask questions like salespeople. Not "retrieve prior responses regarding data-migration methodology" but "what do we usually say when the client pushes back on timelines?" Testing with real users, real phrasing, real deadline pressure changed the retrieval tuning more than any benchmark.
The pattern I suspect generalises: in enterprise LLM systems, the model is the smallest component. Permissions, retrieval, content hygiene and trust design are the product. The demo takes a week; the system takes a quarter — and the quarter is where the value is.
For anyone considering something similar, start with one question: where does the knowledge actually live, and who is allowed to see which parts of it? Your answer is your architecture. The model comes last.
Reading: https://azure.microsoft.com/en-us/blog/chatgpt-is-now-available-in-azure-openai-service/