Agents suck – but it can be better
Watching Agents try to complete tasks is like watching a drunk person trying to solve a Rubik's cube. Entertaining? Yes. Reliable? Not always!
However. Instead of asking an LLM to "do the whole thing" (which is indeed prone to inconsistency) and thus letting LLMs run wild like unsupervised toddlers maintaining the "do not press" buttons at a nuclear facility. A FLAT approach puts control and predictability to LLM interactions by treating them more like traditional programming constructs but enhanced with LLM's natural language understanding. like:
- Binary decisions (gates) - Limited classification (using match/case) - Structured data extraction (using Pydantic models) - well typed function calling
Anywho, Would love to hear your thoughts on an experiment F.L.A.T (Frameworkless LLM Agent... Thing) https://github.com/mindsdb/flat-ai
Showcasing that it is possible to leverage the power of LLMs in Agents though absolute simplicity:
I work in a company that sells chatbots. We use LLMs to understand the intention of the user: start a flow, set a slot value, cancel a current flow...
After that, all the logic to handle the identified action and the response is implemented using normal backend code.
We sometimes use an LLM for the response when the user asks for a FAQ or maybe for rephrasing an output sentence.
How is this different from just using smaller prompts?
i.e One could just use their own prompt for the binary classifier right?
great question, i think rather than smaller inputs, you should be able to have as much context as needed in. Its more aboue smaller outputs, that are well typed and allow you to control workflow via standard coding mechanics. Therefore getting the best of both worlds, llms power of understanding unstructured info (which python sucks at) and being able to control things which python is great at.