Automation agent workflow for business research needs.

PROBLEM

My challenge is figuring out how to enable an LLM to understand my research needs, particularly within the context of business requirements research.

HYPOTHESIS

When conducting research, we humans require an understanding of the business domain and the necessary tools such as business documentation or prior work experience (for foundational knowledge) and web browsers (as the operational tool). Yet, it goes beyond that; we must be able to interact with website elements, take research notes, and synthesize the findings. The same applies to LLMs; my reasoning leads to the conclusion that AI requires tools, business documents, and an iterative process regarding the specific business being researched.

EXPERIMENT

I provide comprehensive business documentation known as RAG in the world of LLMs along with the right tools. I then build an integrated system that combines RAG, external MCP tools (for browsing and data fetching/web scraping used by the LLMs), and a notes function.

FAILURE AND NEW ITERATION

Based on the tests I conducted, it turns out that RAG, external MCP tools, and Notes alone are insufficient. Midway through the research process, the model loses its ability to maintain short-term context memory because the prompts being processed are extensive and numerous. Therefore, the entire interconnected system requires optimization: RAG, MCP tools, KV cache management, layer placement, and finally deterministic instructions (system prompts) that the model can easily grasp without stifling its creativity. "An additional step can be taken if necessary (​​domain-specific fine-tuning using a private dataset rather than a public one) If this is done, you will obtain output that precisely matches your requirements."

"An additional step can be taken if necessary (​​domain-specific fine-tuning using a private dataset rather than a public one) If this is done, you will obtain output that precisely matches your requirements."

ITERATION

When all these elements come together as a unified whole, the results are highly effective, even if the process takes a bit longer. However, this is a reasonable trade-off, considering that the work is being performed by local LLMs on consumer-grade hardware rather than by massive frontier models with unlimited resources.

EVIDENCE

You can review the screenshots I provided earlier. They allow you to assess various aspects such as token usage, the number of tool calls made per request, and the number of URLs or websites accessed before the final output is generated. I have also included charts illustrating how the overall system configuration optimizes results; for instance, comparing the model's coherence when using only RAG and tools versus the performance gains achieved by adding KV cache management optimization, and so on.

WORKFLOW COMPLEXITY

33 tool calling

TOOL SUCCESS RATE

~99%

TOTAL RESEARCH TIME

~51 Minutes On 1x Turn

SELF-CORRECTION

Active

RESULT

Only after everything is optimized as a unified whole can truly optimal results be realized. The output is spot on adhering to instructions and incorporating self correction, all achieved with just a single request.

WHAT I LEARNED

To achieve optimal results, I shouldn't keep thinking that "this model lacks sophistication" or "it turns out the model isn't that good." That mindset is actually flawed because it relies on a perspective limited to a single environment. In contrast, when looking at current AI applications such as ChatGPT, Claude, Gemini, Qwen Studio, and others engineers don't focus solely on the "model" itself, they consider the entire operational infrastructure, recognizing that the model is just one of many components requiring optimization.