Super AI Engineer Season 6, Level 1 Hackathon
Answer 100 Thai multiple-choice questions about an online electronics shop, using only its knowledge base of 118 Markdown documents (857 chunks) and a Thai LLM through the ThaiLLM API. Each question has 10 choices; choice 9 means "not in the knowledge base" and choice 10 "unrelated to the shop", so the system also has to recognize when no answer exists.
Questions differ in kind — a single spec lookup, a comparison of products, a price or points calculation that needs a policy file, a long story with the real question buried at the end — so one retriever is not enough.
The provided baseline used a weaker multilingual embedding model, naive fixed-length chunking, and no reranking. Swapped in BAAI/bge-m3 — a stronger multilingual embedding model with solid Thai support — and fused BM25 keyword retrieval (tokenized with PyThaiNLP) with dense retrieval using Reciprocal Rank Fusion, so exact matches (like product codes) and semantic matches both surface reliably.
Four helper functions (LLM call, BM25 and dense retrieval, answer parsing) started from the starter kit; the intent classifier, query decomposition, forced retrieval, prompt modes, run loop and debug helpers were written from scratch.
Fusion weights depend on the intent, so product-code lookups lean on BM25 and paraphrased questions on dense search. Every product named in a question is force-retrieved through a reverse index, which is what makes comparison questions answerable. Prompts are engineered to make the model answer "not in knowledge base" or "not related" correctly instead of guessing — the failure mode that hurts naive RAG systems most on edge-case questions.
Wrote a self-authored analysis document laying out the diagnosis and the fix, citing external research (including ACL 2024 work on RAG best practices) to justify each choice rather than tuning by trial and error alone.