Building Twigmate: A Practical Case Study of an LLM+RAG-Based Onboarding Chatbot at Twigfarm

Background

At Twigfarm, we recognized the challenge new employees face when adjusting to a new company. From learning about organizational culture and tools to understanding policies and workflows, there’s a lot to absorb in a short time. Traditional onboarding often lacks consistency and scalability.

To address this, we developed Twigmate — an AI-powered onboarding chatbot designed specifically to support new hires with immediate, accurate, and contextual responses using internal documents and company knowledge.

This project branched off from our existing Twigfarm Copilot, focusing on narrowing the scope to deliver more relevant answers tailored to onboarding needs.

‍

Tech Stack Overview

We designed Twigmate with a practical, serverless architecture to optimize both performance and maintainability:

• Large Language Model (LLM): GPT-4o
  • Chosen for its strong performance-to-cost ratio and adaptability
• RAG (Retrieval-Augmented Generation) for grounding answers in company documentation
• Infrastructure: AWS (us-west-2)
  • S3 (static web hosting and file storage)
  • Lambda + API Gateway (backend endpoints)
  • Bedrock (LLM access)
• Frontend: React (with Tailwind CSS planned for UI consistency)
• Vector Search: FAISS-based retrieval for document chunking and indexing

Employees can access Twigmate through a hosted URL and even view and manage source files via a keyboard shortcut (Ctrl + Shift + .).

‍

Problem Definition

The original Copilot chatbot had a broad knowledge scope, which occasionally led to irrelevant or overly generic answers for onboarding-specific questions.

This created confusion for new employees asking simple, repetitive questions such as:

“Where can I find information about vacation policies?”
“How do I join a Slack channel?”
“What’s the process for expense reimbursement?”

Without precise filtering, responses lacked clarity and sometimes delivered outdated or unrelated information.

‍

‍

Solution Process

1. Narrowing the Retrieval Scope

We limited document retrieval to a curated set of onboarding-related PDFs:

• Employee handbooks
• HR policies
• Tool setup guides
• Internal FAQs

This reduced hallucinations and significantly improved answer relevance.

2. LLM Benchmark Testing

We tested eight LLMs using 13 real onboarding queries. Each was evaluated on:

• Answer accuracy
• Prompt compliance
• Language fluency
• Information clarity
• Hallucination control

‍

‍

Result: GPT-4o was selected as the base model for production due to its performance consistency and manageable cost.

3. Question Set Curation

We compiled a list of common questions from actual new hires and ran multiple prompt tuning iterations for each to ensure the chatbot:

• Provided clear, context-specific answers
• Followed a consistent format
• Handled unknowns gracefully (“Sorry, I couldn’t find that info.”)

‍

‍

Outcomes

• Increased Accuracy: Answer precision rose to over 90% in most onboarding scenarios.
• Faster Response: Average response time dropped from 5–10 minutes (human reply) to under 3 seconds (chatbot).
• User Satisfaction: Internal feedback survey showed a satisfaction rating of 4.6/5.0 among new hires.
• Cost Control: Average per-response cost for GPT-4o was under ₩30, making it affordable for daily use.

‍

‍

Lessons Learned & Next Steps

Lessons

• A smaller, high-quality domain often works better than a general-purpose knowledge base.
• RAG reduces hallucinations significantly — but only if the source docs are well structured.
• Prompt engineering matters as much as the model itself.
• LLM usage costs can be predictable with careful input/output design.

Next Steps

• Integrate Slack, Notion, and Jira to answer tool-specific questions dynamically.
• Implement role-based access control (e.g., restrict HR data to admins).
• Add session memory for follow-up questions and conversational continuity.

‍

‍

🔗 References

• Twigmate Access URL
• AWS Bedrock Documentation
• Meta AI – LlamaIndex: A Retrieval-Augmented Architecture
• Vector Database: FAISS by Facebook

This project showcases a real-world application of LLM+RAG for internal knowledge access. It demonstrates how companies can move beyond general-purpose chatbots and instead build domain-specific, high-utility assistants that serve real organizational needs. If you’re considering building one for your team, starting with onboarding is a great first step.

‍

‍