Turn Your Website Into a Knowledge Base

Guide

1. 1

   What this does

   You paste your website address. FlowCastle finds the pages, reads them, strips the menus and cookie banners, and saves each page as a document in a knowledge base. Attach that base to an AI Chat Agent block and the bot answers from your real pages — your prices, your services, your delivery rules.

   It is the fastest way to fill a knowledge base, and usually the most accurate one, because the text is already the text you show customers.

2. 2

   Start an import

   Open Settings → AI Settings and scroll to Knowledge bases. Press Import from website. You will see it in two places: at the top of the section (imports into a new base) and on a base you already selected (adds pages to that one).

   The form asks for five things:

   • Website address — https://example.com. A deeper address imports only that section: give it https://example.com/docs and nothing outside /docs is touched.
   • Where to put the pages — add them to the base you selected, or create a new one.
   • Knowledge base name — leave it empty and the base is named after the domain.
   • Include blog and articles — on by default. Blog posts are what answer "how do I…" questions. Turn it off if you want a smaller base with only the commercial pages.
   • Page limit — 200 by default, 500 at most. If the site has more eligible pages than the limit, the important ones are kept: home, pricing, about, contacts, services, FAQ, then features and docs, then everything else.

   Press Find pages.

3. 3

   It asks before it reads

   Discovery runs first and then stops. The card shows Found 137 pages on example.com. Import them? with a sample of the addresses it plans to read.

   This step exists so you can catch a wrong domain, a staging site or a shop with ten thousand product pages before anything is fetched. Press Import 137 pages to continue, or Cancel import.

   How the pages are found, in order:

   1. robots.txt — read first, both for the rules and for the sitemap links.
   2. The sitemaps it points at, including sitemap indexes.
   3. If there is no sitemap at all: the links on the home page, one level deep. The card warns you when this happens, because the result is usually thinner.

   Pages that robots.txt disallows are never fetched, and the card tells you how many were skipped for that reason. The crawler identifies itself as FlowCastleKBBot/1.0.

4. 4

   While it runs

   The card shows the stage — Finding pages, Reading pages, Saving pages, Indexing, then Ready — and a counter like 42 of 137 pages read.

   You can leave the page. The import keeps running on the server, and the card picks it back up when you return.

5. 5

   What gets skipped, and why

   Not every page belongs in a knowledge base. The importer drops the ones that would only add noise:

   What you see	What happened
   Too little text	Under 60 words after the menus and footer were removed. Usually a gallery, a redirect page or an empty category.
   Duplicate of another page	The same text already came in from another address.
   Rendered by JavaScript	The page arrives empty and fills itself in the browser. There is nothing to read in the HTML.
   Over the page limit	Eligible, but the page limit was already spent.
   Could not be fetched	Timed out, returned an error, was not an HTML page, or was too big (2 MB per page).

   Cart, checkout, login, search, tag and author pages are filtered out before fetching, along with images and files.

   When the import finishes, Pages that were not imported lists every one of them with the reason. If that list is long and full of Rendered by JavaScript, your site is a single-page app — the importer says so directly, and the answer is to paste the key text in as a document instead.

6. 6

   Contact details are collected once

   Email addresses and phone numbers usually sit in the footer of every page. Importing them 600 times would make the bot answer "here are our contacts" to almost anything.

   So they are collected across the whole site and written into one document: Contact details for example.com. It is a normal document — open it, edit it, delete a number that should not be public.

7. 7

   Attach it to the agent

   The base is not connected to anything by itself. Open your flow, click the AI Chat Agent block, go to Settings → Knowledge, and pick the base under Knowledge bases.

   Wait for indexing to finish before you test. A document is only searched once it reaches Ready — the bot cannot answer from a page it has not indexed yet, and a test run too early looks like the import failed.

   The rest of the setup — system prompt, model, intents, what happens when nothing matches — is in The AI Chat Agent Node.

8. 8

   Keep it current with Re-crawl

   Websites change. Press Re-crawl on the base and the whole thing runs again with the same settings.

   It is not a re-import from scratch:

   • Pages that did not change are left alone and not re-indexed.
   • Pages that changed are updated in place.
   • Pages that no longer exist on the site are deleted from the base.

   So a re-crawl of a mostly-static site is cheap and finishes quickly. Documents you wrote by hand are never touched — only pages that came from that website.

9. 9

   Ask the AI builder instead

   You can also just say it in the AI chat: "make an assistant for example.com". The builder starts the same import, tells you what it is reading, and attaches the finished base to the agent it builds. It runs in the background, so you can keep chatting while it works.

   Agents connected over MCP have the same tool: import_website_into_knowledge_base starts the crawl and returns a job id, and get_website_import reports progress.

10. 10

    Limits worth knowing

    • 500 pages per import, 200 by default.
    • 2 MB per page, 20 MB per import in total.
    • 20 seconds per request, three pages at a time.
    • http and https only. Private and internal addresses are refused.
    • If the site asks for a crawl delay longer than 2 seconds, 2 seconds is used and you are told.
11. 11

    When a website is the wrong source

    Import the site when the pages already say what you want the bot to say. Write documents by hand when they do not.

    Your public pages are written to sell. A support bot often needs the boring version — the exact return window, the exact delivery times, the sentence you are comfortable sending to a customer at 2am. Import the site first, then add those few documents by hand. The two live side by side in the same base.

12. 12

    Next up

    One bot, several languages — including the pages you just imported: Multilingual Bots: One Bot, Many Languages.