What I have now heard from a couple of recent clients, is the importance of actually tracking progress in AI search. As with any parts of marketing you want to creating great messaging material, content and so on; but you also need to know whether that effort is actually working or not.

I use a simple spreadsheet for doing this, attached below:

To use this spreadsheet, just manually work through each of the columns below, for each of the queries where you think you should appear *:

Query
→ Type the exact search query or question you used (e.g., “What is Generative Engine Optimization?”).

Date
→ Enter the date you performed the search (format: DD/MM/YYYY).

Platform (Google / Perplexity / ChatGPT etc.)
→ Note which platform or search engine you used (e.g., Google, Perplexity, ChatGPT).

Mode (Normal / Incognito / VPN)
→ Indicate how you accessed the platform (e.g., Incognito, Signed-in, VPN).

Category (SEO / AEO / GEO)
→ Choose which category the result fits into:

• SEO = Traditional search results
• AEO = AI or answer-engine snippet
• GEO = Full generative or LLM-based output

Result Source (Regular / AI Overview / AI Mode)
→ Identify where the result appeared:

• Regular Search = Standard search results
• AI Overview = Google’s AI summary panel
• AI Mode = Full generative output replacing the usual page

Result Type (Direct Link / Snippet / Generative Synthesis)
→ Note how your content appeared:

• Direct Link = SEO link in results
• Snippet = Text summary or mention
• Generative Synthesis = LLM or AI-generated paragraph referencing your content

Did you appear? (Yes / No)
→ Simply record whether your site, name, or content appeared.

How did it appear? (Quoted / Mentioned in Narrative)
→ Describe how you appeared:

• Quoted = Direct quotation or citation
• Mentioned in Narrative = Indirect or paraphrased mention

Screenshot File
→ Add the filename of your saved screenshot (e.g., Screenshots/7.png).

* Or, for help getting this set up for yourself, give me a shout on Contact Us to find out more!

---

When building search tools, intelligent assistants, or AI-driven Q&A systems, one of the most foundational decisions you’ll make is how to retrieve relevant content. Most systems historically use keyword-based search—great for basic use cases, but easily confused by natural language or synonyms.

That’s where embedding-based retrieval comes in.

In this guide, I’ll break down:

• The difference between keyword and embedding-based retrieval
• Real-world pros and cons
• A step-by-step implementation using OpenAI and Pinecone
• An alternative local setup using Chroma

---

Keyword Search vs. Embedding Search

Keyword-Based Retrieval

How it works:
Searches for exact matches between your query and stored content. Works best when both use the same words.

Example:
Query: "What is vector search?"
Returns docs with the exact phrase "vector search".

Pros:

• Very fast and low-resource
• Easy to explain why a match was returned
• Great for structured and exact-match data

Cons:

• Doesn’t understand synonyms or phrasing differences
• Fails if the words aren’t an exact match

Embedding-Based Retrieval (Semantic Search)

How it works:
Both queries and documents are converted into dense vectors using machine learning models (like OpenAI’s text-embedding-ada-002). The system compares their semantic similarity, not just their words.

Example:
Query: "How does semantic search work?"
Returns docs about “meaning-based search” even if the words are different.

Pros:

• Understands intent, not just keywords
• Great for unstructured content and natural queries
• Can surface more relevant results even if phrasing is varied

Cons:

• More computationally intensive
• Results are harder to explain (based on vector math)
• Requires pre-trained models and a vector database

---

Feature Comparison Table

Feature	Keyword-Based Retrieval	Embedding-Based Retrieval
Search Logic	Matches words exactly	Matches by meaning
Flexibility	Low	High
Speed	Fast	Slower
Resource Use	Low	Higher
Explainability	High	Low
Best For	Structured search	Chatbots, recommendation, unstructured data
Common Tools	Elasticsearch, Solr	Pinecone, Chroma, FAISS

---

Setting Up Embedding-Based Retrieval

Let’s build a basic semantic search system using:

• OpenAI (text-embedding-ada-002)
• Pinecone (hosted vector DB)
• Chroma (optional local alternative)

1. Choose Your Tools

Embedding model:
OpenAI’s text-embedding-ada-002 or a local Hugging Face model.

Vector database:
Cloud: Pinecone (scalable, managed)
Local: Chroma (open-source, lightweight)

2. Install Required Libraries

pip install openai pinecone-client chromadb

3. Set API Keys

export OPENAI_API_KEY="your-openai-key"
export PINECONE_API_KEY="your-pinecone-key"

In Python:

import openai
openai.api_key = "your-openai-key"

4. Generate Embeddings

def get_embedding(text):
    response = openai.Embedding.create(
        input=text,
        model="text-embedding-ada-002"
    )
    return response['data'][0]['embedding']

documents = [
    {"id": "1", "text": "This is an introduction to embedding-based search."},
    {"id": "2", "text": "Embedding-based retrieval finds similar meanings."},
]

for doc in documents:
    doc['embedding'] = get_embedding(doc["text"])

5. Store in Pinecone

import pinecone

pinecone.init(api_key="your-pinecone-key", environment="us-east-1")

index_name = "embeddings-index"
pinecone.create_index(index_name, dimension=1536)

index = pinecone.Index(index_name)

to_upsert = [(doc['id'], doc['embedding'], {"text": doc["text"]}) for doc in documents]
index.upsert(vectors=to_upsert)

6. Perform a Semantic Search

query = "How does semantic search work?"
query_embedding = get_embedding(query)

results = index.query(query_embedding, top_k=5, include_metadata=True)

for match in results["matches"]:
    print(f"ID: {match['id']} | Score: {match['score']}")
    print(f"Text: {match['metadata']['text']}\n")

Optional: Use Chroma for Local Embedding Search

import chromadb

client = chromadb.Client()
collection = client.create_collection("documents")

for doc in documents:
    collection.add(
        documents=[doc["text"]],
        embeddings=[doc["embedding"]],
        ids=[doc["id"]]
    )

query_result = collection.query(query_texts=["How does embedding retrieval work?"], n_results=5)
print(query_result)

---

Evaluate the Results

Once you’re set up:

• Check result relevance
• Tune your top_k or switch models if needed
• Add keyword filtering for hybrid search

You now have a foundation for building:

• Intelligent assistants
• Internal knowledge base search
• Chatbots that retrieve based on meaning

---

What’s Next?

You can scale this up to thousands or millions of documents. Consider:

• Crawling blogs, docs, or Notion pages
• Combining embeddings with filters or metadata
• Using hybrid keyword + embedding pipelines for speed and precision

---

This is the part where all the content sources came together into a centralized system I could actually interact with.

This post is a cleaned-up record of what I built, what worked, what didn’t, and what I planned next. If you’ve ever tried to unify fragmented notes, decks, blogs, and structured documents into a searchable system, this might resonate 🙂

---

What I Built

There were two main components at the heart of the system:

1. Batch Processing Script
   PopulateChatSystemDataRepository.py — this was run manually to gather and format all source data into a single repository. My plan was to automate it later.
2. Continuous Scanner
   A lightweight background service monitored for new blog posts and updates.

At that point, the batch script did the heavy lifting, though I intended to shift it onto Google Cloud Run to handle scale.

---

Where the Data Lived

The sources I processed included:

• PowerPoint files
  These were manually selected and hardcoded into the script — a reasonable tradeoff given how few I needed to track.
• RSS Feeds
  • My blog at bjrees.com
  • A few curated industry insight feeds
• OneNote Notebooks, such as:
  • Project documentation (e.g. Skynet, The Oracle)
  • Notes from a Cambridge Judge Business School programme
  • Third-party and personal research logs
• iCloud Backups
  These contained archived slide decks and supporting materials.

All of this data was funneled into a staging area for eventual vector embedding and retrieval.

---

Microsoft Graph API + OneNote

To pull content from OneNote, I used the Microsoft Graph API. First, I installed the required libraries:

pip install msal requests

• msal handled authentication via Azure Active Directory
• requests allowed me to interact with the Graph API endpoints

Once I authenticated, I could enumerate and query notebooks like this:

python ExtractNotes.py

After logging in via a Microsoft-generated URL, I could successfully extract content from all the notebooks I needed.

---

Licensing Curveballs

At the time, I hit a snag: my Microsoft 365 Family plan didn’t include SharePoint Online, which was required to query OneNote via the Graph API.

I weighed my options:

1. Pay for a Business Standard plan (~£9.40/month)
2. Try and use my home license in some way, even thought it didn’t seem to have what I needed for OneNote

I went with option 2, supported by a one-month free trial of Microsoft Business Basic to help validate the approach.

---

Google Sheets as the Backbone

The ingestion script used a JSON keyfile to interact with Google Sheets. It opened the sheet like this:

client.open_by_key(sheet_id).sheet1

Sheets acted as a live database — but I ran into 429 rate-limit errors, especially when repeatedly reading the same files. To solve this, I built a basic checkpointing system so the script would:

• Cache previously processed records
• Avoid re-downloading the same content every time
• Track progress and only fetch new entries on each run

---

The GitHub Reset

After a short break from the project, I realized the codebase had grown too complex. I had introduced a lot of logic to deal with throttling and retries, but it made everything harder to understand.

So I rolled back to a much earlier commit and started again from a simpler foundation.

It was the right move.

---

What Came Next

Here’s what I tackled after that cleanup:

• Migrated the whole project to an old home laptop
• Simplified the ingestion pipeline
• Ensured each run processed only new data, not the full archive
• Finalized access and querying via Microsoft Graph API for OneNote and SharePoint content

---

Reflections

Skynet began as a chatbot experiment, but evolved into something bigger — a contextual knowledge system that drew from years of notes, presentations, and personal writing.

Stage 2 was about turning chaos into structure. The next phase was even more exciting: embeddings, retrieval, and building a system that could answer real questions, grounded in my own work.

---

Read Stage 1 if you missed the start.

---