Build a self-updating RAG pipeline: step-by-step checklist
This checklist is based on the RAG 101: Stop using “chatbots” and start building engines featuring Maddy Osman's workflow, published on the AI Lab by ActiveCampaign.
Get the checklist
By the end of this checklist, you’ll have a self-updating retrieval-augmented generation (RAG) pipeline that turns your content library into a searchable knowledge base your AI agents can draw from. It’s based on the workflow Maddy, who runs the content agency The Blogsmith, built in n8n after outgrowing an earlier version she ran in Zapier. [Total build time isn’t specified in the source post—budget a few focused sessions to configure both halves of the workflow below.]
Before you start
- A content library to draw from: blog posts, podcast transcripts, or other written assets, stored somewhere a workflow tool can watch (the source used a Google Drive folder)
- A workflow automation account: the source used n8n after starting on Zapier
- A Pinecone account with a vector index created: Pinecone is the vector database used to store and search embeddings
- An OpenAI API key: used to generate the embeddings for each content chunk
- A Google Sheet: used to track chunks and their metadata between the two halves of the pipeline
The workflow
Phase 1: Capture new content automatically
After this phase, you’ll have: a workflow that picks up new content the moment it’s published, with no manual upload step.
- Connect a Google Drive folder as your trigger source: set up a Google Drive trigger that fires when a new file lands in a watched folder
- Test the trigger with a sample file: add a test document (e.g., a podcast transcript) and confirm the workflow picks it up automatically
Phase 2: Chunk content for retrieval
After this phase, you’ll have: a custom code step that reliably splits any new document into retrievable chunks.
- Add a download-and-extract step: pull the file from Drive and extract its raw text
- Build a custom code node to chunk the text: split the document into compact sections (the source used roughly 600 characters per chunk) with slight overlap between them, so ideas don’t get cut off at the boundaries
- Match your chunking approach to your content type: what works for podcast transcripts may not work for documentation, research, or support content—decide this upfront rather than reworking it later
“I’m not a developer; I asked ChatGPT to help me write the code.”
Phase 3: Track metadata and organize chunks
After this phase, you’ll have: every chunk logged in a spreadsheet with the metadata needed to trace it back to its source.
- Check for a source URL on each document: if one isn’t already attached, add a step that searches for the corresponding page (e.g., the podcast episode page) using the document title, then retrieves the URL
- Attach consistent metadata to each chunk: source URL, publish date, and content type at minimum
- Merge and append chunks to a Google Sheet: each row should represent one chunk plus its metadata, ready for embedding
Phase 4: Embed, store, and keep the system clean
After this phase, you’ll have: a live vector database that stays current and doesn’t reprocess the same content twice.
- Build a second, manually-triggered workflow: pull new rows from the Google Sheet and download the latest version of each document
- Process chunks in parallel: use a map-and-fan-out step so multiple chunks embed at once instead of one at a time
- Send each chunk to OpenAI for embedding, then store it in Pinecone: use a Default Data Loader to convert the Sheet rows into the structured metadata and vector records Pinecone needs
- Add a duplicate check: mark each embedded chunk as processed in the Google Sheet so it’s never embedded twice, which also keeps OpenAI API costs down
Quick reference
- Total time: Not specified in the source post—treat this as a multi-session build, not a single sitting
- Tools needed: n8n (or a comparable automation tool), Google Drive, Google Sheets, Pinecone, OpenAI, ChatGPT (optional, for writing the chunking code)
- Key output: A self-updating knowledge base that AI agents can query for answers grounded in your own published work
Related
More data from the AI Lab.