Intersections - Mind, Technology, and Society

This post about an exciting experiment I worked on last year: "House of Wisdom" --an AI-driven application designed for contextually rich conversations grounded in books, research reports, and laws, with answers accompanied by Mermaid diagrams to spark insights with cognitive ease.

https://github.com/shuruheel/house-of-wisdom

Genesis

Once you have the full text data for a given set of books and reports, it's common to interface with it using some type of a chunking strategy. What this means is that you put all of the text in your knowledge base, so when you ask a question, the system finds chunks most relevant to your question, and includes those chunks in the context passed to the model.

However, knowledge is complicated. It is not organized in chunks. It has other Platonic forms. So I had an idea: could I define some of these Platonic forms of knowledge, transform the raw text data into these forms, populate a graph database based on these forms, and retrieve relevant node networks rather than text chunks to improve reasoning performance.

Forms of Data

For books, I defined a graph schema that included nodes for Stories, Events, Entities, Concepts (with relationships), and Claims, with relevant metadata for each node.

Then I created a data processing pipeline that:

Processed the full text data chunk by chunk
Transformed each chunk into a JSON object based on the graph schema using an LLM
Populated a Neo4j graph database with nodes and relationships from the transformed JSON files
Added embeddings to key nodes for efficient retrieval To be clear, when I say data processing pipeline, I mean a messy folder with a lot of Python files But I did make use of batch processing above.

I repeated this process for research reports and laws, with their own graph schemas, making use of batch processing.

I was initially using the Neo4j Aura DB (cloud service) but soon realized that it would get very expensive, so I secured a license for startups, set up an EC2 instance, and installed and configured Neo4j on it.

Frontend Architecture

Next.js 13 & App Router

Using the newer App Router organizes the API endpoints in a more intuitive folder structure.
This keeps services modular and easy to maintain. Async Chain-of-Thought Processing
Multiple chain-of-thought questions are processed concurrently, improving response time Mermaid Diagram Rendering
Some complexities just need a visual. To illustrate a flow or depict reasoning, the application generates chain-of-thought in a Mermaid syntax.
Through a custom React component, those diagrams are rendered, letting users see an actual "map" of reasoning. Neo4j Graph Database
Graphs capture context better than typical relational data for certain problem domains. Concluding Thoughts

My biggest takeaway was that with the help of generative models, it has become so much easier to write the code for such applications, which adds a premium on creative and cross-disciplinary thinking.

My second biggest takeaway was that this could be adapted across domains, albeit with some domain-specific graph schema definition. In particular, this approach would work well for health, law, geopolitics, and diplomacy.

Finally, I realized that some improvements could be made to the initial graph database schema that I had set for the extraction of knowledge from books. These weren't quite the perfect Platonic forms of knowledge, but a close enough approximation.

Please leave your thoughts in comments and I will respond to you.