Getting topic based data from Pinecone which contains multiple pdf files

Question

I am trying to build a chatbot where I need the bot to response to user query from the data that is stored in vector database like pinecone. Here I want to store pdf of different topic, like different car model of a particular brand. How can I get proper result from pinecone if user ask a particular question about a model?

Loaded the multiple pdf using SimpleDirectoryLoader to store in pinecone and used the query engine to get the similar vector data as response.

from llama_index.core import VectorStoreIndex,SimpleDirectoryReader, StorageContext
from llama_index.vector_stores.pinecone import PineconeVectorStore

documents=SimpleDirectoryReader("direcory_name").load_data()

vector_store = PineconeVectorStore(pinecone_index=pinecone_index)
storage_context = StorageContext.from_defaults(vector_store=vector_store)
index = VectorStoreIndex.from_documents(documents=documents, storage_context=storage_context)

from llama_index.core.retrievers import VectorIndexRetriever
from llama_index.core.query_engine import RetrieverQueryEngine
from llama_index.core.indices.postprocessor import SimilarityPostprocessor

retriever=VectorIndexRetriever(index=index,similarity_top_k=4)
postprocessor=SimilarityPostprocessor(similarity_cutoff=0.80)

query_engine=RetrieverQueryEngine(retriever=retriever,
                              node_postprocessors=[postprocessor])

Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it's hard to tell exactly what you're asking. — Community
– Community Bot, Commented Apr 10, 2024 at 6:41

devvie · Accepted Answer · 2024-04-18 20:19:17Z

0

This is not a full proof solution, these are totally based on knowledge I have on llamaindex.

You can try adding data in form of chunks by chunks i mean meaningful chunking, you can explore chunking strategies.
You can try adding metadata along with the data you are upserting to the pinecone vector data base. This can be used to add metadata filters while extracting chunks, when feeding it to a query retriever.
Experiment variation in chunksizes, try maybe different similarity techniques, Model paramters, relationship between nodes, type of retrievers etc

These are just the foundations or base of developing your chat bot to work according to what you expect. You can explore more from documentation and the sample example notebooks from llama_index. They have quite a plenty of notebooks on github.

answered Apr 18, 2024 at 20:19

devvie

485 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Getting topic based data from Pinecone which contains multiple pdf files

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related