{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "
\n", "

\n", " \"arize\n", "
\n", " Docs\n", " |\n", " GitHub\n", " |\n", " Community\n", "

\n", "
\n", "

Evaluating and Improving a LlamaIndex Search and Retrieval Application

\n", "\n", "Imagine you're an engineer at Arize AI and you've built and deployed a documentation question-answering service using LlamaIndex. Users send questions about Arize's core product via a chat interface, and your service retrieves documents from your documentation in order to generate a response to the user. As the engineer in charge of evaluating and maintaining this system, you want to evaluate the quality of the responses from your service.\n", "\n", "Phoenix helps you:\n", "- identify gaps in your documentation\n", "- detect queries for which the LLM gave bad responses\n", "- detect failures to retrieve relevant context\n", "\n", "In this tutorial, you will:\n", "\n", "- Download an pre-indexed knowledge base of the Arize documentation and run a LlamaIndex application\n", "- Visualize user queries and knowledge base documents to identify areas of user interest not answered by your documentation\n", "- Find clusters of responses with negative user feedback\n", "- Identify failed retrievals using cosine similarity, Euclidean distance, and LLM-assisted ranking metrics\n", "\n", "Parts of this notebook require an [OpenAI API key](https://platform.openai.com/account/api-keys) to run. If you don't have an OpenAI key, you can still run Phoenix by skipping cells preceded by the 💭 emoji.\n", "\n", "\n", "## Chatbot Architecture\n", "\n", "Your chatbot was built using LlamaIndex's low-level API. The architecture of your chatbot is shown below and can be explained in five steps.\n", "\n", "![llama-index chatbot architecture](http://storage.googleapis.com/arize-phoenix-assets/assets/docs/notebooks/llama-index-knowledge-base-tutorial/llama_index_chatbot_architecture.png)\n", "\n", "1. The user sends a query about Arize to your service.\n", "1. `langchain.embeddings.OpenAIEmbeddings` makes a request to the OpenAI embeddings API to embed the user query using the `text-embedding-ada-002` model.\n", "1. `llama_index.retrievers.RetrieverQueryEngine` does a similarity search against the entries of your index knowledge base for the two most similar pieces of context by cosine similarity.\n", "1. `llama_index.indices.query.ResponseSynthesizer` generates a response by formatting the query and retrieved context into a single prompt and sending a request to OpenAI chat completions API with the `gpt-3.5-turbo`.\n", "2. The response is returned to the user.\n", "\n", "Phoenix makes your search and retrieval system *observable* by capturing the inputs and outputs of these steps for analysis, including:\n", "\n", "- your query embeddings\n", "- the retrieved context and similarity scores for each query\n", "- the generated response that is return to the user\n", "\n", "With that overview in mind, let's dive into the notebook." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 1. Install Dependencies and Import Libraries" ] }, { "cell_type": "markdown", "metadata": {}, "source": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Install Phoenix and LlamaIndex." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "%pip install -Uq \"arize-phoenix[evals,embeddings]\" \"openai>=1\" \"llama-index<0.10.0\" gcsfs 'httpx<0.28'" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Import libraries." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import os\n", "import textwrap\n", "from datetime import timedelta\n", "from getpass import getpass\n", "\n", "import numpy as np\n", "import pandas as pd\n", "from gcsfs import GCSFileSystem\n", "from IPython.display import YouTubeVideo\n", "from llama_index import (\n", " ServiceContext,\n", " StorageContext,\n", " load_index_from_storage,\n", ")\n", "from llama_index.callbacks import CallbackManager, OpenInferenceCallbackHandler\n", "from llama_index.callbacks.open_inference_callback import as_dataframe\n", "from llama_index.embeddings.openai import OpenAIEmbedding\n", "from llama_index.graph_stores.simple import SimpleGraphStore\n", "from llama_index.llms import OpenAI\n", "\n", "import phoenix as px\n", "from phoenix.evals.retrievals import (\n", " classify_relevance,\n", " compute_precisions_at_k,\n", ")\n", "\n", "pd.set_option(\"display.max_colwidth\", 1000)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 2. Configure Your OpenAI API Key" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "💭 Configure your OpenAI API key." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "if not (openai_api_key := os.getenv(\"OPENAI_API_KEY\")):\n", " openai_api_key = getpass(\"🔑 Enter your OpenAI API key: \")\n", " os.environ[\"OPENAI_API_KEY\"] = openai_api_key" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 3. Download Your Knowledge Base" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Download your pre-built index from cloud storage and instantiate your storage context." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "file_system = GCSFileSystem(project=\"public-assets-275721\")\n", "index_path = \"arize-phoenix-assets/datasets/unstructured/llm/llama-index/arize-docs/index/\"\n", "storage_context = StorageContext.from_defaults(\n", " fs=file_system,\n", " persist_dir=index_path,\n", " graph_store=SimpleGraphStore(), # prevents unauthorized request to GCS\n", ")" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "Download and unzip a pre-built knowledge base index consisting of chunks of the Arize documentation." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 4. Run Your Question-Answering Service\n", "\n", "💭 Start a LlamaIndex application from your downloaded index. Use the `OpenInferenceCallbackHandler` to store your data in [OpenInference format](https://github.com/Arize-ai/openinference), an open standard for capturing and storing AI model inferences that enables production LLMapp servers to seamlessly integrate with LLM observability solutions such as Arize and Phoenix." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "callback_handler = OpenInferenceCallbackHandler()\n", "service_context = ServiceContext.from_defaults(\n", " llm=OpenAI(model_name=\"gpt-3.5-turbo\", temperature=0),\n", " embed_model=OpenAIEmbedding(model=\"text-embedding-ada-002\"),\n", " callback_manager=CallbackManager(handlers=[callback_handler]),\n", ")\n", "index = load_index_from_storage(\n", " storage_context,\n", " service_context=service_context,\n", ")\n", "query_engine = index.as_query_engine()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "💭 Ask a few questions of your question-answering service and view the responses." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "max_line_length = 80\n", "for query in [\n", " \"How do I get an Arize API key?\",\n", " \"Can I create monitors with an API?\",\n", " \"How do I need to format timestamps?\",\n", " \"What is the price of the Arize platform\",\n", "]:\n", " print(\"Query\")\n", " print(\"=====\")\n", " print()\n", " print(textwrap.fill(query, max_line_length))\n", " print()\n", " response = query_engine.query(query)\n", " print(\"Response\")\n", " print(\"========\")\n", " print()\n", " print(textwrap.fill(str(response), max_line_length))\n", " print()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 5. Load Your Data Into Pandas Dataframes" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "To use Phoenix, you must load your data into pandas dataframes. " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "💭 Your query data is saved in a buffer on the callback handler you defined in step 4. Load the data from the buffer into a dataframe." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "query_data_buffer = callback_handler.flush_query_data_buffer()\n", "sample_query_df = as_dataframe(query_data_buffer)\n", "sample_query_df" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The columns of the dataframe are:\n", "\n", "- **:id.id:**: the query ID\n", "- **:timestamp.iso_8601:**: the time at which the query was made\n", "- **:feature.text:prompt**: the query text\n", "- **:feature.[float].embedding:prompt**: the embedding representation of the query\n", "- **:prediction.text:response**: the final response presented to the user\n", "- **:feature.[str].retrieved_document_ids:prompt**: the list of IDs of the retrieved documents\n", "- **:feature.[float].retrieved_document_scores:prompt**: the lists of cosine similarities between the query and retrieved documents\n", "\n", "The column names are in OpenInference format and describe the category, data type and intent of each column." ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "Running queries against a large dataset takes a long time. Download a dataframe containing query data." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "query_df = pd.read_parquet(\n", " \"http://storage.googleapis.com/arize-phoenix-assets/datasets/unstructured/llm/llama-index/arize-docs/query_data_complete3.parquet\",\n", ")\n", "query_df.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In addition to the columns of the previous dataframe, this data has a few additional fields:\n", "\n", "- **:tag.float:user_feedback:** approval or rejection from the user (-1 means thumbs down, +1 means thumbs up)\n", "- **:tag.str:openai_relevance_0:** a binary classification (relevant vs. irrelevant) by GPT-4 predicting whether the first retrieved document is relevant to the query\n", "- **:tag.str:openai_relevance_1:** a binary classification (relevant vs. irrelevant) by GPT-4 predicting whether the second retrieved document is relevant to the query\n", "\n", "We'll go over how to compute the relevance classifications in section 6." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Next load your knowledge base into a dataframe." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "def storage_context_to_dataframe(storage_context: StorageContext) -> pd.DataFrame:\n", " \"\"\"Converts the storage context to a pandas dataframe.\n", "\n", " Args:\n", " storage_context (StorageContext): Storage context containing the index\n", " data.\n", "\n", " Returns:\n", " pd.DataFrame: The dataframe containing the index data.\n", " \"\"\"\n", " document_ids = []\n", " document_texts = []\n", " document_embeddings = []\n", " docstore = storage_context.docstore\n", " vector_store = storage_context.vector_store\n", " for node_id, node in docstore.docs.items():\n", " document_ids.append(node.hash) # use node hash as the document ID\n", " document_texts.append(node.text)\n", " document_embeddings.append(np.array(vector_store.get(node_id)))\n", " return pd.DataFrame(\n", " {\n", " \"document_id\": document_ids,\n", " \"text\": document_texts,\n", " \"text_vector\": document_embeddings,\n", " }\n", " )\n", "\n", "\n", "database_df = storage_context_to_dataframe(storage_context)\n", "database_df = database_df.drop_duplicates(subset=[\"text\"])\n", "database_df.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The columns of your dataframe are:\n", "\n", "- **document_id:** the ID of the chunked document\n", "- **text:** the chunked text in your knowledge base\n", "- **text_vector:** the embedding vector for the text, computed during the LlamaIndex build using \"text-embedding-ada-002\" from OpenAI" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The query and database datasets are drawn from different distributions; the queries are short questions while the database entries are several sentences to a paragraph. The embeddings from OpenAI's \"text-embedding-ada-002\" capture these differences and naturally separate the query and context embeddings into distinct regions of the embedding space. When using Phoenix, you want to \"overlay\" the query and context embedding distributions so that queries appear close to their retrieved context in the Phoenix point cloud. To achieve this, we compute a centroid for each dataset that represents an average point in the embedding distribution and center the two distributions so they overlap." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "database_centroid = database_df[\"text_vector\"].mean()\n", "database_df[\"text_vector\"] = database_df[\"text_vector\"].apply(lambda x: x - database_centroid)\n", "\n", "query_centroid = query_df[\":feature.[float].embedding:prompt\"].mean()\n", "query_df[\":feature.[float].embedding:prompt\"] = query_df[\":feature.[float].embedding:prompt\"].apply(\n", " lambda x: x - query_centroid\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 6. Run LLM-Assisted Evaluations\n", "\n", "Cosine similarity and Euclidean distance are reasonable proxies for retrieval quality, but they don't always work perfectly. A novel idea is to use LLMs to measure retrieval quality by simply asking the LLM whether each retrieved document is relevant to the corresponding query.\n", "\n", "💭 Use OpenAI to predict whether each retrieved document is relevant or irrelevant to the query.\n", "\n", "⚠️ It's strongly recommended to use GPT-4 for evaluations if you have access." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "evals_model_name = \"gpt-3.5-turbo\"\n", "# evals_model_name = \"gpt-4\" # use GPT-4 if you have access\n", "query_texts = sample_query_df[\":feature.text:prompt\"].tolist()\n", "list_of_document_id_lists = sample_query_df[\":feature.[str].retrieved_document_ids:prompt\"].tolist()\n", "document_id_to_text = dict(zip(database_df[\"document_id\"].to_list(), database_df[\"text\"].to_list()))\n", "\n", "first_document_texts, second_document_texts = [\n", " [\n", " document_id_to_text[document_ids[document_index]]\n", " for document_ids in list_of_document_id_lists\n", " ]\n", " for document_index in [0, 1]\n", "]\n", "first_document_relevances, second_document_relevances = [\n", " [\n", " classify_relevance(query_text, document_text, evals_model_name)\n", " for query_text, document_text in zip(query_texts, first_document_texts)\n", " ]\n", " for document_texts in [first_document_texts, second_document_texts]\n", "]\n", "\n", "\n", "sample_query_df = sample_query_df.assign(\n", " retrieved_document_text_0=first_document_texts,\n", " retrieved_document_text_1=second_document_texts,\n", " relevance_0=first_document_relevances,\n", " relevance_1=second_document_relevances,\n", ")\n", "sample_query_df[\n", " [\n", " \":feature.text:prompt\",\n", " \"retrieved_document_text_0\",\n", " \"retrieved_document_text_1\",\n", " \"relevance_0\",\n", " \"relevance_1\",\n", " ]\n", "].rename(columns={\":feature.text:prompt\": \"query_text\"})" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 7. Compute Ranking Metrics\n", "\n", "Now that you know whether each piece of retrieved context is relevant or irrelevant to the corresponding query, you can compute precision@k for k = 1, 2 for each query. This metric tells you what percentage of the retrieved context is relevant to the corresponding query.\n", "\n", "precision@k = (# of top-k retrieved documents that are relevant) / (k retrieved documents)\n", "\n", "If your precision@2 is greater than zero for a particular query, your LlamaIndex application successfully retrieved at least one relevant piece of context with which to answer the query. If the precision@k is zero for a particular query, that means that no relevant piece of context was retrieved.\n", "\n", "Compute precision@k for k = 1, 2 and view the results." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "first_document_relevances = [\n", " {\"relevant\": True, \"irrelevant\": False}.get(rel)\n", " for rel in query_df[\":tag.str:openai_relevance_0\"].tolist()\n", "]\n", "second_document_relevances = [\n", " {\"relevant\": True, \"irrelevant\": False}.get(rel)\n", " for rel in query_df[\":tag.str:openai_relevance_1\"].tolist()\n", "]\n", "list_of_precisions_at_k_lists = [\n", " compute_precisions_at_k([rel0, rel1])\n", " for rel0, rel1 in zip(first_document_relevances, second_document_relevances)\n", "]\n", "precisions_at_1, precisions_at_2 = [\n", " [precisions_at_k[index] for precisions_at_k in list_of_precisions_at_k_lists]\n", " for index in [0, 1]\n", "]\n", "query_df[\":tag.float:openai_precision_at_1\"] = precisions_at_1\n", "query_df[\":tag.float:openai_precision_at_2\"] = precisions_at_2\n", "query_df[\n", " [\n", " \":tag.str:openai_relevance_0\",\n", " \":tag.str:openai_relevance_1\",\n", " \":tag.float:openai_precision_at_1\",\n", " \":tag.float:openai_precision_at_2\",\n", " ]\n", "]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 7. Launch Phoenix" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Define your knowledge base dataset with a schema that specifies the meaning of each column (features, predictions, actuals, tags, embeddings, etc.). See the [docs](https://docs.arize.com/phoenix/) for guides on how to define your own schema and API reference on `phoenix.Schema` and `phoenix.EmbeddingColumnNames`." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# get a random sample of 500 documents (including retrieved documents)\n", "# this will be handled by by the application in a coming release\n", "num_sampled_point = 500\n", "retrieved_document_ids = set(\n", " [\n", " doc_id\n", " for doc_ids in query_df[\":feature.[str].retrieved_document_ids:prompt\"].to_list()\n", " for doc_id in doc_ids\n", " ]\n", ")\n", "retrieved_document_mask = database_df[\"document_id\"].isin(retrieved_document_ids)\n", "num_retrieved_documents = len(retrieved_document_ids)\n", "num_additional_samples = num_sampled_point - num_retrieved_documents\n", "unretrieved_document_mask = ~retrieved_document_mask\n", "sampled_unretrieved_document_ids = set(\n", " database_df[unretrieved_document_mask][\"document_id\"]\n", " .sample(n=num_additional_samples, random_state=0)\n", " .to_list()\n", ")\n", "sampled_unretrieved_document_mask = database_df[\"document_id\"].isin(\n", " sampled_unretrieved_document_ids\n", ")\n", "sampled_document_mask = retrieved_document_mask | sampled_unretrieved_document_mask\n", "sampled_database_df = database_df[sampled_document_mask]" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "database_schema = px.Schema(\n", " prediction_id_column_name=\"document_id\",\n", " prompt_column_names=px.EmbeddingColumnNames(\n", " vector_column_name=\"text_vector\",\n", " raw_data_column_name=\"text\",\n", " ),\n", ")\n", "database_ds = px.Inferences(\n", " dataframe=sampled_database_df,\n", " schema=database_schema,\n", " name=\"database\",\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Define your query dataset. Because the query dataframe is in OpenInference format, Phoenix is able to infer the meaning of each column without a user-defined schema by using the `phoenix.Dataset.from_open_inference` class method." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "query_ds = px.Inferences.from_open_inference(query_df)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Launch Phoenix. Follow the instructions in the cell output to open the Phoenix UI." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "(session := px.launch_app(primary=query_ds, corpus=database_ds)).view()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 8. Surface Problematic Clusters and Data Points\n", "\n", "Phoenix helps you:\n", "\n", "- visualize your embeddings\n", "- color the resulting point cloud using evaluation metrics\n", "- cluster the points and surface up problematic clusters based on whatever metric you care about\n", "\n", "Follow along with the tutorial walkthrough [here](https://youtu.be/hbQYDpJayFw?t=1782), or view the video in your notebook by running the cell below. The video will show you how to investigate your query and knowledge base and identify problematic clusters of data points using Phoenix." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "start_time_in_seconds = int(timedelta(hours=0, minutes=29, seconds=42).total_seconds())\n", "YouTubeVideo(\"hbQYDpJayFw\", start=start_time_in_seconds, width=560, height=315)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Congrats! You've identified a problematic cluster of queries. You now have tools at your disposal to investigate clusters of queries where your search and retrieval application is performing poorly based on:\n", "\n", "- query percentage\n", "- user feedback\n", "- LLM-assisted ranking metrics\n", "\n", "As an actionable next step, you should augment your knowledge base to include information about the pricing and cost of the Arize platform, since your users seem especially interested in this topic." ] } ], "metadata": { "language_info": { "name": "python" } }, "nbformat": 4, "nbformat_minor": 4 }