{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "
\n", "

\n", " \"phoenix\n", "
\n", " Docs\n", " |\n", " GitHub\n", " |\n", " Community\n", "

\n", "
\n", "

LLM Ops - Tracing, Evaluation, and Analysis

\n", "\n", "In this tutorial we will learn how to build, observe, evaluate, and analyze a LLM powered application. \n", "\n", "It has the following sections:\n", "\n", "1. Understanding LLM-powered applications\n", "2. Observing the application using traces\n", "3. Evaluating the application using LLM Evals\n", "4. Exploring and and troubleshooting the application using UMAP projection and clustering\n", "\n", "\n", "## Understanding LLM powered applications\n", "\n", "Building software with LLMs, or any machine learning model, is [fundamentally different](https://karpathy.medium.com/software-2-0-a64152b37c35). Rather than compiling source code into binary to run a series of commands, we need to navigate datasets, embeddings, prompts, and parameter weights to generate consistent accurate results. LLM outputs are probabilistic and therefore don't produce the same deterministic outcome every time.\n", "\n", "\n", "\n", "There's a lot that can go into building an LLM application, but let's focus on the architecture. Below is a diagram of one possible architecture. In this diagram we see that the application is built around an LLM but that there are many other components that are needed to make the application work.\n", "\n", "\n", "\n", "The complexity that is involved in building an LLM application is why observability is so important. Observability is the ability to understand the internal state of a system by examining its inputs and outputs. Each step of the response generation process needs to be monitored, evaluated and tuned to provide the best possible experience. Not only that, certain tradeoffs might need to be made to optimize for speed, cost, or accuracy. In the context of LLM applications, we need to be able to understand the internal state of the system by examining telemetry data such as LLM Traces.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Observing the application using traces\n", "\n", "LLM Traces and Observability lets us understand the system from the outside, by letting us ask questions about that system without knowing its inner workings. Furthermore, it allows us to easily troubleshoot and handle novel problems (i.e. “unknown unknowns”), and helps us answer the question, “Why is this happening?”\n", "\n", "LLM Traces are designed to be a category of telemetry data that is used to understand the execution of LLMs and the surrounding application context such as retrieval from vector stores and the usage of external tools such as search engines or APIs. It lets you understand the inner workings of the individual steps your application takes wile also giving you visibility into how your system is running and performing as a whole.\n", "\n", "Traces are made up of a sequence of `spans`. A span represents a unit of work or operation (think a span of time). It tracks specific operations that a request makes, painting a picture of what happened during the time in which that operation was executed.\n", "\n", "LLM Tracing supports the following span kinds:\n", "\n", "\n", "\n", "\n", "By capturing the building blocks of your application while it's running, phoenix can provide a more complete picture of the inner workings of your application. To illustrate this, let's look at an example LLM application and inspect it's traces." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Traces and Spans in action\n", "\n", "Let's build a relatively simple LLM-powered application that will answer questions about Arize AI. This example uses LlamaIndex for RAG and OpenAI as the LLM but you could use any LLM you would like. The details of the application are not important, but the architecture is. Let's get started." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "%pip install -qq arize-phoenix==7.3.2 llama-index-core==0.12.8 llama-index-llms-ollama==0.5.0 llama-index-embeddings-ollama==0.5.0 llama-index-readers-file==0.4.1 gcsfs==2024.12.0 nest_asyncio==1.6.0 openinference-instrumentation-llama_index==3.1.2 litellm==1.55.12" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import phoenix as px\n", "\n", "session = px.launch_app()" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from openinference.instrumentation.llama_index import LlamaIndexInstrumentor\n", "\n", "from phoenix.otel import register\n", "\n", "tracer_provider = register()\n", "LlamaIndexInstrumentor().instrument(skip_dep_check=True, tracer_provider=tracer_provider)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "You'll need to make sure you have ollama installed and running both models `ollama/mistral:7b` and `ollama/nomic-embed-text`.\n", "\n", "To do this, run the following commands in your terminal:\n", "\n", "```bash\n", "ollama pull mistral:7b\n", "ollama pull nomic-embed-text\n", "```\n", "\n", "Then we'll need to set the `OLLAMA_API_BASE` environment variable to point to your local ollama instance. The port `11434` is the default port for ollama." ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [], "source": [ "ollama_model = \"mistral:7b\"\n", "ollama_embed_model = \"nomic-embed-text\"" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [], "source": [ "import os\n", "\n", "os.environ[\"OLLAMA_API_BASE\"] = \"http://localhost:11434\"" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [], "source": [ "from llama_index.core import (\n", " Settings,\n", " VectorStoreIndex,\n", ")\n", "from llama_index.core.llama_dataset import download_llama_dataset\n", "from llama_index.embeddings.ollama import OllamaEmbedding\n", "from llama_index.llms.ollama import Ollama\n", "\n", "# download and install dependencies for benchmark dataset\n", "rag_dataset, documents = download_llama_dataset(\"PaulGrahamEssayDataset\", \"./data\")\n", "\n", "Settings.llm = Ollama(model=ollama_model)\n", "Settings.embed_model = OllamaEmbedding(model_name=ollama_embed_model)\n", "\n", "# build basic RAG system\n", "index = VectorStoreIndex.from_documents(documents=documents)\n", "query_engine = index.as_query_engine()" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "response = query_engine.query(\"What did the Paul Graham do at Y Combinator?\")\n", "print(response)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's test a few more queries to see if the application is working as expected." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from tqdm import tqdm\n", "\n", "queries = [\n", " \"What was Paul Graham's role at Y Combinator?\",\n", " \"What are Paul Graham's views on startups?\",\n", " \"What does Paul Graham think about programming languages?\",\n", " \"How did Paul Graham become successful as an entrepreneur?\",\n", "]\n", "\n", "for query in tqdm(queries):\n", " response = query_engine.query(query)\n", " print(f\"Query: {query}\")\n", " print(f\"Response: {response}\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now that we've run the application a few times, let's take a look at the traces in the UI" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "print(\"The Phoenix UI:\", session.url)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The UI will give you an interactive troubleshooting experience. You can sort, filter, and search for traces. You can also view the detail of each trace to better understand what happened during the response generation process.\n", "\n", "" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In addition to being able to view the traces in the UI, you can also query for traces to use as pandas dataframes in your notebook." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "spans_df = px.Client().get_spans_dataframe()\n", "spans_df[[\"name\", \"span_kind\", \"attributes.input.value\", \"attributes.retrieval.documents\"]].head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "With just a few lines of code, we have managed to gain visibility into the inner workings of our application. We can now better understand how things like retrieval, prompts, and parameter weights could be affecting our application. But what can we do with this information? Let's take a look at how to use LLM evals to evaluate our application." ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [], "source": [ "if (\n", " input(\"The tutorial is about to move on to the evaluation section. Continue [Y/n]?\")\n", " .lower()\n", " .startswith(\"n\")\n", "):\n", " assert False, \"notebook stopped\"" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Evaluating the application using LLM Evals\n", "\n", "Evaluation should serve as the primary metric for assessing your application. It determines whether the app will produce accurate responses based on the data sources and range of queries.\n", "\n", "While it's beneficial to examine individual queries and responses, this approach is impractical as the volume of edge-cases and failures increases. Instead, it's more effective to establish a suite of metrics and automated evaluations. These tools can provide insights into overall system performance and can identify specific areas that may require scrutiny.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's first convert our traces into a workable datasets" ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [], "source": [ "from phoenix.session.evaluation import get_qa_with_reference, get_retrieved_documents\n", "\n", "retrieved_documents_df = get_retrieved_documents(px.Client())\n", "queries_df = get_qa_with_reference(px.Client())" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can now use Phoenix's LLM Evals to evaluate these queries. LLM Evals uses an LLM to grade your application based on different criteria. For this example we will use the evals library to see if any `hallucinations` are present and if the `Q&A Correctness` is good (whether or not the application answers the question correctly)." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import nest_asyncio\n", "\n", "from phoenix.evals import (\n", " HALLUCINATION_PROMPT_RAILS_MAP,\n", " HALLUCINATION_PROMPT_TEMPLATE,\n", " QA_PROMPT_RAILS_MAP,\n", " QA_PROMPT_TEMPLATE,\n", " LiteLLMModel,\n", " llm_classify,\n", ")\n", "\n", "nest_asyncio.apply()\n", "\n", "# Check if the application has any indications of hallucinations\n", "hallucination_eval = llm_classify(\n", " dataframe=queries_df,\n", " model=LiteLLMModel(model=\"ollama/\" + ollama_model),\n", " template=HALLUCINATION_PROMPT_TEMPLATE,\n", " rails=list(HALLUCINATION_PROMPT_RAILS_MAP.values()),\n", " provide_explanation=True, # Makes the LLM explain its reasoning\n", ")\n", "hallucination_eval[\"score\"] = (\n", " hallucination_eval.label[~hallucination_eval.label.isna()] == \"factual\"\n", ").astype(int)\n", "\n", "# Check if the application is answering questions correctly\n", "qa_correctness_eval = llm_classify(\n", " dataframe=queries_df,\n", " model=LiteLLMModel(model=\"ollama/\" + ollama_model),\n", " template=QA_PROMPT_TEMPLATE,\n", " rails=list(QA_PROMPT_RAILS_MAP.values()),\n", " provide_explanation=True, # Makes the LLM explain its reasoning\n", ")\n", "\n", "qa_correctness_eval[\"score\"] = (\n", " qa_correctness_eval.label[~qa_correctness_eval.label.isna()] == \"correct\"\n", ").astype(int)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "hallucination_eval.head()" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "qa_correctness_eval.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "As you can see from the results, one of the queries was flagged as a hallucination. Let's log these results to the phoenix server to view the hallucinations in the UI." ] }, { "cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [], "source": [ "from phoenix.trace import SpanEvaluations\n", "\n", "px.Client().log_evaluations(\n", " SpanEvaluations(eval_name=\"Hallucination\", dataframe=hallucination_eval),\n", " SpanEvaluations(eval_name=\"QA Correctness\", dataframe=qa_correctness_eval),\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now that we have the hallucinations logged, let's take a look at them in the UI. You will notice that the traces that correspond to hallucinations are clearly marked in the UI and you can now query for them!" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "print(\"The Phoenix UI:\", session.url)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We've now identified that there are certain queries that are resulting in hallucinations or incorrect answers. Let's see if we can use LLM Evals to identify if these issues are caused by the retrieval process for RAG. We are going to use an LLM to grade whether or not the chunks retrieved are relevant to the query." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from phoenix.evals import (\n", " RAG_RELEVANCY_PROMPT_RAILS_MAP,\n", " RAG_RELEVANCY_PROMPT_TEMPLATE,\n", " llm_classify,\n", ")\n", "\n", "retrieved_documents_eval = llm_classify(\n", " dataframe=retrieved_documents_df,\n", " model=LiteLLMModel(model=\"ollama/\" + ollama_model),\n", " template=RAG_RELEVANCY_PROMPT_TEMPLATE,\n", " rails=list(RAG_RELEVANCY_PROMPT_RAILS_MAP.values()),\n", " provide_explanation=True,\n", ")\n", "\n", "retrieved_documents_eval[\"score\"] = (\n", " retrieved_documents_eval.label[~retrieved_documents_eval.label.isna()] == \"relevant\"\n", ").astype(int)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "retrieved_documents_eval.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Looks like we are getting a lot of irrelevant chunks of text that might be polluting the prompt sent to the LLM. Let's log these evals to phoenix, at which point phoenix will automatically calculate retrieval metrics for us." ] }, { "cell_type": "code", "execution_count": 21, "metadata": {}, "outputs": [], "source": [ "from phoenix.trace import DocumentEvaluations\n", "\n", "px.Client().log_evaluations(\n", " DocumentEvaluations(eval_name=\"Relevance\", dataframe=retrieved_documents_eval)\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "If we once again visit the UI, we will now see that Phoenix has aggregated up retrieval metrics (`precision`, `ndcg`, and `hit`). We see that our hallucinations and incorrect answers directly correlate to bad retrieval!" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "print(\"The Phoenix UI:\", session.url)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "\n", "There are many more evaluations metrics that can be used to make determinations about the quality of your application. Phoenix supports evaluating not only traces but individual spans (such as retrievers) as well as performing retrieval analysis for your document chunks. Evaluation metrics can also be customized for your specific use-case. For more details, consult the phoenix docs." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Conclusion\n", "\n", "Phoenix is a powerful companion for building LLM-powered applications. It allows you to observe, evaluate, and explore your application at every step of you development process. For more details, consult the [phoenix docs](https://docs.arize.com/phoenix)." ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.16" } }, "nbformat": 4, "nbformat_minor": 2 }