{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Evaluating Traces Quickstart\n",
    "This guide will walk you through the process of evaluating traces captured in Phoenix, and exporting the results to the Phoenix UI.\n",
    "\n",
    "This process is similar to the [evaluation quickstart guide](https://docs.arize.com/phoenix/evaluation/evals), but instead of creating your own dataset or using an existing external one, you'll export a trace dataset from Phoenix and log the evaluation results to Phoenix."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Install dependencies & Set environment variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "pip install -q \"arize-phoenix>=4.29.0\"\n",
    "pip install -q openai 'httpx<0.28'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "from getpass import getpass\n",
    "\n",
    "import dotenv\n",
    "\n",
    "dotenv.load_dotenv()\n",
    "\n",
    "if not (openai_api_key := os.getenv(\"OPENAI_API_KEY\")):\n",
    "    openai_api_key = getpass(\"🔑 Enter your OpenAI API key: \")\n",
    "\n",
    "os.environ[\"OPENAI_API_KEY\"] = openai_api_key"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Connect to Phoenix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Check if PHOENIX_API_KEY is present in the environment variables.\n",
    "# If it is, we'll use the cloud instance of Phoenix. If it's not, we'll start a local instance.\n",
    "# A third option is to connect to a docker or locally hosted instance.\n",
    "# See https://docs.arize.com/phoenix/setup/environments for more information.\n",
    "\n",
    "import os\n",
    "\n",
    "if \"PHOENIX_API_KEY\" in os.environ:\n",
    "    os.environ[\"PHOENIX_CLIENT_HEADERS\"] = f\"api_key={os.environ['PHOENIX_API_KEY']}\"\n",
    "    os.environ[\"PHOENIX_COLLECTOR_ENDPOINT\"] = \"https://app.phoenix.arize.com\"\n",
    "    print(\"Using cloud instance of Phoenix.\")\n",
    "else:\n",
    "    import phoenix as px\n",
    "\n",
    "    px.launch_app().view()\n",
    "    print(\"Using local instance of Phoenix.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have Phoenix configured, we can register that instance with OpenTelemetry, which will allow us to collect traces from our application here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from phoenix.otel import register\n",
    "\n",
    "tracer_provider = register(project_name=\"evaluating_traces_quickstart\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prepare trace dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For the sake of making this guide fully runnable, we'll briefly generate some traces and track them in Phoenix. Typically, you would have already captured traces in Phoenix and would skip to \"Download trace dataset from Phoenix\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "pip install -q openinference-instrumentation-openai"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from openinference.instrumentation.openai import OpenAIInstrumentor\n",
    "\n",
    "OpenAIInstrumentor().instrument(tracer_provider=tracer_provider)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from openai import OpenAI\n",
    "\n",
    "# Initialize OpenAI client\n",
    "client = OpenAI()\n",
    "\n",
    "\n",
    "# Function to generate a joke\n",
    "def generate_joke():\n",
    "    response = client.chat.completions.create(\n",
    "        model=\"gpt-3.5-turbo\",\n",
    "        messages=[\n",
    "            {\"role\": \"system\", \"content\": \"You are a helpful assistant that generates jokes.\"},\n",
    "            {\"role\": \"user\", \"content\": \"Tell me a joke.\"},\n",
    "        ],\n",
    "    )\n",
    "    joke = response.choices[0].message.content\n",
    "    return joke\n",
    "\n",
    "\n",
    "# Generate 5 different jokes\n",
    "jokes = []\n",
    "for _ in range(5):\n",
    "    joke = generate_joke()\n",
    "    jokes.append(joke)\n",
    "    print(f\"Joke {len(jokes)}:\\n{joke}\\n\")\n",
    "\n",
    "print(f\"Generated {len(jokes)} jokes and tracked them in Phoenix.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Download trace dataset from Phoenix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import phoenix as px\n",
    "\n",
    "spans_df = px.Client().get_spans_dataframe(project_name=\"evaluating_traces_quickstart\")\n",
    "spans_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Generate evaluations\n",
    "\n",
    "Now that we have our trace dataset, we can generate evaluations for each trace. Evaluations can be generated in many different ways. Ultimately, we want to end up with a set of labels and/or scores for our traces.\n",
    "\n",
    "You can generate evaluations using:\n",
    "- Plain code\n",
    "- Phoenix's [built-in LLM as a Judge evaluators](https://docs.arize.com/phoenix/evaluation/how-to-evals/running-pre-tested-evals)\n",
    "- Your own [custom LLM as a Judge evaluator](https://docs.arize.com/phoenix/evaluation/how-to-evals/bring-your-own-evaluator)\n",
    "- Other evaluation packages\n",
    "\n",
    "As long as you format your evaluation results properly, you can upload them to Phoenix and visualize them in the UI.\n",
    "\n",
    "Let's start with a simple example of generating evaluations using plain code. OpenAI has a habit of repeating jokes, so we'll generate evaluations to label whether a joke is a repeat of a previous joke."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a new DataFrame with selected columns\n",
    "eval_df = spans_df[[\"context.span_id\", \"attributes.llm.output_messages\"]].copy()\n",
    "eval_df.set_index(\"context.span_id\", inplace=True)\n",
    "\n",
    "# Create a list to store unique jokes\n",
    "unique_jokes = set()\n",
    "\n",
    "\n",
    "# Function to check if a joke is a duplicate\n",
    "def is_duplicate(joke_data):\n",
    "    joke = joke_data[0][\"message.content\"]\n",
    "    if joke in unique_jokes:\n",
    "        return True\n",
    "    else:\n",
    "        unique_jokes.add(joke)\n",
    "        return False\n",
    "\n",
    "\n",
    "# Apply the is_duplicate function to create the new column\n",
    "eval_df[\"label\"] = eval_df[\"attributes.llm.output_messages\"].apply(is_duplicate)\n",
    "\n",
    "# Convert boolean to integer (0 for False, 1 for True)\n",
    "eval_df[\"label\"] = eval_df[\"label\"]\n",
    "\n",
    "# Reset unique_jokes list to ensure correct results if the cell is run multiple times\n",
    "unique_jokes.clear()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "eval_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now have a DataFrame with a column for whether each joke is a repeat of a previous joke. Let's upload this to Phoenix."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Upload evaluations to Phoenix\n",
    "\n",
    "Our evals_df has a column for the span_id and a column for the evaluation result. The span_id is what allows us to connect the evaluation to the correct trace in Phoenix. Phoenix will also automatically look for columns named \"label\" and \"score\" to display in the UI."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "eval_df[\"score\"] = eval_df[\"label\"].astype(int)\n",
    "eval_df[\"label\"] = eval_df[\"label\"].astype(str)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "eval_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from phoenix.trace import SpanEvaluations\n",
    "\n",
    "px.Client().log_evaluations(SpanEvaluations(eval_name=\"Duplicate\", dataframe=eval_df))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You should now see evaluations in the Phoenix UI!\n",
    "\n",
    "From here you can continue collecting and evaluating traces, or move on to one of these other guides:\n",
    "* If you're interested in more complex evaluation and evaluators, start with [how to use LLM as a Judge evaluators](https://docs.arize.com/phoenix/evaluation/how-to-evals/running-pre-tested-evals)\n",
    "* If you're ready to start testing your application in a more rigorous manner, check out [how to run structured experiments](https://docs.arize.com/phoenix/datasets-and-experiments/how-to-experiments/run-experiments)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "phoenix",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}