Integrate PostHog telemetry and enhance project planning capabilities (#70)
* feat: ๐ integrate PostHog telemetry and add plan generation standards **โจ Features** - Introduce a comprehensive `TelemetryService` using PostHog to track extension activation, feature usage, and performance metrics across Node.js and Web environments. - Instrument chat providers, inline completions, and configuration commands to capture detailed event data, including token usage and error types. - Implement an observability hook system via `PostHogHook` to automatically report request completion and message trimming events. **๐ Documentation** - Add `plan-generation.instructions.md` to define strict standards for generating autonomous, code-explicit implementation plans. **๐งน Chores** - Update project dependencies to include `posthog-js` and `posthog-node` while bumping the version to `1.6.0-dev4`. - Configure `esbuild` and `tsconfig` path aliases to support platform-specific telemetry adapters. **๐งช Testing** - Add unit tests for `TelemetryService` to verify correct event capturing and settings respect. * feat: ๐ Upgrade planning agent and integrate PostHog error tracking - **โจ Features** - Refactor `Project Planner` agent to adopt standardized `plan-generation.instructions.md` and switch from `.plan.md` to `.prompt.md` files. - Implement full PostHog error tracking with automated source map uploads via `npm run posthog:sourcemaps`. - Add configuration schema for multiple LiteLLM backends. - Add `CHANGELOG.md` support in VS Code packaging. - **๐งน Chores** - Update `esbuild` to support external source map generation and improved build logging. - Update project version to `1.6.0-dev7`. - Enhance `TelemetryService` to support exception capture, user identification, and feature flag checks. - Update `Logger` to automatically report errors to `TelemetryService`. - Add unit tests for telemetry adapters. * feat: add centralized telemetry and update pre-release version ๐ - **โจ Features** - Initialize centralized `TelemetryService` and wire observability hooks into core request paths for consistent tracking. - Enrich telemetry data with dimensions like OS, extension version, and UI context to improve event analysis. - **๐ Fixes** - Update `post-session-validation.sh` to handle stdin gracefully only when input is provided. - **๐งน Chores** - Bump version to `1.6.0-dev8` and remove outdated `baseUrl` from `tsconfig.json`. * feat: ๐ฐ๏ธ Implement comprehensive telemetry and add project planning agent - **โจ Features** - Add `project-planner` agent prompt to streamline implementation planning for new tasks. - Implement feature-usage reporting across core commands and providers, including chat, completions, and model picking. - Add `ConfigManager` integration to capture feature toggle snapshots on startup and report toggle changes in real-time. - Wire command execution tracking into model management and reset commands for better usage visibility. - Enhance `TelemetryService` with exception capture including caller metadata for better error traceability. - **๐งน Chores** - Update extension version to `1.6.0-dev10` and consolidate development changes. - Remove redundant PostHog documentation from README as telemetry is now fully integrated. - Cleanup unused code paths and fix minor shell script status codes. - **๐งช Tests** - Add unit tests for `ConfigManager` toggle reporting and verify telemetry event payloads. - Expand test coverage for `TelemetryService` to validate snapshot and feature usage event tracking. * feat: ๐ improve telemetry identification and bump version - **โจ Features** - Use `vscode.env.sessionId` as the primary distinct ID for better crash reporting correlation. - Standardize `extension_version` property usage across all telemetry events and identity calls. - **๐งช Tests** - Add test case to verify that `identify` calls correctly capture the session ID and extension version. - **๐งน Chores** - Bump package version to 1.6.0-dev11. * test: ๐งช update telemetry unit tests to use sessionId for distinctId - **๐งช Tests** - Update `TelemetryService` tests to use `vscode.env.sessionId` instead of `machineId` as the source for `distinctId`. - Remove redundant `identify` test case to clean up the test suite. * chore: ๐ ๏ธ Add implementation prompt, update version, and expand test coverage **โจ Features** - Add `implement-plan.prompt.md` to support standardized plan implementation workflows. **๐งน Chores** - Bump extension version to `1.6.0-dev12`. **๐งช Tests** - Add tests to `liteLLMStreamInterpreter` for tool call deduplication, response format edge cases, and Gemini native format. - Add connectivity, token counting, and chat edge case tests for `LiteLLMClient`. - Expand `ResponsesAdapter` testing for combined text/tool calls, tool name synthesis, and complex message content handling. - Add `ResponsesClient` tests for anonymous tool calls, delta variations, and usage reporting safety. - Add comprehensive token budget and V2 message normalization unit tests in `tokenUtils` and `utils`. - Add robust tests for `manageConfig` commands and `modelPicker` error scenarios. - Improve test coverage for telemetry adapters and base provider functionality, including error handling and model tagging. - Ensure extension activation properly handles edge cases like missing metadata or registration failures. * feat: ๐ enhance provider compatibility and robust request validation **โจ Features** - Support native streaming and vision modalities in `LiteLLMProviderBase` by adding relevant tags to the model info. - Improve `ResponsesClient` reliability by adding error handling to `DataPart` usage reports. - Update `stripMarkdownCodeBlocks` to specifically target triple backticks, preserving single backticks within text. **๐ Fixes** - Enhance `validateRequest` to strictly check for empty message content across all request messages. - Allow `assistant` role string in `trimV2MessagesForBudget` for broader compatibility. - Ensure `isContinuation` logic properly breaks the loop to prevent message processing errors. **๐งน Chores** - Bump project version to `1.6.0-dev13`. - Update unit tests to reflect updated logic for progress reporting, message budget trimming, and validation. * refactor: ๐ reduce telemetry verbosity - **๐งน Chores** - Comment out excessive telemetry tracking across providers to reduce noise in logs. - Temporarily disable successful request telemetry in `PostHogHook` and `LiteLLMTelemetry` to streamline output. - Bump project version to `1.6.0-dev14`. * test: ๐งช update fetch stubbing in LiteLLM client tests - **๐งช Tests** - Refactor fetch stubbing to use `returns` with resolved promises for better control over mock behavior. - Ensure consistent async handling during rate limit retry testing. * feat: ๐ Enhance telemetry system and tracking - **โจ Features** - Add documentation and automation prompts to standardize telemetry implementation across the codebase. - Implement aggregated feature usage tracking in `TelemetryService` to reduce payload frequency by batching events every 15 minutes. - Add specific tracking for model/provider usage and feature adoption to gain insights into user behavior. - **๐ Fixes** - Refactor `captureFeatureUsed` to support aggregation, ensuring performance is not degraded by high-volume events. - **๐งน Chores** - Increment project version to `1.6.0-dev15`. - Update `extension.ts` to trigger initial feature adoption events on activation. - Update `LiteLLMChatProvider` and `LiteLLMProviderBase` to capture model usage metrics during requests and discovery. - Add extensive unit tests for aggregated telemetry and new tracking methods in `telemetryService.test.ts`. * feat: ๐ Add changelog automation and improve telemetry observability - **โจ Features** - Add a new automation prompt to generate `CHANGELOG.md` updates from completed plan files. - **๐ Enhancements** - Propagate `request_id` across telemetry events to allow for better request lifecycle correlation in PostHog. - Improve `TelemetryService` initialization to prioritize `machineId` over `sessionId` for more stable user identification. - **๐ Fixes** - Ensure commit-message generation errors are surfaced in logs and telemetry instead of being silently swallowed. - Fix test suite cleanup by introducing a Sinon sandbox to prevent cross-test interference. - **๐งช Testing** - Add comprehensive regression tests to verify `request_id` propagation and error reporting in telemetry. * feat: ๐ Add support for multiple LiteLLM instances ### โจ Features - Implement support for multiple backend and LiteLLM instances in the legacy method. - Integrate telemetry for non-identifiable data collection and reporting. ### ๐ ๏ธ Bug Fixes - โ๏ธ Fix undefined silent flag, resolving configuration issues for LiteLLM instances. - ๐ ๏ธ Address nuke command failures that leave stale models during configuration resets. - ๐ ๏ธ Stabilize telemetry ID alignment and re-scope unhandled exception management. ### ๐ Telemetry & Observability - ๐ Add feature usage reporting for chat, completions, and model picker events. * feat: ๐ Add Model Capability Overrides configuration to improve VS Code support # โจ Features - Introduce configuration for manual capability overrides (toolCalling, imageInput) to address inaccuracies in LiteLLM's auto-detection. Configure using `litellm-connector.modelCapabilitiesOverrides`. # ๐ ๏ธ Bug Fixes - Ensure commit-message generation failure visibility in both local logs and telemetry. * test: add unit tests for PostHogHook functionality ๐ **๐งช Tests** - Introduce unit tests for the `PostHogHook` class to ensure correct registration of hooks and functionality for success and error paths. - Validate `captureRequestCompleted` and `captureRequestFailed` calls based on the metadata passed in the context. - Ensure that `captureTrimExecuted` is called correctly based on the trim conditions and defaults to zero when metadata is missing. - Include tests for the safe disposal of the hook to prevent errors on multiple calls to dispose. * chore: update CI workflows and coverage thresholds ๐ **๐ฆ CI Changes** - Removed deprecated branch `geth-dev` and its patterns from workflows to streamline CI setup. - Retained only necessary branches to trigger CI processes. **๐ง Coverage Updates** - Increased coverage threshold from 10% to 20% in `codecov.yml` to improve code quality standards.
A
amwdrizz committed
ce057fc524141ca287e91344d5a584bb22749012
Parent: ab65341
Committed by GitHub <noreply@github.com>
on 4/9/2026, 1:49:17 AM