SIGN IN SIGN UP

Integrate PostHog telemetry and enhance project planning capabilities (#70)

* feat: ๐Ÿ“Š integrate PostHog telemetry and add plan generation standards

**โœจ Features**
- Introduce a comprehensive `TelemetryService` using PostHog to track extension activation, feature usage, and performance metrics across Node.js and Web environments.
- Instrument chat providers, inline completions, and configuration commands to capture detailed event data, including token usage and error types.
- Implement an observability hook system via `PostHogHook` to automatically report request completion and message trimming events.

**๐Ÿ“ Documentation**
- Add `plan-generation.instructions.md` to define strict standards for generating autonomous, code-explicit implementation plans.

**๐Ÿงน Chores**
- Update project dependencies to include `posthog-js` and `posthog-node` while bumping the version to `1.6.0-dev4`.
- Configure `esbuild` and `tsconfig` path aliases to support platform-specific telemetry adapters.

**๐Ÿงช Testing**
- Add unit tests for `TelemetryService` to verify correct event capturing and settings respect.

* feat: ๐Ÿš€ Upgrade planning agent and integrate PostHog error tracking

- **โœจ Features**
  - Refactor `Project Planner` agent to adopt standardized `plan-generation.instructions.md` and switch from `.plan.md` to `.prompt.md` files.
  - Implement full PostHog error tracking with automated source map uploads via `npm run posthog:sourcemaps`.
  - Add configuration schema for multiple LiteLLM backends.
  - Add `CHANGELOG.md` support in VS Code packaging.

- **๐Ÿงน Chores**
  - Update `esbuild` to support external source map generation and improved build logging.
  - Update project version to `1.6.0-dev7`.
  - Enhance `TelemetryService` to support exception capture, user identification, and feature flag checks.
  - Update `Logger` to automatically report errors to `TelemetryService`.
  - Add unit tests for telemetry adapters.

* feat: add centralized telemetry and update pre-release version ๐Ÿš€

- **โœจ Features**
  - Initialize centralized `TelemetryService` and wire observability hooks into core request paths for consistent tracking.
  - Enrich telemetry data with dimensions like OS, extension version, and UI context to improve event analysis.
- **๐Ÿ› Fixes**
  - Update `post-session-validation.sh` to handle stdin gracefully only when input is provided.
- **๐Ÿงน Chores**
  - Bump version to `1.6.0-dev8` and remove outdated `baseUrl` from `tsconfig.json`.

* feat: ๐Ÿ›ฐ๏ธ Implement comprehensive telemetry and add project planning agent

- **โœจ Features**
  - Add `project-planner` agent prompt to streamline implementation planning for new tasks.
  - Implement feature-usage reporting across core commands and providers, including chat, completions, and model picking.
  - Add `ConfigManager` integration to capture feature toggle snapshots on startup and report toggle changes in real-time.
  - Wire command execution tracking into model management and reset commands for better usage visibility.
  - Enhance `TelemetryService` with exception capture including caller metadata for better error traceability.

- **๐Ÿงน Chores**
  - Update extension version to `1.6.0-dev10` and consolidate development changes.
  - Remove redundant PostHog documentation from README as telemetry is now fully integrated.
  - Cleanup unused code paths and fix minor shell script status codes.

- **๐Ÿงช Tests**
  - Add unit tests for `ConfigManager` toggle reporting and verify telemetry event payloads.
  - Expand test coverage for `TelemetryService` to validate snapshot and feature usage event tracking.

* feat: ๐Ÿš€ improve telemetry identification and bump version

- **โœจ Features**
  - Use `vscode.env.sessionId` as the primary distinct ID for better crash reporting correlation.
  - Standardize `extension_version` property usage across all telemetry events and identity calls.
- **๐Ÿงช Tests**
  - Add test case to verify that `identify` calls correctly capture the session ID and extension version.
- **๐Ÿงน Chores**
  - Bump package version to 1.6.0-dev11.

* test: ๐Ÿงช update telemetry unit tests to use sessionId for distinctId

- **๐Ÿงช Tests**
  - Update `TelemetryService` tests to use `vscode.env.sessionId` instead of `machineId` as the source for `distinctId`.
  - Remove redundant `identify` test case to clean up the test suite.

* chore: ๐Ÿ› ๏ธ Add implementation prompt, update version, and expand test coverage

**โœจ Features**
- Add `implement-plan.prompt.md` to support standardized plan implementation workflows.

**๐Ÿงน Chores**
- Bump extension version to `1.6.0-dev12`.

**๐Ÿงช Tests**
- Add tests to `liteLLMStreamInterpreter` for tool call deduplication, response format edge cases, and Gemini native format.
- Add connectivity, token counting, and chat edge case tests for `LiteLLMClient`.
- Expand `ResponsesAdapter` testing for combined text/tool calls, tool name synthesis, and complex message content handling.
- Add `ResponsesClient` tests for anonymous tool calls, delta variations, and usage reporting safety.
- Add comprehensive token budget and V2 message normalization unit tests in `tokenUtils` and `utils`.
- Add robust tests for `manageConfig` commands and `modelPicker` error scenarios.
- Improve test coverage for telemetry adapters and base provider functionality, including error handling and model tagging.
- Ensure extension activation properly handles edge cases like missing metadata or registration failures.

* feat: ๐Ÿš€ enhance provider compatibility and robust request validation

**โœจ Features**
- Support native streaming and vision modalities in `LiteLLMProviderBase` by adding relevant tags to the model info.
- Improve `ResponsesClient` reliability by adding error handling to `DataPart` usage reports.
- Update `stripMarkdownCodeBlocks` to specifically target triple backticks, preserving single backticks within text.

**๐Ÿ› Fixes**
- Enhance `validateRequest` to strictly check for empty message content across all request messages.
- Allow `assistant` role string in `trimV2MessagesForBudget` for broader compatibility.
- Ensure `isContinuation` logic properly breaks the loop to prevent message processing errors.

**๐Ÿงน Chores**
- Bump project version to `1.6.0-dev13`.
- Update unit tests to reflect updated logic for progress reporting, message budget trimming, and validation.

* refactor: ๐Ÿ“‰ reduce telemetry verbosity

- **๐Ÿงน Chores**
  - Comment out excessive telemetry tracking across providers to reduce noise in logs.
  - Temporarily disable successful request telemetry in `PostHogHook` and `LiteLLMTelemetry` to streamline output.
  - Bump project version to `1.6.0-dev14`.

* test: ๐Ÿงช update fetch stubbing in LiteLLM client tests

- **๐Ÿงช Tests**
  - Refactor fetch stubbing to use `returns` with resolved promises for better control over mock behavior.
  - Ensure consistent async handling during rate limit retry testing.

* feat: ๐Ÿš€ Enhance telemetry system and tracking

- **โœจ Features**
  - Add documentation and automation prompts to standardize telemetry implementation across the codebase.
  - Implement aggregated feature usage tracking in `TelemetryService` to reduce payload frequency by batching events every 15 minutes.
  - Add specific tracking for model/provider usage and feature adoption to gain insights into user behavior.
- **๐Ÿ› Fixes**
  - Refactor `captureFeatureUsed` to support aggregation, ensuring performance is not degraded by high-volume events.
- **๐Ÿงน Chores**
  - Increment project version to `1.6.0-dev15`.
  - Update `extension.ts` to trigger initial feature adoption events on activation.
  - Update `LiteLLMChatProvider` and `LiteLLMProviderBase` to capture model usage metrics during requests and discovery.
  - Add extensive unit tests for aggregated telemetry and new tracking methods in `telemetryService.test.ts`.

* feat: ๐Ÿš€ Add changelog automation and improve telemetry observability

- **โœจ Features**
  - Add a new automation prompt to generate `CHANGELOG.md` updates from completed plan files.
- **๐Ÿš€ Enhancements**
  - Propagate `request_id` across telemetry events to allow for better request lifecycle correlation in PostHog.
  - Improve `TelemetryService` initialization to prioritize `machineId` over `sessionId` for more stable user identification.
- **๐Ÿ› Fixes**
  - Ensure commit-message generation errors are surfaced in logs and telemetry instead of being silently swallowed.
  - Fix test suite cleanup by introducing a Sinon sandbox to prevent cross-test interference.
- **๐Ÿงช Testing**
  - Add comprehensive regression tests to verify `request_id` propagation and error reporting in telemetry.

* feat: ๐Ÿš€ Add support for multiple LiteLLM instances

### โœจ Features
- Implement support for multiple backend and LiteLLM instances in the legacy method.
- Integrate telemetry for non-identifiable data collection and reporting.

### ๐Ÿ› ๏ธ Bug Fixes
- โš™๏ธ Fix undefined silent flag, resolving configuration issues for LiteLLM instances.
- ๐Ÿ› ๏ธ Address nuke command failures that leave stale models during configuration resets.
- ๐Ÿ› ๏ธ Stabilize telemetry ID alignment and re-scope unhandled exception management.

### ๐Ÿ“Š Telemetry & Observability
- ๐Ÿ“ˆ Add feature usage reporting for chat, completions, and model picker events.

* feat: ๐ŸŽ‰ Add Model Capability Overrides configuration to improve VS Code support

# โœจ Features
- Introduce configuration for manual capability overrides (toolCalling, imageInput) to address inaccuracies in LiteLLM's auto-detection. Configure using `litellm-connector.modelCapabilitiesOverrides`.

# ๐Ÿ› ๏ธ Bug Fixes
- Ensure commit-message generation failure visibility in both local logs and telemetry.

* test: add unit tests for PostHogHook functionality ๐ŸŽ‰

**๐Ÿงช Tests**
- Introduce unit tests for the `PostHogHook` class to ensure correct registration of hooks and functionality for success and error paths.
- Validate `captureRequestCompleted` and `captureRequestFailed` calls based on the metadata passed in the context.
- Ensure that `captureTrimExecuted` is called correctly based on the trim conditions and defaults to zero when metadata is missing.
- Include tests for the safe disposal of the hook to prevent errors on multiple calls to dispose.

* chore: update CI workflows and coverage thresholds ๐ŸŽ‰

**๐Ÿ“ฆ CI Changes**
- Removed deprecated branch `geth-dev` and its patterns from workflows to streamline CI setup.
- Retained only necessary branches to trigger CI processes.

**๐Ÿ”ง Coverage Updates**
- Increased coverage threshold from 10% to 20% in `codecov.yml` to improve code quality standards.
A
amwdrizz committed
ce057fc524141ca287e91344d5a584bb22749012
Parent: ab65341
Committed by GitHub <noreply@github.com> on 4/9/2026, 1:49:17 AM