AI Admin Rating (Private Beta)

Overview

AI Admin Rating provides a human-in-the-loop review workflow for Ask AI sessions. Admins can browse recent questions and answers, review statuses, ratings, and user feedback, add comments to coordinate with users or data owners, and monitor overall quality. Use it to evaluate accuracy, clarity, and usefulness.

Why this matters

• Increase trust in Ask AI with direct visibility into conversation quality and usage.

• Demonstrate value to leadership with real conversations and summary metrics.

• Make usage and quality visible so you can spot where users struggle and take action - then triage quickly and route fixes:

  • Catalog/metadata improvements: When metadata quality is low, data stewards should improve asset descriptions, tags, collections, and other metadata to ensure assets are well-documented and discoverable.

  • Prompting guidance: If a question is underspecified, suggest adding context or rephrasing.

  • Ask AI behavior: When issues are caused by incorrect prompt handling in Ask AI, flag them as Select Star Issues and route to Support.

Key features

• Purpose-built evaluation: Human-in-the-loop review of Ask AI conversations.

• Status management: Admins can set workflow statuses.

• Evidence view: Analyze the question, reasoning steps, generated SQL, and responses.

• Quality tracking: Overall metrics (total sessions, pass rate, average rating) to monitor progress. Use filters to identify patterns across sessions.

Getting Started

1. Ensure your organization role is Admin.

2. Make sure AI features are enabled in your org.

3. Open Ask AI → AI Admin Rating.

Basic Workflow

Review the summary – check total sessions, Pass/Need Review counts, and average rating. Click a chat to see more details about a specific conversation:

Analyze the conversation and leave a comment if you find any issues:

Take action – set a workflow status and add a comment if needed:

Ratings & Statuses

• Pass - Set automatically for ratings 4–5. Use when the answer is acceptable for business use.

• Need Review - Set automatically for ratings 1–3. Use when follow‑up is required to correct or clarify.

• In Review - Admin has picked it up to investigate. Signals ownership to avoid duplicate work.

• Issue Found - Admin flags an incorrect/low‑quality/risky answer. Add a note and coordinate with the user or data owner to resolve.

• Select Star Issue - Technical/model problem. Route to owner or Select Star Support.

• Resolved - Admin confirms the conversation is acceptable after fixes/re‑test.

FAQ

• Can admins edit others' sessions? Admins can view all sessions but cannot change ratings.

• Can admins add comments to any sessions? Yes. Admins can add comments on sessions they do not own.

• Who gets notified when someone is mentioned? Only admins receive notifications when they are mentioned. Mentions of other users do not trigger notifications.