How to Create Backfillable Qualitative Questions

Introduction

Backfillable Qualitative Questions (QQs) allow you to use AI to analyze historical ticket conversations at scale. Instead of manually reviewing thousands of tickets, you can define a question — such as "How satisfied was the customer with the resolution?" or "Did the agent follow the greeting script?" — and let the AI evaluate each ticket conversation automatically. Backfills can be run on regular qualitative questions as well as Basic Score and Two-Choice questions.

With backfillable questions, you can:

Analyze historical data — Run your question against tickets that have already been resolved, not just new ones going forward.
Surface trends — Identify patterns in customer sentiment, agent behavior, or process compliance across large volumes of tickets.
Power dashboard widgets — Use backfillable questions as data sources in your Dynamic Dashboards for reporting and visualization.

This guide walks you through every step of creating a backfillable qualitative question, from initial setup to monitoring your first backfill run.

Before You Begin

Before you can create backfillable qualitative questions, make sure all of the following are in place:

Your user account has an Admin or Owner role — The question creation tools on the AI Analysis page are only accessible to admins and owners.
Your user has permission to create qualitative questions — Have your account admin enable this permission for your user profile. Your user account must also be in an active state. This permission also lets you activate your own questions.
AI Question Creation is enabled for your account — Ask your Qvasa account manager to enable AI Question Creation on your account. This is a one-time setup step, and it is required for the Enable for Backfill option.

Note: If you do not have permission to create questions, the + Create Question button simply will not appear on the AI Analysis page — you will not see an error message. Similarly, if AI Question Creation is not enabled for your account, the Enable for Backfill option will not appear. If something you expect to see is missing, check with your account admin or Qvasa account manager to verify that all prerequisites are met.

Planning Tips

Before diving in, take a few minutes to plan your question:

Decide what you want to measure. Are you measuring customer sentiment, agent adherence to a script, whether a specific topic was discussed, or something else?
Choose the right question type. If you need a graduated scale (High / Medium / Low), use a Basic Score Question. If you need a simple yes/no determination, use a Two-Choice Question. If you need your own set of categories, use a Custom Classifier.
Draft your response options in advance. Option values become permanently locked once the question runs against real data (when you activate it for live data or run your first backfill), so plan them carefully before you begin.

Step 1: Choose Your Question Type

Navigate to the AI Analysis page and click the + Create Question button at the top of the page.

This takes you to the Create a Question page, where you choose a question type:

Basic Score Question — 4-point scale: High, Medium, Low, Unknown
Two-Choice Question — Binary classification with primary option and fallback
Custom Classifier — Your own categories (up to 25) with one fallback

Basic Score Question

A Basic Score Question has four response options, each with a built-in score:

Option	Score
High	3
Medium	2
Low	1
Unknown	0

In the form, the option panels are shaded in blue by score intensity (darker blue = higher score), with gray for Unknown.

Best for: Measuring sentiment, satisfaction, quality, or any dimension where you need a graduated scale.

Example questions:

"How satisfied was the customer with the resolution?"
"How well did the agent handle the customer's complaint?"
"How positive was the customer's tone during the conversation?"

Two-Choice Question

A Two-Choice Question has two response options: a Primary Option and a Fallback Option.

Best for: Binary determinations — yes/no, present/absent, compliant/non-compliant.

Example questions:

"Did the agent offer a discount or credit?"
"Was a product return initiated during this conversation?"
"Did the customer mention a competitor by name?"

Custom Classifier

A Custom Classifier lets you define your own categories — the AI classifies each ticket into exactly one of them. You can add up to 25 categories, and exactly one of them must be marked as the Fallback (the category the AI picks when nothing else fits).

Best for: Contact reason categorization, topic classification, or any use case where the built-in structures don't fit.

Example questions:

"What was the primary reason the customer contacted support?"
"Which product line does this ticket relate to?"

Backfills are not limited to Basic Score and Two-Choice questions — regular qualitative questions, including Custom Classifiers, can also be backfilled. Any ticket question that has been enabled for backfill appears in the backfill dropdown on dashboard widgets.

Click the card for your chosen question type to proceed to the creation form.

Step 2: Create Your Question

The creation form includes a progress indicator at the top showing four steps: Create Question > Review Options > AI Review > Activate.

At the top of the form, a "What should the AI analyze?" toggle lets you choose between Tickets (the AI reads each ticket's messages to answer this question) and Bots (the AI reads each bot interaction's transcript when it ends). Backfills only run against tickets, so leave this set to Tickets for a backfillable question.

Fill out the following fields:

Question Name

This is the display name that appears in dashboards and reports.
Maximum 40 characters.
Must be unique within your account — no two questions can share the same name.
Choose something descriptive and concise (e.g., "Customer Satisfaction", "Discount Offered", "Escalation Needed").

Question for the AI

This is the prompt that the AI will use to evaluate each ticket conversation.
Be specific and unambiguous. The clearer your question, the more accurate the AI's responses will be.
Good example: "Based on the conversation, was the customer satisfied with the resolution provided by the agent? Consider the customer's final messages and any explicit feedback."
Weak example: "Was the customer happy?"

Additional Prompt Context (Optional)

Use this field to provide extra guidance to the AI.
This is helpful for domain-specific terminology, edge cases, or clarifying what should be considered.
Example: "Consider only the customer's messages, not the agent's. If the customer did not explicitly express satisfaction or dissatisfaction, classify as Unknown."

Response Options

Below the question fields, you define the response options:

Basic Score Questions show four panels (High, Medium, Low, Unknown), each with its score.
Two-Choice Questions show two panels: Primary Option and Fallback Option.
Custom Classifiers show a Categories section where you add up to 25 category rows with the + Add Category button, and mark exactly one as the fallback ("Picked when nothing fits").

Each option requires:

Option Value (Required) — A machine-readable identifier used internally for data storage and reporting.
- Must use lowercase letters, numbers, and underscores only (e.g., very_satisfied, agent_offered_discount).
- Must be unique across your entire account — not just within this question, but across all questions in your account.
- Maximum 60 characters.
Description (Optional but recommended) — A human-readable explanation that helps the AI understand when to select this option.
- Example for a "high" option: "The customer explicitly expressed satisfaction, thanked the agent, or indicated the issue was resolved to their liking."

Validation Rules

All option values are required.
Option values must be unique across your entire account.
Option values must contain only lowercase letters, numbers, and underscores.
Basic Score questions must have exactly 4 options — one for each score level.
Two-Choice questions must have exactly 2 options, and they must be different.
Custom Classifiers can have 1 to 25 categories, and exactly one must be marked as the fallback before the question can be activated or backfilled.

If any validation fails, you will see an error message explaining what needs to be corrected.

When you are ready, click Create Question (or Create Classifier for a Custom Classifier).

Step 3: Review and Refine Options

After creating your question, a confirmation message will appear: "Question created! Review your options below and submit for AI review when ready." You will land on the review page for step 2: Review and refine response options.

On this page you can:

Edit the question name, AI prompt, and additional context in the Question Configuration section.
Edit option values (until the question has run against real data).
Edit option descriptions at any time.
Add options to a Custom Classifier with the inline "Add new option" row.
Configure Message Filtering — control which message types the AI includes when analyzing tickets, with toggles to exclude system messages, agent messages, bot messages, customer messages, or the AI ticket summary.

Take the time to review everything carefully. Use Save and Continue to proceed to the AI Review step, or Save and Exit to come back later.

Step 4: Submit for AI Review

The AI Review step is required before you can activate a question you created or enable it for backfill.

What the AI Reviews

When you submit your question for AI review, the AI evaluates your question and options for:

Clarity and specificity — Is the question clear enough for consistent AI interpretation?
Mutual exclusivity — Are the options distinct enough that only one should apply per ticket?
Coverage — Do the options cover all likely scenarios?
Description quality — Are the option descriptions helpful and practical?

Custom Classifiers are reviewed with a rubric tailored to category classification — including how well your categories cover likely contact reasons, how distinct they are from each other, and whether the fallback is well-defined.

During the Review

The review typically takes a few seconds to a couple of minutes (up to about 2 minutes).
The page will show real-time progress updates.

Note: It is recommended that you stay on the page during the AI review. The review runs in the background, so if you do navigate away, you can return and refresh the page after about a minute to check the status. If the review fails, you will see an "AI Review Failed" alert with a Try Again button.

Interpreting the Results

After the review completes, you will see feedback with an overall score out of 100:

Excellent (80 and above) — Your question is well-structured and ready to go.
Needs Work (60–79) — The AI identified areas to refine. Review the suggestions and consider updating your question, descriptions, or option definitions.
Poor (below 60) — Significant issues were found. You should address the feedback before proceeding.

The feedback will include specific strengths, issues, and suggestions. Use these to refine your question if needed — you can make changes and click Re-run AI Review to get updated feedback.

Once the AI review is complete, click Continue to Activation to proceed to the final step.

Step 5: Activate Your Question

On the final step you will see a Configuration Complete confirmation and two paths forward. You can activate your question yourself — there is no waiting on the Qvasa team.

Activate for Live Data

To run your question on incoming tickets going forward, click Choose Run States & Activate. This takes you to the question's management page, where you pick when the question should run:

First time in New / Open / Pending / On-hold / Solved status — run the question the first time a ticket enters the selected status(es).
On Final Ticket Close — run the question when a ticket is finally closed.

Select at least one run state, then click Activate. A confirmation will show that your question is now active and will run on live data.

Enable for Backfill

To run your question against historical tickets instead, use Enable for Backfill Only on the final step (or the Enable for Backfill action on the question's management page).

What happens when you enable for backfill:

The question appears in the backfill dropdown on dashboard widgets that support QQ backfills.
The question is listed under Backfillable Ticket Workflows on the AI Analysis page, and its source changes to "Backfillable Custom" in the system.

A confirmation message will appear: "Backfill enabled! This question now appears in the backfill dropdown on dashboard widgets."

Note: The Enable for Backfill option only appears if AI Question Creation is enabled for your account and your user has the required permission. Backfill is only available for ticket questions (not bot questions), and the AI review must be completed first.

Live data and backfill are separate modes. Activating a question for live data turns backfill mode off (and vice versa, enabling backfill takes the question out of live mode). A backfill-enabled question can later be activated for live data from its management page.

When Option Values Lock

Option values become permanently locked once the question runs against real data — that is, when you activate it for live data or when your first backfill run is created. Changing values after that would corrupt existing results. You can still edit the question name, AI prompt, additional context, option names and descriptions, and message filtering at any time — and you can clone the question to start a new draft.

Step 6: Run a Backfill

Now that your question is enabled for backfill, you can run it against historical tickets.

Navigate to your Dynamic Dashboard.
Open a dashboard widget that supports QQ backfill.
In the AI Backfill form, use the Select Qualitative Question dropdown to choose your question. The dropdown lists every ticket question that has been enabled for backfill — including regular qualitative questions and Custom Classifiers, not just Basic Score and Two-Choice questions.
Click Start Backfill.

The system will respond with:

A success message confirming the backfill was initiated.
A backfill ID for tracking.
Your queue position — backfills are processed one at a time per account in a first-in, first-out (FIFO) order. If another backfill is already running, yours will be queued.
The current status of the backfill.

Note: Only one backfill can be actively processing per account at a time. If you trigger multiple backfills, they will be queued and processed in the order they were created. Remember that your first backfill run permanently locks the question's option values.

Step 7: Monitor Your Backfill

Navigate to the Backfill Management page to monitor your backfill progress.

Daily Limit

At the top of the page, a banner displays your daily usage:

"Daily Limit: X / 30,000 tickets processed today (Y%). Limit resets at midnight UTC."

Important: Each account can process a maximum of 30,000 tickets per day across all backfills. When the daily limit is reached, any in-progress backfill will be automatically paused and will resume the following day after midnight UTC.

Backfill Status

The management page shows a table with the following information for each backfill:

Column	Description
Status	Current state of the backfill (see below)
Qualitative Question	The qualitative question being evaluated
Dashboard	The dashboard widget that triggered the backfill
Tickets	Total number of tickets in the backfill set
Progress	Percentage of tickets processed
Created By	The user who triggered the backfill
Created At	When the backfill was initiated
ETA	Estimated time to completion
Actions	Available actions for the backfill

Status Badges

Status	Color	Meaning
Queued	Blue	Waiting for another backfill to finish
Processing	Yellow	Actively working — shows the current step, such as "Building ticket set" or "Processing ticket"
Completed	Green	All tickets have been processed
Error	Red	Something went wrong — contact your admin
Paused	Orange	Daily limit reached; will auto-resume tomorrow
Cancelled	Gray	Manually cancelled by a user

Available Actions

Cancel — Available for backfills that are Queued, Processing, or Paused. Cancelling a backfill stops future processing, but any tickets already analyzed will retain their results.
Delete — Available for backfills that are Completed, Errored, or Cancelled. Removes the backfill record from the list.

Note: If a backfill appears to be stuck (no progress for an extended period), contact your Qvasa account manager for assistance.

Managing Your Question Later

Every question has a management page where you can see its current state and move it through its lifecycle. To reach it, click the Manage State icon next to the question on the AI Analysis page.

From the management page you can:

See which settings are locked and why, and which remain editable.
Adjust when the question runs (run states) for active ticket questions.
Change state — submit for AI Review, Enable for Backfill, Activate for Live Data, Convert to Nested Question, Deactivate, or Clone.
Edit Prompts & Options — jump back to the options review page.

Tip: If a question's option values are locked and you need different ones, use Clone to start a new draft based on the existing question.

Tips for Writing Effective Questions

Craft a Clear, Specific AI Prompt

Be explicit about what to look for. Instead of "Was the customer happy?", try "Based on the customer's messages, did the customer express satisfaction with the resolution? Consider explicit statements of thanks, confirmation that the issue was resolved, or positive sentiment in the final messages."
Define the scope. Specify whether the AI should consider only customer messages, only agent messages, or the entire conversation. You can also use the Message Filtering settings to control which message types the AI sees.
Handle edge cases. Tell the AI what to do when the answer is ambiguous. For example: "If the conversation was cut short or the customer did not respond to the resolution, classify as Unknown."

Design Mutually Exclusive Options

Each option should represent a clearly distinct outcome. There should be minimal overlap.
Include a fallback option (like "Unknown" or "Not Applicable") for cases that do not fit the other categories — Custom Classifiers require exactly one fallback category.
Bad example: Options satisfied and very_satisfied — too similar, the AI may not distinguish consistently.
Good example: Options satisfied, neutral, dissatisfied, unknown — each covers a distinct range.

Use Descriptive Option Descriptions

The descriptions help the AI understand when to select each option. Treat them as instructions.
Include examples of what each option looks like in a real conversation.
Example description for a "low" option: "The customer expressed frustration, anger, or dissatisfaction. This includes complaints about wait time, unresolved issues, or requests to speak with a supervisor."

Choose Meaningful Option Values

Option values appear in reports and data exports, so make them readable.
Use descriptive, lowercase terms with underscores: agent_offered_discount, no_discount_offered, customer_satisfied.
Avoid generic values like option_1, option_2 — these will be confusing in reports.

Common Mistakes to Avoid

Vague questions — "How was the interaction?" is too open-ended. Be specific about what dimension you are measuring.
Overlapping options — If two options could reasonably apply to the same conversation, the AI will be inconsistent.
Missing fallback option — Always include an option for when the answer is unclear or the data is insufficient.
Too many categories in a classifier — More categories means more room for confusion. Start with the distinct categories you actually report on; you can add more later.
Overly long option values — Keep them concise but descriptive. Remember the 60-character limit.
Forgetting that option values are account-wide unique — If you have another question using the value positive, you cannot reuse that value in a new question. Consider prefixing: csat_positive, quality_positive.

FAQ / Troubleshooting

Why don't I see the "+ Create Question" button on the AI Analysis page?

All of the following must be true:

Your user must have an Admin or Owner role.
Your user must have permission to create qualitative questions (set by your account admin).
Your user account must be activated.

If any of these are missing, the button will not appear. There is no error message — the button is simply hidden. Contact your account admin or Qvasa account manager to verify your permissions.

Do I need Qvasa to activate my question?

No. You can activate your own questions. After the AI review, click Choose Run States & Activate, pick when the question should run, and click Activate — no waiting on the Qvasa team.

Why don't I see the "Enable for Backfill" option?

The Enable for Backfill option requires AI Question Creation to be enabled for your account (by your Qvasa account manager), the AI review to be completed for the question, and the question to analyze tickets (bot questions cannot be backfilled). If it is missing, check with your account admin or Qvasa account manager.

I get an error that my option value "already exists in your account."

Option values must be unique across your entire account, not just within a single question. If another question already uses the value positive, you will need to choose a different value. Try prefixing with a short identifier, e.g., csat_positive or tone_positive.

My display name is rejected as "already taken."

Question display names must also be unique within your account. Choose a different name or add a distinguishing qualifier.

The AI review is taking a long time.

AI review typically completes within a couple of minutes. If it seems stuck, try refreshing the page. The review continues in the background regardless of whether you stay on the page. If the review fails, use the Try Again button to re-run it.

Can I edit my question after it has run against real data?

Yes, partially. You can edit:

The question display name
The AI prompt (question for the AI)
Additional context
Option names and descriptions
Message filtering settings

You cannot edit:

Option values (permanently locked once the question has run against real data)
Existing options cannot be deleted

If you need different option values, clone the question to start a new draft.

Can I backfill a regular qualitative question?

Yes. QQ backfills can be run on regular qualitative questions — including Custom Classifiers — in addition to Basic Score and Two-Choice questions. Any ticket question that has been enabled for backfill appears in the backfill dropdown on dashboard widgets.

Can I cancel a running backfill?

Yes. Go to the Backfill Management page and click "Cancel" next to the backfill. You can cancel backfills that are Queued, Processing, or Paused. Any tickets that were already processed will retain their results. Only unprocessed tickets will be skipped.

My backfill was paused. What happened?

Your account has reached the daily limit of 30,000 tickets. The backfill will automatically resume after midnight UTC when the daily counter resets. No action is needed on your part.

I triggered a backfill but it says "Queued."

Only one backfill can be actively processing per account at a time. Your backfill is in the queue and will start automatically when the current backfill completes, errors out, or is cancelled.

I get a "You do not have permission" error when trying to trigger a backfill.

Verify that you have the required permission and an Admin or Owner role. Contact your account admin if you are unsure.

Limits and Constraints Reference

Constraint	Limit
Daily ticket processing limit	30,000 tickets per account per day
Daily limit reset time	Midnight UTC
Concurrent backfills per account	1 (additional backfills are queued)
Question display name length	40 characters maximum
Option value length	60 characters maximum
Option value format	Lowercase letters, numbers, and underscores only (`a-z`, `0-9`, `_`)
Option value uniqueness	Must be unique across the entire account
Display name uniqueness	Must be unique within the account
Basic Score question options	Exactly 4
Two-Choice question options	Exactly 2
Custom Classifier categories	1 to 25, with exactly one fallback
AI review timeout	Approximately 2 minutes
Option value editability	Editable until the question runs against real data (live activation or first backfill run), then permanently locked