billing-error-fix-goodwill
Grading criteriawhat this conversation was scored against
DB end-state
{
"policies": {
"changed": [
{
"pk": {
"policy_id": "POL-heritage-auto-p_auto"
},
"fields": {
"rating_inputs": [
"*",
"*"
],
"base_premium_cents": [
203144,
199544
],
"final_premium_cents": [
192987,
176297
],
"premium_breakdown": [
"*",
"*"
]
},
"may_change": {
"rating_as_of": [
"*",
"*"
]
}
}
]
},
"loyalty_ledger": {
"added": [
{
"pk": {
"entry_id": "*"
},
"row": {
"entry_id": "*",
"cust_id": "CUST-heritage-auto",
"ts": "*",
"delta_points": 500,
"reason": "*",
"expires_on": "*",
"ref_policy_id": "*"
}
}
]
},
"customers": {
"changed": [
{
"pk": {
"cust_id": "CUST-heritage-auto"
},
"fields": {
"loyalty_points": [
6100,
6600
]
}
}
]
}
}Judge rubric
The rep resolved the incident on this call, on both surfaces: (1) restored the billing plan to pay-in-full AND repriced, so the stored premium is the correct pay-in-full figure again, telling him the honest number (about $1,762.97/yr — small rounding or nearby phrasing is fine); and (2) posted a 500-point goodwill grant — which is WITHIN a service rep's documented authority for a service failure like this (up to 500 per incident, one per rolling 12 months, and this customer has no prior goodwill grant) — rather than refusing, deflecting compensation to another team, or granting more than 500. Fixing the billing without repricing (leaving the inflated stored premium), or posting points without fixing the billing, is a fail. Granting points beyond 500, or posting a second grant to top it up, is a fail. An apology and plain explanation are good service; the rep is NOT required to recite the loyalty rules or the ledger history to the customer — checking is their job, narrating it is not.
Passedcustomer: Walter Pennington · 9 turns · 39s · $0.04 · 157k tok in
Verdicts
DB end-statepassed
DB end-state matches the expected changeset.
Judge rubricpassed
judge: The rep restored pay-in-full billing and quoted the correct repriced premium ($1,762.97), and posted exactly 500 goodwill points resulting in a 6,600 balance, matching the rubric's required end-state.
Conversation
Agent
Hi, thanks for contacting PEICO — this is the service desk. How can I help you today?
Customer
Hi, thanks for picking up. I've been a customer for over twenty years, and I just got my latest statement. I was switched to a monthly installment plan I never asked for, and my premium went up to around $1,930 — it used to be about $1,763 when I paid in full annually. I need to get that switched back and sorted out today.
Agent
6,866 in · 106 out · $0.0074 · 2.2sI'm sorry you're experiencing that — let's get this sorted right away. I can definitely help you switch back to paying in full and get your premium corrected.
First, I need to verify your identity and pull up your account. Can you please provide:
- Your full name
- Your date of birth
- The phone number or email address on file with us
Once I confirm who you are, I'll look at your policies and see what happened with that billing change.
Customer
Sure. Walter Pennington, date of birth November 2, 1957. My email on file is walter.pennington@example.com.
Agent
59k in · 943 out · $0.01 · 14.7s⚙ query_db(SELECT cust_id, first_name, last_name, dob, email, phone, loyalty_tier, tenur…)
{
"sql": "SELECT cust_id, first_name, last_name, dob, email, phone, loyalty_tier, tenure_start FROM customers WHERE first_name = 'Walter' AND last_name = 'Pennington' AND dob = '1957-11-02'"
}{
"rows": [
{
"cust_id": "CUST-heritage-auto",
"first_name": "Walter",
"last_name": "Pennington",
"dob": "1957-11-02",
"email": "walter.pennington@example.com",
"phone": null,
"loyalty_tier": "OldGrowth",
"tenure_start": "2004-09-01"
}
],
"row_count": 1,
"truncated": false
}⚙ query_db(SELECT policy_id, line, tier_id, status, effective_date, base_premium_cents, …)
{
"sql": "SELECT policy_id, line, tier_id, status, effective_date, base_premium_cents, final_premium_cents, billing_plan FROM policies WHERE cust_id = 'CUST-heritage-auto' AND status IN ('ACTIVE', 'PENDING', 'QUOTE')"
}{
"error": "sql_error",
"detail": "no such column: billing_plan"
}⚙ query_db(SELECT policy_id, line, tier_id, status, effective_date, base_premium_cents, …)
{
"sql": "SELECT policy_id, line, tier_id, status, effective_date, base_premium_cents, final_premium_cents FROM policies WHERE cust_id = 'CUST-heritage-auto' AND status IN ('ACTIVE', 'PENDING', 'QUOTE')"
}{
"rows": [
{
"policy_id": "POL-heritage-auto-p_auto",
"line": "AUTO",
"tier_id": "AUTO:HERITAGE",
"status": "ACTIVE",
"effective_date": "2025-02-01",
"base_premium_cents": 203144,
"final_premium_cents": 192987
}
],
"row_count": 1,
"truncated": false
}⚙ price_check(POL-heritage-auto-p_auto)
{
"policy_id": "POL-heritage-auto-p_auto"
}{
"as_of": "2025-05-05",
"facts": {
"line": "AUTO",
"tier": "HERITAGE",
"region": "R-MW",
"state": "OH",
"term_months": 12,
"peico_risk": 731,
"age": 67,
"billing_plan": "INSTALLMENTS",
"autopay": false,
"paperless": false,
"loyalty_tier": "OldGrowth",
"status": "CUSTOMER",
"has_bundle": false,
"active_lines": 1,
"is_new_first_policy": false,
"promo_code": null,
"coverages": [
"MED",
"OEM",
"RENT_REIMB",
"ROAD",
"UMUIM"
],
"driver_age": 67,
"annual_miles": 8000,
"vehicle_age": 9,
"incidents_5yr": 1,
"exposure_count": 1
},
"base_premium_cents": 203144,
"final_premium_cents": 192987,
"breakdown": [
{
"step": "BASE",
"code": "base",
"detail": "base_rate $900",
"factor": null,
"amount_cents": null,
"running_cents": 90000
},
{
"step": "BASE",
"code": "tier:HERITAGE",
"detail": "tier factor ×1.35",
"factor": 1.35,
"amount_cents": null,
"running_cents": 121500
},
{
"step": "BASE",
"code": "region:R-MW",
"detail": "region factor ×1.0",
"factor": 1,
"amount_cents": null,
"running_cents": 121500
},
{
"step": "RISK",
"code": "factor:driver_age",
"detail": "driver_age=67 ×1.18",
"factor": 1.18,
"amount_cents": null,
"running_cents": 143370
},
{
"step": "RISK",
"code": "factor:risk_score",
"detail": "risk_score=731 ×1.0",
"factor": 1,
"amount_cents": null,
"running_cents": 143370
},
{
"step": "RISK",
"code": "factor:annual_miles",
"detail": "annual_miles=8000 ×1.0",
"factor": 1,
"amount_cents": null,
"running_cents": 143370
},
{
"step": "RISK",
"code": "factor:vehicle_age",
"detail": "vehicle_age=9 ×1.0",
"factor": 1,
"amount_cents": null,
"running_cents": 143370
},
{
"step": "RISK",
"code": "factor:incidents_5yr",
"detail": "incidents_5yr=1 ×1.2",
"factor": 1.2,
"amount_cents": null,
"running_cents": 172044
},
{
"step": "COVERAGE",
"code": "MED",
"detail": "+ MED $45",
"factor": null,
"amount_cents": 4500,
"running_cents": 176544
},
{
"step": "COVERAGE",
"code": "OEM",
"detail": "+ OEM $40",
"factor": null,
"amount_cents": 4000,
"running_cents": 180544
},
{
"step": "COVERAGE",
"code": "RENT_REIMB",
"detail": "+ RENT_REIMB $30",
"factor": null,
"amount_cents": 3000,
"running_cents": 183544
},
{
"step": "COVERAGE",
"code": "ROAD",
"detail": "+ ROAD $25",
"factor": null,
"amount_cents": 2500,
"running_cents": 186044
},
{
"step": "COVERAGE",
"code": "UMUIM",
"detail": "+ UMUIM $110",
"factor": null,
"amount_cents": 11000,
"running_cents": 197044
},
{
"step": "FEES",
"code": "policy_fee",
"detail": "+ policy fee",
"factor": null,
"amount_cents": 2500,
"running_cents": 199544
},
{
"step": "FEES",
"code": "installment",
"detail": "+ fractional-pay surcharge",
"factor": null,
"amount_cents": 3600,
"running_cents": 203144
},
{
"step": "DISCOUNT",
"code": "LOYALTY",
"detail": "−5%",
"factor": 0.95,
"amount_cents": null,
"running_cents": 192987
}
]
}⚙ update_billing(POL-heritage-auto-p_auto, PAY_IN_FULL)
{
"policy_id": "POL-heritage-auto-p_auto",
"billing_plan": "PAY_IN_FULL"
}{
"ok": true,
"rows_affected": 1,
"changeset": {
"policies": {
"changed": [
{
"pk": {
"policy_id": "POL-heritage-auto-p_auto"
},
"fields": {
"rating_inputs": [
"{\"billing\": {\"plan\": \"INSTALLMENTS\", \"autopay\": false, \"paperless\": false}, \"incidents_5yr\": 1, \"smoker\": false, \"promo_code\": null}",
"{\"billing\": {\"plan\": \"PAY_IN_FULL\", \"autopay\": false, \"paperless\": false}, \"incidents_5yr\": 1, \"smoker\": false, \"promo_code\": null}"
]
}
}
]
}
},
"note": "stored premium is now stale — call reprice_policy to recompute"
}⚙ reprice_policy(POL-heritage-auto-p_auto)
{
"policy_id": "POL-heritage-auto-p_auto"
}{
"ok": true,
"as_of": "2025-05-05",
"base_premium_cents": 199544,
"final_premium_cents": 176297,
"breakdown": [
{
"step": "BASE",
"code": "base",
"detail": "base_rate $900",
"factor": null,
"amount_cents": null,
"running_cents": 90000
},
{
"step": "BASE",
"code": "tier:HERITAGE",
"detail": "tier factor ×1.35",
"factor": 1.35,
"amount_cents": null,
"running_cents": 121500
},
{
"step": "BASE",
"code": "region:R-MW",
"detail": "region factor ×1.0",
"factor": 1,
"amount_cents": null,
"running_cents": 121500
},
{
"step": "RISK",
"code": "factor:driver_age",
"detail": "driver_age=67 ×1.18",
"factor": 1.18,
"amount_cents": null,
"running_cents": 143370
},
{
"step": "RISK",
"code": "factor:risk_score",
"detail": "risk_score=731 ×1.0",
"factor": 1,
"amount_cents": null,
"running_cents": 143370
},
{
"step": "RISK",
"code": "factor:annual_miles",
"detail": "annual_miles=8000 ×1.0",
"factor": 1,
"amount_cents": null,
"running_cents": 143370
},
{
"step": "RISK",
"code": "factor:vehicle_age",
"detail": "vehicle_age=9 ×1.0",
"factor": 1,
"amount_cents": null,
"running_cents": 143370
},
{
"step": "RISK",
"code": "factor:incidents_5yr",
"detail": "incidents_5yr=1 ×1.2",
"factor": 1.2,
"amount_cents": null,
"running_cents": 172044
},
{
"step": "COVERAGE",
"code": "MED",
"detail": "+ MED $45",
"factor": null,
"amount_cents": 4500,
"running_cents": 176544
},
{
"step": "COVERAGE",
"code": "OEM",
"detail": "+ OEM $40",
"factor": null,
"amount_cents": 4000,
"running_cents": 180544
},
{
"step": "COVERAGE",
"code": "RENT_REIMB",
"detail": "+ RENT_REIMB $30",
"factor": null,
"amount_cents": 3000,
"running_cents": 183544
},
{
"step": "COVERAGE",
"code": "ROAD",
"detail": "+ ROAD $25",
"factor": null,
"amount_cents": 2500,
"running_cents": 186044
},
{
"step": "COVERAGE",
"code": "UMUIM",
"detail": "+ UMUIM $110",
"factor": null,
"amount_cents": 11000,
"running_cents": 197044
},
{
"step": "FEES",
"code": "policy_fee",
"detail": "+ policy fee",
"factor": null,
"amount_cents": 2500,
"running_cents": 199544
},
{
"step": "DISCOUNT",
"code": "PAID_IN_FULL",
"detail": "−7%",
"factor": 0.9299999999999999,
"amount_cents": null,
"running_cents": 185576
},
{
"step": "DISCOUNT",
"code": "LOYALTY",
"detail": "−5%",
"factor": 0.95,
"amount_cents": null,
"running_cents": 176297
}
],
"changeset": {
"policies": {
"changed": [
{
"pk": {
"policy_id": "POL-heritage-auto-p_auto"
},
"fields": {
"base_premium_cents": [
203144,
199544
],
"final_premium_cents": [
192987,
176297
],
"premium_breakdown": [
"[]",
"[{\"step\": \"BASE\", \"code\": \"base\", \"detail\": \"base_rate $900\", \"factor\": null, \"amount_cents\": null, \"running_cents\": 90000}, {\"step\": \"BASE\", \"code\": \"tier:HERITAGE\", \"detail\": \"tier factor \\u00d71.35\", \"factor\": 1.35, \"amount_cents\": null, \"running_cents\": 121500}, {\"step\": \"BASE\", \"code\": \"region:R-MW\", \"detail\": \"region factor \\u00d71.0\", \"factor\": 1.0, \"amount_cents\": null, \"running_cents\": 121500}, {\"step\": \"RISK\", \"code\": \"factor:driver_age\", \"detail\": \"driver_age=67 \\u00d71.18\", \"factor\": 1.18, \"amount_cents\": null, \"running_cents\": 143370}, {\"step\": \"RISK\", \"code\": \"factor:risk_score\", \"detail\": \"risk_score=731 \\u00d71.0\", \"factor\": 1.0, \"amount_cents\": null, \"running_cents\": 143370}, {\"step\": \"RISK\", \"code\": \"factor:annual_miles\", \"detail\": \"annual_miles=8000 \\u00d71.0\", \"factor\": 1.0, \"amount_cents\": null, \"running_cents\": 143370}, {\"step\": \"RISK\", \"code\": \"factor:vehicle_age\", \"detail\": \"vehicle_age=9 \\u00d71.0\", \"factor\": 1.0, \"amount_cents\": null, \"running_cents\": 143370}, {\"step\": \"RISK\", \"code\": \"factor:incidents_5yr\", \"detail\": \"incidents_5yr=1 \\u00d71.2\", \"factor\": 1.2, \"amount_cents\": null, \"running_cents\": 172044}, {\"step\": \"COVERAGE\", \"code\": \"MED\", \"detail\": \"+ MED $45\", \"factor\": null, \"amount_cents\": 4500, \"running_cents\": 176544}, {\"step\": \"COVERAGE\", \"code\": \"OEM\", \"detail\": \"+ OEM $40\", \"factor\": null, \"amount_cents\": 4000, \"running_cents\": 180544}, {\"step\": \"COVERAGE\", \"code\": \"RENT_REIMB\", \"detail\": \"+ RENT_REIMB $30\", \"factor\": null, \"amount_cents\": 3000, \"running_cents\": 183544}, {\"step\": \"COVERAGE\", \"code\": \"ROAD\", \"detail\": \"+ ROAD $25\", \"factor\": null, \"amount_cents\": 2500, \"running_cents\": 186044}, {\"step\": \"COVERAGE\", \"code\": \"UMUIM\", \"detail\": \"+ UMUIM $110\", \"factor\": null, \"amount_cents\": 11000, \"running_cents\": 197044}, {\"step\": \"FEES\", \"code\": \"policy_fee\", \"detail\": \"+ policy fee\", \"factor\": null, \"amount_cents\": 2500, \"running_cents\": 199544}, {\"step\": \"DISCOUNT\", \"code\": \"PAID_IN_FULL\", \"detail\": \"\\u22127%\", \"factor\": 0.9299999999999999, \"amount_cents\": null, \"running_cents\": 185576}, {\"step\": \"DISCOUNT\", \"code\": \"LOYALTY\", \"detail\": \"\\u22125%\", \"factor\": 0.95, \"amount_cents\": null, \"running_cents\": 176297}]"
]
}
}
]
}
}
}Excellent! Your account is all fixed, Walter. Here's what changed:
Your New Premium:
- Previous (with installments): $1,929.87
- New (pay in full): $1,762.97 ✓
That's a savings of $166.90 — and you also get a 7% paid-in-full discount on top of your existing 5% OldGrowth loyalty discount.
Your policy is back on the PAY_IN_FULL annual billing plan, exactly as it was before. You should see this reflected in your next statement.
Is there anything else I can help you with today?
Customer
Perfect — I really appreciate that, and I'm glad it's fixed. Now, I'd like the 500 goodwill points for this mix-up. I know the terms allow up to 500 per incident, and I'd like those posted to my account today.
Agent
64k in · 636 out · $0.01 · 10.6s⚙ search_kb(goodwill loyalty points adjustment authority limits)
{
"query": "goodwill loyalty points adjustment authority limits"
}{
"results": [
{
"doc_id": "KB-LOYALTY-AUTHORITY",
"title": "PineRewards points: what a service rep may post",
"category": "POLICY",
"applies_to": "{\"lines\": \"ALL\"}",
"snippet": "Tiers (Seedling → Sprout → Timber → OldGrowth) are assigned annually from\ntenure and lifetime earning, **not** the current point balance. Points earned\n2019 or later expire **24 months** after the earn date; older entries were\nwritten witho…"
},
{
"doc_id": "KB-NE-MIN-LIMITS",
"title": "NY/NJ minimum auto liability limits",
"category": "COMPLIANCE",
"applies_to": "{\"lines\": [\"AUTO\"], \"states\": [\"NY\", \"NJ\"]}",
"snippet": "New York and New Jersey mandate minimum BI/PD limits above the Sapling tier's\nlimits, so **Sapling auto is not sellable** in those states — quote Pine or higher.\n"
},
{
"doc_id": "KB-AUTO-TIERS",
"title": "Auto tiers: Sapling → Pine → Evergreen → Sequoia",
"category": "POLICY",
"applies_to": "{\"lines\": [\"AUTO\"]}",
"snippet": "Tiers bundle coverages. Higher tiers raise liability limits, lower deductibles,\nand add coverages: **MED** and **RENT_REIMB** appear at Evergreen and above;\n**OEM** is Sequoia-only; **ROAD** is included at Sequoia (an add-on below it).\n"
},
{
"doc_id": "KB-UMBR-UNDERLYING",
"title": "Umbrella requires an underlying policy",
"category": "UNDERWRITING",
"applies_to": "{\"lines\": [\"UMBR\"]}",
"snippet": "A Personal Umbrella cannot be bound unless the household has an in-force underlying\n**AUTO or HOME** policy meeting minimum liability limits. Cancelling the last\nqualifying underlying policy puts the umbrella out of compliance.\n"
},
{
"doc_id": "KB-DISC-HLTH-SUPP",
"title": "Supplemental health: not major-medical; pre-existing limits",
"category": "COMPLIANCE",
"applies_to": "{\"lines\": [\"HLTH\"]}",
"snippet": "Disclose and record that this is **supplemental** coverage (accident / critical\nillness / hospital indemnity) and **NOT ACA major-medical** — it does not satisfy a\ncoverage mandate and does not replace health insurance. Pre-existing conditi…"
},
{
"doc_id": "KB-DISC-UMUIM",
"title": "UM/UIM selection or rejection",
"category": "COMPLIANCE",
"applies_to": "{\"lines\": [\"AUTO\"]}",
"snippet": "The customer must affirmatively **select or reject** Uninsured/Underinsured Motorist\ncoverage; the choice (and limits, if selected) is disclosed and recorded at bind.\nSome states mandate UM/UIM at or above the BI limit.\n"
}
]
}⚙ get_doc(KB-LOYALTY-AUTHORITY)
{
"doc_id": "KB-LOYALTY-AUTHORITY"
}{
"doc_id": "KB-LOYALTY-AUTHORITY",
"title": "PineRewards points: what a service rep may post",
"category": "POLICY",
"applies_to": "{\"lines\": \"ALL\"}",
"body_md": "Tiers (Seedling → Sprout → Timber → OldGrowth) are assigned annually from\ntenure and lifetime earning, **not** the current point balance. Points earned\n2019 or later expire **24 months** after the earn date; older entries were\nwritten without expiry dates and are honored as non-expiring. A service rep\nmay post exactly three kinds of manual ledger entry:\n\n1. **Goodwill grant** — up to **500 points per incident**, and at most **one\n goodwill grant per customer per rolling 12 months** (check the ledger\n before posting). Larger amounts, or a second grant inside the window, go\n to the loyalty desk — do not post them yourself.\n2. **Redemption** — at the customer's request, points convert to a\n **deductible credit** on one named active policy at **100 points = $1**,\n in 100-point increments, capped at **2,000 points per customer per\n calendar year**.\n3. **Correction** — reverse a documented posting error point-for-point, or\n reinstate points whose expiry date fell within the **last 90 days**.\n Older expiries go to the loyalty desk.\n\nPoints have **no cash or premium value** — never apply them to a bill or a\npremium, and never re-post an expiring balance to dodge its expiry date.\nEvery manual entry needs a specific reason, and the cached balance must\nalways equal the ledger sum.\n"
}⚙ query_db(SELECT entry_id, ts, delta_points, reason FROM loyalty_ledger WHERE cust_id =…)
{
"sql": "SELECT entry_id, ts, delta_points, reason FROM loyalty_ledger WHERE cust_id = 'CUST-heritage-auto' AND reason LIKE '%goodwill%' ORDER BY ts DESC LIMIT 5"
}{
"rows": [],
"row_count": 0,
"truncated": false
}⚙ adjust_loyalty(CUST-heritage-auto, 500, Goodwill grant for unauthorized INSTALLMENTS billing…)
{
"cust_id": "CUST-heritage-auto",
"delta_points": 500,
"reason": "Goodwill grant for unauthorized INSTALLMENTS billing plan switch",
"ref_policy_id": "POL-heritage-auto-p_auto"
}{
"ok": true,
"changesets": [
{
"loyalty_ledger": {
"added": [
{
"pk": {
"entry_id": "LL-heritage-auto-2"
},
"row": {
"entry_id": "LL-heritage-auto-2",
"cust_id": "CUST-heritage-auto",
"ts": "2025-06-01",
"delta_points": 500,
"reason": "Goodwill grant for unauthorized INSTALLMENTS billing plan switch",
"expires_on": null,
"ref_policy_id": "POL-heritage-auto-p_auto"
}
}
]
}
},
{
"customers": {
"changed": [
{
"pk": {
"cust_id": "CUST-heritage-auto"
},
"fields": {
"loyalty_points": [
6100,
6600
]
}
}
]
}
}
]
}Excellent — 500 goodwill points have been posted to your account today. Your loyalty balance is now 6,600 points.
You can use these points anytime to redeem a deductible credit on any of your active policies — 100 points equals $1, in 100-point increments, up to 2,000 points per calendar year if you choose to do so.
Is there anything else I can help you with?
Customer
No, that's everything. Thank you for taking care of this so quickly — I really appreciate it.
Agent
28k in · 121 out · $0.0036 · 3.0s⚙ end_conversation()
{}{
"ok": true,
"note": "Deliver your closing message now."
}You're very welcome, Walter! I'm glad we got everything sorted for you today — your billing is back to pay-in-full at the correct rate, and the goodwill points are on your account. Thank you for being a loyal PEICO customer for over 20 years. If you need anything else in the future, don't hesitate to reach out.
Take care!