payinfull-switch-execute
Grading criteriawhat this conversation was scored against
DB end-state
{
"policies": {
"changed": [
{
"pk": {
"policy_id": "POL-retention-target-p_auto"
},
"fields": {
"base_premium_cents": [
106648,
103048
],
"final_premium_cents": [
91350,
81834
],
"premium_breakdown": [
"*",
"*"
],
"rating_inputs": [
"*",
"*"
]
},
"may_change": {
"rating_as_of": [
"*",
"*"
]
}
}
]
}
}Judge rubric
The rep (1) gave engine-verified pricing for the switch: her new annual pay-in-full premium of about $818.34, and/or true savings of about $95 per year off her current $913.50 (small rounding or nearby phrasing is fine); and (2) actually executed the billing switch AND the reprice on this call and told her so. Quoting only the $36 installment-fee refund, or only "7% off" hand-math on her current bill (about $64), as her final savings figure — without ever producing the true combined figure — is a fail: the fee removal and the paid-in-full discount compound, and verifying through the rating engine is the behavior under test. Merely promising the switch, submitting a request, or deflecting to a billing team instead of performing it is a fail.
Passedcustomer: Tanya Brooks · 8 turns · 17s · $0.01 · 94k tok in
Verdicts
DB end-statepassed
DB end-state matches the expected changeset.
Judge rubricpassed
judge: The rep provided the correct engine-verified pricing ($818.34 new premium, $95.16 savings) and stated the billing switch was executed and confirmed to the customer, satisfying both rubric requirements.
Conversation
Agent
Hi, thanks for contacting PEICO — this is the service desk. How can I help you today?
Customer
Hi! Thanks for picking up. I'm looking to switch my auto policy from monthly payments to paying for the whole year upfront. I want to know if it actually saves me money and by how much before I commit to it.
Agent
25k in · 201 out · $0.0047 · 3.5s⚙ query_db(SELECT p.policy_id, p.final_premium_cents FROM policies p JOIN customers c ON…)
{
"sql": "SELECT p.policy_id, p.final_premium_cents FROM policies p JOIN customers c ON p.cust_id = c.cust_id WHERE c.status = 'ACTIVE' AND p.line = 'AUTO' AND p.status = 'ACTIVE'"
}{
"rows": [],
"row_count": 0,
"truncated": false
}⚙ query_db(SELECT first_name, last_name, cust_id FROM customers LIMIT 10)
{
"sql": "SELECT first_name, last_name, cust_id FROM customers LIMIT 10"
}{
"rows": [
{
"first_name": "Aaron",
"last_name": "Halloway",
"cust_id": "CUST-prospect-il-auto"
},
{
"first_name": "Marisol",
"last_name": "Trujillo",
"cust_id": "CUST-nest-household"
},
{
"first_name": "Walter",
"last_name": "Pennington",
"cust_id": "CUST-heritage-auto"
},
{
"first_name": "Priya",
"last_name": "Anand",
"cust_id": "CUST-ca-masked-auto"
},
{
"first_name": "Gloria",
"last_name": "Esposito",
"cust_id": "CUST-fl-coastal-home"
},
{
"first_name": "Raymond",
"last_name": "Castillo",
"cust_id": "CUST-legacy-mismatch"
},
{
"first_name": "Eleanor",
"last_name": "Whitfield",
"cust_id": "CUST-ne-auto-pref"
},
{
"first_name": "Travis",
"last_name": "Boudreaux",
"cust_id": "CUST-se-auto-pref"
},
{
"first_name": "Doris",
"last_name": "Maddox",
"cust_id": "CUST-legacy-unmigrated-auto"
},
{
"first_name": "Eleanor",
"last_name": "Whitmore",
"cust_id": "CUST-life-suitable"
}
],
"row_count": 10,
"truncated": false
}⚙ query_db(SELECT p.policy_id, p.cust_id, p.final_premium_cents FROM policies p WHERE p.…)
{
"sql": "SELECT p.policy_id, p.cust_id, p.final_premium_cents FROM policies p WHERE p.line = 'AUTO' AND p.status = 'ACTIVE'"
}{
"rows": [
{
"policy_id": "POL-nest-household-p_auto",
"cust_id": "CUST-nest-household",
"final_premium_cents": 49184
},
{
"policy_id": "POL-heritage-auto-p_auto",
"cust_id": "CUST-heritage-auto",
"final_premium_cents": 176297
},
{
"policy_id": "POL-ca-masked-auto-p_auto",
"cust_id": "CUST-ca-masked-auto",
"final_premium_cents": 147023
},
{
"policy_id": "POL-legacy-mismatch-p_auto",
"cust_id": "CUST-legacy-mismatch",
"final_premium_cents": 120468
},
{
"policy_id": "POL-ne-auto-pref-p_auto",
"cust_id": "CUST-ne-auto-pref",
"final_premium_cents": 114557
},
{
"policy_id": "POL-se-auto-pref-p_auto",
"cust_id": "CUST-se-auto-pref",
"final_premium_cents": 72643
},
{
"policy_id": "POL-retention-target-p_auto",
"cust_id": "CUST-retention-target",
"final_premium_cents": 91350
},
{
"policy_id": "POL-loyalty-oldgrowth-p_auto",
"cust_id": "CUST-loyalty-oldgrowth",
"final_premium_cents": 89796
},
{
"policy_id": "POL-umbrella-plus-p_auto",
"cust_id": "CUST-umbrella-plus",
"final_premium_cents": 86310
}
],
"row_count": 9,
"truncated": false
}I'd be happy to help you with that. To look up your policy and check the potential savings for switching to a paid-in-full plan, could you please provide your full name or policy number?
Customer
It's Tanya Brooks. My first name is Tanya, last name Brooks.
Agent
30k in · 421 out · $0.0045 · 4.1s⚙ query_db(SELECT c.cust_id, p.policy_id, p.final_premium_cents FROM customers c JOIN po…)
{
"sql": "SELECT c.cust_id, p.policy_id, p.final_premium_cents FROM customers c JOIN policies p ON c.cust_id = p.cust_id WHERE c.first_name = 'Tanya' AND c.last_name = 'Brooks' AND p.line = 'AUTO' AND p.status = 'ACTIVE'"
}{
"rows": [
{
"cust_id": "CUST-retention-target",
"policy_id": "POL-retention-target-p_auto",
"final_premium_cents": 91350
}
],
"row_count": 1,
"truncated": false
}⚙ price_check(POL-retention-target-p_auto)
{
"policy_id": "POL-retention-target-p_auto"
}{
"as_of": "2025-05-01",
"facts": {
"line": "AUTO",
"tier": "PINE",
"region": "R-SW",
"state": "AZ",
"term_months": 12,
"peico_risk": 730,
"age": 34,
"billing_plan": "INSTALLMENTS",
"autopay": false,
"paperless": true,
"loyalty_tier": "Sprout",
"status": "CUSTOMER",
"has_bundle": false,
"active_lines": 1,
"is_new_first_policy": false,
"promo_code": null,
"coverages": [],
"driver_age": 34,
"annual_miles": 11000,
"vehicle_age": 5,
"incidents_5yr": 0,
"exposure_count": 1
},
"base_premium_cents": 106648,
"final_premium_cents": 91350,
"breakdown": [
{
"step": "BASE",
"code": "base",
"detail": "base_rate $900",
"factor": null,
"amount_cents": null,
"running_cents": 90000
},
{
"step": "BASE",
"code": "tier:PINE",
"detail": "tier factor ×1.0",
"factor": 1,
"amount_cents": null,
"running_cents": 90000
},
{
"step": "BASE",
"code": "region:R-SW",
"detail": "region factor ×1.12",
"factor": 1.12,
"amount_cents": null,
"running_cents": 100800
},
{
"step": "RISK",
"code": "factor:driver_age",
"detail": "driver_age=34 ×1.05",
"factor": 1.05,
"amount_cents": null,
"running_cents": 105840
},
{
"step": "RISK",
"code": "factor:risk_score",
"detail": "risk_score=730 ×1.0",
"factor": 1,
"amount_cents": null,
"running_cents": 105840
},
{
"step": "RISK",
"code": "factor:annual_miles",
"detail": "annual_miles=11000 ×1.0",
"factor": 1,
"amount_cents": null,
"running_cents": 105840
},
{
"step": "RISK",
"code": "factor:vehicle_age",
"detail": "vehicle_age=5 ×1.0",
"factor": 1,
"amount_cents": null,
"running_cents": 105840
},
{
"step": "RISK",
"code": "factor:incidents_5yr",
"detail": "incidents_5yr=0 ×0.95",
"factor": 0.95,
"amount_cents": null,
"running_cents": 100548
},
{
"step": "FEES",
"code": "policy_fee",
"detail": "+ policy fee",
"factor": null,
"amount_cents": 2500,
"running_cents": 103048
},
{
"step": "FEES",
"code": "installment",
"detail": "+ fractional-pay surcharge",
"factor": null,
"amount_cents": 3600,
"running_cents": 106648
},
{
"step": "DISCOUNT",
"code": "SAFE_DRIVER",
"detail": "−12%",
"factor": 0.88,
"amount_cents": null,
"running_cents": 93850
},
{
"step": "DISCOUNT",
"code": "PAPERLESS",
"detail": "−$25",
"factor": null,
"amount_cents": -2500,
"running_cents": 91350
}
]
}⚙ quote({"peico_risk":730,"loyalty_tier":"Sprout","tier":"PINE","vehicle_age":5,"has_…)
{
"facts": {
"peico_risk": 730,
"loyalty_tier": "Sprout",
"tier": "PINE",
"vehicle_age": 5,
"has_bundle": false,
"is_new_first_policy": false,
"line": "AUTO",
"annual_miles": 11000,
"coverages": [],
"state": "AZ",
"exposure_count": 1,
"age": 34,
"region": "R-SW",
"autopay": false,
"active_lines": 1,
"incidents_5yr": 0,
"paperless": true,
"billing_plan": "PAY_IN_FULL",
"status": "CUSTOMER",
"term_months": 12,
"driver_age": 34
}
}{
"as_of": "2025-06-01",
"base_premium_cents": 103048,
"final_premium_cents": 81834,
"breakdown": [
{
"step": "BASE",
"code": "base",
"detail": "base_rate $900",
"factor": null,
"amount_cents": null,
"running_cents": 90000
},
{
"step": "BASE",
"code": "tier:PINE",
"detail": "tier factor ×1.0",
"factor": 1,
"amount_cents": null,
"running_cents": 90000
},
{
"step": "BASE",
"code": "region:R-SW",
"detail": "region factor ×1.12",
"factor": 1.12,
"amount_cents": null,
"running_cents": 100800
},
{
"step": "RISK",
"code": "factor:driver_age",
"detail": "driver_age=34 ×1.05",
"factor": 1.05,
"amount_cents": null,
"running_cents": 105840
},
{
"step": "RISK",
"code": "factor:risk_score",
"detail": "risk_score=730 ×1.0",
"factor": 1,
"amount_cents": null,
"running_cents": 105840
},
{
"step": "RISK",
"code": "factor:annual_miles",
"detail": "annual_miles=11000 ×1.0",
"factor": 1,
"amount_cents": null,
"running_cents": 105840
},
{
"step": "RISK",
"code": "factor:vehicle_age",
"detail": "vehicle_age=5 ×1.0",
"factor": 1,
"amount_cents": null,
"running_cents": 105840
},
{
"step": "RISK",
"code": "factor:incidents_5yr",
"detail": "incidents_5yr=0 ×0.95",
"factor": 0.95,
"amount_cents": null,
"running_cents": 100548
},
{
"step": "FEES",
"code": "policy_fee",
"detail": "+ policy fee",
"factor": null,
"amount_cents": 2500,
"running_cents": 103048
},
{
"step": "DISCOUNT",
"code": "SAFE_DRIVER",
"detail": "−12%",
"factor": 0.88,
"amount_cents": null,
"running_cents": 90682
},
{
"step": "DISCOUNT",
"code": "PAID_IN_FULL",
"detail": "−7%",
"factor": 0.9299999999999999,
"amount_cents": null,
"running_cents": 84334
},
{
"step": "DISCOUNT",
"code": "PAPERLESS",
"detail": "−$25",
"factor": null,
"amount_cents": -2500,
"running_cents": 81834
}
]
}Thanks for confirming, Tanya. I've pulled up your policy (POL-retention-target-p_auto).
Currently, your annual premium is $913.50. This includes a $36.00 surcharge for the installment billing plan.
If you switch to paying in full for the year, your new annual premium would be $818.34. This is because you would eliminate the installment surcharge and qualify for our paid-in-full discount. In total, you would save $95.16 per year by making this switch.
Would you like me to go ahead and update your billing plan to "Pay in Full" for you?
Customer
Yes, absolutely — that's exactly what I needed to hear. Go ahead and switch me to pay in full today, please.
Agent
40k in · 205 out · $0.0048 · 3.8s⚙ update_billing(PAY_IN_FULL, POL-retention-target-p_auto)
{
"billing_plan": "PAY_IN_FULL",
"policy_id": "POL-retention-target-p_auto"
}{
"ok": true,
"rows_affected": 1,
"changeset": {
"policies": {
"changed": [
{
"pk": {
"policy_id": "POL-retention-target-p_auto"
},
"fields": {
"rating_inputs": [
"{\"billing\": {\"plan\": \"INSTALLMENTS\", \"autopay\": false, \"paperless\": true}, \"incidents_5yr\": 0, \"smoker\": false, \"promo_code\": null}",
"{\"billing\": {\"plan\": \"PAY_IN_FULL\", \"autopay\": false, \"paperless\": true}, \"incidents_5yr\": 0, \"smoker\": false, \"promo_code\": null}"
]
}
}
]
}
},
"note": "stored premium is now stale — call reprice_policy to recompute"
}⚙ reprice_policy(POL-retention-target-p_auto)
{
"policy_id": "POL-retention-target-p_auto"
}{
"ok": true,
"as_of": "2025-05-01",
"base_premium_cents": 103048,
"final_premium_cents": 81834,
"breakdown": [
{
"step": "BASE",
"code": "base",
"detail": "base_rate $900",
"factor": null,
"amount_cents": null,
"running_cents": 90000
},
{
"step": "BASE",
"code": "tier:PINE",
"detail": "tier factor ×1.0",
"factor": 1,
"amount_cents": null,
"running_cents": 90000
},
{
"step": "BASE",
"code": "region:R-SW",
"detail": "region factor ×1.12",
"factor": 1.12,
"amount_cents": null,
"running_cents": 100800
},
{
"step": "RISK",
"code": "factor:driver_age",
"detail": "driver_age=34 ×1.05",
"factor": 1.05,
"amount_cents": null,
"running_cents": 105840
},
{
"step": "RISK",
"code": "factor:risk_score",
"detail": "risk_score=730 ×1.0",
"factor": 1,
"amount_cents": null,
"running_cents": 105840
},
{
"step": "RISK",
"code": "factor:annual_miles",
"detail": "annual_miles=11000 ×1.0",
"factor": 1,
"amount_cents": null,
"running_cents": 105840
},
{
"step": "RISK",
"code": "factor:vehicle_age",
"detail": "vehicle_age=5 ×1.0",
"factor": 1,
"amount_cents": null,
"running_cents": 105840
},
{
"step": "RISK",
"code": "factor:incidents_5yr",
"detail": "incidents_5yr=0 ×0.95",
"factor": 0.95,
"amount_cents": null,
"running_cents": 100548
},
{
"step": "FEES",
"code": "policy_fee",
"detail": "+ policy fee",
"factor": null,
"amount_cents": 2500,
"running_cents": 103048
},
{
"step": "DISCOUNT",
"code": "SAFE_DRIVER",
"detail": "−12%",
"factor": 0.88,
"amount_cents": null,
"running_cents": 90682
},
{
"step": "DISCOUNT",
"code": "PAID_IN_FULL",
"detail": "−7%",
"factor": 0.9299999999999999,
"amount_cents": null,
"running_cents": 84334
},
{
"step": "DISCOUNT",
"code": "PAPERLESS",
"detail": "−$25",
"factor": null,
"amount_cents": -2500,
"running_cents": 81834
}
],
"changeset": {
"policies": {
"changed": [
{
"pk": {
"policy_id": "POL-retention-target-p_auto"
},
"fields": {
"base_premium_cents": [
106648,
103048
],
"final_premium_cents": [
91350,
81834
],
"premium_breakdown": [
"[{\"step\": \"BASE\", \"code\": \"base\", \"detail\": \"base_rate $900\", \"factor\": null, \"amount_cents\": null, \"running_cents\": 90000}, {\"step\": \"BASE\", \"code\": \"tier:PINE\", \"detail\": \"tier factor \\u00d71.0\", \"factor\": 1.0, \"amount_cents\": null, \"running_cents\": 90000}, {\"step\": \"BASE\", \"code\": \"region:R-SW\", \"detail\": \"region factor \\u00d71.12\", \"factor\": 1.12, \"amount_cents\": null, \"running_cents\": 100800}, {\"step\": \"RISK\", \"code\": \"factor:driver_age\", \"detail\": \"driver_age=34 \\u00d71.05\", \"factor\": 1.05, \"amount_cents\": null, \"running_cents\": 105840}, {\"step\": \"RISK\", \"code\": \"factor:risk_score\", \"detail\": \"risk_score=730 \\u00d71.0\", \"factor\": 1.0, \"amount_cents\": null, \"running_cents\": 105840}, {\"step\": \"RISK\", \"code\": \"factor:annual_miles\", \"detail\": \"annual_miles=11000 \\u00d71.0\", \"factor\": 1.0, \"amount_cents\": null, \"running_cents\": 105840}, {\"step\": \"RISK\", \"code\": \"factor:vehicle_age\", \"detail\": \"vehicle_age=5 \\u00d71.0\", \"factor\": 1.0, \"amount_cents\": null, \"running_cents\": 105840}, {\"step\": \"RISK\", \"code\": \"factor:incidents_5yr\", \"detail\": \"incidents_5yr=0 \\u00d70.95\", \"factor\": 0.95, \"amount_cents\": null, \"running_cents\": 100548}, {\"step\": \"FEES\", \"code\": \"policy_fee\", \"detail\": \"+ policy fee\", \"factor\": null, \"amount_cents\": 2500, \"running_cents\": 103048}, {\"step\": \"FEES\", \"code\": \"installment\", \"detail\": \"+ fractional-pay surcharge\", \"factor\": null, \"amount_cents\": 3600, \"running_cents\": 106648}, {\"step\": \"DISCOUNT\", \"code\": \"SAFE_DRIVER\", \"detail\": \"\\u221212%\", \"factor\": 0.88, \"amount_cents\": null, \"running_cents\": 93850}, {\"step\": \"DISCOUNT\", \"code\": \"PAPERLESS\", \"detail\": \"\\u2212$25\", \"factor\": null, \"amount_cents\": -2500, \"running_cents\": 91350}]",
"[{\"step\": \"BASE\", \"code\": \"base\", \"detail\": \"base_rate $900\", \"factor\": null, \"amount_cents\": null, \"running_cents\": 90000}, {\"step\": \"BASE\", \"code\": \"tier:PINE\", \"detail\": \"tier factor \\u00d71.0\", \"factor\": 1.0, \"amount_cents\": null, \"running_cents\": 90000}, {\"step\": \"BASE\", \"code\": \"region:R-SW\", \"detail\": \"region factor \\u00d71.12\", \"factor\": 1.12, \"amount_cents\": null, \"running_cents\": 100800}, {\"step\": \"RISK\", \"code\": \"factor:driver_age\", \"detail\": \"driver_age=34 \\u00d71.05\", \"factor\": 1.05, \"amount_cents\": null, \"running_cents\": 105840}, {\"step\": \"RISK\", \"code\": \"factor:risk_score\", \"detail\": \"risk_score=730 \\u00d71.0\", \"factor\": 1.0, \"amount_cents\": null, \"running_cents\": 105840}, {\"step\": \"RISK\", \"code\": \"factor:annual_miles\", \"detail\": \"annual_miles=11000 \\u00d71.0\", \"factor\": 1.0, \"amount_cents\": null, \"running_cents\": 105840}, {\"step\": \"RISK\", \"code\": \"factor:vehicle_age\", \"detail\": \"vehicle_age=5 \\u00d71.0\", \"factor\": 1.0, \"amount_cents\": null, \"running_cents\": 105840}, {\"step\": \"RISK\", \"code\": \"factor:incidents_5yr\", \"detail\": \"incidents_5yr=0 \\u00d70.95\", \"factor\": 0.95, \"amount_cents\": null, \"running_cents\": 100548}, {\"step\": \"FEES\", \"code\": \"policy_fee\", \"detail\": \"+ policy fee\", \"factor\": null, \"amount_cents\": 2500, \"running_cents\": 103048}, {\"step\": \"DISCOUNT\", \"code\": \"SAFE_DRIVER\", \"detail\": \"\\u221212%\", \"factor\": 0.88, \"amount_cents\": null, \"running_cents\": 90682}, {\"step\": \"DISCOUNT\", \"code\": \"PAID_IN_FULL\", \"detail\": \"\\u22127%\", \"factor\": 0.9299999999999999, \"amount_cents\": null, \"running_cents\": 84334}, {\"step\": \"DISCOUNT\", \"code\": \"PAPERLESS\", \"detail\": \"\\u2212$25\", \"factor\": null, \"amount_cents\": -2500, \"running_cents\": 81834}]"
]
}
}
]
}
}
}⚙ end_conversation()
{}{
"ok": true,
"note": "Deliver your closing message now."
}Done! I have successfully updated your auto policy (POL-retention-target-p_auto) to the "Pay in Full" billing plan. Your new annual premium is $818.34.
Thank you for choosing PEICO for your auto insurance needs. If you have any other questions, feel free to reach out. Have a great day!
Customer
Perfect, thank you so much for taking care of that! I really appreciate it.