← Back to Renude Testing Suite

Historical Trends

Every run, over time

Runs

9

Queries tested

3101

Overall pass rate

61%

Date range

2026-09-01 — 2026-09-04

Overall assessment

One LLM pass reading across every run's results and summary — where the system is weak, and why that's likely happening. Regenerated only when new run data shows up.

The system's performance is inconsistent and trending downward, characterized by a failure to prioritize direct answers over generic conversational flows. The chatbot frequently defaults to irrelevant quiz sequences or deflects queries, indicating a significant gap in its ability to retrieve and synthesize specific brand guidelines and product data.

Over-reliance on generic quiz flows
The bot is configured with an over-eager fallback mechanism that triggers a quiz flow even when specific, answerable product information is available in the knowledge base. This suggests a prompt-level instruction gap where the 'conversational' objective is being prioritized over the 'informational' objective.
Missing mandatory brand-specific safety and usage guidelines
The consistent omission of critical details like layering rules, SPF requirements, and pregnancy exclusions points to a retrieval or grounding gap. The model is likely failing to prioritize these 'must-include' safety constraints during the generation phase, suggesting they are not sufficiently weighted in the system prompt or retrieval context.
Inability to handle out-of-scope queries
The failure to provide contact information or support links for out-of-scope queries indicates a lack of a robust 'handoff' protocol. The system lacks a clear instruction set for identifying when a query exceeds its capability, resulting in dead-end responses rather than helpful redirection.
Poor product data granularity
The bot struggles to distinguish between individual products and bundled sets, suggesting that the underlying product database lacks sufficient metadata or that the retrieval system is not effectively parsing the relationship between these entities.

Pass rate over time

0%25%50%75%100% nipfab1212_20260901_154119 — 2026-09-01 15:42 — 64% (14/22 passed) — click to openbiz16_20260901_163736 — 2026-09-01 16:37 — 0% (0/3 passed) — click to openall-qs-all-profiles_20260901_160703 — 2026-09-01 17:05 — 65% (1315/2009 passed) — click to opennip-fab-inventory-p1_20260901_170833 — 2026-09-01 17:08 — 82% (9/11 passed) — click to openrodial-test-report_20260902_085105 — 2026-09-02 09:01 — 41% (154/372 passed) — click to openbarefaced-testing-impaired-barrier_20260902_121212 — 2026-09-02 12:20 — 59% (152/259 passed) — click to opentest-furqan_20260902_123716 — 2026-09-02 12:38 — 73% (16/22 passed) — click to openbiz21_20260904_161714 — 2026-09-04 16:18 — 43% (23/54 passed) — click to openbiz27_20260904_162512 — 2026-09-04 16:26 — 48% (26/54 passed) — click to open 48%

Query volume over time

nipfab1212_20260901_154119 — 22 queries — click to openbiz16_20260901_163736 — 3 queries — click to openall-qs-all-profiles_20260901_160703 — 2304 queries — click to opennip-fab-inventory-p1_20260901_170833 — 11 queries — click to openrodial-test-report_20260902_085105 — 372 queries — click to openbarefaced-testing-impaired-barrier_20260902_121212 — 259 queries — click to opentest-furqan_20260902_123716 — 22 queries — click to openbiz21_20260904_161714 — 54 queries — click to openbiz27_20260904_162512 — 54 queries — click to open

Recurring failure themes

Matched by exact theme text across runs' findings — a rephrased version of the same root cause won't merge with an older one.

omission of specific brand-required details or product ranges
1 of 9 run(s) · 17 querie(s) · last seen 2026-09-04
missing mandatory brand-specific guidance or product details
1 of 9 run(s) · 11 querie(s) · last seen 2026-09-04
missing mandatory contact information or support links
1 of 9 run(s) · 5 querie(s) · last seen 2026-09-04
provision of incorrect or outdated factual information
1 of 9 run(s) · 5 querie(s) · last seen 2026-09-04
unnecessary redirection to quiz flow
1 of 9 run(s) · 5 querie(s) · last seen 2026-09-04
failure to provide required customer support contact information or links
1 of 9 run(s) · 5 querie(s) · last seen 2026-09-04
unnecessary deflection to quiz flow
1 of 9 run(s) · 4 querie(s) · last seen 2026-09-04
incorrect or missing product recommendations
1 of 9 run(s) · 4 querie(s) · last seen 2026-09-04
failure to provide requested product recommendations or alternatives
1 of 9 run(s) · 3 querie(s) · last seen 2026-09-02
inaccurate or incomplete FAQ information
1 of 9 run(s) · 3 querie(s) · last seen 2026-09-04
inability to distinguish between individual and bundle product specifications
1 of 9 run(s) · 2 querie(s) · last seen 2026-09-02
inclusion of irrelevant information not related to the user query
1 of 9 run(s) · 1 querie(s) · last seen 2026-09-02

All runs

DateRunBusinessServerQueriesPass rate
2026-09-04 16:26 biz27_20260904_162512 27 staging 54 48%
2026-09-04 16:18 biz21_20260904_161714 21 staging 54 43%
2026-09-02 12:38 test-furqan_20260902_123716 16 staging 22 73%
2026-09-02 12:20 barefaced-testing-impaired-barrier_20260902_121212 17 staging 259 59%
2026-09-02 09:01 rodial-test-report_20260902_085105 12 staging 372 41%
2026-09-01 17:08 nip-fab-inventory-p1_20260901_170833 16 staging 11 82%
2026-09-01 17:05 all-qs-all-profiles_20260901_160703 16 staging 2304 65%
2026-09-01 16:37 biz16_20260901_163736 16 staging 3 0%
2026-09-01 15:42 nipfab1212_20260901_154119 16 staging 22 64%