AI experience · cv.beltsov.ru
Roman BeltsovAI in QA and QA for AI
QA leader. Brought LLMs into testing at Kaspersky and shipped my own Android app with a local LLM: eval, guardrails, ~1,000 autotests.
- 25 mininstead of 4 h per test case — LLM grounded in internal docs
- 1 monthinstead of a year — 100% of the API under autotests
- 2 sinstead of 24–28 s — LLM reply on the phone, no server
- +30%utilization of expensive test benches, no new purchases
Kaspersky — LLMs in a QA team
Testing Group Manager · 2025–2026 · team of up to 15
- Test design: 4 h → 25 min per case
- 100% of the API (40 endpoints, 500+ tests) automated in a month — a year by hand
- RAG over internal docs: local LightRAG (chunking, embedding, reranker) + MCP, data stays inside
- Bench monitoring against “forgot to release”: +30% utilization
- Measured e2e payoff: a fraction of a percent of defects → checks moved down the pyramid
How it was done
- How generation works
- Local LightRAG over internal documentation (chunking, embedding, reranker), connected to the LLM via MCP.
- The LLM finds requirements in the docs and writes test cases and API autotests from them.
- An engineer reviews every result: the LLM speeds up, a human decides.
- Qwen3.6-35B locally on a MacBook (oMLX) — when data cannot go to the cloud.
- Benches
- Usage monitoring and booking (React + FastAPI). Benches no longer sit forgotten.
I Hear You — an Android app with a local LLM
Own product · solo with AI agents · 2026 — present
Published on Google Play ↗- LLM (Gemma) and speech recognition run on the phone, no server
- LLM inference optimization: reply 24–28 s → 1.5–2.6 s
- Evaluation-driven development: LLM quality regression, LLM-as-a-Judge, golden datasets, DeepEval
- Guardrails in code: crisis messages, prompt leaks
- 646 commits, 30+ releases, 207 requirements, ~1,000 autotests: AI agents (Claude Code) write the code, I accept by tests
- Product metrics in GA4 with no user text
How it was done
- Speed
- Measurement showed 60–80% of the time went to reprocessing the system prompt every turn.
- GPU with CPU fallback, context cache (first word in 0.5–1.2 s), GPU init cache (12.7 → 4.6 s), warm-up.
- Gemma 4 E4B was thought to need a device with 12 GB of RAM. Tested it: on 8 GB it runs on the GPU (reply in 5–14 s) — replaced the memory threshold with a GPU check.
- Quality evaluation
- Prompt × parameter matrix in one command, only changed cells recomputed.
- The judge reads each conversation three times and decides by majority; judge model pinned.
- Rule-based counters without a judge: paraphrasing, clichés, a bare question instead of an answer.
- Judge cost cut 4–5×: Cyrillic was being sent escaped.
- Prompt experiments
- One variable at a time, frozen scenarios, repeats.
- A short context prompt instead of an empty one: reply 2,164 → 281 chars, 4–6 wins of 6 with the blind judge.
- Hypotheses that hurt metrics were rejected (paraphrasing 10 → 47).
- Process and quality gate
- OpenSpec: 207 requirements, every test links to a requirement, CI rejects a test without one.
- 868 unit, 136 on-device e2e, 5 Maestro flows; the release build is run before every release.
- GA4: 42 of 47 “users” cleaned out as bots and test devices.
Tools
LLM evaluation
- Evaluation-driven development
- LLM quality regression
- LLM-as-a-Judge
- DeepEval
- golden datasets
- Python
LLM and RAG
- RAG
- LightRAG
- MCP
- OpenRouter
- prompt engineering
- inference optimization
- Sherpa-ONNX
Local and cloud LLMs
- Qwen
- Gemma 3/4
- Llama
- Mistral
- oMLX
- LM Studio
- llama.cpp
- LiteRT-LM
- Claude (Opus, Sonnet, Haiku)
- MiniMax
- GLM
AI-assisted development
- AI agents
- Claude Code
- Codex
- OpenCode
- Hermes Agent
- OpenClaw
- MiniMax Code
- OpenSpec
Tests and releases
- JUnit
- Maestro
- PyTest
- Google Play API
- GA4
- Crashlytics
How I test AI
- Case sets and a judge with repeats, not equality asserts
- Measure first, then decide
- One variable at a time, judge model pinned
- Code holds the model’s boundaries, not the prompt
- A small sample is a direction, not a result