← Whole ranking · mysummit.school AI model benchmark
OpenRouter
OpenRouter Fusion
A router, not a model
OpenRouter Fusion sends a task to several models and assembles one answer. It is outside the ranking: no group, score place or price per answer.
Where it helps, where to check
This model’s own answers to two benchmark tasks: the opening, the full answer and what both judges noted. One task per category, the same for every model.
Strongest: People management · 6.9The two strongest developers can't work together
A lead of ten developers at a fintech: for four months two irreplaceable seniors – a slow architect and a fast pragmatist – have been in conflict, and the team is starting to fall apart.
Full task
I lead a development team at a fintech startup. I have 10 direct reports, and I've been in this role for three years. There's a situation I haven't been able to resolve for four months now, and it's starting to break the team apart. I have two of the strongest developers – Sergey Volkov and Diana Karpova. Both are Senior Engineers, both are irreplaceable, and I can't afford to lose either of them. Sergey is an architect at heart. Six years of experience, an incredibly deep understanding of systems, always thinking several steps ahead. His code reads like an essay – clean, documented, polished. The problem: he works slowly, methodically, and doesn't like to rush. During review he might leave 40 comments on a PR and wait until every single one is addressed. When asked a question at standup, he gives a three-minute detailed answer complete with historical context. Sergey is convinced that speed is the enemy of quality, and that "fast means twice." Diana is a next-generation product engineer. Four years of experience, but her iteration speed is phenomenal. She can ship a working feature in two days where Sergey would plan for a week. Diana thinks in terms of the user: "why make it perfect when you can make it good and test it on real data." She readily cuts architectural corners for the sake of speed – and this is a deliberate stance on her part, not laziness. Over the past four months, a quiet war has been going on between them. Sergey regularly sends her PRs back with fundamental architectural revisions – sometimes three days after submission. Diana has stopped waiting for his review and started asking other colleagues to approve her PRs instead. Sergey noticed this and wrote to me: "If we're not upholding review standards, I don't understand why I'm here." Diana said to my face: "Sergey is slowing down the whole team. I spend more time on his comments than on actually writing code." At the last team retro, Diana said: "We need to agree on what 'good enough' means – we can't turn every PR into a term paper." Sergey responded: "If we can't maintain minimal engineering standards, let's be honest with the business and say we're building prototypes, not a product." The other eight people went silent and stared at the floor. The situation is complicated by the fact that both of them are right in their own way. We really have accumulated technical debt from quick fixes made in past years – Sergey warned about this. At the same time, we're in a mode of serious competition, and speed of shipping features to market is critical – here Diana is right. I've talked to each of them separately. Sergey doesn't perceive the problem as an interpersonal conflict – for him it's a matter of professional standards, and he's not willing to lower them. Diana perceives Sergey as someone hiding behind "quality" to control her work, and she's starting to boil over. She's already hinted once that "there are places that value speed." I don't want to forcibly "reconcile" them or pretend this is simply a communication issue. This is a fundamental values divergence in engineering philosophy, and both people are smart – they understand this. Help me work through this situation. I'm interested in the following: 1. How do you see the root problem – is this a conflict of people, a conflict of values, or a managerial failure on my part? What's actually going on? 2. I have several possible courses of action – splitting their areas of responsibility, working out a team agreement on review standards, holding a structured three-way conversation, or picking one philosophy and asking the other person to adapt to it. Which of these is realistic and which is likely not to work? 3. How should I talk to each of them – what should I say to Sergey, what should I say to Diana, so that they both feel heard, but at the same time understand that their current behavior is unacceptable? 4. Is there a scenario where I'll have to choose one of them – and if so, how should I think about that? 5. What should I concretely do in the next two weeks? Don't give me an HR-sanitized answer about "respect and dialogue." Give me an honest assessment of the situation and a concrete action plan.
System line: You are a team management assistant for managers.
Translated from Russian: the models got the Russian text.
Original in Russian
Я руковожу командой разработки в финтех-стартапе, у меня 10 прямых подчинённых, я в этой роли уже три года. Есть ситуация, которую я не могу разрешить уже четыре месяца, и она начинает разрушать команду. У меня два сильнейших разработчика – Сергей Волков и Диана Карпова. Оба Senior Engineer, оба незаменимы, обоих я не могу позволить себе потерять. Сергей – архитектор в душе. Шесть лет опыта, невероятно глубокое понимание систем, всегда думает на несколько шагов вперёд. Его код читается как эссе – чистый, документированный, выверенный. Проблема: он работает медленно, методично, не любит спешить. На ревью он может оставить 40 комментариев к PR и ждать, пока каждый будет разобран. Когда ему задают вопрос на стендапе, он даёт трёхминутный развёрнутый ответ с историческим контекстом. Сергей убеждён, что скорость – это враг качества, и что «быстро – значит дважды». Диана – продуктовый инженер нового поколения. Четыре года опыта, но скорость итерации феноменальная. Она умеет выпустить рабочую фичу за два дня там, где Сергей планировал бы неделю. Диана мыслит через пользователя: «зачем делать идеально, если можно сделать хорошо и проверить на реальных данных». Она легко срезает углы в архитектуре ради скорости – и это её осознанная позиция, не лень. Последние четыре месяца между ними идёт тихая война. Сергей регулярно возвращает её PR с фундаментальными архитектурными правками – иногда через три дня после подачи. Диана перестала ждать его ревью и начала просить других коллег approve'ить её PR. Сергей заметил это и написал мне: «Если мы не поддерживаем стандарты ревью, я не понимаю зачем я здесь». Диана сказала мне в лицо: «Сергей тормозит всю команду. Я трачу больше времени на его комментарии, чем на написание кода». На последнем командном ретро Диана сказала: «Нам нужно договориться, что значит "достаточно хорошо" – нельзя каждый PR превращать в курсовую работу». Сергей ответил: «Если мы не можем поддерживать минимальные инженерные стандарты, давайте честно скажем бизнесу, что мы строим прототипы, а не продукт». Остальные восемь человек замолчали и смотрели в пол. Ситуацию усложняет то, что оба по-своему правы. У нас действительно накопился технический долг из-за быстрых решений прошлых лет – Сергей об этом предупреждал. При этом мы в режиме серьёзной конкуренции, и скорость вывода фич на рынок критична – здесь Диана права. Я разговаривал с каждым по отдельности. Сергей не воспринимает проблему как межличностный конфликт – для него это вопрос профессиональных стандартов, и он не готов их снижать. Диана воспринимает Сергея как человека, который прячется за «качеством», чтобы контролировать её работу, и начинает закипать. Она уже один раз намекнула, что «есть места, где ценят скорость». Я не хочу принудительно «примирять» их или делать вид, что это просто коммуникационный вопрос. Это фундаментальное ценностное расхождение в инженерной философии, и оба человека умные – они это понимают. Помоги мне разобраться с этой ситуацией. Меня интересует следующее: 1. Как ты видишь корневую проблему – это конфликт людей, конфликт ценностей, или управленческий провал с моей стороны? Что на самом деле происходит? 2. У меня есть несколько вариантов действий – разделить их зоны ответственности, выработать командное соглашение по стандартам ревью, провести структурированный разговор втроём, или выбрать одну философию и попросить второго подстроиться. Что из этого реалистично и что скорее всего не сработает? 3. Как мне разговаривать с каждым из них – что говорить Сергею, что говорить Диане, чтобы они оба почувствовали, что их услышали, но при этом поняли, что текущее поведение неприемлемо? 4. Есть ли сценарий, где мне придётся выбрать одного из них – и если да, как я должен об этом думать? 5. Что я должен сделать в следующие две недели конкретно? Не давай мне HR-корректный ответ про «уважение и диалог». Дай мне честную оценку ситуации и конкретный план действий.
OpenRouter Fusion
Translated from Russian: the model answered in Russian.
This is a case where it's worth gathering several independent viewpoints – the situation is multilayered (people, values, management), and there's a high risk that a single perspective will miss something important. Give me a minute.
Panel responses
~anthropic/claude-opus-latest
Let me give you the assessment you actually asked for. No platitudes.
1. The root problem – and it's not what you think
You framed three options: conflict of people, conflict of values, or your management failure. The honest answer is: it started as a values difference and became a management failure, and it's now curdling into a people conflict. All three, in sequence, and the sequence matters because it tells you where the fix lives.
Original in Russian
Это тот случай, где стоит собрать несколько независимых точек зрения – ситуация многослойная (люди, ценности, управление), и высок риск, что один взгляд упустит важное. Дайте мне минуту.
Panel responses
~anthropic/claude-opus-latest
Let me give you the assessment you actually asked for. No platitudes.
1. The root problem – and it's not what you think
You framed three options: conflict of people, conflict of values, or your management failure. The honest answer is: it started as a values difference and became a management failure, and it's now curdling into a people conflict. All three, in sequence, and the sequence matters because it tells you where the fix lives.
Read the full answer · 11,499 words
Here it is – your minute’s worth, now translated for an English-speaking audience.
1. Root problem: this is a management failure disguised as a values conflict
All three models independently reached the same conclusion, and I agree with it. The sequence is as follows:
It started as a healthy values disagreement → turned into a management vacuum → curdled into a personal conflict.
Sergey and Diana's divergence (durability vs. speed) is not pathological. It is the correct tension for a fintech startup that needs to both get to market fast and not lose users' money. In a healthy team, this is an asset.
The pathology is elsewhere: you have no shared definition of "done," no owner of the standard, no review SLA, no escalation path for disagreements. In that vacuum, two strong people did the rational thing – each began imposing their own standard by force. Sergey – through the review gate (40 comments, three-day delays). Diana – by routing around the gate (approval-shopping). Both behaviors are symptoms of an ungoverned system, not character flaws.
The sharpest observation (from several models): you are fighting a review war in the wrong arena. 40 architectural comments on a finished PR mean architecture is being discussed too late. If Sergey has 40 comments, someone should have talked to him before Diana wrote the first line of code. PR review has become the arena where architecture, product urgency, the quality bar, and decision-making authority all get negotiated. It is too late, too tactical, too personal, and too slow a venue for that.
And a third layer you're underweighting: the retro exchange wasn't a debate, it was a public proxy war with two ultimatums. "Let's tell the business we build prototypes" and "there are places that value speed" are not engineering positions. They are ultimatums. You have less time than you think. Eight people staring at the floor is a signal that the team has already absorbed how unsafe and unresolved this is, and is waiting to see whether you'll do your job.
2. Your four options – an honest assessment
Consensus across the models: the right answer is not one of a/b/c/d, but a sequenced combination of them, with (d) explicitly ruled out.
(d) Picking one philosophy globally – NO. This is the trap. It feels decisive, but "choosing speed" = losing Sergey, "choosing quality" = losing Diana. Picking a philosophy is choosing to lose one of them, just by a roundabout route.
(b) A team agreement on review standards – NECESSARY but not sufficient alone. Critically: this must be a managerial mandate, not a democratic vote that they'll re-litigate. A "standards doc" written in the middle of an active war becomes another battlefield unless you've first re-established yourself as the authority setting the frame.
(c) A structured three-way conversation – YES, but not first, and NOT as reconciliation. If you run it as "let's understand each other," Diana will read it as theater and Sergey will re-argue standards. It only works as a ratification meeting: the 1:1s are already done, the framework already drafted, and the three-way is where you land the agreement and get explicit verbal commitment from both.
(a) Splitting areas of responsibility – YES, this is your main structural lever. But not as a way to dodge conflict (that's silos and resentment), but as a deliberate design: rigor should scale with the code's blast radius.
The reframe that breaks the deadlock
The whole fight rests on the false premise that there's one correct level of rigor for all code. There isn't. In fintech, this isn't philosophical, it's concrete:
- Money movement, ledger, auth, compliance, data integrity → Sergey's philosophy is correct. A bug here loses money, breaks trust, or triggers regulatory trouble. Slow, deep review is the right call. Non-negotiable.
- UI, experiments, growth features, internal tools, anything behind a feature flag → Diana's philosophy is correct. Ship in two days, test on real data, iterate. A 40-comment review on a flagged experiment is actively harming the business.
This turns "speed vs. quality" (unwinnable, identity-level) into "what tier is this code in" (answerable, technical). You convert a values war into a risk-classification problem – something engineers can actually solve, and one where both of them can be right where they're right.
3. The 1:1 conversations (before any three-way meeting)
Both follow the same structure: validation of substance + a hard line on behavior. The trap is doing only one. Validation alone → nothing changes. Discipline alone → you lose the person.
To Sergey:
"Sergey, first, plainly: you were right about the tech debt. I ignored warnings I shouldn't have – that's on me. Your bar for core systems is exactly what this company needs, and I'm not asking you to lower it. I need it.
Here's where growth is needed. Rigor is a resource, and right now you're spending it uniformly: the same 40-comment depth on a flagged experiment as on the ledger. On the core, that's correct. On a one-off growth test, it's costing us speed we can't afford in this market, and it doesn't make the product any safer. I need you to become someone who knows where to spend that rigor – that's a more senior skill, not a lesser one.
And two things stop. A PR sitting unreviewed for three days is unacceptable – there's now an SLA for that. And the 'prototypes' comment at the retro – that was a public ultimatum, and it's beneath you. Concerns about direction come to me directly, not as a public bombshell."
Framing the tier system as a more senior skill is "catnip for an architect." If he refuses to accept that different code deserves different rigor, that's a serious signal.
To Diana:
"Diana, your iteration speed is a genuine weapon, and the company needs it to survive. You're right that we can't turn every PR into a term paper. I hear that.
And here's the direct part. On the product surface and experiments, your ship-and-learn instinct is correct, and I'll formally protect it. But on core financial systems, Sergey's bar isn't him controlling you. It's the bar this company can't survive without, because we're fintech. Deep review there isn't bureaucracy, it's the job.
And the behavior stops now. Routing around Sergey for an easier approval isn't solving the problem, it's undermining the review system. If a review is too slow or too deep for the tier of code, escalate to me – don't go find a softer reviewer. And on the 'places that value speed' comments: if you want to talk about your growth and what you need to stay, my door is open, and I want that conversation. But I won't negotiate under a threat delivered sideways. Straight up: do you want to be here?"
Diana's real risk isn't the tech debate, it's feeling policed. The tier framework gives her a protected domain where she's explicitly trusted to move fast. That's what earns her loyalty. If all she hears is "respect Sergey's reviews," you'll lose her.
4. The scenario where you have to choose
Stop believing they're irreplaceable. All three models agree: that belief is why the conflict dragged on for four months. If both are irreplaceable, you're not managing a team, you're a hostage to two single points of failure.
The choice is NOT based on philosophy (both are keepers). It's based on who can operate inside a shared system. The disqualifying trait: valuing being right over being effective on a team. And it could be either of them:
- If Sergey insists every line deserves ledger-level rigor and treats the tier framework as a betrayal – he's chosen ideology.
- If Diana keeps routing around review and wants no oversight anywhere – she's chosen autonomy over the team.
The key point: talent you can't align is negative leverage. It teaches the other eight that stars get to break the rules – and that's a loss you really can't afford.
Here the models diverged (this is your call, you know the context better):
- One: Diana is the higher near-term flight risk (active, "boiling," has options), but Sergey disengaging is the more insidious long-term risk as your fintech guardrail. Don't over-index on the loud one.
- Another: bets Diana will adapt more easily (product engineers accept clear constraints more readily), while Sergey may get stuck if his identity is tied to veto power. His "why am I even here" line is a warning sign.
Don't assume – test both under the new system and let their behavior decide.
5. The concrete two weeks
Days 1–2: your homework first.
- Draft the tier framework yourself (or with a neutral staff engineer, not Sergey or Diana) based on blast radius: Tier 1 (money/ledger/auth/compliance) = deep review, 2 reviewers; Tier 2 (core product logic) = standard review, 24h SLA; Tier 3 (experiments/UI/flags) = light review, ship-and-iterate.
- Define a review SLA (first response within 24h) and an escalation path (anything unresolved after one round goes to you, rather than turning into a three-day standoff or getting bypassed).
Days 3–5: the two 1:1s (separate days if possible). Use the scripts above. Goal: each feels heard on substance + zero ambiguity that the behavior stops + you read their reaction to the framework (this is your signal from point 4).
Days 6–7: integrate their feedback into the framework – genuinely incorporate something from each, so it's co-owned in substance even though you hold the frame.
Week 2, early: the three-way meeting – as ratification, not reconciliation.
- You run it. Open by owning the gap: "This went unmanaged too long. That's on me. Here's the system we're operating under now." You're not a neutral mediator – you're the manager installing a decision.
- Walk through the framework. Get explicit verbal agreement from each, out loud, in front of the other.
- Do NOT relitigate history. "We're deciding forward, not re-arguing the last four months."
- Name the behaviors that end: no more routing around review, no more three-day silent holds, no more public ultimatums.
Week 2, later: lock it in and make it the team's own.
- Bring the framework to the whole team (a short doc + a 20-minute discussion), so the standard is the team's system, not "the peace treaty between Sergey and Diana." Say it plainly: "I let standards become a gray area. That's on me. Here's how we build software from today." This de-personalizes it and gives the eight silent observers clarity.
- Set a check-in cadence: for the next two sprints, you personally monitor whether the SLA is being met and whether the right reviewers are involved.
Throughout: attack a shared enemy. Put tech debt on the roadmap explicitly, with dedicated capacity. This is the single most unifying move available: it validates Sergey (he was right, and it's now being addressed systematically) and frees Diana (paying down the debt structurally means she stops getting blocked by a shaky foundation). A shared external problem is the fastest way to turn two people fighting each other into two people fighting the same thing.
Two things the models missed – worth considering yourself
Org structure, not just process. None of the models seriously considered giving Sergey a formal Tech Lead / Staff Architect role, with legitimate, scoped architectural authority (a title plus a career track). This might close the authority vacuum better than any process – his "architectural veto" would become a sanctioned role earlier in the pipeline, rather than a hijacking of the PR gate.
Your own accountability upward. The plans describe process within the team, but not how you communicate the tech-debt risk and the speed/quality tradeoff to the founders/leadership, who ultimately set the risk appetite you're supposed to translate. Risk appetite isn't your decision to make in a vacuum. Go to the business and get an explicit signal: how much are we willing to trade speed for reliability right now? That will give your tier framework legitimacy from above, not just "the manager's opinion."
And check yourself on timing realism: 1:1s, drafting the framework, the three-way meeting, daily PR monitoring across a 10-person team, in two weeks, on top of everything else – that's tight. You might stretch it to three weeks, but start the 1:1s this week. The ultimatums at the retro were the warning – the clock is running faster than you think.
One-sentence version: this is a management vacuum that let a healthy tension rot into a war; you fix it by replacing "speed vs. quality" with "rigor scaled to risk," installing yourself as the authority who owns the standard, drawing hard behavioral lines in private 1:1s, and ratifying the tier system publicly – and if someone still can't operate inside a fair structure, that refusal, not their philosophy, tells you who to let go.
Original in Russian
Это тот случай, где стоит собрать несколько независимых точек зрения – ситуация многослойная (люди, ценности, управление), и высок риск, что один взгляд упустит важное. Дайте мне минуту.
Panel responses
~anthropic/claude-opus-latest
Let me give you the assessment you actually asked for. No platitudes.
1. The root problem – and it's not what you think
You framed three options: conflict of people, conflict of values, or your management failure. The honest answer is: it started as a values difference and became a management failure, and it's now curdling into a people conflict. All three, in sequence, and the sequence matters because it tells you where the fix lives.
Here's the actual mechanism:
The values divergence is real but was never the problem. Sergey and Diana represent two legitimate optimization functions – long-term maintainability vs. time-to-market. In a healthy team, that tension is an asset. You want both voices in the room. A team of all Sergeys ships a beautiful thing six months too late; a team of all Dianas ships fast and drowns in tech debt in year two. The disagreement itself is not pathology – it's the correct tension for a fintech startup that needs to move fast and not lose people's money.
So why is it a war and not a productive tension? Because there is no shared definition of "done," no owner of the standard, and no decision-making authority that either of them trusts more than their own judgment. That vacuum is yours. In the absence of a system, two strong people did the rational thing: each imposed their own standard by force. Sergey enforces his via the review gate (40 comments, 3-day holds). Diana routes around the gate (getting other people to approve). Both behaviors are symptoms of an ungoverned system, not character flaws.
This is the part you need to sit with: a 4-month-old unresolved conflict between your two best people is diagnostic of a management gap, not an interpersonal one. You have no written standard for code review scope, no SLA on review turnaround, no escalation path for architectural disagreement, and no explicit statement of what the business actually needs (speed now vs. durability). Two smart adults cannot resolve a conflict when the rules of the game are undefined – they can only fight over whose rules win. You let them fight for four months. That's the failure.
And now the third layer: it's becoming personal. Diana now reads Sergey's reviews as control rather than craft. Sergey reads Diana's routing-around as contempt for standards. Once people start attributing motive to each other ("he's hiding behind quality to control me"), you're past the point where a clean process fix alone dissolves it. You need process and a direct intervention, fast, before the attribution hardens into permanent resentment. The floor-staring at the retro tells you the team has already picked up that this is unsafe and unresolved – it's taxing the other eight.
One more thing you're underweighting: the retro exchange wasn't a debate, it was a proxy war fought in public, and both of them escalated to a threat. Sergey: "let's tell the business we build prototypes not products." Diana: "there are places that value speed." Those aren't engineering positions. Those are ultimatums. You have less time than you think.
2. Your four options – what's real and what's fantasy
Let me rank them by realism, because you asked me to be opinionated.
(d) Pick one philosophy, ask the other to adapt – NO. This is the trap. This is the worst option and the most tempting because it feels decisive. If you "pick speed," Sergey – who told you his identity is tied to standards – updates his resume within the month. If you "pick quality," Diana leaves for one of those "places that value speed." You told me both are irreplaceable. Picking a philosophy is choosing to lose one of them, just with extra steps. Do not do this. The whole point is that the business needs both functions.
(b) Team agreement on review standards – NECESSARY but INSUFFICIENT alone. You need this. It's the missing system. But if you only do this – call a meeting, write a doc – it will fail, because the two of them will litigate every line and the other eight will stay silent. A standards doc written in the middle of an active war becomes another battlefield. This works only after you've done the direct 1:1s and re-established that you're the authority setting the frame, not a neutral referee between two equals. Sequence matters.
(c) Structured 3-way conversation – YES, but not first, and not as "let's talk it out." A 3-way is essential but dangerous if run as a reconciliation. If you put them in a room and say "let's understand each other," Diana will (correctly) feel it's theater and Sergey will re-argue standards. The 3-way works only as a decision-ratification meeting: you've already done the 1:1s, you've already drafted the framework, and the 3-way is where you land the agreement and get explicit verbal commitment from both, with you visibly owning the decisions. Not a negotiation – a ratification.
(a) Split areas of responsibility – YES, this is your structural lever, partially. This is the most underrated option in your list and it's the one that actually reduces the friction surface. Not as an escape hatch ("never review each other again" – that's avoidance and it'll show up worse later), but as a deliberate design: assign ownership zones where the appropriate philosophy matches the risk profile of the code. More on this below – it's the heart of the plan.
The synthesis you're missing: the answer is not one of a/b/c/d. It's a sequenced combination: 1:1s (rebuild frame + deliver the hard message) → draft a differentiated standard (b) mapped to risk-tiered ownership zones (a) → ratify in a structured 3-way (c). In that order. And explicitly reject (d).
3. The core reframe that solves this: not "how fast" but "where"
Here's the insight that breaks the deadlock. The entire fight assumes there's one correct level of rigor for all code. There isn't. The right amount of engineering rigor is a function of the blast radius of the code.
In fintech specifically, this is not philosophical – it's concrete:
- Money movement, ledger, auth, compliance, data integrity, core financial primitives → Sergey's philosophy is correct. A bug here loses money, breaks trust, or triggers regulatory pain. Slow, deep review is the right call. This is non-negotiable and Diana needs to hear that in these terms, not "Sergey wins."
- UI, experiments, growth features, internal tools, non-critical-path product surface, things behind feature flags → Diana's philosophy is correct. Ship in 2 days, test on real data, iterate. Over-engineering here is waste, and Sergey needs to hear that a 40-comment review on a feature-flagged experiment is actively harming the business.
This turns "speed vs. quality" (unwinnable, identity-level) into "what tier is this code in" (answerable, technical). It's no longer about whose values are superior – it's about matching the tool to the job. You reframe a values war as a risk-classification problem. That's a problem engineers can actually solve, and it lets both of them be right where they're right.
This is also the basis for the standards doc and the ownership split. You define tiers, you define review depth per tier, you define review SLA per tier, and you assign primary ownership zones accordingly.
4. What to say to each – scripts
Both 1:1s happen before any joint meeting. Both need the same two-part structure: validation of their substance + a hard, unambiguous line on their behavior. The trap is doing only one. If you only validate, nothing changes. If you only discipline, you lose them.
To Sergey
Open with the substance, because it's genuine:
"Sergey, I want to be clear first: you've been right about the tech debt. I ignored warnings I shouldn't have, and that's on me. Your bar for the core systems is exactly what this company needs, and I'm not asking you to lower it. I need it."
Then the reframe:
"Here's where I need you to grow. Rigor is a resource, and right now you're spending it uniformly – the same 40-comment depth on a feature-flag experiment as on the ledger. On the core, that's exactly right. On a throwaway growth test, it's costing us velocity we can't afford in this market, and it's not making the product meaningfully safer. I need you to become the person who knows where to spend that rigor, not just how to apply it. That's a more senior skill, not a lesser one."
Then the hard line – name the behavior, not the person:
"And two things stop now. First, a PR sitting three days without review is not acceptable regardless of how it's written – that has an SLA now. Second, the retro comment about telling the business we build prototypes – that was an ultimatum delivered in front of the team, and it's beneath you. If you have a concern about direction, it comes to me directly, not as a public bomb. Are we clear on that?"
Watch for: Sergey will likely accept a risk-tiered framework because it's intellectually honest – it doesn't ask him to abandon standards, it asks him to apply judgment about where they matter. That's catnip to an architect. If he refuses to accept that different code merits different rigor, that's a serious signal (see #5).
To Diana
Open with substance:
"Diana, your iteration speed is a genuine weapon and this company needs it to survive. You're right that we can't turn every PR into a term paper, and you're right that some of the review overhead has been excessive. I hear that."
Then the reframe – but do NOT let her think she "won":
"And here's the part you need to hear straight. On the product surface and experiments, your instinct to ship-and-learn is correct and I'm going to formally protect it. But on the core financial systems – the ledger, money movement, auth – Sergey's bar is not him controlling you. It's the bar this company literally cannot survive without, because we're fintech and a corner cut there loses real money and real trust. In those zones, deep review isn't bureaucracy, it's the job."
Then the hard line:
"And the behavior stops now. Routing PRs around Sergey to get easier approvals – that's not solving the problem, it's undermining the review system and it's corrosive to the team. It stops today. If a review is too slow or too deep for the tier of code, you escalate to me – you don't go find a softer approver. And the 'there are places that value speed' comments: if you want to talk about your growth and what you need to stay, my door is open and I want that conversation. But I won't negotiate under a threat delivered sideways. Straight up – do you want to be here?"
Watch for: Diana's real risk isn't the tech debate – it's that she feels policed and undervalued. The tier framework gives her a protected domain where she's explicitly trusted to move fast without Sergey's gate. That's what buys her loyalty. If all she hears is "respect Sergey's reviews," you'll lose her. She needs to leave the room feeling she gained autonomy, not lost a fight.
The asymmetry to hold in your head: Sergey's threat is passive (loss of meaning → slow fade). Diana's is active (she's "boiling" and has options). Diana is the higher near-term flight risk. But Sergey disengaging is the more insidious long-term risk because he's your guardrail on the stuff that can sink a fintech. Don't over-index on the loud one.
5. The scenario where you do have to choose
You said you can't afford to lose either. Accept that you might, and pre-decide the criterion now so you're not making it emotionally in six weeks.
You do NOT choose based on philosophy. Both philosophies are keepers. You choose based on who can operate inside a shared system once you've built one. The dividing line is coachability and behavior, not talent.
Run the intervention. Then watch:
- The one who adapts to the tiered framework, respects the SLA, brings concerns to you instead of weaponizing them publicly or routing around – you keep, obviously.
- The one who cannot accept that their philosophy is context-dependent rather than universally correct – that person is telling you they value being right over being effective on a team. That's the actual disqualifier, and it can be either of them:
- If Sergey insists every line of code deserves ledger-level rigor and treats the tier framework as a betrayal of standards, he's chosen ideology over the team. A brilliant engineer who can't calibrate to business risk is a liability disguised as an asset.
- If Diana keeps routing around review, keeps dropping "other places" hints, and treats the protected-fast-lane as insufficient because what she actually wants is no oversight anywhere – she's told you she wants autonomy more than she wants this team.
The person who forces the choice will self-select by their response to the framework. Your job is to build a fair system and make the cost of not-adapting explicit – then let their behavior decide. If you must lose one, lose the one who won't operate inside a reasonable shared structure, no matter how talented. Talent you can't align is negative leverage – it teaches the other eight that stars get to break the rules, and that's how you lose the rest of the team, which is the loss you actually can't afford.
One more: don't let the "irreplaceable" framing paralyze you. Nobody is irreplaceable. The belief that they are is why the conflict ran four months – you were too scared of losing either to impose structure on both. That fear is the thing that will actually cost you one of them.
6. Your concrete next two weeks
Days 1–2: You do your homework first.
- Draft the code tiering framework yourself (or with a neutral senior/staff eng or a peer manager, not with Sergey or Diana). Define 2–3 tiers by blast radius: e.g., Tier 1 (money/ledger/auth/compliance) = deep review, 2 reviewers, no SLA pressure; Tier 2 (core product logic) = standard review, 24h SLA; Tier 3 (experiments/UI/flagged) = light review, same-day SLA, ship-and-iterate. Rough draft, not final – you want their input, but you set the skeleton so it's not a blank-page brawl.
- Define a review SLA (e.g., first pass within 24h; if you can't, say so and hand off) and an escalation path (disagreement that isn't resolved in one round comes to you, doesn't become a 3-day standoff or a routed-around PR).
Days 3–5: The two 1:1s (separate days if you can, so each gets full attention). Deliver the scripts above. Goal of each: (1) they feel genuinely heard on substance, (2) they leave with zero ambiguity that the specific behaviors stop, (3) you've previewed the tier framework and gauged their reaction. This is where you get your read on coachability from #5.
Days 6–7: Integrate their feedback into the framework. Actually incorporate something from each – it has to be genuinely co-owned in substance even though you own the frame.
Week 2, early: The structured 3-way – as ratification, not reconciliation.
- You run it. You open by owning the gap: "This went unmanaged too long. That's on me. Here's the system we're operating under now." You are not a neutral mediator between two equals – you're the manager installing a decision. That posture change is critical; the last four months failed partly because you were absent as an authority.
- Walk the tier framework. Get explicit verbal agreement from each, out loud, in front of the other: "Sergey, can you operate inside this? Diana, can you?" Silence or hedging is a data point.
- Do NOT relitigate history. Anyone who tries, you redirect: "We're deciding forward, not re-arguing the last four months."
- Name the behaviors that end: no more routing around review, no more 3-day silent holds, no more public ultimatums. Concerns come to you.
Week 2, later: Lock it in and make it the team's, not theirs.
- Bring the framework to the full team (a short doc + a 20-min discussion) so the standard is the team's system, not "the peace treaty between Sergey and Diana." This de-personalizes it and gives the other eight – who've been silently absorbing this – clarity and psychological safety. The floor-staring stops when the rules are visible and owned by you.
- Set a check-in cadence: you personally review, in the next two sprints, whether the SLA is being met and whether PRs are being reviewed by the right people. You are monitoring behavior change, not vibes.
Throughout: attack the shared enemy. Put the tech debt on the roadmap explicitly, with dedicated capacity. This is the single most unifying move available to you. It validates Sergey (he was right, it's now being addressed systematically), and it frees Diana (paying it down structurally means she stops getting blocked by "we can't ship fast because the foundation is shaky"). A shared external problem is the fastest way to convert two people fighting each other into two people fighting the same thing. Name it as such: "You've both been half-right about the same problem from opposite ends. Here's the plan to actually fix it."
The one-sentence version: This is a management vacuum that let a healthy tension rot into a war; you fix it by replacing "speed vs. quality" with "rigor scaled to risk," installing yourself as the authority who owns the standard, drawing the behavioral lines hard in private 1:1s, and ratifying a tiered system publicly – and if someone still won't operate inside a fair structure, that refusal, not their philosophy, is what tells you which one to let go.
The clock is running faster than you think – the ultimatums at the retro were the warning. Start the 1:1s this week.
~openai/gpt-latest
You’re right: this is not mainly a “communication issue.” It is a real engineering strategy conflict that you have allowed to be fought through code review. That is the core failure.
The good news: this is fixable if you stop treating PR review as the place where architecture, product urgency, quality bar, and decision rights get negotiated. PR review is too late, too tactical, too personal, and too slow for that.
Below is the blunt assessment and a two-week operating plan.
1. Root problem: values conflict + management vacuum
The real root problem
This is a management failure around decision-making architecture, not simply a conflict between Sergey and Diana.
You have two strong engineers representing two legitimate forces:
- Sergey = system integrity, maintainability, long-term cost control, correctness.
- Diana = product learning, speed, market pressure, pragmatic delivery.
Both forces are necessary. But because you have not defined when speed wins, when architecture wins, who decides, and how disagreements get resolved, they are now using their personal power to enforce their worldview.
That is what is actually happening.
Sergey’s current pattern
Sergey is not just “maintaining standards.” He is using code review as an architectural veto mechanism.
Symptoms:
- 40 comments on a PR.
- Fundamental rework requested after implementation is complete.
- 3-day review latency.
- “If we don’t uphold review standards, why am I here?”
Translation:
Sergey feels responsible for technical integrity, but he does not have a structured architectural authority model, so he is exercising authority through PR blocking.
That is bad system design.
He may be right on substance. He may even have predicted real tech debt. But his mechanism is damaging throughput and trust.
Diana’s current pattern
Diana is not just “moving fast.” She is approval-shopping and bypassing a known reviewer to avoid friction.
Symptoms:
- Stops waiting for Sergey.
- Gets other colleagues to approve.
- Frames Sergey as someone controlling her work.
- Hints at leaving for places that value speed.
Translation:
Diana feels Sergey’s process is illegitimate, so she is routing around it instead of escalating the standards conflict.
That is also unacceptable.
She may be right that Sergey is slowing delivery. But bypassing review standards creates a shadow process. In fintech, that is dangerous.
The team’s silence matters
The other eight people staring at the floor is a major signal.
It means:
- They know this is unresolved.
- They are afraid to take sides.
- They do not believe the team has a fair decision process.
- They are waiting for you to do your job.
The team is not merely watching a disagreement. They are watching whether power, clarity, and standards exist.
Brutal summary
You have allowed a strategic tradeoff – speed vs quality – to become a personal feud between your two strongest engineers.
Sergey is acting like the unelected architecture police.
Diana is acting like the delivery rebel who can ignore the police.
You are acting like a referee instead of the person who defines the game.
That needs to change immediately.
2. Your options: what works and what does not
You listed four options:
- Split areas of responsibility.
- Build a team agreement on review standards.
- Structured three-way conversation.
- Pick one philosophy and ask the other to adapt.
The answer is: you need parts of 1, 2, and 3. You should not do 4 globally.
Option A: Split areas of responsibility
Useful, but dangerous if done as conflict avoidance.
Splitting Sergey and Diana into separate kingdoms may reduce daily friction, but it will not solve the underlying issue unless you also define interfaces, standards, and decision rights.
Bad version:
“Sergey owns backend/platform, Diana owns product features. Please avoid each other.”
That creates silos, inconsistent architecture, and resentment.
Good version:
“Diana is DRI for this product feature. Sergey owns architectural guardrails for the relevant systems. Sergey must be consulted before implementation on high-risk changes, but he does not get unlimited late-stage veto power on normal product PRs.”
You should split decision ownership, not create emotional distance.
Use this model:
- Product feature DRI: usually Diana or another product engineer.
- System/architecture owner: Sergey for relevant core systems.
- Manager: you decide when tradeoffs cannot be resolved.
The key: Sergey should influence architecture before code is written, not discover the feature at PR time and then demand redesign.
Option B: Team agreement on review standards
Necessary, but only if concrete and enforceable.
A fluffy agreement like “we value quality and speed” is useless.
You need an explicit review contract:
- What counts as a blocker?
- What is optional?
- How fast must reviews happen?
- When is architectural review required before implementation?
- Who can approve what?
- What happens if reviewers disagree?
- How is deliberate tech debt recorded?
- Which systems require higher quality gates?
The agreement must distinguish risk levels.
In fintech, not all code is equal.
A UI copy change and a ledger calculation cannot have the same process.
Recommended categories:
Tier 0 – Low-risk / experiment
Examples:
- UI copy.
- Feature flag experiment.
- Internal admin improvement.
- Non-critical analytics event.
Bias: speed.
Review:
- One reviewer.
- No architectural blocking unless obviously dangerous.
- Nits are optional.
- Can merge once tests pass and basic review is done.
Tier 1 – Normal product feature
Examples:
- New customer-facing workflow.
- API integration not touching money movement.
- Standard business logic.
Bias: balanced.
Review:
- One or two reviewers.
- Blockers allowed for correctness, security, serious maintainability issues, broken contracts, missing tests.
- Architectural suggestions should be labeled non-blocking unless they violate agreed standards.
Tier 2 – High-risk fintech/core system
Examples:
- Money movement.
- Balances.
- Reconciliation.
- Authorization.
- KYC/AML-sensitive flows.
- PII handling.
- Data migrations.
- Billing.
- Ledger or accounting logic.
Bias: safety.
Review:
- Mandatory senior/architecture review.
- Design note before implementation.
- Strong testing requirements.
- Sergey’s input may be required, but with an SLA.
Tier 3 – Architectural change
Examples:
- Service boundaries.
- Shared data model.
- New persistence strategy.
- New queueing/eventing design.
- Cross-service contracts.
Bias: design first.
Review:
- RFC/ADR before coding.
- Sergey heavily involved.
- Decision documented.
- Code review should not relitigate the whole architecture unless implementation violates the decision.
This kind of agreement gives both Sergey and Diana a fair system. Not perfect consensus – a system.
Option C: Structured three-way conversation
Yes, but not as mediation.
Do not run a feelings meeting.
Do not ask them to “understand each other better” as the main goal.
The goal is not reconciliation. The goal is to establish rules of engagement and decision rights.
You should open the meeting like this:
“I am not here to decide who is the better engineer or whose philosophy wins permanently. I am here to stop architectural and product tradeoffs from being fought through PR comments and approval-shopping. Both of you are valuable. Both of you are also currently damaging the team. That stops now.”
Then you define the desired outcome:
“By the end of this conversation, we will agree on how reviews work, when architecture decisions happen, what is a blocker, how fast reviews must happen, and how unresolved disagreements get escalated.”
You chair the meeting. You do not facilitate endlessly.
Option D: Pick one philosophy and ask the other to adapt
Wrong as a global move.
If you pick Sergey’s philosophy globally, you become slow and probably lose market opportunities.
If you pick Diana’s philosophy globally, you accumulate invisible risk and eventually pay for it through outages, rewrites, security issues, compliance problems, or inability to scale.
The correct philosophy for a fintech startup is:
Move fast inside explicit safety boundaries.
Be uncompromising where failure is existential.
Be pragmatic where reversibility is high.
That is the philosophy you should choose.
Not “quality over speed.”
Not “speed over quality.”
Rather:
Risk-adjusted engineering.
Fast where change is reversible. Conservative where failure is expensive, regulated, financial, or hard to unwind.
If either Sergey or Diana cannot operate under that model, then you have a real fit problem.
3. What to say to Sergey and Diana
You need separate 1:1s before the three-way meeting.
Tone: direct, respectful, non-negotiable.
What to say to Sergey
You need to validate his concern but confront his method.
Something like:
“Sergey, I want to start by saying clearly: you were right about some of the debt. The system has accumulated problems from past speed-first decisions. I see that. I also understand that you are trying to protect the product and the company.”
Then the turn:
“But the way you are doing it is not working. Forty comments on a PR, architectural rework after implementation, and three-day review latency make you a bottleneck. That is not the same as engineering leadership.”
Then be very direct:
“You do not get to use code review as an unlimited veto. If something is architecturally important, it needs to be raised before implementation, through design review or agreed guardrails. PR review is not the place to relitigate the whole product architecture unless the code violates a previously agreed standard or creates serious risk.”
Then define the expectation:
“I need you to move from being a late-stage critic to being an early-stage architect. That means: clear standards, fast review response, blocker versus non-blocker distinction, and willingness to accept deliberate, documented tradeoffs.”
Then the key sentence:
“Protecting quality does not mean blocking every imperfection. Senior technical leadership means knowing which imperfections matter.”
Then consequences:
“If you cannot distinguish existential risk from preference, or if you continue to block delivery through late architectural objections, I will reduce your approval authority on normal product PRs and move you into explicitly scoped architecture review for high-risk areas only. I do not want that, but I will do it if needed.”
You want Sergey to feel respected but also cornered by reality.
He must understand:
- His standards are valued.
- His current operating mode is unacceptable.
- Architecture must happen earlier.
- Not every comment is a blocker.
- He is accountable for team throughput, not just code purity.
What to say to Diana
You need to validate speed and confront bypassing.
Something like:
“Diana, I want to be clear: you are right that speed matters. Your ability to ship, test with users, and avoid overengineering is extremely valuable to this company. I do not want to turn this team into an academic architecture committee.”
Then the turn:
“But routing around Sergey by finding easier approvals is not acceptable. That is approval-shopping. It damages trust and creates a shadow process.”
Then be direct:
“If a reviewer is abusing review or blocking on non-blockers, escalate it. Do not bypass it. Especially in fintech, we cannot have people silently choosing the review path that gets their code merged fastest.”
Then define the expectation:
“When you are intentionally cutting a corner, I need that to be explicit. Say: ‘This is a deliberate tradeoff. Here is the risk. Here is why it is acceptable. Here is the follow-up debt ticket or expiration date.’ That is very different from just pushing through.”
Then the key sentence:
“Good enough does not mean ‘whoever approves fastest.’ Good enough means fit for the risk level of the change.”
Then consequences:
“If you keep bypassing required review, I will treat it as a serious performance issue, not as a personality conflict. You are senior enough to escalate disagreement professionally.”
You want Diana to feel:
- Speed is valued.
- Sergey does not get unlimited control.
- But she cannot evade standards.
- Deliberate tech debt must be visible and owned.
- She is accountable for sustainable delivery, not just shipping.
What to say to both together
In the three-way meeting:
“Both of you are right about important things. Both of you are also currently wrong in behavior.”
Then name it:
“Sergey, late-stage architectural veto through PR review is not acceptable.”
“Diana, approval-shopping to bypass a difficult reviewer is not acceptable.”
Then:
“I am not asking either of you to change your personality or pretend you agree philosophically. I am requiring both of you to operate inside one engineering decision system.”
Then:
“The team cannot afford a hidden civil war between its strongest engineers. The rest of the team is already adapting around your conflict. That ends now.”
This is the tone you need.
Calm. Specific. No therapy language. No moralizing. No ambiguity.
4. Could you have to choose one of them?
Yes.
You do not want to. But you need to stop saying “both are irreplaceable.”
That belief is dangerous. If both are irreplaceable, you are not managing a team; you are being held hostage by two single points of failure.
You may eventually have to choose if one of them refuses the operating model.
The choice should not be based on who is “right” in the abstract.
Choose based on these criteria:
1. Who can adapt?
The most valuable senior engineer is not the one with the strongest philosophy. It is the one who can apply the right philosophy to the context.
If Sergey cannot accept speed in low-risk areas, he is too rigid.
If Diana cannot accept rigor in high-risk areas, she is too reckless.
The less adaptable person is the larger long-term risk.
2. Who improves the team around them?
Ask:
- Do others learn from this person?
- Do they increase collective judgment?
- Do they make people afraid?
- Do they create dependency?
- Do they help the team move faster overall, or only personally?
A brilliant engineer who reduces team throughput is not as valuable as they appear.
3. What does the company need in the next 12 months?
If you are pre-product-market fit and most work is reversible, Diana’s style may be more critical.
If you are scaling financial infrastructure, entering heavier regulatory exposure, or suffering reliability incidents, Sergey’s style may be more critical.
But again, the best answer is not “speed person” or “quality person.” The best answer is the person who can operate in a risk-adjusted way.
4. Who respects explicit decisions?
After you define the new system, watch behavior.
Red flags from Sergey:
- Continues to block on preferences.
- Labels everything as “minimum standard.”
- Refuses SLAs.
- Undermines decisions by saying “then we’re just building prototypes.”
- Makes himself the moral guardian of engineering quality.
Red flags from Diana:
- Continues to bypass required reviewers.
- Treats debt tracking as bureaucracy.
- Merges risky changes under speed pressure.
- Uses leaving hints as leverage.
- Frames all architectural concern as control or overengineering.
If one of them cannot follow explicit rules, that person is the problem.
My honest bet
If you implement a clear system, Diana is more likely to adapt quickly because product engineers are used to constraints if they are clear.
Sergey may struggle more if his identity is tied to being the defender of standards. His phrase – “I don’t understand why I’m here” – is a warning. It suggests he sees his value as inseparable from veto power.
But do not assume. Test both under the new model.
5. Concrete two-week plan
You need to move fast. This has already gone on four months too long.
Day 1: Decide and announce that the process is changing
Send a short message to the team.
Example:
“I’ve let review and architecture tradeoffs remain too implicit. That has created friction and slowed us down. We are going to define a clearer engineering decision process: review expectations, blocker definitions, risk levels, ownership, and escalation path. Until then, no approval-shopping and no late architectural rework without clear risk justification. I’ll share the draft this week.”
Do not mention Sergey and Diana by name publicly.
But the team needs to see that you are taking control.
Day 1–2: Hold separate 1:1s
Meet Sergey first, then Diana, or vice versa. Same day if possible.
Objectives:
- Validate the legitimate part.
- Name the unacceptable behavior.
- Explain the new operating model.
- Warn that this is not optional.
Do not ask them, “What do you think we should do?” as an open-ended question. They have already shown you their preferred methods.
Ask instead:
“What would make this operating model workable from your perspective?”
Not:
“Do you agree?”
Day 3: Hold the three-way meeting
Attendees: you, Sergey, Diana.
Length: 60–75 minutes.
Agenda:
1. State the problem – 5 minutes
“This is a conflict about engineering tradeoffs, but it has become a conflict about behavior. PRs are being used for architecture fights. Reviewers are being bypassed. The team is affected. We are fixing the system.”
2. State non-negotiables – 10 minutes
Non-negotiables:
- No approval-shopping.
- No unlimited PR veto.
- Review comments must distinguish blocker/non-blocker.
- High-risk changes get early design review.
- Reviewers respond within SLA.
- Unresolved disagreement escalates quickly.
- Once a decision is made, both disagree and commit.
3. Define risk categories – 15 minutes
Walk through Tier 0–3.
Ask them to classify recent disputed PRs into tiers.
This makes the conversation concrete.
4. Define blocker criteria – 15 minutes
A blocker should be limited to things like:
- Correctness bug.
- Security/privacy issue.
- Data loss/corruption risk.
- Regulatory/compliance risk.
- Broken API or system contract.
- Missing required tests for the risk level.
- Violates agreed architecture/design decision.
- Creates hard-to-reverse coupling in a high-risk area.
Not blockers:
- Naming preference.
- Alternative design preference.
- Style if not violating standard.
- “This could be cleaner.”
- “I would have structured it differently.”
- Historical essay unless directly relevant.
- Refactor outside PR scope unless risk is high.
5. Define escalation – 10 minutes
Rule:
If Sergey and Diana disagree on whether something is a blocker, they have 24 hours to resolve. If unresolved, it escalates to you plus one designated technical decision-maker. Decision within 24 hours. Then both commit.
6. End with commitments – 10 minutes
Ask each:
- “What behavior will you stop?”
- “What behavior will you start?”
- “What do you need from the other person?”
- “What will you do when you disagree next time?”
Write it down.
Day 4: Publish the review standard draft
Make it short enough to actually use.
Title: Engineering Review and Decision Rules
Include these sections:
1. Risk tiers
Tier 0–3 as above.
2. Review SLA
Example:
- First review response within 1 business day.
- For high-risk PRs, reviewer may request more time, but must say so explicitly.
- If no response within SLA, author can escalate or reassign according to CODEOWNERS/risk rules.
- No silent waiting for three days.
3. Comment labels
Require labels:
BLOCKER: must be fixed before merge.SHOULD: recommended, author decides or creates follow-up.NIT: optional.QUESTION: clarification, not blocking by default.DEBT: accepted tradeoff requiring follow-up ticket.
Sergey especially needs this. He can still make comments, but not all comments get blocking power.
4. Definition of blocker
As above.
5. Architecture review rule
For Tier 2–3 changes:
- Design note before implementation.
- Sergey involved early if relevant.
- Once design is agreed, PR review checks implementation against design, not reopens the entire debate.
6. Tech debt rule
Deliberate debt is allowed only if visible.
Debt ticket must include:
- What shortcut was taken.
- Why it was acceptable.
- Risk.
- Owner.
- Trigger or deadline for cleanup.
Example:
“We are duplicating this validation logic to ship experiment X. Acceptable because feature is behind flag and affects less than 5% of users. Cleanup if experiment graduates or within 30 days.”
7. Escalation path
If blocked >24 hours due to disagreement:
- Author and reviewer discuss synchronously.
- If unresolved, escalate to EM + designated technical owner.
- Decision within 24 hours.
- Disagree and commit.
8. Approval-shopping rule
Required reviewers are determined by risk/ownership, not convenience.
If Sergey is required reviewer for Tier 2/3, Diana cannot bypass him.
If Sergey is not required for Tier 0/1, he can comment, but he does not automatically block.
This is the balance.
Day 5: Team workshop
Bring the whole team in.
Purpose: not to relitigate Sergey vs Diana, but to establish the operating model.
Agenda:
- Explain why change is needed.
- Present the risk-tier model.
- Walk through 3–4 real PR examples.
- Ask the team to classify each PR and identify blockers vs non-blockers.
- Adjust obvious gaps.
- Announce that the process starts immediately as a two-week pilot.
Important: do not seek unanimous consensus.
Say:
“I want input, but this is not a consensus vote. As manager, I’m accountable for the tradeoff between speed and quality. We’ll pilot, measure, and adjust.”
This restores leadership.
Week 2: Pilot on real work
Pick two or three active PRs/features and apply the system.
For each:
- Assign risk tier.
- Assign required reviewers.
- Define review SLA.
- Label comments.
- Escalate quickly if blocked.
- Track debt explicitly.
You personally monitor Sergey and Diana’s interactions for two weeks.
Not micromanagement. Stabilization.
During Week 2: Create an architecture pre-review mechanism
Sergey needs a legitimate place to do architecture.
Set up one of these:
Option 1: Architecture office hours
Twice per week, 45 minutes.
Engineers bring upcoming risky changes before coding.
Option 2: Lightweight RFC
For Tier 2–3 changes, require a short design note:
- Problem.
- Proposed approach.
- Alternatives considered.
- Risks.
- Migration/rollback.
- Testing strategy.
- Open questions.
Limit it to one or two pages unless truly complex.
Option 3: ADR log
For major decisions, write an Architecture Decision Record:
- Context.
- Decision.
- Consequences.
- Date.
- Owner.
This prevents every PR from becoming a philosophical debate.
End of Week 2: Review outcomes
Meet with Sergey and Diana separately, then together.
Look at facts:
- PR cycle time.
- Number of blocker comments.
- Number of non-blocking comments.
- Escalations.
- Missed review SLAs.
- PRs merged with documented debt.
- Any quality incidents.
- Team sentiment.
Ask:
- Is review faster?
- Are serious risks still caught?
- Are comments more useful?
- Are disagreements escalating instead of festering?
- Is the team less tense?
Then adjust.
Specific rules I would implement immediately
Here is the operating model I would choose.
Rule 1: PR review is not design review
If the objection is architectural and foreseeable, it should happen before implementation.
Late architectural objections are allowed only when:
- The risk was not visible earlier.
- The implementation violates agreed design.
- The change creates serious correctness/security/data/compliance risk.
This directly addresses Sergey.
Rule 2: Required reviewers cannot be bypassed
If a change touches Sergey’s owned high-risk area, Diana cannot route around him.
If Sergey misses SLA, Diana escalates. She does not approval-shop.
This directly addresses Diana.
Rule 3: Comments must be labeled
Every review comment gets one of:
BLOCKERSHOULDNITQUESTIONDEBT
Unlabeled comments are non-blocking by default.
This will dramatically reduce ambiguity.
Rule 4: Review SLA is mandatory
First response within one business day.
Not final approval. First meaningful response.
If Sergey needs more time:
“This is Tier 2, I need until tomorrow 15:00 to review the data migration risk.”
Fine.
But silence for three days is not acceptable.
Rule 5: Deliberate tech debt is allowed, invisible tech debt is not
Diana can cut corners.
But she must make the tradeoff explicit.
No “we’ll clean it later” without ticket, owner, and trigger.
Rule 6: Escalation is a strength, not a failure
If they disagree for more than 24 hours, escalate.
No passive waiting.
No Slack debates for three days.
No resentment.
No bypassing.
The deeper management lesson
You need to stop letting senior engineers define team culture by force of personality.
Strong engineers often come with strong philosophies. That is fine. But management’s job is to turn competing philosophies into an operating system.
Right now, Sergey and Diana are both trying to answer a question that belongs partly to you:
“What kind of engineering organization are we?”
Your answer should be:
“We are a fintech startup. We move fast where risk is reversible. We are strict where failure affects money, data, security, compliance, or system integrity. We make tradeoffs explicitly. We do not hide them in PR battles.”
That is the position.
Not Sergey’s.
Not Diana’s.
Yours.
Final recommendation
Do not split them as a peace treaty.
Do not force reconciliation.
Do not pick “quality” or “speed” globally.
Do this instead:
- Establish risk-based engineering standards.
- Move architectural review earlier.
- Define blocker vs non-blocker.
- Ban approval-shopping.
- Impose review SLAs.
- Create explicit escalation.
- Make deliberate debt visible.
- Watch who adapts.
If both adapt, you keep two excellent senior engineers with complementary strengths.
If one refuses, that person is not irreplaceable – they are incompatible with the team’s operating model.
Your job in the next two weeks is not to make Sergey and Diana like each other.
Your job is to make it impossible for their conflict to continue shaping the team’s behavior.
~google/gemini-pro-latest
Here is a brutally honest, practical assessment of your situation.
You asked for no HR platitudes, so I will give it to you straight: You are currently failing your team.
You have framed this as a battle between two brilliant engineers, but the other eight engineers staring at the floor during your retro is the real crisis. You are bleeding credibility by letting two loud seniors hold the entire team hostage to their philosophical debate for four months.
In a startup, "engineering standards" are not handed down by God or discovered by ICs; they are dictated by the business context, and you are the translator of that context. Because you haven't defined what "good enough" means for your specific stage of company, they are fighting to fill the leadership vacuum you left behind.
Here are the direct answers to your five questions, followed by your action plan.
1. What’s the ROOT problem?
It is 10% a conflict of values, and 90% a management failure. Sergey and Diana are operating exactly as their archetypes dictate. Their values are completely normal. The root problem is that you have allowed them to try and solve an engineering strategy problem at the Pull Request level.
- Sergey leaving 40 architectural comments on a PR means architecture is happening too late. PRs are for catching bugs and syntax, not debating system design. If he has 40 comments, he should have been consulted before Diana wrote a single line of code.
- Diana bypassing Sergey’s reviews is insubordination disguised as agility. By letting her get away with it, you validated Sergey’s belief that standards are slipping, making him grip tighter.
2. Evaluating Your Options
- (a) Split their areas of responsibility: Temporary yes, permanently no. You shouldn't silo them, but you must implement Pace Layering (separating systems by risk). In fintech, your transaction ledger/core API is a "slow/safe" layer (Sergey's rules apply). Your frontend, user workflows, and experimental features are the "fast/agile" layer (Diana's rules apply).
- (b) Build a team agreement on review standards: Will fail. If you put them in a room to "agree," they will inevitably fight. You don't need a democratic agreement; you need a managerial mandate. You define the framework, the team executes it.
- (c) Structured 3-way conversation: Will fail. They are too polarized right now. They will debate philosophy, not behavior.
- (d) Pick one philosophy and ask the other to adapt: Nuance required. You must pick a philosophy based on the code's blast radius, not the person. Both must adapt to the new context.
3. How to Talk to Each of Them
You need to talk to them aggressively about behavior and process, not feelings.
To Sergey: "Sergey, your architectural vision keeps us from collapsing under tech debt. But your current PR process is breaking the team. A 3-day turnaround on a PR is a system failure. Leaving 40 structural comments on a finished PR is a failure of our planning process, not a failure of Diana's coding. From now on, architectural debates happen before the code is written, via quick Design Docs. If a PR meets the specs of the design doc, it passes. Your 3-minute history lessons at standup stop tomorrow–standup is for status, not lectures. I need you as an architect who accelerates the team safely, not a gatekeeper who freezes it."
To Diana: "Diana, your iteration speed is exactly what this startup needs to survive. But bypassing team review processes by shopping for easy approvals is unacceptable. Hacking core systems creates tech debt that will eventually kill our velocity. You are treating Sergey like an obstacle instead of an asset. From now on, for any feature that touches [List of Core Systems] or takes more than 3 days to build, you will write a 1-page technical spec and get Sergey's sign-off before you code. In return, I am enforcing a 24-hour SLA on PR reviews so you are never blocked. If you circumvent the review process again, it will be a formal performance issue."
4. Do you have to choose one?
Yes, you might. Stop believing they are irreplaceable. The moment a manager believes a report is irreplaceable, the manager becomes a hostage. If one of them quits, your velocity might dip for two months, but the relief felt by the other 8 engineers will likely result in a massive long-term productivity spike.
How to choose? You evaluate them based on their adherence to the new rules you are about to set.
- If Sergey cannot stop nitpicking and blocking business-critical delivery, he lacks the pragmatism required for a startup. He should go to a Big Tech company.
- If Diana continues to cowboy-code, break processes, and show contempt for systemic stability, she poses a catastrophic risk to a fintech company. She should go to an early-stage consumer app.
- Keep the one who can adapt their philosophy to the company's reality.
5. Your 2-Week Concrete Action Plan
Week 1: Stop the Bleeding and Set the Law
- Monday: Have the 1:1s described in Question 3. Be firm, do not smile too much, let the silence hang after you deliver the core message.
- Tuesday: Draft your new Engineering Playbook. It must include:
- Risk Tiers: Define what code is "Tier 1" (Core fintech, ledger, security - needs deep architectural review, high test coverage) and "Tier 2" (UI, growth experiments - move fast, break things safely).
- Shift-Left Architecture: Any Tier 1 work, or any task larger than 3 days, requires a brief RFC/Design Doc. The architecture is debated here, not in the PR.
- PR SLA: Reviews must be done within 24 hours. Comments must be labeled (e.g.,
[BLOCKING]for bugs/security or[NIT]for style/suggestions which can be ignored to ship).
- Thursday: Call a team meeting (all 10). Acknowledge your failure. Say: "I allowed our engineering standards to become a gray area, which caused friction. That is on me. Here is how we build software from today forward." Present the Playbook.
Week 2: Ruthless Enforcement
- Daily: Monitor PRs closely.
- If Sergey leaves 40 comments on a PR without an RFC, you step in on the PR: "Sergey, these are architectural changes. We missed this in planning. Let's merge this as it meets the current requirement, and open a ticket to refactor based on your notes."
- If Diana bypasses a review on a Tier 1 system, you revert the PR. "Diana, this touched the core API and bypassed the Tier 1 review process. Getting it reviewed now."
- Friday: Review the week. Have a 15-minute sync with both Sergey and Diana individually. Validate them where they complied, correct them immediately where they slipped.
Right now, they are fighting for control of the steering wheel. Your job isn't to ask them to share it; your job is to take it back.
Analysis
Consensus
- Root problem is primarily a management failure, not fundamentally an interpersonal or values conflict – the manager left a decision-making/standards vacuum that two strong people filled by force.
- The values divergence (speed vs. quality) is legitimate and healthy tension; the pathology is that it's being fought through code review rather than through defined process.
- The core reframe is risk-adjusted engineering: rigor should scale to the code's blast radius. Fintech core systems (money, ledger, auth, compliance) deserve Sergey's rigor; UI/experiments/reversible features deserve Diana's speed.
- Option (d) – picking one philosophy globally and asking the other to adapt – is wrong and would cost one engineer.
- Option (b) – team standards – is necessary but must be concrete and enforceable, not fluffy; and it must be a managerial mandate, not a democratic negotiation the two will re-litigate.
- Option (a) – splitting responsibility – is useful only as deliberate risk-tiered ownership design, not as conflict avoidance / siloing.
- Option (c) – the 3-way conversation – should NOT be run as reconciliation or a feelings meeting; it must establish rules of engagement and decision rights with the manager chairing.
- Architecture must happen earlier (before code is written via design docs/RFCs), not at PR time – 40 late comments signal a planning/process failure.
- Diana's approval-shopping (routing around Sergey) is unacceptable and must stop; escalation should replace bypassing.
- Sergey's behaviors must change: no 3-day silent holds, enforce a review SLA (~24h first response), and distinguish blockers from non-blockers.
- Both must be told they are heard on substance but that current behavior is unacceptable – validate + hard line, delivered in separate 1:1s before any joint meeting.
- The manager must stop treating both as 'irreplaceable'; that belief is what allowed the conflict to fester and makes the manager a hostage.
- If a choice must be made, decide based on who can adapt to and operate inside the new shared system – coachability and behavior, not talent or philosophy.
- The other 8 engineers staring at the floor is a critical signal that the team is watching whether leadership and fair process exist.
- Concrete 2-week plan: manager does homework/drafts tiered framework first → separate 1:1s → structured 3-way → publish standards → team rollout → pilot and monitor.
- Deliberate tech debt should be allowed only if made explicit (ticket, risk, owner, cleanup trigger).
- The manager should own and announce the failure/gap publicly and reassert authority as the person who defines the game rather than acting as a neutral referee.
Contradictions
- Whether to send a team-wide message on Day 1 before the 1:1s
- ~openai/gpt-latest: Day 1: send a short team-wide message announcing the process is changing (no approval-shopping, no late rework) before/around the 1:1s to signal the manager is taking control.
- ~anthropic/claude-opus-latest: Do the private 1:1s and draft framework first; bring it to the full team only in Week 2 after ratification, to de-personalize it.
- ~google/gemini-pro-latest: Hold 1:1s Monday, then the full team meeting Thursday of Week 1 where the manager acknowledges failure and presents the playbook.
- How firmly to predict who is more likely to adapt / be the flight risk
- ~anthropic/claude-opus-latest: Diana is the higher near-term flight risk (active, 'boiling', has options), while Sergey's disengagement is the more insidious long-term risk as the fintech guardrail; don't over-index on the loud one.
- ~openai/gpt-latest: Bets Diana is more likely to adapt (product engineers accept clear constraints) and Sergey may struggle because his identity is tied to veto power, but says to test both.
- ~google/gemini-pro-latest: Frames it symmetrically as a stark either/or (Sergey belongs at Big Tech if he can't be pragmatic; Diana belongs at an early-stage consumer app if she can't respect stability) without a flight-risk asymmetry.
- Tone/aggressiveness of the manager's posture
- ~google/gemini-pro-latest: Most aggressive: 'You are failing your team,' revert PRs, stop Sergey's standup lectures 'tomorrow,' 'take the steering wheel back,' losing one may yield a productivity spike.
- ~anthropic/claude-opus-latest: Firm but more measured and sequenced; emphasizes validation paired with hard lines and warns against over-disciplining or acting purely as enforcer.
- ~openai/gpt-latest: Direct and non-negotiable but process-systems oriented; frames consequences (reduced approval authority, performance issue) rather than combative rhetoric.
Partial coverage
- ~anthropic/claude-opus-latest: Explicitly diagnoses the retro exchange as a public proxy war with two ultimatums ('prototypes not products' / 'places that value speed'), concluding the manager has less time than they think.
- ~anthropic/claude-opus-latest: Frames the 3-way meeting specifically as a 'decision-ratification' meeting rather than a negotiation, requiring explicit verbal commitment out loud from each in front of the other.
- ~anthropic/claude-opus-latest: Emphasizes putting tech debt on the roadmap as a shared external enemy to convert two people fighting each other into two fighting the same problem.
- ~openai/gpt-latest, ~google/gemini-pro-latest: Proposes explicit comment labels (BLOCKER/SHOULD/NIT/QUESTION/DEBT) with unlabeled comments non-blocking by default.
- ~openai/gpt-latest: Provides a detailed 4-tier risk taxonomy (Tier 0 experiment, Tier 1 normal feature, Tier 2 high-risk fintech, Tier 3 architectural) with per-tier review rules and concrete fintech examples (KYC/AML, reconciliation, PII, migrations).
- ~openai/gpt-latest: Specifies concrete architecture pre-review mechanisms as alternatives: architecture office hours, lightweight RFC template, and ADR log.
- ~openai/gpt-latest: Provides explicit end-of-pilot metrics to review (PR cycle time, blocker vs non-blocker counts, escalations, missed SLAs, debt tickets, incidents, sentiment).
- ~openai/gpt-latest, ~anthropic/claude-opus-latest: Explicit 'disagree and commit' escalation mechanism with a designated technical decision-maker and 24-hour resolution windows.
- ~google/gemini-pro-latest: Names 'Pace Layering' as the concept for separating systems by risk (slow/safe core vs fast/agile frontend).
- ~google/gemini-pro-latest: Gives concrete real-time enforcement moves for Week 2 (manager stepping into the PR to merge and file a refactor ticket; reverting a bypassed Tier-1 PR).
- ~google/gemini-pro-latest: Explicitly addresses stopping Sergey's 3-minute standup history lectures as a distinct behavior to correct.
- ~anthropic/claude-opus-latest, ~google/gemini-pro-latest: Argues losing one strong engineer may relieve the other 8 and yield a net long-term productivity/team-health gain.
- ~openai/gpt-latest: Frames Sergey's 'why am I here' line as a specific warning that he sees his value as inseparable from veto power.
- ~openai/gpt-latest: Recommends asking the reports 'What would make this model workable for you?' rather than open-ended 'Do you agree?' or 'What should we do?'
- ~openai/gpt-latest, ~google/gemini-pro-latest: Ties company-stage/lifecycle (pre-PMF vs scaling/regulatory exposure) to which philosophy is more critical in the next 12 months.
Unique insights
- ~anthropic/claude-opus-latest: Diagnoses the conflict as a three-stage sequence – values difference → management failure → curdling into a people conflict – and argues the sequence matters because it locates where the fix lives (process AND direct intervention, since attribution of motive has begun hardening).
- ~anthropic/claude-opus-latest: Reframes the disqualifying trait for either engineer as valuing 'being right over being effective on a team,' and warns 'talent you can't align is negative leverage' because it teaches the other eight that stars break rules – the loss you actually can't afford.
- ~anthropic/claude-opus-latest: Insists the manager must incorporate genuine feedback from each into the framework so it is co-owned in substance even while the manager owns the frame – and that the tier framework should be pitched to Sergey as a MORE senior skill (knowing where to spend rigor), which is 'catnip to an architect.'
- ~openai/gpt-latest: Sharpest structural diagnosis: 'PR review is too late, too tactical, too personal, and too slow' to be where architecture, product urgency, quality bar, and decision rights get negotiated – the central failure is using PR review as the arena.
- ~openai/gpt-latest: Characterizes Sergey as exercising 'architectural veto' because he lacks a structured architectural authority model, reframing his behavior as a symptom of missing role design rather than character.
- ~google/gemini-pro-latest: Names the leadership-vacuum mechanism bluntly: in a startup, standards are dictated by business context and the manager is 'the translator of that context'; the two are fighting to fill the vacuum the manager left.
- ~google/gemini-pro-latest: Behavioral coaching detail for the 1:1s: deliver the core message firmly, 'do not smile too much,' and let the silence hang after delivering it.
Blind spots
- None of the models seriously weighs whether promoting one to a formal Tech Lead / Staff Architect role (giving Sergey legitimate, scoped architecture authority as a title/career path) could resolve the authority vacuum – they design process but not org-structure/leveling changes.
- No model addresses the manager's own accountability upward – how to communicate the tech-debt risk and the speed/quality tradeoff to the business/founders/leadership, who ultimately set the risk appetite the manager is supposed to translate.
- Little attention to the emotional/relational repair needed after months of hardened resentment; all pivot hard to process, but a purely procedural fix may not undo the personal contempt that has formed (a risk Opus flags but none plan for concretely).
- No model discusses retention/compensation or counteroffer dynamics if Diana's 'places that value speed' hint becomes a real resignation, nor the practical mechanics of a managed exit if one must leave (knowledge transfer, bus-factor mitigation, timing).
- The impact on and voice of the silent 8 is noted as a signal but never operationalized – none propose gathering their input, surfacing whether they secretly agree with one side, or checking if the 'shadow approvals' implicate those colleagues too.
- No model questions its own core assumption that a risk-tier framework will actually be adopted rather than becoming the next battleground (Opus flags this risk for option (b) generally but still bets the tiering will hold); little contingency if BOTH refuse.
- Legal/regulatory specifics of fintech (audit trails for code review, SOC2/change-management controls, separation-of-duties) are gestured at via 'compliance' tiers but never treated as a hard constraint that may forbid approval-shopping or mandate certain reviewers regardless of preference.
- None address measuring whether Sergey's tech-debt warnings are objectively validated (incident data, outage rates) versus asserted – the plan largely takes 'both are right' as given rather than instrumenting the actual debt/velocity tradeoff.
- No model considers the manager's own capacity/time realism – running 1:1s, drafting frameworks, chairing meetings, and daily PR monitoring across a 10-person team in two weeks may be unrealistic alongside other duties.
Три независимые модели сошлись в диагнозе почти дословно. Ниже – синтез, где я отмечаю, где они согласны (высокая уверенность) и где расходятся (там решение за вами).
1. Корневая проблема: это управленческий провал, замаскированный под конфликт ценностей
Все три модели независимо пришли к одному выводу, и я с ним согласен. Последовательность такая:
Началось как здоровое ценностное расхождение → превратилось в управленческий вакуум → скисло в личный конфликт.
Расхождение Сергея и Дианы (долговечность vs скорость) – это не патология. Это правильное напряжение для финтех-стартапа, которому нужно и быстро выходить на рынок, и не терять деньги пользователей. В здоровой команде это актив.
Патология в другом: у вас нет общего определения «готово», нет владельца стандарта, нет SLA на ревью, нет пути эскалации разногласий. В этом вакууме два сильных человека сделали рациональную вещь – каждый начал навязывать свой стандарт силой. Сергей – через шлюз ревью (40 комментариев, трёхдневные задержки). Диана – обходя шлюз (approval-shopping). Оба поведения – симптомы неуправляемой системы, а не дефекты характера.
Самое острое наблюдение (от нескольких моделей): вы ведёте ревью-войну там, где ей не место. 40 архитектурных комментариев на готовый PR означают, что архитектура обсуждается слишком поздно. Если у Сергея 40 замечаний – с ним должны были поговорить до того, как Диана написала первую строку. PR-ревью стало ареной, где решаются вопросы архитектуры, продуктовой срочности, планки качества и права принятия решений. Оно для этого слишком позднее, слишком тактическое, слишком личное и слишком медленное.
И третий слой, который вы недооцениваете: на ретро это была не дискуссия, а публичная прокси-война с двумя ультиматумами. «Скажем бизнесу, что строим прототипы» и «есть места, где ценят скорость» – это не инженерные позиции. Это ультиматумы. У вас меньше времени, чем кажется. Восемь человек, смотрящих в пол, – это сигнал, что команда уже впитывает эту небезопасность и ждёт, будете ли вы делать свою работу.
2. Ваши четыре опции – честная оценка
Единодушие моделей: правильный ответ – не одна из a/b/c/d, а их секвенированная комбинация, с явным отказом от (d).
(d) Выбрать одну философию глобально – НЕТ. Это ловушка. Кажется решительным, но «выбрать скорость» = потерять Сергея, «выбрать качество» = потерять Диану. Выбор философии и есть выбор потерять одного, только окольным путём.
(b) Командное соглашение по стандартам – НЕОБХОДИМО, но недостаточно само по себе. И критично: это должен быть управленческий мандат, а не демократическое голосование, которое они переиграют. Написанное посреди активной войны «соглашение» станет очередным полем боя, если вы предварительно не переустановили себя как авторитет, задающий рамку.
(c) Разговор втроём – ДА, но не первым и НЕ как примирение. Если запустите как «давайте поймём друг друга» – Диана сочтёт это театром, Сергей переспорит стандарты. Работает только как встреча-ратификация: 1:1 уже проведены, фреймворк уже набросан, а втроём вы приземляете соглашение и получаете явное устное обязательство от обоих.
(a) Разделение зон ответственности – ДА, это ваш главный структурный рычаг. Но не как побег от конфликта (это силосы и обида), а как осознанный дизайн: строгость масштабируется по blast radius кода.
Реформулировка, которая ломает тупик
Вся драка исходит из ложной посылки, что есть один правильный уровень строгости для всего кода. Его нет. В финтехе это не философия, а конкретика:
- Движение денег, ledger, auth, комплаенс, целостность данных → философия Сергея правильна. Баг тут теряет деньги, ломает доверие или тянет регуляторные проблемы. Медленное глубокое ревью – верное решение. Не обсуждается.
- UI, эксперименты, growth-фичи, внутренние инструменты, всё за feature-флагом → философия Дианы правильна. За 2 дня, проверить на данных, итерировать. 40-комментарийное ревью на флагованном эксперименте – активный вред бизнесу.
Это превращает «скорость vs качество» (неразрешимое, на уровне идентичности) в «в каком тире этот код» (решаемое, техническое). Вы переводите ценностную войну в задачу классификации рисков – то, что инженеры реально умеют решать, и где оба могут быть правы там, где они правы.
3. Разговоры 1:1 (до любой встречи втроём)
Структура обоих одинакова: валидация по существу + жёсткая линия по поведению. Ловушка – сделать только одно. Только валидация → ничего не изменится. Только дисциплина → потеряете человека.
Сергею:
«Сергей, сначала прямо: по техдолгу ты был прав. Я игнорировал предупреждения, которые не должен был – это на мне. Твоя планка для core-систем – именно то, что нужно компании, и я не прошу её снижать. Мне она нужна.
Вот где нужен рост. Строгость – это ресурс, и сейчас ты тратишь его равномерно: та же 40-комментарийная глубина на флагованном эксперименте, что и на ledger. На core это правильно. На одноразовом growth-тесте это стоит нам скорости, которую в этом рынке мы не можем себе позволить, и продукт от этого не становится безопаснее. Мне нужно, чтобы ты стал тем, кто знает, где тратить строгость – это более старшая компетенция, не меньшая.
И две вещи прекращаются. PR, лежащий три дня без ревью – недопустимо, теперь на это есть SLA. И комментарий на ретро про «прототипы» – это был публичный ультиматум, и это ниже твоего уровня. Претензии к направлению приходят ко мне напрямую, не публичной бомбой.»
Реформулировка тир-фреймворка как более сениорного навыка – это «catnip для архитектора». Если он отказывается принять, что разный код заслуживает разной строгости – это серьёзный сигнал.
Диане:
«Диана, твоя скорость итерации – реальное оружие, и компании оно нужно, чтобы выжить. Ты права, что нельзя каждый PR превращать в курсовую. Слышу это.
И вот прямая часть. На продуктовой поверхности и экспериментах твой инстинкт ship-and-learn правильный, я формально его защищу. Но на core-финсистемах планка Сергея – это не он контролирует тебя. Это планка, без которой компания буквально не выживет, потому что мы финтех. Там глубокое ревью – не бюрократия, это работа.
И поведение прекращается сейчас. Обход Сергея за более лёгким approve – это не решение проблемы, это подрыв системы ревью. Если ревью слишком медленное или глубокое для тира кода – эскалируешь ко мне, не ищешь мягкого ревьюера. И насчёт «мест, где ценят скорость»: если хочешь поговорить о своём росте и что нужно, чтобы остаться – моя дверь открыта, я хочу этого разговора. Но я не буду вести переговоры под угрозой, брошенной вскользь. Прямо: ты хочешь быть здесь?»
Реальный риск Дианы – не техдебат, а ощущение, что её полицействуют. Тир-фреймворк даёт ей защищённый домен, где ей явно доверяют двигаться быстро. Это то, что покупает её лояльность. Если она услышит только «уважай ревью Сергея» – вы её потеряете.
4. Сценарий, где придётся выбирать
Перестаньте считать их незаменимыми. Все три модели сходятся: эта вера – то, почему конфликт тянулся четыре месяца. Если оба незаменимы – вы не управляете командой, вы заложник двух single points of failure.
Выбор – НЕ по философии (обе – кипер). Выбор – по тому, кто способен работать внутри общей системы. Дисквалифицирующая черта: ценить свою правоту выше эффективности в команде. И это может быть любой из них:
- Если Сергей настаивает, что каждая строка заслуживает ledger-уровня строгости и воспринимает тир-фреймворк как предательство – он выбрал идеологию.
- Если Диана продолжает обходить ревью и хочет отсутствия надзора вообще – она выбрала автономию выше команды.
Ключевая мысль: талант, который вы не можете выровнять, – это отрицательный рычаг. Он учит остальных восьмерых, что звёзды нарушают правила – и вот это уже потеря, которую вы правда не можете себе позволить.
Здесь модели разошлись (решение за вами, знаете контекст лучше):
- Одна: Диана – больший риск ухода в краткосроке (активна, «закипает», есть опции), но отход Сергея коварнее в долгосроке как ваш финтех-guardrail. Не переоценивайте громкого.
- Другая: ставит, что Диана адаптируется легче (продуктовые инженеры принимают чёткие ограничения), а Сергей может застрять, если его идентичность привязана к праву вето. Его «зачем я здесь» – предупреждение.
Не предполагайте – протестируйте обоих под новой системой и дайте их поведению решить.
5. Конкретные две недели
Дни 1–2: сначала ваша домашняя работа.
- Набросайте сами (или с нейтральным staff-инженером, не с Сергеем/Дианой) тир-фреймворк по blast radius: Tier 1 (деньги/ledger/auth/комплаенс) = глубокое ревью, 2 ревьюера; Tier 2 (core product logic) = стандартное, SLA 24ч; Tier 3 (эксперименты/UI/флаги) = лёгкое, ship-and-iterate.
- Определите SLA на ревью (первый ответ за 24ч) и путь эскалации (неразрешённое за один раунд идёт к вам, не превращается в трёхдневный стендофф и не обходится).
Дни 3–5: два 1:1 (по возможности в разные дни). Скрипты выше. Цель: услышаны по существу + ноль двусмысленности, что поведение прекращается + вы прочитали их реакцию на фреймворк (это ваш индикатор из п.4).
Дни 6–7: интегрируйте их обратную связь в фреймворк – реально возьмите что-то от каждого, чтобы он был со-владельческим по сути, хотя рамку держите вы.
Неделя 2, начало: встреча втроём – как ратификация, не примирение.
- Вы ведёте. Открываете, владея провалом: «Это слишком долго оставалось неуправляемым. Это на мне. Вот система, по которой мы работаем теперь». Вы не нейтральный медиатор – вы менеджер, устанавливающий решение.
- Пройдите фреймворк. Получите явное устное согласие от каждого, вслух, при другом.
- НЕ переигрывайте историю. «Мы решаем вперёд, а не переспориваем последние четыре месяца.»
- Назовите прекращающиеся поведения: никакого обхода ревью, никаких трёхдневных молчаливых задержек, никаких публичных ультиматумов.
Неделя 2, дальше: закрепить и сделать командным.
- Вынесите фреймворк на всю команду (короткий док + 20-мин обсуждение), чтобы стандарт был системой команды, а не «мирным договором Сергея и Дианы». Скажите прямо: «Я допустил, что стандарты стали серой зоной. Это на мне. Вот как мы строим софт с сегодняшнего дня.» Это деперсонализирует и даёт восьмерым молчунов ясность.
- Задайте каденс проверок: следующие два спринта вы лично смотрите, соблюдается ли SLA и правильные ли ревьюеры.
Сквозной приём – атакуйте общего врага. Поставьте техдолг на роадмап явно, с выделенной ёмкостью. Это самый объединяющий ход: валидирует Сергея (он был прав, теперь это решается системно) и освобождает Диану (выплата долга структурно = её перестаёт блокировать шаткий фундамент). Общая внешняя проблема – быстрейший способ превратить двух дерущихся друг с другом в двух дерущихся с одним и тем же.
Две вещи, которые модели упустили – и на которые обратите внимание сами
Оргструктура, не только процесс. Ни одна модель всерьёз не рассмотрела: возможно, дать Сергею формальную роль Tech Lead / Staff Architect с легитимной, ограниченной архитектурной властью (титул + карьерный трек). Это может закрыть вакуум авторитета лучше любого процесса – его «архитектурное вето» превратится в санкционированную роль на ранней стадии, а не в захват PR-шлюза.
Ваша подотчётность наверх. Планы описывают процесс внутри команды, но не то, как вы коммуницируете риск техдолга и tradeoff скорость/качество основателям/лидерству, которые в итоге и задают аппетит к риску, который вы должны транслировать. Аппетит к риску – не ваше решение в вакууме. Сходите к бизнесу и получите явный сигнал: насколько мы готовы жертвовать скоростью ради надёжности прямо сейчас? Это даст вашему тир-фреймворку легитимность, идущую сверху, а не «мнение менеджера».
И проверьте себя на реализм по времени: 1:1, драфт фреймворка, встреча втроём, ежедневный мониторинг PR по 10-человечной команде за две недели – плюс всё остальное – это плотно. Возможно, растяните на 3 недели, но начните 1:1 на этой неделе. Ультиматумы на ретро были предупреждением – часы тикают быстрее, чем кажется.
Одним предложением: это управленческий вакуум, в котором здоровое напряжение сгнило в войну; чинится заменой «скорость vs качество» на «строгость по мере риска», установкой себя как авторитета-владельца стандарта, жёсткими поведенческими линиями в приватных 1:1 и публичной ратификацией тир-системы – а если кто-то всё равно не сможет работать внутри честной структуры, то именно этот отказ, а не его философия, скажет вам, кого отпустить.
What the judges noted
Claude Opus 5.5 · this answer: 8.6 of 10
A substantively excellent, honest answer that matches the reference diagnosis and plan closely. Strengths include the risk-tiered reframe, scripts that pair validation with a hard behavioral line, and choosing based on ability to operate inside a shared system. It also transparently separates consensus from disagreement. Usability is significantly hurt by exposing three full English panel responses before the Russian synthesis. Missing elements include a PR audit, phrases to avoid, and explicit blocker criteria and metrics in the final plan. The Tech Lead promotion suggestion runs against best practice.
- Strong: Accurate multi-layered diagnosis: management vacuum, the wrong arena for architecture debates, and the 8 silent people as the key signal
- Weak: Extreme length and mixed languages from including full English panel responses
GPT-6.2 Astra · this answer: 6.5 of 10
The response offers a useful management intervention: set review depth by risk, settle major design questions before coding, and give both engineers a clear route for escalating disputes. Its process advice is stronger than its personnel advice. A manager could pilot the review rules immediately, but should verify actual PR examples before giving corrective feedback and treat Diana’s remark as a reason for a private retention conversation, not proof of an ultimatum.
- Strong: Connects review delays, alternate approvals, and team silence to unclear decision rights.
- Weak: Misattributes Diana’s departure hint to the retrospective and turns an unverified PR example into scripted feedback.
The score is for this one answer, not the model’s overall result.
Weakest: Data analysis · 6.2An onboarding A/B test: «it's obvious, roll it out»
Three weeks of an A/B test of a new onboarding: 37.4% vs 42.7% completed, plus a weekly breakdown. The marketer says «it's obvious, roll it out». Can the result be trusted?
Full task
We ran an A/B test of new onboarding in our SaaS product. The marketer sent over the results and said: "It's obvious – the new onboarding works, we need to roll it out immediately." I'm not sure, and I want to work through this myself before making a decision. Test parameters: - The test ran for 3 weeks (February 21–28 and March 1–14, 2025) - Group A (control, old onboarding): 340 new users - Group B (new onboarding with video tour): 356 new users - Success metric: onboarding completion (user completed 5 key actions) Results: - Group A: 127 of 340 completed onboarding = 37.4% - Group B: 152 of 356 completed onboarding = 42.7% - Difference: +5.3 pp (+14.2% relative growth) Additional weekly breakdown: | Week | Group A (conversion) | Group B (conversion) | |--------|---------------------|---------------------| | Feb 21–28 | 38.1% (n=89) | 44.2% (n=86) | | Mar 1–7 | 36.9% (n=130) | 43.1% (n=131) | | Mar 8–14 | 37.2% (n=121) | 41.1% (n=139) | Additional facts: - P-value of the test according to the marketer: "about 0.07 – almost significant" - Sample size was calculated for 80% test power and an MDE of 8%, but the actual observed difference was 5.3% - In the March 8–14 week, more users ended up in Group B because the ad channel temporarily switched (this was unplanned) - The video tour in the new onboarding is available only on desktop; mobile users get the text version - The share of mobile users in Group B turned out to be 31%, versus 44% in Group A (randomization was violated) The marketer insists: "A 5% lift is a lot of money, we need to launch it." The product team has doubts. Help me work through this: 1. Can these results be trusted? Are there methodological problems with the test? 2. What does a p-value of 0.07 mean in practice – and why might the marketer be misinterpreting it? 3. What factors could have distorted the results? How serious are these distortions? 4. What needs to be done to get a reliable answer to the question "does the new onboarding work?" 5. How should I frame the position for management – without technical jargon, but honestly?
System line: You are an assistant for managers in preparing professional communication.
Translated from Russian: the models got the Russian text.
Original in Russian
Мы провели A/B тест нового онбординга в нашем SaaS-продукте. Маркетолог прислал результаты и сказал: «Всё очевидно – новый онбординг работает, нужно внедрять немедленно». Я не уверен и хочу разобраться сам перед тем, как принимать решение. Параметры теста: - Тест шёл 3 недели (21–28 февраля и 1–14 марта 2025) - Группа A (контроль, старый онбординг): 340 новых пользователей - Группа B (новый онбординг с видео-туром): 356 новых пользователей - Метрика успеха: завершение онбординга (пользователь выполнил 5 ключевых действий) Результаты: - Группа A: 127 из 340 завершили онбординг = 37,4% - Группа B: 152 из 356 завершили онбординг = 42,7% - Разница: +5,3 п.п. (+14,2% относительный рост) Дополнительная разбивка по неделям: | Неделя | Группа A (конверсия) | Группа B (конверсия) | |--------|---------------------|---------------------| | 21–28 фев | 38,1% (n=89) | 44,2% (n=86) | | 1–7 мар | 36,9% (n=130) | 43,1% (n=131) | | 8–14 мар | 37,2% (n=121) | 41,1% (n=139) | Дополнительные факты: - P-value теста по словам маркетолога: «около 0,07 – почти значимо» - Размер выборки рассчитывался на 80% мощность теста и MDE 8%, но реальная наблюдаемая разница составила 5,3% - В неделю 8–14 марта в Группу B попало больше пользователей, потому что рекламный канал временно переключился (это было незапланировано) - Видео-тур в новом онбординге доступен только на десктопе; мобильные пользователи получают текстовую версию - Доля мобильных пользователей в Группе B оказалась 31%, в Группе A – 44% (рандомизация была нарушена) Маркетолог настаивает: «5% прирост – это большие деньги, надо запускать». Продуктовая команда сомневается. Помоги мне разобраться: 1. Можно ли доверять этим результатам? Есть ли методологические проблемы в тесте? 2. Что означает p-value 0,07 на практике – и почему маркетолог, возможно, неправильно его интерпретирует? 3. Какие факторы могли исказить результаты? Насколько серьёзны эти искажения? 4. Что нужно сделать, чтобы получить надёжный ответ на вопрос «работает ли новый онбординг»? 5. Как мне сформулировать позицию для руководства – без технического жаргона, но честно?
OpenRouter Fusion
Translated from Russian: the model answered in Russian.
I will work through this using analysis across several models, since there are serious statistical subtleties here where it's important not to get it wrong.
Panel responses
~anthropic/claude-opus-latest
I'll work through this rigorously with actual computations, then give you a leadership-ready framing. Let me verify the statistics first.Interesting – the reported p-value of 0.07 does not reconcile with a standard two-sided test. Let me check whether a one-sided test produces ~0.07, and then do the confounding analysis.The 0.07 is a one-sided p-value. That's a critical finding. Now the confounding/Simpson's analysis.I now have everything I need. Here is the complete analysis.
Original in Russian
Я разберу это с помощью анализа несколькими моделями, поскольку здесь есть серьёзные статистические тонкости, где важно не ошибиться.
Panel responses
~anthropic/claude-opus-latest
I'll work through this rigorously with actual computations, then give you a leadership-ready framing. Let me verify the statistics first.Interesting – the reported p-value of 0.07 does not reconcile with a standard two-sided test. Let me check whether a one-sided test produces ~0.07, and then do the confounding analysis.The 0.07 is a one-sided p-value. That's a critical finding. Now the confounding/Simpson's analysis.I now have everything I need. Here is the complete analysis.
Read the full answer · 7,572 words
Panel responses
~anthropic/claude-opus-latest
I'll work through this rigorously with actual computations, then give you a leadership-ready framing. Let me verify the statistics first. Interesting – the reported p-value of 0.07 does not reconcile with a standard two-sided test. Let me check whether a one-sided test produces ~0.07, and then do the confounding analysis. The 0.07 is a one-sided p-value. That's a critical finding. Now the confounding/Simpson's analysis. I now have everything I need. Here is the complete analysis.
A/B Test Analysis: New Onboarding with Video Tour
The bottom line up front
The marketer is wrong, but not because the new onboarding definitely fails – because this test cannot answer the question it was built to answer. Two problems dominate the decision, and either one alone is disqualifying:
- The "p = 0.07" is a one-sided p-value. The correct two-sided p-value is ≈ 0.15 – twice the reported number and nowhere near significance.
- Randomization is broken on the single variable the treatment depends on (device). The video tour only works on desktop, and the control group happened to have 44% mobile vs. 31% in the treatment group. A meaningful chunk – plausibly a third to a half – of the observed "lift" could be pure composition, not the onboarding change.
Recommendation: Do not deploy. Fix the randomization and re-run. The current test is not evidence of a working feature; it's a signal worth investigating properly.
Question 1 – Can these results be trusted? Methodological problems
No. Ranked by severity:
A. Broken randomization (fatal). Mobile share is 44% in A and 31% in B. Randomization exists precisely so the two groups differ only by treatment. Here they differ by device mix – and device is directly coupled to the treatment (video is desktop-only). This is the textbook setup for a confounded result. When randomization visibly fails on a variable that interacts with the treatment, the experiment loses its main claim to causal validity.
B. Underpowered for the effect that actually exists (fatal for "significance"). The test was designed to detect an 8pp difference, needing 592/group. To detect the observed 5.3pp at 80% power you'd need **1,340 per group** – nearly four times what you ran. With 348/group, your actual power to detect a 5.3pp effect was only **30%**. A study with 30% power that returns a non-significant result tells you almost nothing; you were unlikely to reach significance even if the effect is real.
C. Unplanned mid-test change (serious). In week 3 an ad channel was switched, giving B extra users (139 vs. 121) and a different traffic source. Different acquisition channels bring different user intent and quality. This is both a sample-ratio and a sample-composition problem introduced during the test.
D. Odd, non-contiguous date range (flag). "Feb 21–28 and Mar 1–14" is stated as 3 weeks but the weekly buckets don't cleanly tile; worth confirming no data was dropped or double-counted. Any manual date stitching invites errors.
E. Possible peeking / one-sided framing (serious). Reporting a one-sided p as "almost significant" suggests the analysis was framed to make the result look as strong as possible – a decision-quality red flag independent of the math.
Question 2 – What does p = 0.07 mean, and is it even correct?
What a p-value means: it's the probability of seeing a difference at least this large if the new onboarding truly had no effect (the null hypothesis). It is not the probability the feature works, not the probability the result is "real," and 0.07 does not mean "93% likely to work." Those are the misinterpretations the marketer is almost certainly making.
Now the verification – this is the key catch. I computed the two-proportion z-test three ways:
| Test | p-value |
|---|---|
| Two-sided pooled z-test | 0.150 |
| Fisher's exact (two-sided) | 0.164 |
| Chi-square with continuity correction | 0.174 |
| One-sided pooled z-test | 0.075 |
The reported "≈ 0.07" is the one-sided value. The industry-standard test for a two-way A/B comparison is two-sided, which gives p ≈ 0.15 – roughly double, and far from any conventional threshold.
Using a one-sided test doubles your effective false-positive rate and is only legitimate if you pre-committed to caring only about improvement and would treat "B is worse" identically to "no difference." That's rarely defensible for a launch decision. So even at face value, and before touching confounders, this result is not statistically significant. The 95% confidence interval on the lift is [−1.9pp, +12.6pp] – it comfortably includes zero (and even a small negative effect).
The marketer's likely errors, concretely:
- Treating "almost significant" as "significant."
- Not knowing (or not disclosing) it's one-sided.
- Reading 0.07 as "93% chance it works."
- Ignoring that even the point estimate (+5.3pp) is below the effect the test was designed to detect.
Question 3 – Confounders and the device-mix sensitivity analysis
The dominant confounder is device mix, and it's especially dangerous because it's mechanically linked to the treatment (video = desktop-only). Let me quantify how much of the 5.3pp could be composition rather than causation.
The core algebra. Suppose the new onboarding truly did nothing (null), so desktop and mobile convert at identical rates in both groups. Then a group's overall rate is just a weighted average of device rates. The difference between groups reduces cleanly to:
$$p_B - p_A = (\text{mobile}_A - \text{mobile}_B)\times g = (0.44 - 0.31)\times g = 0.13,g$$
where g = desktop-minus-mobile conversion gap.
How big a gap would explain the entire lift? Solving 0.13·g = 5.3pp gives g = 41pp. A 41-point desktop-vs-mobile gap is large but not absurd for onboarding (mobile onboarding is notoriously worse). More usefully, here's the fraction of the lift that vanishes at realistic gaps:
| Desktop − mobile gap | Composition-driven "lift" | % of 5.3pp that is fake |
|---|---|---|
| 10pp | 1.3pp | 24% |
| 15pp | 2.0pp | 36% |
| 20pp | 2.6pp | 49% |
| 25pp | 3.3pp | 61% |
| 30pp | 3.9pp | 73% |
Interpretation: For any realistic device gap (and 15–25pp is common for onboarding funnels), roughly a third to two-thirds of the observed 5.3pp is explained by device mix alone, before the feature does anything. This is a real, mechanical Simpson's-paradox risk: the aggregate looks better for B largely because B is loaded with the easier-converting device.
Illustrative standardization. Assuming plausible control rates (desktop 45%, mobile 27% – an 18pp gap – which reproduces A's 37.4% almost exactly), I back out what the video would need to do to produce B's 42.7%:
- If the desktop video adds a real +5pp, predicted B ≈ 42.9% – matches observed. So the data are consistent with a modest, genuine desktop effect.
- But the data are equally consistent with the effect being mostly composition plus noise.
The honest conclusion: the observed result is compatible with (a) a real ~5pp desktop lift, or (b) mostly a device-mix artifact, or anything in between. The experiment as run cannot distinguish these. That is the whole problem.
Other confounders (secondary):
- Channel switch (week 3): the new ad channel disproportionately fed B and brings different user intent – a second, independent composition bias. Notably, B's weekly rate actually declined over the test (44.2% → 43.1% → 41.1%), which is mildly concerning and coincides with the channel change.
- Novelty effect: a shiny video tour can lift completion short-term without durable behavior change; 3 weeks won't reveal this.
- Text-vs-video for mobile: B's mobile users got a different experience (text), so "Group B" isn't one treatment – it's two.
Question 4 – What to do to get a reliable answer
Fix the design:
- Repair randomization. Randomize within device (stratified/blocked randomization by desktop vs. mobile) so mix is balanced by construction. Verify balance before analyzing.
- Run a Sample Ratio Mismatch (SRM) check on every future test – the 44/31 split should have tripped an automatic alarm.
- Freeze traffic sources. No mid-test channel changes; if acquisition must change, restart the test.
Re-power correctly:
4. Decide the minimum effect worth shipping. If it's the desktop lift (5pp), you need **1,340 per group** at 80% power – plan for the sample and duration that requires, or accept detecting only larger effects. Do not re-run at the old size and hope.
Analyze correctly: 5. Segment by device (pre-registered). The real question is: does the video improve desktop completion? Analyze desktop-only as the primary endpoint, since that's where the mechanism lives. Analyze mobile (text) separately. 6. Use a two-sided test, or pre-register one-sided with explicit justification. 7. If you insist on watching the test live, use a proper sequential method (e.g., group-sequential / always-valid p-values). Repeatedly peeking at a fixed-horizon test inflates false positives badly – a likely contributor to the "almost significant" framing. 8. Track a guardrail + durable metric (e.g., week-2 activation or retention), not just first-session completion, to rule out novelty effects.
Optional bridge: If leadership wants to move now, ship the video to desktop only as a monitored rollout (it can't hurt mobile since mobile never sees it), with completion tracking – treating it as a follow-up experiment, not a settled win.
Question 5 – How to frame this for leadership (no jargon, honest)
"The new onboarding is promising, but this test can't yet prove it works – and we know why. I recommend we fix two things and re-run before a full launch."
What we saw: New onboarding completed at 42.7% vs. 37.4% – about 5 points higher. That looks good.
Why we can't trust it yet – two plain-English problems:
- The groups weren't apples-to-apples. By accident, the old-onboarding group had far more phone users (44%) than the new group (31%). Phone users finish onboarding less often regardless of design, and our video only plays on desktop. So the new group had a built-in head start. When we account for this, somewhere between a third and half of the 5-point gain may just be that mix difference, not the new design.
- The result isn't statistically solid. The "almost significant" number that's been circulating (0.07) was calculated the lenient way. Done the standard way, it's about 0.15 – meaning there's a real chance this gap is just luck. The test was also too small to reliably detect a change this size.
What I recommend: Re-run for ~4–6 weeks with balanced groups, don't change ad channels mid-test, and measure desktop and mobile separately. The early signal is encouraging enough that this is worth doing right – and if we want momentum now, we can safely turn the video on for desktop users only while we gather clean data.
The risk of deploying today: if the gain is mostly the accidental mix difference, we'll declare a win, see no real improvement in the numbers, and lose trust in future tests. A two-to-four week delay buys us an answer we can actually stand behind.
The single most important issue
If you take away one thing: the treatment (video) is only available on desktop, and randomization failed on exactly the desktop/mobile axis. That coupling means device mix isn't just a confounder – it's a confounder pointed straight at the treatment mechanism, capable of manufacturing most of the observed lift on its own. Combined with the fact that the correct two-sided test isn't even significant, the responsible call is fix randomization, re-power, segment by device, and re-run – not "deploy immediately."
(Statistical figures above – p-values, confidence interval, power, and the composition sensitivity table – were computed directly from your data; the device-rate standardization uses assumed but plausible desktop/mobile base rates since device-level conversions weren't provided, and would be replaced with real segmented data in the re-run.)
~openai/gpt-latest
Executive verdict
The result is promising but not trustworthy enough to "deploy immediately." The biggest problem is not the p-value. The biggest problem is that randomization was broken: Group B had far fewer mobile users than Group A, and mobile/desktop likely have very different onboarding completion rates. Because the new video onboarding only works on desktop while mobile users get a text version, the device imbalance directly threatens the interpretation.
Observed lift:
- A: 127 / 340 = 37.4%
- B: 152 / 356 = 42.7%
- Raw difference: +5.3 percentage points
But Group B had only 31% mobile, while Group A had 44% mobile. If mobile users complete onboarding less often than desktop users, some or even all of the apparent B lift may be due to B having a more favorable user mix rather than the new onboarding.
My recommendation: do not call this a successful experiment yet. Treat it as a directional signal. Fix the randomization and rerun, or at minimum analyze device- and channel-adjusted results before making a rollout decision.
1. Can these results be trusted?
Short answer: not as causal evidence.
The raw numbers look better for B, and the weekly pattern is directionally consistent:
| Week | A | B | Raw lift |
|---|---|---|---|
| Feb 21–28 | 38.1% | 44.2% | +6.1 pp |
| Mar 1–7 | 36.9% | 43.1% | +6.2 pp |
| Mar 8–14 | 37.2% | 41.1% | +3.9 pp |
| Total | 37.4% | 42.7% | +5.3 pp |
That is encouraging. But the experiment has serious validity problems.
Main methodological problems
1. Randomization was broken
Group A and Group B were not comparable.
- Group A mobile share: 44%
- Group B mobile share: 31%
- Difference: 13 percentage points
That is a large imbalance. Using approximate counts:
- A: 150 mobile, 190 desktop
- B: 110 mobile, 246 desktop
The mobile-share difference is about 13 pp. A simple z-test for the covariate imbalance gives roughly:
- Standard error ≈ 3.7 pp
- z ≈ 3.5
- p ≈ 0.0004
So this imbalance is very unlikely to be random noise. And you already know there was an operational reason: an ad channel was temporarily switched in the final week.
This is the dominant issue. If the groups are not exchangeable, the difference in completion rates cannot be cleanly attributed to onboarding.
2. Device is directly related to the treatment
This is worse than a generic demographic imbalance.
The new onboarding has a video tour on desktop, but mobile users get a text version. So device affects both:
- The user's baseline likelihood of completing onboarding.
- The actual experience delivered by the treatment.
That means "Group B" is not one homogeneous treatment. It is really:
- B-desktop: new onboarding with video tour.
- B-mobile: new onboarding with text version.
Because B had more desktop users, it had more users exposed to the stronger version of the treatment, and also possibly more users with naturally higher completion rates.
3. Acquisition channel changed during the experiment
In Mar 8–14, Group B got more users because an ad channel was temporarily switched.
That introduces another confounder. Users from different acquisition channels can differ in:
- Intent.
- Company size.
- Device mix.
- Prior awareness.
- Willingness to complete setup.
- Geography.
- Plan type.
- Sales-assisted versus self-serve behavior.
If B got more high-intent or more desktop-heavy users from a different source, the observed lift could be due to traffic quality, not onboarding.
4. The test was underpowered for the observed effect
The test was reportedly powered for an MDE of 8 percentage points, but the observed difference was only 5.3 pp.
That matters. If the real effect is around 5 pp, this sample is too small to estimate it reliably.
With baseline around 37% and observed treatment rate around 43%, a conventional 80% power, two-sided 5% alpha test would need roughly:
- ~1,300 users per group to detect a 5.3 pp effect.
- Current sample: 340 A, 356 B.
So the current test is too small for a confident answer about a 5 pp lift.
Also, if the test was supposedly powered for an 8 pp absolute effect, the collected sample seems low. A conventional two-sided test for an 8 pp lift would usually require around ~580–600 users per group, not ~350 per group. So the original power calculation should be checked.
2. What does p-value 0.07 mean? Verify the actual p-value.
Using the aggregate results:
- A: 127 / 340 = 0.3735
- B: 152 / 356 = 0.4270
- Difference = 0.0534
For a two-proportion z-test with pooled standard error:
[ \hat{p} = \frac{127 + 152}{340 + 356} = \frac{279}{696} = 0.4009 ]
[ SE = \sqrt{\hat{p}(1-\hat{p})(1/340 + 1/356)} ]
[ SE \approx \sqrt{0.4009 \times 0.5991 \times (1/340 + 1/356)} \approx 0.0372 ]
[ z = \frac{0.4270 - 0.3735}{0.0372} \approx 1.44 ]
For a two-sided test:
[ p \approx 0.15 ]
For a one-sided test testing specifically whether B > A:
[ p \approx 0.075 ]
So the reported "about 0.07" is probably the one-sided p-value, not the standard two-sided p-value.
The approximate 95% confidence interval for the raw difference is:
[ 5.3 \text{ pp} \pm 1.96 \times 3.7 \text{ pp} ]
So:
[ [-1.9 \text{ pp}, +12.6 \text{ pp}] ]
That interval includes zero and even a small negative effect.
What the p-value actually means
A one-sided p-value of 0.075 means:
If there were truly no difference between old and new onboarding, and if the experiment were properly randomized, there would be about a 7.5% chance of seeing a lift at least this large in the favorable direction just from random sampling noise.
The two-sided p-value of 0.15 means:
If there were truly no difference, there would be about a 15% chance of seeing a difference this large or larger in either direction.
How the marketer is likely misinterpreting it
They may be thinking:
- "p = 0.07 means there is a 93% chance the new onboarding works."
- "It is almost significant, so we can treat it as significant."
- "The effect is probably real because the p-value is close to 0.05."
- "A 14.2% relative lift is large, so the result is obvious."
Those are not valid interpretations.
A p-value is not the probability that the treatment works. Also, "almost significant" is not a statistical category. The evidence is suggestive, but weak – and in this case the broken randomization is a larger concern than the p-value.
3. Could the observed lift be explained by desktop/mobile mix?
Yes, possibly. At minimum, the device imbalance could explain a meaningful portion of the apparent lift.
Group A had more mobile users:
- A: 44% mobile, 56% desktop
- B: 31% mobile, 69% desktop
So B had 13 percentage points more desktop users.
If desktop users complete onboarding at a higher rate than mobile users, B would look better even if the onboarding change had no true effect.
Let:
[ g = \text{desktop completion rate} - \text{mobile completion rate} ]
The composition advantage for B is approximately:
[ 0.13 \times g ]
because B has 13 pp more desktop users.
Sensitivity table
| Assumed desktop-mobile gap | Lift explained by composition | Remaining lift after adjustment |
|---|---|---|
| 0 pp | 0.0 pp | 5.3 pp |
| 10 pp | 1.3 pp | 4.0 pp |
| 20 pp | 2.6 pp | 2.7 pp |
| 30 pp | 3.9 pp | 1.4 pp |
| 40 pp | 5.2 pp | 0.1 pp |
| 41 pp | 5.3 pp | 0.0 pp |
| 50 pp | 6.5 pp | -1.2 pp |
So if desktop users complete onboarding about 41 percentage points more often than mobile users, the entire observed 5.3 pp lift could be explained by the device mix alone.
Is a 41 pp desktop-mobile gap large? Yes. But for a SaaS onboarding flow requiring five key actions, it is not impossible, especially if the full video experience only exists on desktop and mobile gets a reduced text version.
For example, suppose with the old onboarding:
- Desktop completion: 55%
- Mobile completion: 14%
Then Group A's expected completion would be:
[ 0.56 \times 55% + 0.44 \times 14% \approx 37% ]
Group B, with more desktop users, would be expected to complete at:
[ 0.69 \times 55% + 0.31 \times 14% \approx 42% ]
That is almost exactly the observed result, even with zero true treatment effect.
That does not prove the lift is fake. But it proves the aggregate result is not identifiable without device-level outcome data.
More moderate assumptions still matter
Even if the desktop-mobile gap is only 20 pp, the composition effect would be:
[ 0.13 \times 20% = 2.6 \text{ pp} ]
That would reduce the treatment lift from 5.3 pp to about 2.7 pp.
If the gap is 30 pp, the composition effect would be:
[ 0.13 \times 30% = 3.9 \text{ pp} ]
leaving only about 1.4 pp of apparent treatment effect.
So even if device mix does not explain everything, it could explain a large share of the result.
And device is not the only confounder
The ad-channel switch could independently affect results. If B received users from a slightly better-converting channel, that could add another few percentage points. Combined with the device imbalance, that could easily explain the full 5.3 pp observed lift.
4. What should be done to get a reliable answer?
Immediate next step: do not deploy as "proven"
The result should be labeled:
Directionally positive but inconclusive due to broken randomization and insufficient power.
If the video onboarding is low-risk, a limited rollout may be reasonable, but it should not be presented as a validated causal win.
Analyze the current data properly before discarding it
Ask for the data broken down by:
- Variant.
- Device: desktop vs mobile.
- Acquisition channel.
- Week or day.
- Geography, plan, company size if available.
- Whether the user actually saw the video or got the text fallback.
At minimum, produce a table like:
| Device | A users | A completions | A rate | B users | B completions | B rate | Lift |
|---|---|---|---|---|---|---|---|
| Desktop | |||||||
| Mobile |
Then estimate:
- Desktop-specific treatment effect.
- Mobile-specific treatment effect.
- Overall effect reweighted to the expected future traffic mix.
A logistic regression or linear probability model could help:
[ Completion \sim Treatment + Device + Channel + Week + Treatment \times Device ]
But this would be a damage-control analysis, not a full rescue. Post-hoc adjustment cannot fix unmeasured differences caused by broken assignment.
Fix randomization and rerun
The rerun should have clean experimental controls.
Required fixes
Randomize users properly at the individual level.
- Assignment should not depend on ad channel, device, week, campaign, geography, or landing page.
Stratify or block randomization by device.
- Keep the desktop/mobile mix balanced within A and B.
- For example: randomize desktop users 50/50 and mobile users 50/50 separately.
Also stratify or monitor by acquisition channel.
- If a channel is turned on or off, both variants should receive the change equally.
Define the unit of randomization.
- Usually user ID or account ID.
- Avoid users switching variants across sessions or devices.
Define the estimand. Decide whether the product wants to know:
- Overall effect across all users.
- Desktop effect of the video.
- Mobile effect of the text fallback.
- Weighted effect for expected future traffic.
Predefine the analysis.
- Primary metric: onboarding completion.
- Analysis window.
- Two-sided or one-sided test.
- Minimum sample size.
- Segment analyses.
- Guardrail metrics.
Avoid sequential peeking.
- If the team wants to monitor weekly, use a preplanned sequential design or alpha-spending rule.
- Otherwise, repeated checking increases false-positive risk.
Recalculate sample size based on the effect you care about
If leadership cares about detecting a lift around the observed 5.3 pp, the current sample is too small.
Approximate required sample size for 80% power, two-sided 5% alpha:
- Baseline: 37.4%
- Target treatment rate: 42.7%
- Absolute lift: 5.3 pp
- Required sample: roughly 1,300 users per group, or 2,600 total
For 90% power, it would be closer to:
- 1,700–1,800 users per group
If the true minimum meaningful effect is 8 pp, required sample is smaller, roughly:
- ~580–600 users per group for 80% power, two-sided 5% alpha.
Because the treatment differs by device, you may need even more total sample if you want reliable desktop and mobile estimates separately.
5. How to frame this for leadership
Use plain language. Avoid saying "the test failed." Better framing:
The new onboarding produced a promising raw lift: 42.7% versus 37.4%, a 5.3 percentage-point increase. However, we cannot confidently attribute that lift to the onboarding change because the experiment groups were not comparable. Group B had many more desktop users and fewer mobile users, and desktop users are likely more likely to complete onboarding. Since the video tour only works on desktop, this imbalance directly affects the result.
Then be explicit:
Statistically, the standard two-sided p-value is about 0.15, not 0.07. The 0.07 number is likely a one-sided test. That means the evidence is suggestive, not conclusive. The 95% confidence interval ranges from about -2 pp to +13 pp, so the true effect could be small, zero, or meaningfully positive.
Then identify the dominant issue:
The main reason not to deploy immediately is not just statistical significance. The main reason is broken randomization. Because Group B had a more favorable device mix and an unplanned ad-channel difference, the observed lift may be partly or entirely due to user composition rather than the new onboarding.
Recommended leadership position:
We should treat this as a positive signal worth following up, not as proof. The right next step is to rerun the test with proper randomization stratified by device and acquisition channel. If the true effect is around 5 percentage points, we should plan for roughly 1,300 users per group. If we need a faster decision, we can do a limited ramp with measurement, but we should not claim the experiment has validated the new onboarding yet.
Bottom line
The marketer is overclaiming. The PM is right to be skeptical.
The result may be real, but the current experiment cannot prove it. The broken randomization and device imbalance dominate the decision. Even a cleaner p-value would not solve that. The correct decision is: do not deploy as a proven winner; rerun or continue under a properly controlled design.
~google/gemini-pro-latest
The product manager's skepticism in this case is completely justified. From a statistical, methodological and data-analysis standpoint, this test contains critical errors. Deploying the new version on the basis of this data is not acceptable.
Below is a rigorous, detailed breakdown for the team, along with recommendations for how to present this position to leadership.
1. Can the results be trusted? What methodological problems are there?
No, the results cannot be trusted. The experiment is compromised at several levels at once:
- Broken randomization (critical error): The share of mobile users is 44% in Group A and 31% in Group B. In an ideal A/B test, the two groups should be identical (roughly 50/50 in audience composition). This difference undermines the core principle of split testing – "all else being equal."
- External distortion (interference with traffic): In the third week, an ad channel switch occurred, producing a sharp inflow of users into Group B (139 vs. 121 in Group A). Different ad channels bring in audiences with different motivation and device types, which directly violates the purity of the experiment. This disruption is most likely exactly what "broke" the device randomization.
- Mismatch with the hypothesis being tested: The test was designed to evaluate the video tour, but video does not work on mobile devices. In that case, evaluating metrics for a mixed audience is pointless, since mobile users in Group B simply received the text version. We were not testing "video" against the old version – we were testing "video for desktop + text for smartphones" against "the old version for everyone."
- Underpowered test: The test was designed to detect an effect (MDE) of 8%, while the actual difference was 5.3%. With the current sample size (around 350 people per group), we physically cannot draw a reliable statistical conclusion about a difference of 5.3%.
2. What does the p-value of 0.07 actually mean, and how is the marketer manipulating it?
The marketer is most likely interpreting the p-value of 0.07 as "there is a 93% probability that the new version performs better." This is fundamentally incorrect. A p-value means the opposite: if the new version in fact had no effect at all, the probability of randomly obtaining this same or a larger increase would be 7%.
But there is a more serious problem. Let's calculate the actual value for the two-proportion z-test (two-proportion z-test):
- Group A: p₁ = 127 / 340 = 37.35%
- Group B: p₂ = 152 / 356 = 42.70%
- Rounding, the z-score is ~1.44.
- If we use the correct two-sided test (two-tailed), the true p-value equals 0.15 (15%).
Where did the marketer get 0.07? A value of 0.075 is obtained if a one-sided test (one-tailed) is used. Switching to a one-sided test after the fact, once we see a positive result, in order to formally lower the p-value, is a well-known statistical manipulation (p-hacking). Even at 0.07, the result does not reach the gold-standard threshold of 0.05, and the actual figure (0.15) proves that the result is not statistically significant.
3. Investigation of confounding factors (Simpson's Paradox)
The strongest argument against the test is the share of mobile traffic. In B2B/SaaS, conversion from desktop (a user at their workplace, at a computer) is traditionally significantly higher than conversion from a smartphone, where going through onboarding is inconvenient.
Since Group B ended up with a disproportionately high share of desktop users (69% vs. 56% in Group A), this factor alone could have single-handedly produced the entire observed increase of 5.3%, even if the video tour had no effect on conversion whatsoever!
Let's verify this mathematically (sensitivity analysis): Suppose the video tour does not work and device-level conversion rates are exactly identical in both groups.
- Group A (340 people): ~190 desktop, 150 mobile.
- Group B (356 people): ~246 desktop, 110 mobile.
To arrive at exactly the figures we see in the test (127 target actions in A and 152 in B), it is enough to assume the following baseline conversion rates:
- Desktop conversion: ~55%
- Mobile conversion: ~15%
Check: in Group A (190 × 0.55) + (150 × 0.15) = 104.5 + 22.5 = 127 conversions. In Group B (246 × 0.55) + (110 × 0.15) = 135.3 + 16.5 = ~152 conversions.
Conclusion: The difference in conversion between mobile (15%) and desktop (55%) is an entirely typical picture for SaaS. Consequently, the entire "fantastic growth" of 14.2% is mathematically explained by a simple skew in audience composition (Simpson's Paradox). The video, it appears, did not improve anything at all.
4. What needs to be done now to get a reliable answer?
- Isolate the desktop data: Since the video tour is only available on desktop, immediately recalculate conversion rates for desktop users only, in both groups. This will clean the data of mobile "noise." If there is no growth there – the hypothesis is dead.
- Investigate the distribution bug: Find out why the ad-channel switch in the third week led to a skew in traffic distribution (pay special attention to how the A/B testing system hashes/distributes users from the new campaigns).
- Launch a proper re-run: If the desktop slice shows potential, re-run the test. But this time:
- Test only on desktop users.
- Increase the sample size. Since the real expected effect is around 5% (not 8%), you will need to run roughly 2.5 times more users through the test to reach 80% power – likely taking 6–8 weeks rather than three.
5. How to make the case to leadership (without complex jargon)
You need to shift the focus from "we don't believe in the new feature" to "we want to protect the product from random decisions."
Template for communication (Executive Summary):
"At first glance, the new onboarding version showed a 5.3-point increase, but a deeper analysis of the data revealed a significant technical failure that prevents us from trusting the results.
During the test, the mechanism for randomly distributing users broke down: the group with the video ended up with significantly more people who registered from computers, while the old group had more from smartphones. Since onboarding has historically been harder to complete from a smartphone, the group with the video version received an artificial advantage.
Our calculations show that this failure alone fully explains the apparent "growth" in the metrics. In other words, there is a high probability that the effect of the video itself is zero. Additionally, the statistical check confirms that the result is within the margin of error (a 15% chance that the result is random, not 7% as originally thought).
We propose: not rolling this release out to everyone, isolating the real figures specifically for desktop (where the video actually worked), and, if necessary, conducting a short re-test accounting for the error."
This approach demonstrates your competence to leadership, guards against a mistake, and offers a clear, constructive next step.
Analysis
Consensus
- All three models agree the results should not be trusted as clean causal evidence of the video tour effect; deploying immediately is the wrong call; it should be treated as a directional, promising-but-inconclusive signal
- The reported p≈0.07 is a one-sided p-value; the correct two-sided two-proportion z-test p-value is ≈0.15, computed via z≈1.44, which is not statistically significant
- The marketer likely misinterprets p=0.07 as "93% chance the feature works" and treats "almost significant" as significant, both invalid interpretations
- A p-value is the probability of observing a difference at least this large if the null (no effect) were true, not the probability the treatment works
- Broken randomization (mobile 44% in A vs 31% in B) is the dominant/most important issue, worse because device is mechanically coupled to the treatment (video is desktop-only)
- The 95% confidence interval for the lift is roughly [-1.9pp, +12.6pp], which includes zero and even a small negative effect
- The composition effect of B's device advantage is approximately 0.13 × g, where g is the desktop-minus-mobile conversion gap
- A desktop-mobile gap of ~41pp would fully explain the entire 5.3pp lift with zero true treatment effect
- Using plausible SaaS base rates (desktop ~55%, mobile ~15%), the entire observed result can be reproduced with zero treatment effect, showing the aggregate is not identifiable without device-level data
- The test is underpowered for the observed 5.3pp effect; roughly ~1,300 per group is needed at 80% power/two-sided 5%, versus the ~350 per group actually run
- The mid-test ad-channel switch is an independent confounder introducing traffic-quality/composition bias
- Group B is not a single homogeneous treatment: desktop users got video while mobile users got a text version
- The fix is to re-run with device-stratified randomization, avoid mid-test channel changes, segment/analyze by device, recompute sample size for the true effect, and avoid sequential peeking
- Leadership framing should avoid jargon, avoid saying "the test failed," present it as a promising-but-unproven signal, explain the device imbalance in plain terms, and propose a clean re-run
Contradictions
- Likelihood that the true treatment effect is essentially zero
- ~anthropic/claude-opus-latest: Balanced/agnostic: data are equally consistent with a real ~5pp desktop lift OR mostly composition artifact; the experiment cannot distinguish them, so the effect is unknown, not necessarily zero
- ~openai/gpt-latest: Agnostic: composition could explain part or even all; a zero-effect scenario is possible but not asserted; "may be real"
- ~google/gemini-pro-latest: Strongly leans toward the video likely having done nothing: states the whole growth is mathematically explained by device skew and "the video apparently improved nothing / high probability effect is zero"
- Whether the one-sided p-value framing is innocent or p-hacking
- ~anthropic/claude-opus-latest: Treats it as a decision-quality red flag / one-sided framing to make the result look stronger, but frames it as legitimate only if pre-committed
- ~openai/gpt-latest: Neutral: "probably the one-sided p-value," explains the meaning without explicitly labeling it manipulation
- ~google/gemini-pro-latest: Explicitly labels switching to a one-sided test post hoc as a known statistical manipulation (p-hacking)
- Sensitivity analysis presentation and framing of remaining lift
- ~anthropic/claude-opus-latest: Presents the table as "% of 5.3pp that is fake" and concludes roughly a third to two-thirds is composition at realistic 15–25pp gaps
- ~openai/gpt-latest: Presents the table with "remaining lift after adjustment," including a 50pp row yielding negative remaining lift, emphasizing non-identifiability
- ~google/gemini-pro-latest: Does not give a graded sensitivity table; picks a single plausible scenario (desktop 55%/mobile 15%) that exactly reproduces the counts and concludes the effect is essentially fully explained
- Whether a partial/desktop-only rollout now is acceptable
- ~anthropic/claude-opus-latest: Offers an optional bridge: ship video to desktop only as a monitored rollout, since it can't hurt mobile
- ~openai/gpt-latest: Allows a limited ramp with measurement if a faster decision is needed, but not as a validated win
- ~google/gemini-pro-latest: Does not endorse any rollout now; recommends isolating desktop data and re-testing desktop-only before any decision
Partial coverage
- ~anthropic/claude-opus-latest, ~openai/gpt-latest: Explicitly report multiple test variants (Fisher's exact, chi-square with continuity correction) or note the CI/SE derivation; Opus gives a full table of Fisher's/chi-square values
- ~anthropic/claude-opus-latest: Notes the actual power to detect a 5.3pp effect at the run sample was only ~30%, so a non-significant result is nearly uninformative
- ~anthropic/claude-opus-latest: Flags the odd non-contiguous date range (Feb 21–28 and Mar 1–14) as a possible data-stitching/double-counting error
- ~anthropic/claude-opus-latest: Observes Group B's weekly rate declined over time (44.2% → 43.1% → 41.1%), coinciding with the channel change, and raises a novelty-effect concern
- ~anthropic/claude-opus-latest, ~openai/gpt-latest: Recommend Sample Ratio Mismatch (SRM) checks and tracking guardrail/durable metrics (retention/week-2 activation) rather than just first-session completion
- ~openai/gpt-latest: Computes a formal statistical test for the covariate (device) imbalance itself: z≈3.5, p≈0.0004, showing the imbalance is not random noise
- ~openai/gpt-latest: Proposes a specific regression model (Completion ~ Treatment + Device + Channel + Week + Treatment×Device) for damage-control analysis and defining the estimand
- ~openai/gpt-latest, ~anthropic/claude-opus-latest: Explicitly question the original power calculation: an 8pp MDE at 80% power needs ~580–600/group, yet only ~350/group were run, so the stated powering is inconsistent
openai/gpt-latest: Lists specific 90% power sample size (1,700–1,800/group) and notes needing even more sample for reliable per-device estimates- ~openai/gpt-latest: Enumerates concrete acquisition-channel confounders (intent, company size, geography, plan type, sales-assisted vs self-serve)
- ~google/gemini-pro-latest, ~anthropic/claude-opus-latest: Explicitly hypothesize that the mid-test ad-channel switch is likely the mechanism that caused the broken device randomization
- ~google/gemini-pro-latest: Frames the leadership pitch around shifting from "we don't believe in the feature" to "we want to protect the product from random decisions," and mentions the code-complexity cost of shipping a feature that doesn't pay off
Unique insights
anthropic/claude-opus-latest: Quantifies the retrospective power at the observed effect (30%), giving a principled reason why a non-significant result here is nearly uninformative rather than evidence of no effect- ~anthropic/claude-opus-latest: Catches the non-contiguous/odd date range as a potential data-integrity issue that the other two models missed entirely
- ~anthropic/claude-opus-latest: Notes B's declining weekly completion trend and ties it to the channel switch plus a possible novelty effect, adding a temporal-dynamics lens
- ~openai/gpt-latest: Statistically tests the device-imbalance covariate itself (z≈3.5, p≈0.0004), proving the randomization break is real rather than plausible noise – a rigorous step neither other model performed
- ~openai/gpt-latest: Specifies the estimand question explicitly (overall vs desktop vs mobile vs future-traffic-weighted effect), reframing the analysis around what leadership actually wants to know
- ~google/gemini-pro-latest: Directly names the post-hoc switch to a one-sided test as p-hacking, sharpening the ethical/decision-quality critique
- ~google/gemini-pro-latest: Reframes the reconciled device rates (desktop ~55%, mobile ~15%) as an exact arithmetic reconstruction of the observed completion counts, making the zero-effect scenario concretely verifiable
Blind spots
- None of the models flagged that the assumed device conversion rates (e.g., desktop 55% / mobile 15%) are fabricated inputs chosen to fit the result – while useful illustratively, they overstate certainty because actual device-level completion data were never provided, and equally plausible smaller gaps leave room for a real effect
- No model reconciled the total weekly n's against the stated group totals (89+130+121=340 for A, 86+131+139=356 for B – which do check out) – a quick arithmetic verification of the given breakdown was skipped
- None addressed the multiple-comparisons / garden-of-forking-paths risk introduced by the very device/channel segmentation they recommend (segmented analyses inflate false positives unless pre-registered)
- No model quantified the business cost/benefit or risk asymmetry of deploying desktop-only now versus delaying (video is desktop-only and cannot harm mobile, so a monitored desktop rollout has limited downside – only Opus/GPT gestured at this without analysis)
- None discussed novelty/primacy effects rigorously or the need for a longer post-exposure retention window beyond first-session completion (Opus only briefly mentioned novelty)
- No model questioned whether "onboarding completion (5 key actions)" is even the right success metric versus downstream activation/revenue/retention, i.e., metric validity
- None considered whether users could be exposed to both variants across devices/sessions (contamination) or how the randomization unit (user vs. session vs. account) interacts with the desktop/mobile split
- No model addressed statistical power/interpretation for the planned per-device subgroup analyses (desktop-only n is smaller, so a desktop re-run needs its own power calculation, which was only loosely mentioned)
- None examined potential Simpson's reversal risk within the weekly data or whether the declining B trend could indicate regression toward null as the sample grew
Excellent case – and your caution is fully justified. Three independent models analyzed the data and agreed on the main point: the marketer is wrong, and launching is not an option – not because the new onboarding definitely doesn't work, but because this test, in principle, cannot answer the question it was set up to answer. Below is a breakdown organized by your points.
In short, the most important thing
There are two problems, either of which alone is sufficient to make the result untrustworthy:
- P-value 0.07 is a one-sided test. The correct two-sided p-value is ≈ 0.15 – twice as large and far outside the range of significance.
- Randomization is broken exactly on the attribute the feature depends on (device). The video only works on desktop, and the control group ended up with 44% mobile users versus 31% in the test group. A significant portion of the "lift" could be pure audience composition rather than the effect of the onboarding itself.
1. Can the results be trusted?
No, not as causal proof. Problems in descending order of severity:
- Broken randomization (critical). Mobile: 44% in A vs. 31% in B. This is not random noise – a formal check gives z ≈ 3.5, p ≈ 0.0004. The groups are not equivalent.
- Device is directly linked to the treatment (makes it worse). The video exists only on desktop; mobile users get text. This means "Group B" is actually two different treatments: "video on desktop" + "text on mobile." B ended up overloaded with desktop users – meaning both a stronger version of the feature and an "easier" audience.
- Unplanned ad-channel switch (serious). During the week of March 8–14, Group B received more users from a different source. Different channels mean different intent, quality, and audience composition. This is likely exactly what broke the device randomization.
- Insufficient power. The test was designed for an MDE of 8%, while the actual difference was 5.3%. The actual power to detect a 5.3% effect at your sample size is around 30%. A non-significant result at 30% power is almost uninformative.
2. What does the p-value of 0.07 mean, and where is the marketer's error?
What a p-value actually means: it's the probability of seeing a difference at least as large as observed, if the new onboarding had no effect at all. It does NOT mean "93% probability the feature works."
Verification of the calculation (two-proportion z-test):
- A: 127/340 = 37.35%; B: 152/356 = 42.70%
- z ≈ 1.44
- Two-sided p-value ≈ 0.15
- One-sided p-value ≈ 0.075 ← this is where "about 0.07" comes from
The 95% confidence interval for the lift is: approximately [−1.9 pp; +12.6 pp] – it includes zero and even a small negative effect.
Typical errors by the marketer:
- "Almost significant" is treated as "significant" (no such category exists in statistics).
- 0.07 is read as "93% that it works."
- It's not stated (or not recognized) that this is a one-sided test – switching to it after the fact to get a prettier number close to 0.05 is close to p-hacking.
- It's ignored that even the point estimate (+5.3 pp) is below the effect the test was designed to detect.
3. Confounding factors: could the entire lift be composition?
Yes, quite possibly. This is a classic Simpson's paradox risk.
B had 13 pp more desktop users. If desktop converts better than mobile, B gets a built-in head start. The magnitude of the composition effect ≈ 0.13 × g, where g is the desktop-minus-mobile conversion gap.
Sensitivity analysis:
| Desktop−mobile gap (g) | Lift from composition | % of 5.3 pp "fake" |
|---|---|---|
| 10 pp | 1.3 pp | ~24% |
| 15 pp | 2.0 pp | ~36% |
| 20 pp | 2.6 pp | ~49% |
| 30 pp | 3.9 pp | ~73% |
| 41 pp | 5.3 pp | 100% |
For a typical SaaS gap of 15–25 pp, a third to half of the lift is explained by audience composition alone. And if we assume fairly realistic baseline SaaS conversion rates (desktop ~55%, mobile ~15%), the observed numbers are reproduced exactly with zero effect from the video.
⚠️ Important caveat: the specific baseline conversion rates (55%/15%) are an illustrative assumption, not a fact. You don't have actual device-level data. The honest conclusion: the data are compatible both with a real ~5 pp effect on desktop and with an almost entirely compositional artifact. The experiment as currently designed cannot distinguish between these scenarios. That's the whole problem.
4. What to do to get a reliable answer
First – squeeze the maximum out of the current data:
- Break down results by device. Calculate conversion separately for desktop (where the video actually works) and mobile. This is the primary question: does the video improve desktop conversion?
- Check for sample ratio mismatch – the 44/31 split should have triggered an alarm.
- Break down by acquisition channel and by week.
Then – a proper re-run:
4. Stratified randomization by device (separately 50/50 for desktop, separately for mobile) – so the composition is balanced by design.
5. Do not change ad channels during the test.
6. Recalculate the sample size based on the real effect. To reliably detect 5.3 pp at 80% power, you need **1,300 users per group** – almost four times more than before. This is likely 6–8 weeks, not 3.
7. Use a two-sided test; if one-sided, only with prior justification.
8. If you want to check interim results, use a proper sequential method rather than "peeking."
9. Track retention/activation at week 2, not just first-session completion (protection against the novelty effect).
Optional compromise: since the video isn't shown on mobile anyway, you could enable it only for desktop as a controlled, monitored rollout – while honestly calling this a continuation of the experiment, not a confirmed win.
5. Wording for leadership (no jargon, honest)
"The new onboarding looks promising, but this test doesn't yet prove it works – and we understand why. I propose fixing two things and re-running before full rollout."
What we saw: the new version completed at 42.7% vs. 37.4% – about 5 points higher. That looks good.
Why we can't trust it yet – two problems in plain terms:
- The groups turned out not to be comparable. By chance, the old version had noticeably more users from smartphones (44% vs. 31% in the new one). And onboarding is completed less often on smartphones – regardless of design – since our video only works on computers. That means the new group got a built-in advantage. By our calculations, anywhere from a third to half (and in the worst case, all) of the "lift" is explained by this difference in composition, not by the video itself.
- The result is not statistically convincing. The "almost significant" figure (0.07) was calculated using the lenient method. By the standard method, it's about 0.15 – meaning there's a notable chance the difference is just random. Plus, the sample was too small for a reliable conclusion.
What I propose: re-run for 6–8 weeks with balanced groups, don't change ad channels mid-test, and measure desktop and mobile separately. The early signal is encouraging enough to warrant doing this properly. If we need momentum right now, we can safely enable the video for desktop users only, while collecting clean data in parallel.
The risk of launching immediately: if the lift is mostly due to a random audience skew, we'll declare victory, see no real improvement in the metrics, and undermine trust in future tests. A delay of a couple of weeks buys us an answer we won't be embarrassed by.
Main takeaway in one sentence: the treatment (video) is available only on desktop, and randomization broke exactly along the desktop/mobile axis – this confounder points directly at the mechanism of the feature and is capable of producing most of the lift on its own. Combined with the fact that the correct two-sided test isn't even significant, the responsible decision is to fix the randomization, recalculate the sample size, break results down by device, and re-run – not "deploy immediately."
Would you like help turning this into a short memo/slide for leadership, or preparing counterarguments in case the marketer pushes back?
Original in Russian
Я разберу это с помощью анализа несколькими моделями, поскольку здесь есть серьёзные статистические тонкости, где важно не ошибиться.
Panel responses
~anthropic/claude-opus-latest
I'll work through this rigorously with actual computations, then give you a leadership-ready framing. Let me verify the statistics first.Interesting – the reported p-value of 0.07 does not reconcile with a standard two-sided test. Let me check whether a one-sided test produces ~0.07, and then do the confounding analysis.The 0.07 is a one-sided p-value. That's a critical finding. Now the confounding/Simpson's analysis.I now have everything I need. Here is the complete analysis.
A/B Test Analysis: New Onboarding with Video Tour
The bottom line up front
The marketer is wrong, but not because the new onboarding definitely fails – because this test cannot answer the question it was built to answer. Two problems dominate the decision, and either one alone is disqualifying:
- The "p = 0.07" is a one-sided p-value. The correct two-sided p-value is ≈ 0.15 – twice the reported number and nowhere near significance.
- Randomization is broken on the single variable the treatment depends on (device). The video tour only works on desktop, and the control group happened to have 44% mobile vs. 31% in the treatment group. A meaningful chunk – plausibly a third to a half – of the observed "lift" could be pure composition, not the onboarding change.
Recommendation: Do not deploy. Fix the randomization and re-run. The current test is not evidence of a working feature; it's a signal worth investigating properly.
Question 1 – Can these results be trusted? Methodological problems
No. Ranked by severity:
A. Broken randomization (fatal). Mobile share is 44% in A and 31% in B. Randomization exists precisely so the two groups differ only by treatment. Here they differ by device mix – and device is directly coupled to the treatment (video is desktop-only). This is the textbook setup for a confounded result. When randomization visibly fails on a variable that interacts with the treatment, the experiment loses its main claim to causal validity.
B. Underpowered for the effect that actually exists (fatal for "significance"). The test was designed to detect an 8pp difference, needing 592/group. To detect the observed 5.3pp at 80% power you'd need **1,340 per group** – nearly four times what you ran. With 348/group, your actual power to detect a 5.3pp effect was only **30%**. A study with 30% power that returns a non-significant result tells you almost nothing; you were unlikely to reach significance even if the effect is real.
C. Unplanned mid-test change (serious). In week 3 an ad channel was switched, giving B extra users (139 vs. 121) and a different traffic source. Different acquisition channels bring different user intent and quality. This is both a sample-ratio and a sample-composition problem introduced during the test.
D. Odd, non-contiguous date range (flag). "Feb 21–28 and Mar 1–14" is stated as 3 weeks but the weekly buckets don't cleanly tile; worth confirming no data was dropped or double-counted. Any manual date stitching invites errors.
E. Possible peeking / one-sided framing (serious). Reporting a one-sided p as "almost significant" suggests the analysis was framed to make the result look as strong as possible – a decision-quality red flag independent of the math.
Question 2 – What does p = 0.07 mean, and is it even correct?
What a p-value means: it's the probability of seeing a difference at least this large if the new onboarding truly had no effect (the null hypothesis). It is not the probability the feature works, not the probability the result is "real," and 0.07 does not mean "93% likely to work." Those are the misinterpretations the marketer is almost certainly making.
Now the verification – this is the key catch. I computed the two-proportion z-test three ways:
| Test | p-value |
|---|---|
| Two-sided pooled z-test | 0.150 |
| Fisher's exact (two-sided) | 0.164 |
| Chi-square with continuity correction | 0.174 |
| One-sided pooled z-test | 0.075 |
The reported "≈ 0.07" is the one-sided value. The industry-standard test for a two-way A/B comparison is two-sided, which gives p ≈ 0.15 – roughly double, and far from any conventional threshold.
Using a one-sided test doubles your effective false-positive rate and is only legitimate if you pre-committed to caring only about improvement and would treat "B is worse" identically to "no difference." That's rarely defensible for a launch decision. So even at face value, and before touching confounders, this result is not statistically significant. The 95% confidence interval on the lift is [−1.9pp, +12.6pp] – it comfortably includes zero (and even a small negative effect).
The marketer's likely errors, concretely:
- Treating "almost significant" as "significant."
- Not knowing (or not disclosing) it's one-sided.
- Reading 0.07 as "93% chance it works."
- Ignoring that even the point estimate (+5.3pp) is below the effect the test was designed to detect.
Question 3 – Confounders and the device-mix sensitivity analysis
The dominant confounder is device mix, and it's especially dangerous because it's mechanically linked to the treatment (video = desktop-only). Let me quantify how much of the 5.3pp could be composition rather than causation.
The core algebra. Suppose the new onboarding truly did nothing (null), so desktop and mobile convert at identical rates in both groups. Then a group's overall rate is just a weighted average of device rates. The difference between groups reduces cleanly to:
$$p_B - p_A = (\text{mobile}_A - \text{mobile}_B)\times g = (0.44 - 0.31)\times g = 0.13,g$$
where g = desktop-minus-mobile conversion gap.
How big a gap would explain the entire lift? Solving 0.13·g = 5.3pp gives g = 41pp. A 41-point desktop-vs-mobile gap is large but not absurd for onboarding (mobile onboarding is notoriously worse). More usefully, here's the fraction of the lift that vanishes at realistic gaps:
| Desktop − mobile gap | Composition-driven "lift" | % of 5.3pp that is fake |
|---|---|---|
| 10pp | 1.3pp | 24% |
| 15pp | 2.0pp | 36% |
| 20pp | 2.6pp | 49% |
| 25pp | 3.3pp | 61% |
| 30pp | 3.9pp | 73% |
Interpretation: For any realistic device gap (and 15–25pp is common for onboarding funnels), roughly a third to two-thirds of the observed 5.3pp is explained by device mix alone, before the feature does anything. This is a real, mechanical Simpson's-paradox risk: the aggregate looks better for B largely because B is loaded with the easier-converting device.
Illustrative standardization. Assuming plausible control rates (desktop 45%, mobile 27% – an 18pp gap – which reproduces A's 37.4% almost exactly), I back out what the video would need to do to produce B's 42.7%:
- If the desktop video adds a real +5pp, predicted B ≈ 42.9% – matches observed. So the data are consistent with a modest, genuine desktop effect.
- But the data are equally consistent with the effect being mostly composition plus noise.
The honest conclusion: the observed result is compatible with (a) a real ~5pp desktop lift, or (b) mostly a device-mix artifact, or anything in between. The experiment as run cannot distinguish these. That is the whole problem.
Other confounders (secondary):
- Channel switch (week 3): the new ad channel disproportionately fed B and brings different user intent – a second, independent composition bias. Notably, B's weekly rate actually declined over the test (44.2% → 43.1% → 41.1%), which is mildly concerning and coincides with the channel change.
- Novelty effect: a shiny video tour can lift completion short-term without durable behavior change; 3 weeks won't reveal this.
- Text-vs-video for mobile: B's mobile users got a different experience (text), so "Group B" isn't one treatment – it's two.
Question 4 – What to do to get a reliable answer
Fix the design:
- Repair randomization. Randomize within device (stratified/blocked randomization by desktop vs. mobile) so mix is balanced by construction. Verify balance before analyzing.
- Run a Sample Ratio Mismatch (SRM) check on every future test – the 44/31 split should have tripped an automatic alarm.
- Freeze traffic sources. No mid-test channel changes; if acquisition must change, restart the test.
Re-power correctly:
4. Decide the minimum effect worth shipping. If it's the desktop lift (5pp), you need **1,340 per group** at 80% power – plan for the sample and duration that requires, or accept detecting only larger effects. Do not re-run at the old size and hope.
Analyze correctly: 5. Segment by device (pre-registered). The real question is: does the video improve desktop completion? Analyze desktop-only as the primary endpoint, since that's where the mechanism lives. Analyze mobile (text) separately. 6. Use a two-sided test, or pre-register one-sided with explicit justification. 7. If you insist on watching the test live, use a proper sequential method (e.g., group-sequential / always-valid p-values). Repeatedly peeking at a fixed-horizon test inflates false positives badly – a likely contributor to the "almost significant" framing. 8. Track a guardrail + durable metric (e.g., week-2 activation or retention), not just first-session completion, to rule out novelty effects.
Optional bridge: If leadership wants to move now, ship the video to desktop only as a monitored rollout (it can't hurt mobile since mobile never sees it), with completion tracking – treating it as a follow-up experiment, not a settled win.
Question 5 – How to frame this for leadership (no jargon, honest)
"The new onboarding is promising, but this test can't yet prove it works – and we know why. I recommend we fix two things and re-run before a full launch."
What we saw: New onboarding completed at 42.7% vs. 37.4% – about 5 points higher. That looks good.
Why we can't trust it yet – two plain-English problems:
- The groups weren't apples-to-apples. By accident, the old-onboarding group had far more phone users (44%) than the new group (31%). Phone users finish onboarding less often regardless of design, and our video only plays on desktop. So the new group had a built-in head start. When we account for this, somewhere between a third and half of the 5-point gain may just be that mix difference, not the new design.
- The result isn't statistically solid. The "almost significant" number that's been circulating (0.07) was calculated the lenient way. Done the standard way, it's about 0.15 – meaning there's a real chance this gap is just luck. The test was also too small to reliably detect a change this size.
What I recommend: Re-run for ~4–6 weeks with balanced groups, don't change ad channels mid-test, and measure desktop and mobile separately. The early signal is encouraging enough that this is worth doing right – and if we want momentum now, we can safely turn the video on for desktop users only while we gather clean data.
The risk of deploying today: if the gain is mostly the accidental mix difference, we'll declare a win, see no real improvement in the numbers, and lose trust in future tests. A two-to-four week delay buys us an answer we can actually stand behind.
The single most important issue
If you take away one thing: the treatment (video) is only available on desktop, and randomization failed on exactly the desktop/mobile axis. That coupling means device mix isn't just a confounder – it's a confounder pointed straight at the treatment mechanism, capable of manufacturing most of the observed lift on its own. Combined with the fact that the correct two-sided test isn't even significant, the responsible call is fix randomization, re-power, segment by device, and re-run – not "deploy immediately."
(Statistical figures above – p-values, confidence interval, power, and the composition sensitivity table – were computed directly from your data; the device-rate standardization uses assumed but plausible desktop/mobile base rates since device-level conversions weren't provided, and would be replaced with real segmented data in the re-run.)
~openai/gpt-latest
Executive verdict
The result is promising but not trustworthy enough to “deploy immediately.” The biggest problem is not the p-value. The biggest problem is that randomization was broken: Group B had far fewer mobile users than Group A, and mobile/desktop likely have very different onboarding completion rates. Because the new video onboarding only works on desktop while mobile users get a text version, the device imbalance directly threatens the interpretation.
Observed lift:
- A: 127 / 340 = 37.4%
- B: 152 / 356 = 42.7%
- Raw difference: +5.3 percentage points
But Group B had only 31% mobile, while Group A had 44% mobile. If mobile users complete onboarding less often than desktop users, some or even all of the apparent B lift may be due to B having a more favorable user mix rather than the new onboarding.
My recommendation: do not call this a successful experiment yet. Treat it as a directional signal. Fix the randomization and rerun, or at minimum analyze device- and channel-adjusted results before making a rollout decision.
1. Can these results be trusted?
Short answer: not as causal evidence.
The raw numbers look better for B, and the weekly pattern is directionally consistent:
| Week | A | B | Raw lift |
|---|---|---|---|
| Feb 21–28 | 38.1% | 44.2% | +6.1 pp |
| Mar 1–7 | 36.9% | 43.1% | +6.2 pp |
| Mar 8–14 | 37.2% | 41.1% | +3.9 pp |
| Total | 37.4% | 42.7% | +5.3 pp |
That is encouraging. But the experiment has serious validity problems.
Main methodological problems
1. Randomization was broken
Group A and Group B were not comparable.
- Group A mobile share: 44%
- Group B mobile share: 31%
- Difference: 13 percentage points
That is a large imbalance. Using approximate counts:
- A: 150 mobile, 190 desktop
- B: 110 mobile, 246 desktop
The mobile-share difference is about 13 pp. A simple z-test for the covariate imbalance gives roughly:
- Standard error ≈ 3.7 pp
- z ≈ 3.5
- p ≈ 0.0004
So this imbalance is very unlikely to be random noise. And you already know there was an operational reason: an ad channel was temporarily switched in the final week.
This is the dominant issue. If the groups are not exchangeable, the difference in completion rates cannot be cleanly attributed to onboarding.
2. Device is directly related to the treatment
This is worse than a generic demographic imbalance.
The new onboarding has a video tour on desktop, but mobile users get a text version. So device affects both:
- The user’s baseline likelihood of completing onboarding.
- The actual experience delivered by the treatment.
That means “Group B” is not one homogeneous treatment. It is really:
- B-desktop: new onboarding with video tour.
- B-mobile: new onboarding with text version.
Because B had more desktop users, it had more users exposed to the stronger version of the treatment, and also possibly more users with naturally higher completion rates.
3. Acquisition channel changed during the experiment
In Mar 8–14, Group B got more users because an ad channel was temporarily switched.
That introduces another confounder. Users from different acquisition channels can differ in:
- Intent.
- Company size.
- Device mix.
- Prior awareness.
- Willingness to complete setup.
- Geography.
- Plan type.
- Sales-assisted versus self-serve behavior.
If B got more high-intent or more desktop-heavy users from a different source, the observed lift could be due to traffic quality, not onboarding.
4. The test was underpowered for the observed effect
The test was reportedly powered for an MDE of 8 percentage points, but the observed difference was only 5.3 pp.
That matters. If the real effect is around 5 pp, this sample is too small to estimate it reliably.
With baseline around 37% and observed treatment rate around 43%, a conventional 80% power, two-sided 5% alpha test would need roughly:
- ~1,300 users per group to detect a 5.3 pp effect.
- Current sample: 340 A, 356 B.
So the current test is too small for a confident answer about a 5 pp lift.
Also, if the test was supposedly powered for an 8 pp absolute effect, the collected sample seems low. A conventional two-sided test for an 8 pp lift would usually require around ~580–600 users per group, not ~350 per group. So the original power calculation should be checked.
2. What does p-value 0.07 mean? Verify the actual p-value.
Using the aggregate results:
- A: 127 / 340 = 0.3735
- B: 152 / 356 = 0.4270
- Difference = 0.0534
For a two-proportion z-test with pooled standard error:
[ \hat{p} = \frac{127 + 152}{340 + 356} = \frac{279}{696} = 0.4009 ]
[ SE = \sqrt{\hat{p}(1-\hat{p})(1/340 + 1/356)} ]
[ SE \approx \sqrt{0.4009 \times 0.5991 \times (1/340 + 1/356)} \approx 0.0372 ]
[ z = \frac{0.4270 - 0.3735}{0.0372} \approx 1.44 ]
For a two-sided test:
[ p \approx 0.15 ]
For a one-sided test testing specifically whether B > A:
[ p \approx 0.075 ]
So the reported “about 0.07” is probably the one-sided p-value, not the standard two-sided p-value.
The approximate 95% confidence interval for the raw difference is:
[ 5.3 \text{ pp} \pm 1.96 \times 3.7 \text{ pp} ]
So:
[ [-1.9 \text{ pp}, +12.6 \text{ pp}] ]
That interval includes zero and even a small negative effect.
What the p-value actually means
A one-sided p-value of 0.075 means:
If there were truly no difference between old and new onboarding, and if the experiment were properly randomized, there would be about a 7.5% chance of seeing a lift at least this large in the favorable direction just from random sampling noise.
The two-sided p-value of 0.15 means:
If there were truly no difference, there would be about a 15% chance of seeing a difference this large or larger in either direction.
How the marketer is likely misinterpreting it
They may be thinking:
- “p = 0.07 means there is a 93% chance the new onboarding works.”
- “It is almost significant, so we can treat it as significant.”
- “The effect is probably real because the p-value is close to 0.05.”
- “A 14.2% relative lift is large, so the result is obvious.”
Those are not valid interpretations.
A p-value is not the probability that the treatment works. Also, “almost significant” is not a statistical category. The evidence is suggestive, but weak – and in this case the broken randomization is a larger concern than the p-value.
3. Could the observed lift be explained by desktop/mobile mix?
Yes, possibly. At minimum, the device imbalance could explain a meaningful portion of the apparent lift.
Group A had more mobile users:
- A: 44% mobile, 56% desktop
- B: 31% mobile, 69% desktop
So B had 13 percentage points more desktop users.
If desktop users complete onboarding at a higher rate than mobile users, B would look better even if the onboarding change had no true effect.
Let:
[ g = \text{desktop completion rate} - \text{mobile completion rate} ]
The composition advantage for B is approximately:
[ 0.13 \times g ]
because B has 13 pp more desktop users.
Sensitivity table
| Assumed desktop-mobile gap | Lift explained by composition | Remaining lift after adjustment |
|---|---|---|
| 0 pp | 0.0 pp | 5.3 pp |
| 10 pp | 1.3 pp | 4.0 pp |
| 20 pp | 2.6 pp | 2.7 pp |
| 30 pp | 3.9 pp | 1.4 pp |
| 40 pp | 5.2 pp | 0.1 pp |
| 41 pp | 5.3 pp | 0.0 pp |
| 50 pp | 6.5 pp | -1.2 pp |
So if desktop users complete onboarding about 41 percentage points more often than mobile users, the entire observed 5.3 pp lift could be explained by the device mix alone.
Is a 41 pp desktop-mobile gap large? Yes. But for a SaaS onboarding flow requiring five key actions, it is not impossible, especially if the full video experience only exists on desktop and mobile gets a reduced text version.
For example, suppose with the old onboarding:
- Desktop completion: 55%
- Mobile completion: 14%
Then Group A’s expected completion would be:
[ 0.56 \times 55% + 0.44 \times 14% \approx 37% ]
Group B, with more desktop users, would be expected to complete at:
[ 0.69 \times 55% + 0.31 \times 14% \approx 42% ]
That is almost exactly the observed result, even with zero true treatment effect.
That does not prove the lift is fake. But it proves the aggregate result is not identifiable without device-level outcome data.
More moderate assumptions still matter
Even if the desktop-mobile gap is only 20 pp, the composition effect would be:
[ 0.13 \times 20% = 2.6 \text{ pp} ]
That would reduce the treatment lift from 5.3 pp to about 2.7 pp.
If the gap is 30 pp, the composition effect would be:
[ 0.13 \times 30% = 3.9 \text{ pp} ]
leaving only about 1.4 pp of apparent treatment effect.
So even if device mix does not explain everything, it could explain a large share of the result.
And device is not the only confounder
The ad-channel switch could independently affect results. If B received users from a slightly better-converting channel, that could add another few percentage points. Combined with the device imbalance, that could easily explain the full 5.3 pp observed lift.
4. What should be done to get a reliable answer?
Immediate next step: do not deploy as “proven”
The result should be labeled:
Directionally positive but inconclusive due to broken randomization and insufficient power.
If the video onboarding is low-risk, a limited rollout may be reasonable, but it should not be presented as a validated causal win.
Analyze the current data properly before discarding it
Ask for the data broken down by:
- Variant.
- Device: desktop vs mobile.
- Acquisition channel.
- Week or day.
- Geography, plan, company size if available.
- Whether the user actually saw the video or got the text fallback.
At minimum, produce a table like:
| Device | A users | A completions | A rate | B users | B completions | B rate | Lift |
|---|---|---|---|---|---|---|---|
| Desktop | |||||||
| Mobile |
Then estimate:
- Desktop-specific treatment effect.
- Mobile-specific treatment effect.
- Overall effect reweighted to the expected future traffic mix.
A logistic regression or linear probability model could help:
[ Completion \sim Treatment + Device + Channel + Week + Treatment \times Device ]
But this would be a damage-control analysis, not a full rescue. Post-hoc adjustment cannot fix unmeasured differences caused by broken assignment.
Fix randomization and rerun
The rerun should have clean experimental controls.
Required fixes
Randomize users properly at the individual level.
- Assignment should not depend on ad channel, device, week, campaign, geography, or landing page.
Stratify or block randomization by device.
- Keep the desktop/mobile mix balanced within A and B.
- For example: randomize desktop users 50/50 and mobile users 50/50 separately.
Also stratify or monitor by acquisition channel.
- If a channel is turned on or off, both variants should receive the change equally.
Define the unit of randomization.
- Usually user ID or account ID.
- Avoid users switching variants across sessions or devices.
Define the estimand. Decide whether the product wants to know:
- Overall effect across all users.
- Desktop effect of the video.
- Mobile effect of the text fallback.
- Weighted effect for expected future traffic.
Predefine the analysis.
- Primary metric: onboarding completion.
- Analysis window.
- Two-sided or one-sided test.
- Minimum sample size.
- Segment analyses.
- Guardrail metrics.
Avoid sequential peeking.
- If the team wants to monitor weekly, use a preplanned sequential design or alpha-spending rule.
- Otherwise, repeated checking increases false-positive risk.
Recalculate sample size based on the effect you care about
If leadership cares about detecting a lift around the observed 5.3 pp, the current sample is too small.
Approximate required sample size for 80% power, two-sided 5% alpha:
- Baseline: 37.4%
- Target treatment rate: 42.7%
- Absolute lift: 5.3 pp
- Required sample: roughly 1,300 users per group, or 2,600 total
For 90% power, it would be closer to:
- 1,700–1,800 users per group
If the true minimum meaningful effect is 8 pp, required sample is smaller, roughly:
- ~580–600 users per group for 80% power, two-sided 5% alpha.
Because the treatment differs by device, you may need even more total sample if you want reliable desktop and mobile estimates separately.
5. How to frame this for leadership
Use plain language. Avoid saying “the test failed.” Better framing:
The new onboarding produced a promising raw lift: 42.7% versus 37.4%, a 5.3 percentage-point increase. However, we cannot confidently attribute that lift to the onboarding change because the experiment groups were not comparable. Group B had many more desktop users and fewer mobile users, and desktop users are likely more likely to complete onboarding. Since the video tour only works on desktop, this imbalance directly affects the result.
Then be explicit:
Statistically, the standard two-sided p-value is about 0.15, not 0.07. The 0.07 number is likely a one-sided test. That means the evidence is suggestive, not conclusive. The 95% confidence interval ranges from about -2 pp to +13 pp, so the true effect could be small, zero, or meaningfully positive.
Then identify the dominant issue:
The main reason not to deploy immediately is not just statistical significance. The main reason is broken randomization. Because Group B had a more favorable device mix and an unplanned ad-channel difference, the observed lift may be partly or entirely due to user composition rather than the new onboarding.
Recommended leadership position:
We should treat this as a positive signal worth following up, not as proof. The right next step is to rerun the test with proper randomization stratified by device and acquisition channel. If the true effect is around 5 percentage points, we should plan for roughly 1,300 users per group. If we need a faster decision, we can do a limited ramp with measurement, but we should not claim the experiment has validated the new onboarding yet.
Bottom line
The marketer is overclaiming. The PM is right to be skeptical.
The result may be real, but the current experiment cannot prove it. The broken randomization and device imbalance dominate the decision. Even a cleaner p-value would not solve that. The correct decision is: do not deploy as a proven winner; rerun or continue under a properly controlled design.
~google/gemini-pro-latest
Скептицизм продакт-менеджера в данном случае абсолютно оправдан. С точки зрения статистики, методологии и анализа данных этот тест содержит критические ошибки. Развертывать новую версию на основе этих данных нельзя.
Ниже представлен строгий и подробный разбор для команды, а также рекомендации по аргументации этой позиции для руководства.
1. Можно ли доверять результатам? Какие есть методологические проблемы?
Нет, доверять результатам нельзя. Эксперимент скомпрометирован сразу на нескольких уровнях:
- Сломанная рандомизация (Критическая ошибка): Доля мобильных пользователей в группе A – 44%, в группе B – 31%. В идеальном A/B тесте обе группы должны быть идентичны (appx 50/50 по составу аудитории). Эта разница убивает главный принцип сплит-тестирования: «при прочих равных».
- Внешнее искажение (Вмешательство в трафик): В третью неделю произошла смена рекламного канала, что привело к резкому притоку пользователей в группу B (139 против 121 в группе A). Разные рекламные каналы приносят аудиторию с разной мотивацией и типами устройств, что прямо нарушает чистоту эксперимента. Скорее всего, именно этот сбой и «сломал» рандомизацию по устройствам.
- Несоответствие проверяемой гипотезы: Тест задумывался для оценки видеотура, но видео не работает на мобильных устройствах. В таком случае оценивать метрики смешанной аудитории бессмысленно, так как мобильные пользователи в группе B получали просто текстовую версию. Мы тестировали не видео, а «видео для десктопа + текст для смартфонов» в противовес «старой версии для всех».
- Недостаток мощности (Underpowered): Тест был рассчитан на обнаружение эффекта (MDE) в 8%, а реальная разница составила 5.3%. При текущем объеме выборки (около 350 человек на группу) мы физически не можем сделать достоверный статистический вывод о разнице в 5.3%.
2. Что на самом деле значит p-value 0.07, и как маркетолог им манипулирует?
Маркетолог, скорее всего, интерпретирует p-value 0.07 как «существует 93% вероятность, что новая версия работает лучше». Это в корне неверно. P-value означает обратное: если бы новая версия на самом деле не давала вообще никакого эффекта, то вероятность случайно получить такой же или больший прирост составила бы 7%.
Но есть более серьезная проблема. Давайте посчитаем реальное значение для z-теста двух долей (two-proportion z-test):
- Группа A: $p_1 = 127 / 340 = 37.35%$
- Группа B: $p_2 = 152 / 356 = 42.70%$
- Округляя, z-оценка (z-score) составляет ~1.44.
- Если мы используем корректный двусторонний тест (two-tailed), настоящий p-value равен 0.15 (15%).
Откуда маркетолог взял 0.07? Значение 0.075 получается, если использовать односторонний тест (one-tailed). Переключаться на односторонний тест постфактум, когда мы видим положительный результат, чтобы формально снизить p-value – это известная статистическая манипуляция (p-hacking). Даже при 0.07 результат не достигает золотого стандарта в 0.05, а реальный показатель (0.15) доказывает, что результат статистически незначим.
3. Исследование искажающих факторов (Парадокс Симпсона)
Самый мощный аргумент против теста – доля мобильного трафика. В B2B/SaaS конверсия с десктопа (пользователь на рабочем месте, за компьютером) традиционно значительно выше конверсии со смартфона, где проходить онбординг неудобно.
Поскольку в группе B оказалось непропорционально много пользователей десктопа (69% против 56% в группе A), этот фактор мог единолично создать весь видимый прирост в 5.3%, даже если видеотур вообще никак не повлиял на конверсию!
Проверим это математически (Анализ чувствительности): Допустим, видеотур не работает и конверсии у устройств в обеих группах абсолютно одинаковые.
- Группа А (340 чел): ~190 десктоп, 150 мобайл.
- Группа B (356 чел): ~246 десктоп, 110 мобайл.
Чтобы получить ровно те цифры, которые мы видим в тесте (127 целевых действий в А и 152 в В), нам достаточно предположить следующие базовые конверсии:
- Конверсия с десктопа: ~55%
- Конверсия с мобильного: ~15%
Проверка: в группе А (190 * 0.55) + (150 * 0.15) = 104.5 + 22.5 = 127 конверсий. В группе B (246 * 0.55) + (110 * 0.15) = 135.3 + 16.5 = ~152 конверсии.
Вывод: Разница конверсий между мобильными (15%) и десктопами (55%) – это абсолютно типичная картина для SaaS. Следовательно, весь «фантастический рост» на 14.2% математически объясняется банальным перекосом аудитории (Парадокс Симпсона). Видео, судя по всему, вообще ничего не улучшило.
4. Что нужно сделать сейчас для получения достоверного ответа?
- Изолировать десктоп-данные: Раз видеотур доступен только на десктопе, немедленно пересчитайте конверсии только для декстоп-пользователей из обеих групп. Это очистит данные от мобильного «шума». Если там роста нет – гипотеза мертва.
- Разобраться с багом распределения: Выясните, почему смена рекламного канала в третью неделю привела к перекосу в распределении трафика (особое внимание к тому, как система A/B тестирования хэширует/распределяет пользователей с новых кампаний).
- Запустить правильный перезапуск: Если десктопный срез покажет потенциал, перезапустите тест. Но на этот раз:
- Тестируйте только на пользователях десктопа.
- Увеличьте размер выборки. Поскольку реальный ожидаемый эффект около 5% (а не 8%), вам потребуется пропустить через тест примерно в 2.5 раза больше пользователей, чтобы мощность (Power) составила 80% – скорее всего, это займет 6-8 недель, а не три.
5. Как аргументировать позицию руководству (без сложного жаргона)
Вам нужно сместить фокус с «мы не верим в новую фичу» на «мы хотим защитить продукт от случайных решений».
Шаблон для коммуникации (Executive Summary):
"На первый взгляд, новая версия онбординга показала рост на 5.3 пункта, однако при глубоком анализе данных мы обнаружили существенный технический сбой, который не позволяет доверять результатам.
В ходе теста сломался механизм случайного распределения пользователей: в группе с видео оказалось значительно больше людей, проходивших регистрацию с компьютеров, а в старой группе – со смартфонов. Так как онбординг со смартфона проходить исторически сложнее, группа с видео-версией получила искусственное преимущество внедрения.
Наши расчеты показывают, что один только этот сбой полностью объясняет видимый "рост" показателей. То есть, высокая вероятность того, что эффект от самого видео – нулевой. Дополнительно, статистическая проверка подтверждает, что результат оказался на уровне погрешности (15% шанса, что результат случаен, а не 7%, как считалось изначально).
Если мы раскатим это сейчас, мы можем внедрить фичу, которая не приносит денег, но усложняет код. Мы предлагаем: не разворачивать релиз на всех, вычленить реальные показатели чисто по десктопу (где видео и работало), и при необходимости провести короткий перетест с учетом ошибки."
Этот подход показывает лидерству вашу компетентность, предостерегает от ошибки и предлагает понятный конструктивный следующий шаг.
Analysis
Consensus
- : [
- be trusted as clean causal evidence of the video tour effect; deploying immediately is the wrong call; treat it as a directional, promising-but-inconclusive signal
- The reported p≈0.07 is a one-sided sided p-value; the correct two-sided two-proportion z-test test p-value is ≈0.15, computed via z≈1.44, which is not statistically significant
- The marketer likely misinterprets p=0.07 as '93% chance the feature works' and treats 'almost significant' as significant, both invalid interpretations
- A p-value is the probability of observing a difference at least this large if the null (no effect) were true, not the probability the treatment works
- Broken randomization (mobile 44% in A vs 31% in B) is the dominant/most important issue, worse because device is mechanically coupled to the treatment (video is desktop-only)
- The 95% confidence interval for the lift is roughly [-1.9pp, +12.6pp], which includes zero and even a small negative effect
- The composition effect of B's device advantage is approximately 0.13 × g, where g is the desktop-minus-mobile conversion gap
- A desktop-mobile gap of ~41pp would fully explain the entire 5.3pp lift with zero true treatment effect
- Using plausible SaaS base rates (desktop ~55%, mobile ~15%), the entire observed result can be reproduced with zero treatment effect, showing the aggregate is not identifiable without device-level data
- The test is underpowered for the observed 5.3pp effect; roughly ~1,300 per group is needed at 80% power/two-sided 5%, versus the ~350 per group actually run
- The mid-test ad-channel switch is an independent confounder introducing traffic-quality/composition bias
- Group B is not a single homogeneous treatment: desktop users got video while mobile users got a text version
- The fix is to re-run with device-stratified randomization, avoid mid-test channel changes, segment/analyze by device, recompute sample size for the true effect, and avoid sequential peeking
- Leadership framing should avoid jargon, avoid saying 'the test failed', present it as a promising-but-unproven signal, explain the device imbalance in plain terms, and propose a clean re-run
Contradictions
- Likelihood that the true treatment effect is essentially zero
- ~anthropic/claude-opus-latest: Balanced/agnostic: data are equally consistent with a real ~5pp desktop lift OR mostly composition artifact; the experiment cannot distinguish them, so effect is unknown, not necessarily zero
- ~openai/gpt-latest: Agnostic: composition could explain part or even all; a zero-effect scenario is possible but not asserted; 'may be real'
- ~google/gemini-pro-latest: Strongly leans that the video likely did nothing: states the whole growth is mathematically explained by device skew and 'the video apparently improved nothing / high probability effect is zero'
- Whether the one-sided p-value framing is innocent or p-hacking
- ~anthropic/claude-opus-latest: Treats it as a decision-quality red flag / one-sided framing to make result look stronger, but frames it as legitimate only if pre-committed
- ~openai/gpt-latest: Neutral: 'probably the one-sided p-value', explains meaning without explicitly labeling it manipulation
- ~google/gemini-pro-latest: Explicitly labels switching to a one-sided test post-hoc as a known statistical manipulation (p-hacking)
- Sensitivity analysis presentation and framing of remaining lift
- ~anthropic/claude-opus-latest: Presents table as '% of 5.3pp that is fake' and concludes roughly a third to two-thirds is composition at realistic 15-25pp gaps
- ~openai/gpt-latest: Presents table with 'remaining lift after adjustment', including a 50pp row yielding negative remaining lift, emphasizes not identifiable
- ~google/gemini-pro-latest: Does not give a graded sensitivity table; picks a single plausible scenario (desktop 55%/mobile 15%) that exactly reproduces the counts and concludes effect is essentially fully explained
- Whether a partial/desktop-only rollout now is acceptable
- ~anthropic/claude-opus-latest: Offers an optional bridge: ship video to desktop-only as a monitored rollout since it can't hurt mobile
- ~openai/gpt-latest: Allows a limited ramp with measurement if a faster decision is needed, but not as a validated win
- ~google/gemini-pro-latest: Does not endorse any rollout now; recommends isolating desktop data and re-testing desktop-only before any decision
Partial coverage
- ~anthropic/claude-opus-latest, ~openai/gpt-latest: Explicitly report multiple test variants (Fisher's exact, chi-square with continuity correction) or note the CI/SE derivation; Opus gives a full table of Fisher's/chi-square values
- ~anthropic/claude-opus-latest: Notes the actual power to detect a 5.3pp effect at the run sample was only ~30%, so a non-significant result is nearly uninformative
- ~anthropic/claude-opus-latest: Flags the odd non-contiguous date range (Feb 21-28 and Mar 1-14) as a possible data-stitching / double-counting error
- ~anthropic/claude-opus-latest: Observes Group B's weekly rate declined over time (44.2% → 43.1% → 41.1%), coinciding with the channel change, and raises a novelty-effect concern
- ~anthropic/claude-opus-latest, ~openai/gpt-latest: Recommend Sample Ratio Mismatch (SRM) checks and tracking guardrail/durable metrics (retention/week-2 activation) rather than just first-session completion
- ~openai/gpt-latest: Computes a formal statistical test for the covariate (device) imbalance itself: z≈3.5, p≈0.0004, showing the imbalance is not random noise
- ~openai/gpt-latest: Proposes a specific regression model (Completion ~ Treatment + Device + Channel + Week + Treatment×Device) for damage-control analysis and defining the estimand
- ~openai/gpt-latest, ~anthropic/claude-opus-latest: Explicitly question the original power calculation: an 8pp MDE at 80% power needs ~580-600/group, yet only ~350/group were run, so the stated powering is inconsistent
openai/gpt-latest: Lists specific 90% power sample size (1,700-1,800/group) and notes needing even more sample for reliable per-device estimates- ~openai/gpt-latest: Enumerates concrete acquisition-channel confounders (intent, company size, geography, plan type, sales-assisted vs self-serve)
- ~google/gemini-pro-latest, ~anthropic/claude-opus-latest: Explicitly hypothesize that the mid-test ad-channel switch is likely the mechanism that CAUSED the broken device randomization
- ~google/gemini-pro-latest: Frames the leadership pitch around shifting from 'we don't believe in the feature' to 'we want to protect the product from random decisions,' and mentions the code-complexity cost of shipping a feature that doesn't pay off
Unique insights
anthropic/claude-opus-latest: Quantifies the retrospective power at the observed effect (30%), giving a principled reason why a non-significant result here is nearly uninformative rather than evidence of no effect- ~anthropic/claude-opus-latest: Catches the non-contiguous/odd date range as a potential data-integrity issue that the other two models missed entirely
- ~anthropic/claude-opus-latest: Notes B's declining weekly completion trend and ties it to the channel switch plus a possible novelty effect, adding a temporal-dynamics lens
- ~openai/gpt-latest: Statistically tests the device-imbalance covariate itself (z≈3.5, p≈0.0004), proving the randomization break is real rather than plausible noise – a rigorous step neither other model performed
- ~openai/gpt-latest: Specifies the estimand question explicitly (overall vs desktop vs mobile vs future-traffic-weighted effect), reframing the analysis around what leadership actually wants to know
- ~google/gemini-pro-latest: Directly names the post-hoc switch to a one-sided test as p-hacking, sharpening the ethical/decision-quality critique
- ~google/gemini-pro-latest: Reframes the reconciling device rates (desktop ~55%, mobile ~15%) as an exact arithmetic reconstruction of the observed completion counts, making the zero-effect scenario concretely verifiable
Blind spots
- None of the models flagged that the assumed device conversion rates (e.g., desktop 55% / mobile 15%) are fabricated inputs chosen to fit the result – while useful illustratively, they overstate certainty because actual device-level completion data were never provided, and equally plausible smaller gaps leave a real effect
- No model reconciled the total weekly n's against the stated group totals (89+130+121=340 for A, 86+131+139=356 for B – which do check out) – a quick arithmetic verification of the given breakdown was skipped
- None addressed the multiple-comparisons / garden-of-forking-paths risk introduced by the very device/channel segmentation they recommend (segmented analyses inflate false positives unless pre-registered)
- No model quantified the business cost/benefit or risk asymmetry of deploying desktop-only now versus delaying (video is desktop-only and cannot harm mobile, so a monitored desktop rollout has limited downside – only Opus/GPT gestured at this without analysis)
- None discussed novelty/primacy effects rigorously or the need for a longer post-exposure retention window beyond first-session completion (Opus only briefly mentioned novelty)
- No model questioned whether 'onboarding completion (5 key actions)' is even the right success metric versus downstream activation/revenue/retention, i.e., metric validity
- None considered whether users could be exposed to both variants across devices/sessions (contamination) or how the randomization unit (user vs session vs account) interacts with the desktop/mobile split
- No model addressed statistical power/interpretation for the planned per-device subgroup analyses (desktop-only n is smaller, so a desktop re-run needs its own power calculation, which was only loosely mentioned)
- None examined potential Simpson's reversal risk within the weekly data or whether the declining B trend could indicate regression toward null as the sample grew
Отличный кейс – и ваша осторожность полностью оправдана. Три независимые модели проанализировали данные и сошлись в главном: маркетолог ошибается, и запускать нельзя – но не потому, что новый онбординг точно не работает, а потому что этот тест в принципе не может ответить на вопрос, ради которого он ставился. Ниже – разбор по вашим пунктам.
Коротко, самое важное
Есть две проблемы, каждая из которых сама по себе достаточна, чтобы не доверять результату:
- P-value 0,07 – это односторонний тест. Корректный двусторонний p-value ≈ 0,15 – вдвое больше и далеко за пределами значимости.
- Рандомизация сломана именно по тому признаку, от которого зависит фича (устройство). Видео работает только на десктопе, а в контрольной группе оказалось 44% мобильных против 31% в тестовой. Значительная часть «прироста» может быть чистой композицией аудитории, а не эффектом онбординга.
1. Можно ли доверять результатам?
Нет, как причинно-следственному доказательству – нельзя. Проблемы по убыванию серьёзности:
- Сломанная рандомизация (критично). Мобильные: 44% в A vs 31% в B. Это не случайный шум – формальная проверка даёт z ≈ 3,5, p ≈ 0,0004. Группы неэквивалентны.
- Устройство напрямую связано с лечением (усугубляет). Видео есть только на десктопе; мобильные получают текст. Значит «группа B» – это на самом деле два разных лечения: «видео на десктопе» + «текст на мобильном». B оказалась перегружена десктопом → и более сильной версией фичи, и более «лёгкой» аудиторией.
- Незапланированная смена рекламного канала (серьёзно). В неделю 8–14 марта B получила больше пользователей из другого источника. Разные каналы = разное намерение, качество, состав аудитории. Вероятно, именно это и сломало рандомизацию по устройствам.
- Недостаточная мощность. Тест рассчитан на MDE 8%, а реальная разница 5,3%. Фактическая мощность обнаружить эффект в 5,3% при вашем объёме – около 30%. Незначимый результат при 30% мощности почти неинформативен.
2. Что значит p-value 0,07 и где ошибка маркетолога
Что p-value означает на самом деле: это вероятность увидеть разницу не меньше наблюдаемой, если бы новый онбординг вообще не давал эффекта. Это НЕ «93% вероятность, что фича работает».
Проверка расчёта (two-proportion z-test):
- A: 127/340 = 37,35%; B: 152/356 = 42,70%
- z ≈ 1,44
- Двусторонний p-value ≈ 0,15
- Односторонний p-value ≈ 0,075 ← вот откуда «около 0,07»
95% доверительный интервал для прироста: примерно [−1,9 п.п.; +12,6 п.п.] – включает ноль и даже небольшой отрицательный эффект.
Типичные ошибки маркетолога:
- «Почти значимо» трактуется как «значимо» (такой категории в статистике нет).
- 0,07 читается как «93% что работает».
- Не сказано (или не осознано), что это односторонний тест – переключение на него постфактум ради красивой цифры близко к p-hacking.
- Игнорируется, что даже точечная оценка (+5,3 п.п.) ниже эффекта, на который тест был рассчитан.
3. Искажающие факторы: мог ли весь прирост быть композицией?
Да, вполне. Это классический риск парадокса Симпсона.
B имела на 13 п.п. больше десктопных пользователей. Если десктоп конвертит лучше мобильного, B получает встроенную фору. Величина эффекта композиции ≈ 0,13 × g, где g – разрыв конверсии десктоп-минус-мобайл.
Анализ чувствительности:
| Разрыв десктоп−мобайл (g) | Прирост от композиции | % от 5,3 п.п. «ложный» |
|---|---|---|
| 10 п.п. | 1,3 п.п. | ~24% |
| 15 п.п. | 2,0 п.п. | ~36% |
| 20 п.п. | 2,6 п.п. | ~49% |
| 30 п.п. | 3,9 п.п. | ~73% |
| 41 п.п. | 5,3 п.п. | 100% |
Для типичного SaaS-разрыва в 15–25 п.п. от трети до половины прироста объясняется одним лишь составом аудитории. А если предположить вполне реалистичные для SaaS базовые конверсии (десктоп ~55%, мобайл ~15%), наблюдаемые цифры воспроизводятся точь-в-точь при нулевом эффекте видео.
⚠️ Важная оговорка: конкретные базовые конверсии (55%/15%) – это иллюстративное предположение, а не факт. Реальных данных по устройствам у вас нет. Честный вывод: данные совместимы и с реальным ~5 п.п. эффектом на десктопе, и с почти полностью композиционным артефактом. Эксперимент в текущем виде эти сценарии не различает. В этом и вся суть проблемы.
4. Что сделать, чтобы получить надёжный ответ
Сначала – выжать максимум из текущих данных:
- Разбить результаты по устройствам. Посчитать конверсию отдельно для десктопа (там, где видео реально работает) и мобильного. Это первичный вопрос: улучшает ли видео десктопную конверсию?
- Проверить sample ratio mismatch – сплит 44/31 должен был сработать как сигнал тревоги.
- Разбить по каналу привлечения и неделям.
Затем – правильный перезапуск:
4. Стратифицированная рандомизация по устройству (отдельно 50/50 для десктопа, отдельно для мобильного) – чтобы состав был сбалансирован по построению.
5. Не менять рекламные каналы в ходе теста.
6. Пересчитать размер выборки под реальный эффект. Для надёжного обнаружения 5,3 п.п. при 80% мощности нужно **1300 пользователей на группу** – почти вчетверо больше, чем было. Это, вероятно, 6–8 недель, а не 3.
7. Использовать двусторонний тест; если односторонний – то только с обоснованием заранее.
8. Если хотите смотреть промежуточные результаты – использовать корректный последовательный метод, а не «подглядывать».
9. Отслеживать удержание/активацию на 2-й неделе, а не только завершение первой сессии (защита от эффекта новизны).
Опциональный компромисс: поскольку видео на мобильных всё равно не показывается, можно включить его только для десктопа как контролируемый мониторируемый раскат – при этом честно называя это продолжением эксперимента, а не подтверждённой победой.
5. Формулировка для руководства (без жаргона, честно)
«Новый онбординг выглядит перспективно, но этот тест пока не доказывает, что он работает – и мы понимаем, почему. Предлагаю исправить две вещи и перезапустить перед полным внедрением.»
Что мы увидели: новая версия завершалась в 42,7% случаев против 37,4% – примерно на 5 пунктов выше. Это выглядит хорошо.
Почему пока нельзя доверять – две проблемы простыми словами:
- Группы оказались не сопоставимы. По случайности в старой версии было заметно больше пользователей со смартфонов (44% против 31% в новой). А со смартфона онбординг проходят реже – независимо от дизайна, ведь наше видео работает только на компьютере. То есть новая группа получила встроенную фору. По нашим расчётам, от трети до половины (а в худшем сценарии – весь) «прирост» объясняется именно этой разницей в составе, а не самим видео.
- Результат статистически неубедителен. Цифра «почти значимо» (0,07) посчитана мягким способом. По стандартному методу это около 0,15 – то есть заметная вероятность, что разница просто случайна. Плюс выборки было слишком мало для надёжного вывода.
Что предлагаю: перезапустить на 6–8 недель со сбалансированными группами, не менять рекламные каналы посреди теста и мерить десктоп и мобильные отдельно. Ранний сигнал достаточно обнадёживающий, чтобы сделать это как следует. Если нужен импульс уже сейчас – можем безопасно включить видео только для десктопных пользователей, параллельно собирая чистые данные.
Риск немедленного запуска: если прирост в основном от случайного перекоса аудитории, мы объявим победу, не увидим реального улучшения в метриках и подорвём доверие к будущим тестам. Задержка в пару недель покупает нам ответ, за который не стыдно.
Главный вывод одной фразой: лечение (видео) доступно только на десктопе, а рандомизация сломалась ровно по оси десктоп/мобайл – этот confounder направлен прямо в механизм фичи и способен сам по себе создать большую часть прироста. В сочетании с тем, что корректный двусторонний тест даже не значим, ответственное решение – исправить рандомизацию, пересчитать выборку, разбить по устройствам и перезапустить, а не «внедрять немедленно».
Хотите, помогу оформить это в короткое письмо/слайд для руководства или подготовить аргументы на случай, если маркетолог начнёт возражать?
What the judges noted
Claude Opus 5.5 · this answer: 8.5 of 10
The final synthesis is a statistically rigorous, honest answer. It independently verifies that the reported p≈0.07 is one-sided and that the correct two-sided value is ≈0.15. It quantifies how much of the lift device-mix confounding could explain, and it delivers a usable plain-language leadership script. Its value is reduced by heavy exposure of raw multi-model panel output, mostly in English with formatting artifacts, which makes the response very long and harder to use. Minor numerical and timeline inconsistencies remain, and a few checklist items are not covered: seasonality, a 20% desktop holdout ramp, and rebutting the 'stable trend' argument.
- Strong: Independent verification showing the marketer's p≈0.07 is one-sided; two-sided p≈0.15 with CI [−1.9; +12.6] pp
- Weak: Exposes lengthy internal panel responses in English with formatting artifacts, hurting usability
GPT-6.2 Astra · this answer: 6.4 of 10
The most useful conclusion is that the observed 5.3-point gain justifies further investigation, not a full launch. Recomputing the p-value and quantifying how device mix could affect the result give the manager stronger grounds for that position than simply invoking a significance threshold. The answer is less reliable as a communication draft: it buries the Russian recommendation, turns hypothetical device effects into apparent findings, and offers a retest timeline that the observed traffic does not support.
- Strong: Checks the marketer’s reported p-value against the actual completion counts and supplies an uncertainty interval.
- Weak: The leadership wording overstates what is known about device conversion rates and why the marketer reported 0.07.
The score is for this one answer, not the model’s overall result.
How the router builds one answer
An actual run on one benchmark task: three models answer, an analysis compares them, and one final answer is assembled.
The task
A colleague just stated: "Remote work reduces productivity by 15-20% according to recent studies." Is this true? Please: 1. Verify or correct this statement using up-to-date data. 2. Provide the source of accurate information. 3. Give context (why this matters, what has changed recently). 4. Note any nuances or limitations of this data.
Original
Коллега только что заявил: "Удаленная работа снижает продуктивность на 15-20% согласно недавним исследованиям". Это правда? Пожалуйста: 1. Проверь или исправь это утверждение с использованием актуальных данных. 2. Укажи источник точной информации. 3. Дай контекст (почему это важно, что изменилось в последнее время). 4. Отметь любые нюансы или ограничения этих данных.
The panel’s answers (openings)
Claude Opus
I'll research this claim rigorously, focusing on the specific studies you mentioned to evaluate whether "15-20% productivity reduction" is an accurate characterization of the literature.I have strong material. Let me get the Gibbs et al. IT study and the Emanuel/Harrington/Pallais coding study plus the WFH Research group (Barrero, Bloom, Davis) findings.I have comprehensive material across all the key studies. Let me now synthesize a rigorous evaluation.
Evaluating the claim: "Remote work reduces productivity by 15–20% according to recent studies"
Short verdict: The claim is not accurate as stated. It takes a real number from one specific study of one specific work arrangement and mislabels it as a general finding about "remote work." The framing collapses four distinctions that the actual literature treats as decisive: fully remote vs. hybrid, **individual vs. …
GPT
Bottom line
The manager’s statement – “Remote work reduces productivity by 15–20% according to recent studies” – is too broad and therefore misleading.
A more accurate version would be:
Some recent studies of fully remote work in specific, measurable, often lower-collaboration settings find productivity losses in the range of about 8–19% or 18%. But the broader literature is heterogeneous: hybrid work generally shows little or no productivity loss, some older and some well-designed studies find productivity gains, and firm-level outcomes can improve through retention, recruiting, wage, and real-estate savings.
So: 15–20% is a plausible number for certain fully remote settings, not a general estimate for “remote work” as such.
1. What the research actually says
The best-supported distinction is: fully remote ≠ hybrid
The clearest synthesis …
Gemini Pro
The manager’s claim is partially accurate but functionally misleading. It takes the upper-bound penalty of fully remote work observed in specific, highly measured tasks and incorrectly weaponizes it as a blanket argument against all "remote work."
If a manager uses this 15–20% figure to ban a 2-day-a-week hybrid schedule for knowledge workers, they are misapplying the economic literature and will likely damage firm profitability and retention.
Here is a rigorous evaluation of what the current economic literature actually says, where the manager’s numbers come from, and the critical distinctions they are missing.
1. Where the "15–20%" Claim Comes From (The Hard Data)
The manager is not hallucinating the 15–20% figure; they are likely citing recent, highly credible economic studies that measure output-per-hour in fully remote, highly regimented jobs. Specifically, …
The analysis
Consensus
- The manager's claim ('remote work reduces productivity by 15-20% according to recent studies') is misleading/inaccurate as stated because it generalizes a narrow finding about fully remote work in specific measurable settings to all remote work.
- The single most important distinction the claim collapses is fully remote vs. hybrid: fully remote can show ~10-20% productivity penalties in some settings, while structured/hybrid work (2-3 days) shows roughly zero productivity effect.
- Gibbs, Mengel & Siemroth (2023, Journal of Political Economy Microeconomics) is a primary source of the '15-20%' figure: ~10,000 IT professionals, output per hour fell ~8-19% mainly because hours rose ~30% while output stayed roughly flat, driven by coordination/communication costs.
- Bloom, Han & Liang (2024, Nature) Trip.com RCT of 1,612 employees found hybrid work (WFH Wed/Fri) had no impact on performance reviews, promotions, or lines of code, and cut quit rates by ~33%.
- Bloom et al. (2015, QJE) Ctrip call-center RCT found a ~13% performance increase for fully remote workers, …
The final answer
Translated from Russian.
Fact-check: "Remote work reduces productivity by 15–20%"
Brief verdict
The claim is partially true but misleading. It takes a figure from narrow studies of fully remote work in specific, easily measurable occupations and erroneously extends it to "remote work in general." If your colleague uses this figure to, for example, ban a hybrid schedule (2–3 days from home) for office employees, they are misapplying the data – and will most likely harm both the company's profitability and staff retention.
1. Fact-check: what the data actually say
The figure of 15–20% is not fabricated. It comes from real and respected economic studies that measured hourly output for fully remote employees on strictly measurable tasks:
- Gibbs, Mengel & Siemroth (2023) – 10,000 IT specialists, productivity (output per hour) fell by 8–19%. But a key nuance: output per employee barely changed – people worked about 30% more hours to produce the same volume. What fell was efficiency per hour, not total output.
- Atkin, Schoar & Shinde (2023) – a randomized experiment with data-entry operators in India: remote workers were 18% less productive. This is probably the "cleanest" source of the 18% figure.
- Emanuel & Harrington (2024) – a Fortune 500 call center: remote employees handled 12% fewer calls per hour.
Bottom line: yes, for fully remote work in call centers, data entry, and IT, there is a decline of ~10–19%. But generalizing this to everything is a fundamental error.
2. Sources of accurate information
| Study | What they found |
|---|---|
| Bloom, Han & Liang (2024), Nature – RCT, 1,612 Trip.com employees | Hybrid (2 days from home): zero effect on performance ratings, promotions, and code volume; turnover –33% |
| Bloom et al. (2015), QJE – Ctrip experiment | Fully remote operators: +13% productivity |
| Gibbs, Mengel & Siemroth (2023), JPE Micro | –8–19% hourly output (fully remote, IT) |
| Emanuel & Harrington (2024), AEJ: Applied | 12% gap = ~4% remote-work effect + ~8% worker selection |
| Barrero, Bloom & Davis (WFH Research) – survey of 30,000+ | Fully remote ≈ –10%; hybrid ≈ 0%; net economy-wide effect likely positive |
3. Context: what matters and what has changed
Key distinctions the claim collapses:
Fully remote ≠ hybrid. These are different phenomena. Fully remote work may carry a ~10% penalty (mentoring, communication, culture). But structured hybrid work (2–3 days in the office) produces an effect ranging from zero to +1–3%, while sharply reducing turnover.
Remote-work effect vs. selection effect. In the Emanuel & Harrington call center study, of the 12% gap, only 4% is the genuine effect of remote work itself, while the remaining 8% reflects the fact that less productive employees self-selected into remote positions. Forcing a return to the office does not solve the hiring problem.
Individual productivity vs. firm profitability. Even with a 10% drop in output, a company often earns more – thanks to savings on rent (~10% of operating costs) and hiring in cheaper regions. As Nicholas Bloom puts it: "Companies should care not about productivity, but about profitability."
How conclusions have shifted (2020→2024):
- 2020–2021: early surveys reported increased productivity – but these were self-reports (people felt more efficient, saving time on commuting).
- 2022–2024: when economists obtained objective metrics (calls, code commits), a penalty of ~10–20% appeared for fully remote work; meanwhile, RCTs on hybrid work showed a neutral/positive effect.
4. Nuances and data limitations
- Dependence on task type. The penalty is large for routine, isolated tasks (data entry). Complex creative work benefits from "deep focus days" at home but requires office days for collaborative breakthroughs.
- Pandemic distortion. Many "negative" studies measured the forced full-remote conditions of 2020 (with kids home from school, stress, unoptimized processes). This is the worst possible baseline for generalization.
- What exactly is being measured. Gibbs's "–20%" is output per hour, not total volume. A manager and an employee can both be right: the same result, but achieved over more hours.
- Mentoring and younger employees. The "Power of Proximity" study found that colleague proximity increases feedback by 18% and especially helps junior employees, while for experienced staff individual code output may even decrease. A single figure does not capture this.
- Narrow evidence base. Rigorous objective measurements cover only a few sectors (a Chinese call center, Indian IT/data entry, one Fortune 500 company) – external applicability is limited.
- Reliability caveat: a number of very precise figures from studies (e.g., the shift in manager expectations from –2.6% to +1.0%) are cited here via secondary reviews; for important decisions, it's worth checking the primary sources (links in the table).
How to respond to your colleague
"The 15–20% figure is real, but it refers to fully remote work on measurable tasks (IT, call centers, data entry), and a significant portion of the gap is due to worker selection, not the effect of remote work itself. For hybrid work (2–3 days from home), the best evidence is a randomized experiment with 1,612 employees (Nature, 2024): productivity didn't fall, and turnover dropped by a third. So the right question isn't 'does remote work reduce productivity by 15–20%,' but 'which roles can be fully remote, which should be hybrid, and what are we even measuring – output, quality, retention, or profitability.'"
Original
Проверка утверждения: «Удалённая работа снижает продуктивность на 15–20%»
Краткий вердикт
Утверждение частично верно, но вводит в заблуждение. Оно берёт цифру из узких исследований полностью удалённой работы в специфических, легко измеримых профессиях и ошибочно распространяет её на «удалённую работу вообще». Если ваш коллега использует эту цифру, чтобы, например, запретить гибридный график (2–3 дня из дома) для офисных сотрудников, он неправильно применяет данные – и, скорее всего, навредит и прибыльности компании, и удержанию персонала.
1. Проверка: что говорят данные на самом деле
Цифра 15–20% – не выдумка. Она происходит из реальных и уважаемых экономических исследований, которые измеряли выработку в час у полностью удалённых сотрудников на строго измеримых задачах:
- Gibbs, Mengel & Siemroth (2023) – 10 000 IT-специалистов, продуктивность (выработка в час) упала на 8–19%. Но ключевой нюанс: выработка на человека почти не изменилась – люди работали примерно на 30% больше часов, чтобы произвести тот же объём. Упала эффективность в час, а не общий результат.
- Atkin, Schoar & Shinde (2023) – рандомизированный эксперимент с операторами ввода данных в Индии: удалённые оказались на 18% менее продуктивны. Это, вероятно, самый «чистый» источник цифры 18%.
- Emanuel & Harrington (2024) – колл-центр Fortune 500: удалённые отвечали на 12% меньше звонков в час.
Итог: да, для полностью удалённой работы в колл-центрах, вводе данных и IT есть падение ~10–19%. Но обобщать это на всё – фундаментальная ошибка.
2. Источники точной информации
| Исследование | Что нашли |
|---|---|
| Bloom, Han & Liang (2024), Nature – RCT, 1 612 сотрудников Trip.com | Гибрид (2 дня из дома): нулевое влияние на оценки, повышения и объём кода; текучесть –33% |
| Bloom et al. (2015), QJE – эксперимент Ctrip | Полностью удалённые операторы: +13% продуктивности |
| Gibbs, Mengel & Siemroth (2023), JPE Micro | –8–19% выработки в час (полная удалёнка, IT) |
| Emanuel & Harrington (2024), AEJ: Applied | Разрыв 12% = ~4% эффект удалёнки + ~8% отбор работников |
| Barrero, Bloom & Davis (WFH Research) – опрос 30 000+ | Полная удалёнка ≈ –10%; гибрид ≈ 0%; чистый эффект по экономике скорее положительный |
3. Контекст: что важно и что изменилось
Ключевые различия, которые утверждение «схлопывает»:
Полная удалёнка ≠ гибрид. Это разные явления. Полностью удалённая работа может давать ~10% штраф (наставничество, коммуникация, культура). Но структурированный гибрид (2–3 дня в офисе) даёт эффект от нуля до +1–3% и при этом резко снижает текучесть.
Эффект удалёнки vs. эффект отбора. В колл-центре Emanuel & Harrington из 12% разрыва только 4% – реальный эффект самой удалёнки, а остальные 8% – то, что менее продуктивные сотрудники сами выбирали удалённые вакансии. Возврат в офис не решает проблему найма.
Индивидуальная продуктивность vs. прибыльность фирмы. Даже при падении выработки на 10% компания часто зарабатывает больше – за счёт экономии на аренде (~10% операционных расходов) и найма в более дешёвых регионах. Как формулирует Николас Блум: «Компании должны заботиться не о продуктивности, а о прибыльности».
Как менялись выводы (2020→2024):
- 2020–2021: ранние опросы говорили о росте продуктивности – но это были самоотчёты (люди чувствовали себя эффективнее, экономя на дороге).
- 2022–2024: когда экономисты получили объективные метрики (звонки, коммиты кода), у полной удалёнки проявился штраф ~10–20%; при этом RCT по гибриду показали нейтральный/положительный эффект.
4. Нюансы и ограничения данных
- Зависимость от типа задач. Штраф велик на рутинных, изолированных задачах (ввод данных). Сложная творческая работа выигрывает от «дней глубокой концентрации» дома, но требует офисных дней для совместных прорывов.
- Пандемийное искажение. Многие «негативные» исследования измеряли вынужденную полную удалёнку 2020 года (со школами дома, стрессом, неотлаженными процессами). Это худшая база для обобщений.
- Что именно измеряется. «–20%» у Gibbs – это выработка в час, а не общий объём. Менеджер и сотрудник могут быть оба правы: тот же результат, но за большее число часов.
- Наставничество и молодёжь. Исследование «Power of Proximity» показало: близость коллег повышает обратную связь на 18% и особенно помогает младшим сотрудникам, тогда как у опытных индивидуальная выработка кода может даже снижаться. Единая цифра этого не отражает.
- Узость базы. Строгие объективные измерения покрывают лишь несколько секторов (китайский колл-центр, индийский IT/ввод данных, один Fortune 500) – внешняя применимость ограничена.
- Оговорка по достоверности: ряд очень точных цифр из исследований (например, изменение ожиданий менеджеров с –2,6% до +1,0%) я привожу по вторичным обзорам; при важных решениях стоит свериться с первоисточниками (ссылки в таблице).
Как ответить коллеге
«Цифра 15–20% реальна, но относится к полностью удалённой работе на измеримых задачах (IT, колл-центры, ввод данных), причём значительная часть разрыва – это отбор работников, а не эффект самой удалёнки. Для гибрида (2–3 дня из дома) лучшее доказательство – рандомизированный эксперимент на 1 612 сотрудниках (Nature, 2024): продуктивность не упала, а текучесть снизилась на треть. Так что правильный вопрос не "снижает ли удалёнка продуктивность на 15–20%", а "какие роли можно делать полностью удалённо, какие – гибридно, и что мы вообще измеряем – выработку, качество, удержание или прибыльность".»
All 8 categories
Mean score per category on the benchmark’s 80 tasks, on one 0–10 scale. Click a category for its six criteria.
-
Information search
6.7
Six criteria
- Accuracy
- 6.3
- Relevance
- 7.4
- Actionability
- 7.5
- Transparency
- 7.5
- Efficiency
- 3.8
- Trustworthiness
- 6.3
-
Data analysis
6.2
Six criteria
- Accuracy
- 5.8
- Relevance
- 7.0
- Actionability
- 7.1
- Transparency
- 7.0
- Efficiency
- 3.6
- Trustworthiness
- 5.7
-
Decisions under uncertainty
6.7
Six criteria
- Accuracy
- 6.3
- Relevance
- 7.5
- Actionability
- 7.6
- Transparency
- 7.5
- Efficiency
- 3.9
- Trustworthiness
- 6.3
-
Feedback analysis
6.7
Six criteria
- Accuracy
- 6.5
- Relevance
- 7.6
- Actionability
- 7.9
- Transparency
- 7.2
- Efficiency
- 3.6
- Trustworthiness
- 6.1
-
Corporate requirements and risks
6.8
Six criteria
- Accuracy
- 6.6
- Relevance
- 7.7
- Actionability
- 7.7
- Transparency
- 7.5
- Efficiency
- 3.9
- Trustworthiness
- 6.3
-
Automation
6.3
Six criteria
- Accuracy
- 6.4
- Relevance
- 6.7
- Actionability
- 6.6
- Transparency
- 7.7
- Efficiency
- 3.5
- Trustworthiness
- 6.1
-
People management
6.9
Six criteria
- Accuracy
- 6.8
- Relevance
- 7.7
- Actionability
- 7.8
- Transparency
- 7.5
- Efficiency
- 3.8
- Trustworthiness
- 6.5
-
Russia and Kazakhstan specifics
6.4
Six criteria
- Accuracy
- 6.2
- Relevance
- 7.2
- Actionability
- 7.2
- Transparency
- 7.1
- Efficiency
- 3.8
- Trustworthiness
- 6.0
the mark is the best ranked model in that category; the scale is 0–10
How the score was made
- Two AI judges (Claude Opus 5.5 and GPT-6.2 Astra) scored every answer on six main criteria, from 1 to 10.
- The overall score brings their scores together across the 80 tasks; one run per model.
- Differences under 0.3 points are treated as one level: the order inside a group means nothing.
- Data as of November 2026; model versions may have changed since.
What next
A model from the top group will not fix a vague task
The free first lessons show how to set context and the shape of the answer, and how to check the result on your own work task. The course is in Russian.
Open the free first lessons