You send one candidate message to 24 people and another to 24. The second version gets six replies; the first gets four. It is tempting to save the second as the new template.

But what did you actually learn? If one group contained people actively looking and the other contained people settled in their roles, the message may have had little to do with the difference. If both groups were comparable, two extra replies still leave plenty of uncertainty.

For recruiters working with a small talent pool, the useful question is not simply which message won. It is whether the comparison tells you enough to change how you approach the next candidate. A small test can inform that decision, provided the team designs it before sending.

Choose a question you can answer

“Which message works better?” leaves too much open. Better for connection acceptance, any reply, a conversation about this role, or an interview? Those are different outcomes.

Start with one decision. For example: should the opening sentence explain the work the person would own, or the project they would join? Both versions should disclose the same essential role information. The test changes what leads the invitation, not whether candidates receive an honest description.

Write the hypothesis before editing: “Leading with ownership of the work may make this role easier to evaluate than leading with the project.” If the team cannot name the difference, it will struggle to explain the result.

Use one reviewed pool, not two unrelated searches

Consider a hypothetical recruiter hiring a data integration engineer for a retail business. The role involves connecting order, inventory and warehouse systems. The recruiter has reviewed 48 relevant profiles and is already connected with these people on LinkedIn. Nobody in this group has received an invitation for the role.

Using existing connections makes the example a test of the first role message. It does not mix connection acceptance with message response. If you need to send connection requests first, decide separately whether you are testing that request or the message after acceptance.

The reviewed pool includes people whose recent work centres on retail systems and others whose integration experience comes from logistics. Do not send version A to the first group and B to the second. That would compare two audiences and two messages at once.

Instead, allocate candidates randomly to A or B within each experience group, with an even split where possible. This is a practical application of NIST’s guidance on randomized block designs: account for important known differences, then randomize within those groups. NIST explains a general experimental method, not a recruiting benchmark.

Random allocation does not make a small pool perfectly balanced. It reduces deliberate selection bias. Keep the candidate’s assigned version in a tracker so nobody quietly moves a promising profile into the preferred group.

Change the opening, keep the offer

For this hypothetical role, the team prepares two openings.

Version A: lead with the project

“We’re connecting our order, inventory and warehouse systems so the teams can work from consistent operational data.”

Version B: lead with ownership

“This role would own the integrations between our order, inventory and warehouse systems, including how changes are tested and released.”

The rest of the invitation stays the same: a relevant observation about the person’s experience, the role details, working arrangement, compensation information the team can disclose, and the invitation to discuss the role.

These sentences assume the employer has confirmed the project and ownership. They are examples to adapt, not promises to paste into a real campaign.

Keep the sender, channel, follow-up wording and delays consistent. Run both versions over comparable sending periods. Do not give B a new recruiter profile, an extra follow-up and a better salary range, then credit the opening sentence.

You can compare whole messages if that is the actual decision. Just describe it accurately as a comparison of two message packages. You will not know which individual change produced the difference.

Define a useful response before replies arrive

For this example, the primary outcome is a candidate explicitly agreeing to discuss the advertised role within ten days of the first message. Ten days is an illustrative review window chosen for the example, not a recommended industry standard.

A “thanks for reaching out” is not automatically an agreement to discuss the role. Neither is a connection acceptance. Record those events separately so a friendly acknowledgement does not become a successful recruiting outcome.

For the comparison, keep counts of assigned candidates, messages actually sent, any replies and explicit interest. Use the same definitions and the same observation period for both groups. If some messages fail to send, investigate that imbalance before calling the result a clean message comparison.

For a LinkedIn sequence, connection requests and role messages have different denominators. Report each stage clearly. In this example, everyone is already connected, so the message comparison avoids that additional selection step.

Let Wandify handle the repeated sending

Wandify combines professional search, folders and outreach sequences. Use a reviewed pool, record the allocation, then set up separate campaigns for the two versions. Check that each person appears in only one test group.

The random allocation and experiment record are team-managed steps. This workflow does not assume Wandify has a native A/B randomizer, a statistical significance calculator or an automatic winner-selection feature.

The useful automation is narrower: the campaigns carry out the configured sending steps, so the recruiter does not rewrite and send each invitation individually. Keep the message versions unchanged during the comparison. The team reviews replies, answers questions and takes over the conversation.

If you need help discovering the initial pool, the Wandify AI Search guide covers that part of the workflow. Here, the search is already complete. The decision is what can be learned from the outreach that follows.

Read the result without forcing a winner

Suppose the hypothetical test ends with all 48 messages sent. A gets four replies and B gets six. Of those, three people in A and four in B agree to discuss the role.

B has more replies, but the primary outcome differs by one person. That is a useful description of what happened. It is not, by itself, reliable evidence that B will perform better in the next pool.

A small study can miss a meaningful difference or make a chance difference look persuasive. NIST’s guidance on sample size for testing proportions explains that the sample needed depends on the baseline proportion, the difference you want to detect and the chosen error thresholds. There is no universal number of candidates that makes every outreach test conclusive.

Do not keep checking the dashboard until one version pulls ahead and stop there. Microsoft’s experimentation guidance explains why repeated looks and early stopping need appropriate statistical treatment. That research concerns online product experiments; the warning is relevant here, but its test durations are not recruiting recommendations.

For a simple recruiter-run comparison, agree on the observation window and review point in advance. If the pool is too small to support a confident conclusion, record the result as inconclusive. Choose the clearer, more accurate message for practical reasons rather than claiming a measured uplift.

Keep a test record the next recruiter can use

A reusable record needs only the role and pool definition, the hypothesis, the two exact message versions, allocation method, sender and schedule, outcome definition, review window, counts and decision.

For the example, the decision might read: “B produced four interested replies and A produced three. Evidence is insufficient to declare a winner. Keep B because its description of ownership is clearer; retest only when another comparable pool exists.”

That separates editorial judgment from experimental evidence. It also prevents the team turning one small result into a rule for every role.

Before your next outreach test, use one reviewed candidate pool in Wandify, write down the single change you want to compare, and agree what result would justify changing the message.