Experiment Methodology: How We Run Public Experiments

At SKY Labs, we believe that transparent methodology is more valuable than cherry-picked success stories. Every experiment we run follows the same 5-step process, documented publicly so anyone can learn from our process — not just our conclusions.

The 5-Step SKY Labs Methodology

Define Clear Hypothesis

Every experiment starts with a specific, testable hypothesis. We never start an experiment without knowing exactly what we're testing and why.

Format: "If we change [variable], we expect [outcome] to happen because [reason]."
Must be measurable and specific
Documented before any changes are made

Example Hypothesis: "If we move the email signup form from the sidebar to an inline form within the first 500 words, we expect a 15-25% increase in conversions because readers are more engaged at that point."

Set Success Criteria

Before running any test, we define what would count as success, what would count as failure, and what's inconclusive. These criteria are never changed mid-experiment.

Quantitative thresholds (e.g., "minimum 10% improvement")
Timeframe for evaluation
Statistical significance considerations (given small traffic)

Example Criteria: "Success = 10% increase in CTR over 14 days. Failure = decrease of 5% or more. Inconclusive = change between -5% and +10%."

Run Controlled Test

We change one variable at a time and document everything about the setup, including control conditions when possible.

One variable changed at a time
Control group where applicable
Document all external factors (seasonality, other changes)

Document Everything

Raw data, unexpected events, contradictory signals — everything gets documented. We don't cherry-pick data that supports our hypothesis.

Daily/weekly snapshots
Notes on external events
Full dataset, not just averages

Share Learnings Publicly

We publish complete results — both successes and failures — with full transparency about what worked, what didn't, and what surprised us.

Raw data when possible
Honest assessment of limitations
No fake numbers or revenue screenshots

Experiment Documentation Template

Every SKY Labs experiment follows this template to ensure consistency and transparency.

Hypothesis

Clear statement of what we're testing and expected outcome

"If we change X, we expect Y because Z."

Success Criteria

Specific metrics and thresholds for success/failure

"10% increase in conversion rate over 14 days"

Setup

How the experiment was configured and implemented

"Changed placement, kept all other elements identical"

Results

Raw data, observations, and any unexpected findings

"Day-by-day metrics with notes on external factors"

Learnings

What we learned, regardless of success/failure

"Even failed tests taught us about user behavior patterns"

Limitations

Honest assessment of what conclusions can (and can't) be drawn

"Small sample size means these results are directional, not conclusive"

Pre-Experiment Checklist

Hypothesis written and approved (no starting without it)

Success criteria defined with specific numbers

Baseline data collected (minimum 7 days before change)

Only one variable identified for change

Documentation template prepared

Experiment duration defined (minimum 14 days for traffic experiments)

Potential external factors noted (holidays, platform updates, etc.)

If you're not documenting your failures, you're not really learning. If you're not sharing them, you're not helping anyone else learn either.

— SKY Labs Principle

Related Experiments Using This Methodology

Back to SKY Labs