Microsoft SkillOpt: A Practical Guide to Self-Improving Agent Skills

A reusable skill can preserve a good workflow. It can also preserve a bad assumption. Microsoft’s SkillOpt explores how to improve those instructions using scored examples and controlled revisions. The target language model stays fixed while the skill file changes. Microsoft Research published its overview on June 30, 2026. Microsoft Research overview.

For a creator business, the useful question is practical: could a better instruction file reduce recurring mistakes in article checks, spreadsheet cleanup or website maintenance? This guide proposes a small experiment. It does not report a CreatorStack implementation or measured improvement.

What changed, and what is available?

The official project exposes two paths. The research engine optimizes skills against defined tasks and evaluation splits. SkillOpt-Sleep reviews supported coding-agent sessions and stages proposed memory or skill changes. Sleep remains a preview. Its documentation describes a review-and-adopt step, so “self-improving” should not be read as permission for uncontrolled changes. Project repository, Sleep documentation.

The changelog dates version 0.2.0, which introduced Sleep, to July 2, 2026. Current installation documentation warns that some features on the main branch are newer than that package release. Record the version or commit used for an experiment instead of assuming every example works with every installation. Changelog, installation guide.

What you need before starting

The documented research setup requires Python 3.10 or later and a configured model backend. Sleep also provides a mock backend without credentials. A source checkout is needed for repository assets such as benchmark configuration and data-preparation scripts that are not included in the package wheel. Installation requirements.

For your own trial, prepare a saved baseline skill, representative examples, an evaluator and a spending limit. Keep the model and available tools stable during comparison. Otherwise, a better result may come from the environment changing rather than the instruction edit.

There is no universal experiment price. Your budget must cover the model calls, repetitions and infrastructure you choose. Our recommendation is to set a small trial ceiling first and stop if the evidence does not justify further runs.

1. Choose a task you can actually score

Start with something narrow, such as checking whether an article’s product claims match supplied source excerpts. “Make the agent better at business” is too broad to evaluate consistently.

Write the expected behavior before asking an agent to improve its instructions. For an article checker, useful outcomes might include identifying an unsupported price, distinguishing annual billing from monthly billing and preserving a correct claim. Include examples where no correction is needed; an agent that flags everything is not a useful editor.

Our AI output evaluation guide explains how to turn these expectations into a repeatable rubric.

A small experiment with a meaningful failure

Imagine a skill that checks subscription prices in draft articles. Its recurring error is accepting a monthly equivalent without mentioning annual prepayment. This is a proposed experiment, not a SkillOpt run performed by CreatorStack.

Build cases that separate the actual problem from superficial cues. One source has a true monthly price; another has an annual equivalent; a third has both; a fourth contains a promotional first month. Include a correct draft so the checker cannot succeed by objecting to everything.

A baseline instruction might say “Verify all prices.” A candidate might add: “For every price, preserve currency, billing interval, required upfront payment and promotion period; flag a missing condition that changes the recommendation.” That is a reusable rule. “Whenever you see $16, replace it with $24” is a shortcut tied to one example and should be rejected.

Score the resulting checks at the claim level. Did the agent catch the omitted annual commitment? Did it leave a correct monthly price alone? Did it invent a replacement price when the source did not provide one? Keep the last failure as a blocking condition even if the total number of detected problems rises.

The intermediate artifact you want is a list of proposed instruction edits with the traces that motivated them. The final artifact is a skill you can read and explain. Neither should contain the expected answers from the untouched evaluation cases.

2. Separate learning examples from final checks

Use one set of cases to expose recurring errors, another to select candidate improvements, and an untouched set for the final comparison. Avoid placing nearly identical versions of the same case in every group.

The official first-experiment guide illustrates this separation with SearchQA and an evaluation step for the saved skill artifact. It also explains that the supplied ID manifest needs data materialization; downloading the repository alone does not create a runnable benchmark. First experiment.

For a small creator project, a modest collection of carefully reviewed cases is a sensible pilot. It is not enough to justify broad accuracy claims. Save the actual outputs so you can inspect why a score moved.

3. Keep edits bounded and the gate enabled

SkillOpt runs tasks, examines their execution traces and proposes instruction edits. The documented loop merges related proposals, limits the edits applied and evaluates the resulting candidate. With validation gating enabled, the candidate must improve the configured selection score; disabling the gate force-accepts candidates despite still recording validation. Training-loop documentation.

The epoch-level slow update has a separate control, optimizer.slow_update_gate_with_selection: its guidance can be applied without that selection check. Enabling evaluation.use_gate alone therefore does not establish that every update is gated; inspect both settings.

Our recommendation is to change one task skill at a time. If a proposed rewrite also changes unrelated publishing rules, separate that change for review. A stronger average score should not conceal a failure that matters to your business, such as replacing correct prices with guesses.

4. Use Sleep only after reviewing its data boundary

Sleep can derive tasks from past coding sessions and replay them through a configured backend. Its documentation warns that real backends may send transcript excerpts and derived tasks to a provider. Local harvesting does not mean all subsequent processing stays local, and redaction is not guaranteed to remove every secret. Sleep data boundary.

Use a disposable project or deliberately prepared examples for a first trial. Review the extracted material before involving client work. Keep API credentials, personal details and unpublished customer information out of the sample. This is a practical prerequisite when the source material is your own work history.

5. Read the final skill before adopting it

The research workflow produces a reusable best_skill.md artifact. That file is inspectable text rather than a newly trained model checkpoint. Project overview.

Compare it with the baseline. Look for brittle shortcuts, references to particular test answers, unnecessary length and changed permissions. Then run the untouched cases and a small real task. Record costs, failures and the version you adopted; keep a rollback copy.

A skill that succeeds in one harness may behave differently when tools, models or permissions change. Recheck the result after a material environment change.

Diagnose a score increase before celebrating it

Compare the changed cases, not only the aggregate score. An improvement can come from a better general rule, a more permissive evaluator or accidental exposure to the answer. Freeze the evaluator while comparing candidates, and separately review any proposed evaluator change.

If the new skill becomes much longer, inspect what each addition contributes. A paragraph that fixes one case by repeating its answer may reduce portability. If formerly correct drafts are now overflagged, inspect whether the rule has become too broad. If the same candidate swings between pass and fail, repeat representative cases before interpreting the difference as a stable gain.

Budget for the full optimization loop: target-agent executions, proposal generation, validation and the final untouched check. Save the baseline, candidate, run configuration and outputs together. You need enough evidence to decide whether to adopt the change and enough history to reverse it.

When a simpler approach is enough

If you have only one recurring mistake, first write a clear instruction and test it manually. Our Codex workflow guide is a lighter starting point. Skill optimization becomes more useful when you have repeated tasks, reliable scoring and enough examples to distinguish a real improvement from a lucky answer.

Treat SkillOpt as an experiment you can inspect. Better evidence should determine whether the skill changes—not the confidence of the agent proposing the rewrite.