Skill Issues: Optimizing Lakehouse Agents | Nebius Science Paper Club

A language model that reviews only the final response sees the agent’s account of its work. It cannot establish whether the underlying data matches that account.

Bauplan combines response evaluation with programmatic checks of data state and execution traces, which record tool calls and their results. These checks can confirm expected changes and detect modifications to protected tables or branches. The judge can then assess aspects such as the quality of an explanation.

Git for data applies branches, commits and merges to data. An agent can work on an isolated branch while verifiers inspect the resulting tables and recorded changes.

For example, a verifier can check that a pipeline produced the expected schema and rows, and that a protected branch remained unchanged. If the task requires a merge, it can check that too. The checks establish whether the recorded changes meet the task’s requirements.

In the process described at the session, one model generates a task and another generates Python code to verify the result. The team can inspect, lint and type-check that code before using it. Those checks catch some coding errors; human review must also check whether the verifier tests the intended requirement.

Tasks vary by workflow, difficulty and the detail in the request. This helps test whether agents handle both precise instructions and requests that leave more for the agent to infer.

Skills are written instructions that guide an agent through a workflow. GEPA improves those instructions using feedback from task execution.

Agents attempt tasks in isolated environments. Verifiers score the results, and GEPA uses the scores and execution traces to propose revised skills. It then evaluates new candidates and repeats the process. The model’s weights stay fixed; the instructions change.

The presentation highlighted almost 30% higher reward than manually written skills. The paper reports a 12% average relative improvement across six skills, with gains up to 28.6%. Reward measures the fraction of evaluation checks passed, so this is not a 30-percentage-point increase in task success. The experiment used one model and coding-agent configuration. Read the paper.

Jacopo Tagliabue also described fewer retries as a practical benefit, but the session did not give a separate quantified result for that claim.

Use varied tasks and traces so the optimizer learns reusable instructions rather than adding an exception for each example. Jacopo described penalties for excessive skill length and human review before releasing skills to customers.

Keep evaluation roles separate: use training tasks to revise skills, validation tasks to select candidates and decide when to stop, and a final test set to estimate performance on unseen tasks. The talk also mentioned monitoring test performance for early stopping. That makes the test set part of development, so teams should reserve another untouched set for a final assessment.

Sometimes, but the evidence presented was anecdotal. Jacopo described cases where skills optimized with a stronger model also helped a less capable model. He did not report a transfer rate or establish when transfer works.

Teams should test transferred skills on their target model. Optimizing a separate version may improve performance, but it also adds evaluation cost and maintenance work.

Yes, when those actions produce inspectable data states. Verifiers can check recorded transformations individually, which helps identify where a pipeline went wrong even if the final output looks plausible.

These checks cover only the properties they test. Commits do not establish that every action was appropriate or every requirement was met. Execution traces and human review may still be necessary.