Migrating Five Python Services: What It Actually Looks Like

A ticket arrived from the Security/Infrastructure Team. The title was vague — "update Python because it's already EOL and there are security vulnerabilities " but the body was more specific: a version with at least a year until EOL, plus a couple of major dependencies. In a corporation, such tickets are routine: it's work for a dedicated team, finding things everyone else has been ignoring for years. The wording is almost always the same: the service is outdated, Python is already end of life, update to a version with current security patches. Along with it come library versions, container versions, and whatever else.

In the dashboard, this card looks no different from the rest. But it's not a feature, and treating it as one is a mistake.

A typical feature has at least some predictable scope or at least a vision of what needs to be done: you can outline it and put a number in the card. Migration doesn't work that way — it's more honest to treat it as a research project. Until you dig into it, you can't realistically estimate the volume. Sometimes you update and everything works immediately, but that's luck, not the rule; more often there's a lot of manual work ahead. The Python ecosystem has almost no ready-made tools that would lead you through this painlessly — there are scattered tools of varying usefulness: pyupgrade, ruff, and a couple more. And there's LLM, which people pin hopes on and which I'll discuss separately below.

This particular case was interesting in scale: five services fell under the migration. Python versions from 3.8 to 3.9. Package managers of all kinds: pip with requirements.txt in some places, poetry in others, conda elsewhere. Each has its own degree of dependency on external packages and its own way of managing them.

Any migration starts with an audit. You need to understand the whole picture: which packages the project pulls in and — more importantly — which functions from those packages the code actually uses. Not "we have pandas installed," but "here are these specific pandas calls, and they're the ones at risk."

The ideal audit outcome is boring: all dependencies are alive, resolve cleanly, you fix a couple of numbers in requirements.txt or poetry.lock, run the tests — and it's green. But that's not a migration, it's routine maintenance. In practice, you need to prepare for something different: even living dependencies have accumulated changes inside — structural, mechanical, and most unpleasantly, behavioral. In the worst case, you hit a dead, abandoned library — and then another layer of work gets added: finding a replacement.

In my case, the outcome was suboptimal: pandas from v1 to v2, numpy updates, fastapi, matplotlib, plotly, and small change. Suboptimal — because pandas v1→v2 isn't a number in a lock file, it's a series of behavioral changes that static analysis doesn't catch.

The answers about changes lie in documentation, changelogs, and git repositories. The question is whether you can delegate reading to a tool. There are a few candidates.

pyupgrade

pyupgrade updates syntax between Python versions and saves time on mechanical rewrites of similar lines. For example, it rewrites old formatting:

# before
"{} = {}".format(key, value)

# after pyupgrade (--py39-plus)
f"{key} = {value}"

This is safe. But pyupgrade doesn't understand code — it brings a line to what it considers correct form without accounting for what data passes through it or what order arguments go to the updated function. Hence the first problem: the tool doesn't care about behavior, but it looks reliable.

ruff

ruff works differently. It doesn't pretend to fix things — it shows you where to look: it gathers a report across the project (which line, which object, sometimes with a link to a specific rule). It's a navigator through technical debt, not blind auto-replacement. In some cases it knows about specific migrations — for example, for Airflow 3 there are separate rules (AIR301/AIR302 for breaking changes, AIR311/AIR312 for recommended ones). But that's an exception; for most migrations, there's no such ready knowledge.

The next candidate is LLM agents - in my case, Claude Code, Codex, and experimenting with Letta Code and very lightly with Hermes Agents. I'm an engineer and I want to work with facts, not hype.

A prompt like "update the project to the latest Python version and pull in dependencies" works — but only on small projects where there's little abstraction, reuse is predictable, and debugging is fast. Once a service gets layers and isolation between them, that prompt isn't enough.

Here it's useful to introduce the concept of minimum context — a threshold below which an agent is useless, above which it starts being helpful. From my experience, it includes four things:

• relevant changelog sections (sections themselves, not a link),
• reports from static analyzers with specific line numbers,
• the affected code itself along with tests,
• failure logs — tracebacks or behavioral diffs.

Without any of these pieces, the agent falls into general advice or makes things up. Assembling the set is manual work: changelogs are read by eye, reports are generated by tools. For me, this immediately took 30–40% of the context window.

And even with full context, I couldn't get any agent to strictly stick to a plan. Changes came out incomplete, inaccurate, or marked as applied when they weren't in the code at all. In essence, the agent worked like an enhanced static analyzer with write permissions. Beyond the task, it kept trying to rename variables or change the order of operations to its taste. High level of improvisation, nondeterministic process.

The key thing is that the agent, like a linter, couldn't distinguish behavioral changes from syntactic ones.

This is the most unpleasant class, because the code keeps running. A canonical example is chained assignment with inplace, which in pandas 2.x no longer does what it did:

# pandas v1 — silently worked, column was updated
df["foo"].fillna(0, inplace=True)

# pandas v2 — FutureWarning, the original df is NOT changed
# correct now:
df.fillna({"foo": 0}, inplace=True)
# or
df["foo"] = df["foo"].fillna(0)

This is the essence of Copy-on-Write: chained writes like df["a"][1:3] = 0 go to a temporary object that behaves like a copy, and the original doesn't change. An agent will skip such code — the syntax is valid. Tests will fail silently; or worse, they won't.

The same kind of trap — changed defaults. In pandas 2.0, numeric_only in all reduction methods became False by default, and code that had silently been dropping non-numeric columns for years started failing:

df.mean()                  # v1: computed over numeric
df.mean(numeric_only=True) # v2: specify explicitly

And handling missing values: pd.NA and nullable types appeared, and checking for missing values correctly now uses a function rather than comparison to a specific sentinel:

x is None        # fragile
x is np.nan      # fragile
pd.isna(x)       # reliable: catches None, NaN, and pd.NA

On behavioral changes, you hit the ceiling of all tools at once. A linter sees syntax, not behavior. An agent sees a bit more than a linter, but it's nondeterministic and can't distinguish "works" from "runs." And behavioral changes only show up at runtime, on specific data — and aren't caught by any of the above. From that point on, tools end and manual work begins: you open the traceback (if you're lucky) or diff the output (if not), go to the changelog, find the line, and reconcile old behavior with new by hand.

Starting a migration, you end up in an awkward interim state: on the old environment you can't run the project anymore, on the new one you still can't because it doesn't compile due to syntax errors, broken imports, and so on. While you're rewriting code, it's hard to test a separate piece for workability.

From this come two practical rules. First: the migration lives in a separate git branch. Second: you keep two virtual environments, current and target, and switch between them to compare.

The main safeguard is tests, BUT a passing test asserts only equivalence to an author-chosen baseline, not correctness — it pins a fixed expected value (or a tolerance band) at a finite set of input points someone thought to exercise, and says nothing about the behavior between them. This is the oracle problem: migration "correctness" presupposes an oracle of what right output means, and tests are a weak, partial, human-authored oracle, so if the baseline encodes a bug, a faithful migration reproduces that bug faithfully — and the test goes green over it. Numerical pipelines make this acute: bumping numpy/BLAS/compiler changes float summation order and reduction associativity, so results shift by amounts that an exact assert_equal flags as a false regression while a loose atol/rtol silently masks a real one — the tolerance itself is a confession that you have a guess about an acceptable delta, not a specification. And the deepest dependency is on whoever wrote the suite: a test inherits its author's domain understanding and blind spots and cannot contain knowledge they never had, which is exactly how a chart can render "correctly" against a green suite for six months and only reveal the true behavior when a migration perturbs it. So tests are genuinely valuable — but as a change detector, not a correctness detector; the failure mode isn't trusting tests, it's reading "matches the baseline" as "is correct," and no amount of compute or budget closes that gap, because the missing piece is an oracle that was never put into the system, not more scale applied to the one you have. The contract between code and a dependency can't be touched directly, and tests remain the only golden master that will tell you how the migration went. In my experience, below 75–80% coverage, migration becomes blind work.

What's really missing is a tool that would capture that contract at runtime and produce a report; Python doesn't have one. There's semi-DIY: wrap the boundary module of a dependency in unittest.mock with autospec=True, run tests on the old version recording all calls via mock_calls, then on the new version check that autospec accepts the same signatures. Autospec catches signature changes at runtime. You could call it "contract at runtime," but assembled by hand, without a report and without the full picture.

Transitive Dependencies

There was a temptation to do the migration step by step: first Python, then pandas, then numpy. This only works in a world without transitive dependencies. I had numpy as both a direct dependency and a transitive one — via pandas:

your-service
├── numpy            (direct dependency)
└── pandas ─> numpy  (same numpy, but via pandas)

Updating pandas pulled numpy along with it, so I couldn't untie the knot of "I'll update one today, another next week" — both tasks had to be solved simultaneously. Projects at GEV were large scientific and computational ones where pandas and numpy share the load roughly equally, so this knot wasn't a corner case but central.

One of the five services lived entirely on Airflow v2. Its pipeline solved the task of forecasting and building a wind map 365 days ahead (extrapolation) by which engineers design the turbine. A detail that explains the cost of error: almost every wind turbine is unique and customized for its position — mast length, blade size and angle, generator characteristics. In other words, the pipeline's output isn't just a service response, it's numbers that engineers use to calculate an iron structure.

Meanwhile, Airflow released version v3 — and here "migration" stops being a synonym for "upgrade." Airflow 3 came out in April 2025, and the 2.x branch lasts until April 22, 2026, so the Security Team ticket for such projects is just a matter of time. The problem is that the transition from v2 to v3 for many teams turns out to be not an update but a rewrite. Workers lost direct access to the metadata database — tasks and custom operators now go through a REST API; SubDAG was removed in favor of TaskGroups and Assets; the API path changed from /api/v1 to /api/v2; old keys like execution_date were removed from context (now logical_date); BashOperator and PythonOperator moved to a separate standard provider that needs to be installed explicitly.

Each point is a behavioral change of that class. A linter with AIR301/302 rules will catch syntax, but won't tell you that the task execution architecture is now different. So such a move isn't done in-place; instead, you spin up a new environment alongside and migrate DAGs via blue/green, keeping the old one as a fallback. The same two environments as in the pandas case, just a different scale — architecture is updated, not a lock file.

On the v3 side, there are wins: native DAG versioning, data-aware scheduling via Assets, a rewritten UI — this simplifies testing and maintenance. Hence a practical paradox: technical debt isn't the most interesting work, but a forced migration usually brings a project into a healthier state than it would have reached on its own without an external ticket.

You can't merge and forget a migration. Before merge — a green full run on the target environment and, if possible, a trial run in production via shadow or canary: the golden master only catches regressions on the data you have. After merge, technical debt doesn't disappear — it restarts: a new minor version comes out, deprecation warnings flow in, transitive numpy gets updated — and someday another ticket arrives.

Breaking the cycle is helped by what usually starts such problems in the first place — CI/CD. Running a matrix of versions, alerts on deprecation warnings, regular dependency updates in small chunks instead of one big move. This way debt is paid down evenly and doesn't accumulate to fall on one person.

Migration ends not when code compiles on a new version, but when the next such ticket becomes routine maintenance again, not a research project.