Fix Cloud Composer Upgrade Dependency Loops and google-cloud-aiplatform 'evaluation' Warnings by Reproducing Conflicts Locally

Stuck on a Cloud Composer upgrade with endless pip warnings? Mirror the image locally to expose real dependency conflicts and finish the Airflow 2.10.5 upgrade.

Upgrading a Cloud Composer environment can surface a wall of dependency warnings that obscure the real blockers. A typical case is an upgrade from composer-2.13.9-airflow-2.9.3 to composer-2.13.9-airflow-2.10.5 where pip emits repeated messages around google-cloud-aiplatform and an extra named evaluation. The resolver appears to spin without making progress, and it is not obvious which packages actually collide.

Problem in context

During the upgrade, the dependency resolver repeatedly prints lines like the following:

WARNING: google-cloud-aiplatform 0.7.1 does not provide the extra 'evaluation'
WARNING: google-cloud-aiplatform 0.7.0 does not provide the extra 'evaluation'
WARNING: google-cloud-aiplatform 0.6.0 does not provide the extra 'evaluation'
WARNING: google-cloud-aiplatform 0.5.1 does not provide the extra 'evaluation'
WARNING: google-cloud-aiplatform 0.5.0 does not provide the extra 'evaluation'
WARNING: google-cloud-aiplatform 0.4.0 does not provide the extra 'evaluation'
WARNING: google-cloud-aiplatform 0.3.1 does not provide the extra 'evaluation'

The output suggests a loop in the resolver, but it doesn’t tell you which of your project dependencies are incompatible with the Composer image’s package set.

What’s really happening

The warning spam makes it easy to miss the actual conflict set. In practice, the underlying problem can be specific package pins in your own requirements that don’t align with the versions bundled with the target Composer image. The Composer UI may not show the precise conflict details. Reproducing the install locally with the exact same Python and the image’s package list is what reveals the concrete incompatibilities.

Reproduce locally and surface the true conflicts

First, create a clean virtual environment that matches the Python used by your Composer image. In the case described, the image used Python 3.11.8. Verify your interpreter version before you proceed.

python3.11 -m venv .env_sync
source .env_sync/bin/activate
python --version  # Expecting 3.11.8
pip install --upgrade pip

Next, install the packages that correspond to your exact Composer image. The official version documentation lists both the Python version and the package set for a given Composer build. Prepare a requirements file from that list. A minimal slice that mattered for the conflicts looked like this:

apache-airflow==2.10.5+composer
google-cloud-aiplatform==1.108.0
google-cloud-bigquery==3.35.1
python-dateutil==2.9.0.post0
requests==2.32.4
backoff==2.2.1

If some small packages fail to install locally due to platform or Python version issues that are irrelevant to your project, you can exclude them temporarily to get through the analysis. In the described case, the mysql client package was excluded locally.

Now install the Composer package set together with your own project requirements and watch the logs carefully:

pip install -r composer-locked.txt -r local-requirements.txt

This step exposes conflicts that the Composer dependency resolver may have hidden. For example, the local resolver surfaced the following:

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
mypackage 1.74.0 requires google-cloud-bigquery~=3.16.0, but you have google-cloud-bigquery 3.35.1 which is incompatible.
mypackage 1.74.0 requires python-dateutil~=2.8.2, but you have python-dateutil 2.9.0.post0 which is incompatible.
mypackage 1.74.0 requires requests~=2.31.0, but you have requests 2.32.4 which is incompatible.
analytics-python 1.4.post1 requires backoff==1.10.0, but you have backoff 2.2.1 which is incompatible.

These explicit messages are the actionable part. Adjusting local requirement pins in line with those recommendations resolved the actual blockers. After that, the Composer environment upgraded cleanly to composer-2.14.0-airflow-2.10.5.

Why this approach works

A Composer image bundles a large, tightly-versioned ecosystem around Airflow and its providers. When your project pins clash with those versions, the managed resolver output can be noisy and not point at the real cause. Mirroring the environment locally with the same Python version and the image’s package set forces pip to emit direct, helpful conflict diagnostics. If you encounter packages that are known to be problematic on your workstation but not relevant to your project, excluding them locally helps you focus on genuine incompatibilities. Guidance available on Stack Overflow can also help identify package versions that won’t work locally.

Practical outcome

Instead of trying to silence the evaluation-related warnings, focus on reproducing the complete install locally and aligning your own dependencies with the Composer image. In the case above, reconciling versions of google-cloud-bigquery, python-dateutil, requests and backoff removed the blockers and made the upgrade straightforward.

Takeaways

Treat resolver warning loops as a symptom, not the root cause. Match the Python runtime, install the exact Composer package set together with your project requirements, and rely on the local pip resolver to tell you what truly conflicts. Resolve those concrete version mismatches first; the upgrade should then proceed without surprises.