Shut down the replaced MeterProvider in get_otel_logger()

[frank-ye]

Re-initialising the OTel metrics pipeline inside a long-lived process leaves the previous
pipeline running and exporting.

#64703 resets the SDK's Once() guard so that set_meter_provider() genuinely succeeds on a
second call. That fixed metrics being silently dropped in forked children (#64690). What it does
not do is shut down the provider it replaces.

The outgoing provider owns a PeriodicExportingMetricReader, and that reader's constructor has
already started a daemon thread which exports on its own interval. The provider is built with
shutdown_on_exit=False, so nothing ever reaps it. The abandoned reader therefore keeps
exporting, but its instruments are no longer recorded to, so its cumulative totals freeze. It
republishes those frozen totals indefinitely, under the same resource, with a different
start_time_unix_nano.

A consumer then receives two conflicting cumulative streams for one series: one climbing
correctly, one stuck at whatever the value was when the pipeline was replaced. Backends that must
collapse them to a single series alternate between the two, which reads as a large spurious rate
spike; backends that reject out-of-order samples can drop the counter entirely.

This is invisible under delta temporality, because the abandoned reader honestly reports "nothing
new" forever. It only surfaces once counters are reported as cumulative.

Reproduction

Against opentelemetry-sdk==1.42.1, mimicking get_otel_logger(): build a provider, record 48 to
a counter, reset the Once() guard, install a second provider, record 1000 to the counter.

Before:

A-first-init     batches=  9  last_value=   48  start_time=1787084407400194000
B-second-init    batches=  5  last_value= 1000  start_time=1787084408103536000

A exports occurring AFTER B came online: 5
  same metric name: True   different start_time: True
live reader threads: ['OtelPeriodicExportingMetricReader', 'OtelPeriodicExportingMetricReader']

After:

A exports occurring AFTER B came online: 0
live reader threads: ['OtelPeriodicExportingMetricReader']

The new test in test_otel_logger.py asserts the process does not accumulate exporter threads
across a re-initialisation. It fails without this change (leaked_readers=1) and passes with it.

Fix

Shut down the outgoing provider before installing the replacement. MeterProvider.shutdown()
force-flushes first, so the final datapoints are not lost, which keeps the guarantee #64703 was
written to restore.

---

• Read the Pull Request Guidelines
• Tests added (shared/observability/tests/observability/metrics/test_otel_logger.py)
• Newsfragment added

Was generative AI tooling used to co-author this PR?

• Yes (please specify the tool below)

Claude Code (Opus)

[boring-cyborg]

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide
Here are some useful points:

• Pay attention to the quality of your code (ruff, mypy and type annotations). Our prek-hooks will help you with that.
• In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example Dag that shows how users should use it.
• Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
• Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
• Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
• Be sure to read the Airflow Coding style.
• Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
  Apache Airflow is a community-driven project and together we are making it better 🚀.
  In case of doubts contact the developers at:
  Mailing List: dev@airflow.apache.org
  Slack: https://s.apache.org/airflow-slack