1.2.0

[cassidyjames]

Over the past few months, we've focused on making Osprey:

1. Faster & more resilient with a new execution-plan scheduler and hardened asyncio worker upstreamed from Discord

2. Easier to run anywhere with fewer hard dependencies on specific cloud services, and fixes for rough edges

3. Better to contribute to so working on Osprey itself is both easier and safer

This is a minor release, but there are a couple of changes to be aware of when upgrading; see the upgrade notes below.

Faster & more resilient

We've updated Osprey's engine and coordinator to use an immutable execution-plan scheduler; rather than rebuilding the same graph bookkeeping for every action, Osprey now compiles it once from the ruleset and reuses it so each action does less redundant work. Thanks to Leon Shi (@cmttt) from Discord for contributing this. In testing at Discord, this new scheduler cut engine CPU time per action by about 30% and median latency by about a third at the cost of somewhat higher memory use; we expect other adopters to see similar results, potentially reducing operating costs.

We've also improved the experimental asyncio-native worker thanks to contributions by Ayu (@ayubun) from Discord. HasLabel now works in the async worker; native async UDFs now have configurable execution timeouts (so a stalled call releases worker capacity); and when a UDF is cancelled, the parent execution is cancelled and its outstanding tasks are drained cleanly.

Finally, the coordinator now retries transient Snowflake ID generation failures (like rate limits and server errors) with backoff while still failing fast on permanent errors, and we narrowed some overly broad regular expressions in the string UDFs and UI.

Easier to run anywhere

Osprey no longer requires Google Cloud credentials to start. Pub/Sub publishing is now opt-in, so a fresh docker compose up or any deployment outside of GCP just works. Thanks to Juliet Shen (@julietshen) for this improvement, which is part of ongoing work to make Osprey less tied to any one cloud provider.

Since MinIO has dropped support for their open source edition, we've moved from MinIO to RustFS (an Apache 2.0-licensed S3-compatible object store). Thank you to Emelia Smith (@ThisIsMissEm) and Mark Reitblatt (@reitblatt) for tackling this!

Better to contribute to

Emelia Smith (@ThisIsMissEm) and Ayu (@ayubun) improved how tests run locally, isolating their databases and containers so multiple test stacks can coexist and tests won't wipe out your development database.

We also smoothed out several first-run and cross-platform rough edges in this release: Adarsh Arun (@Adarsh04Arun) fixed a line-ending issue that caused shell scripts to fail on Windows; Forest Savage (@forest-savage1234) contributed a fix to the demo script to make it wait for the worker to actually be ready when demoing; and Mark Reitblatt (@reitblatt) fixed an issue where the example test data generator would peg the CPU by starting a new Kafka producer for every message.

Finally, we moved from issue templates to issue forms thanks to @vedarolap, we've improved and expanded the Osprey documentation including guidance on SML ruleset configuration, and we've updated the docs site with a refreshed look and new "Share" menu in the header.

Upgrade notes

Pub/Sub is now off by default. If you rely on Osprey publishing to Google Cloud Pub/Sub, set OSPREY_PUBSUB_ENABLED=true.

Osprey now uses protobuf 7 and gRPC 1.82. If you consume osprey_rpc alongside older versions of either, you'll need to upgrade them. Thanks to Mark Reitblatt (@reitblatt) for working through the upgrade and its many downstream dependencies.

The local Docker Compose stack uses RustFS instead of MinIO. The service name, ports, and OSPREY_MINIO_* settings are unchanged.

Thank you!

A warm welcome to the five people making their first contribution to Osprey in this release: Emelia Smith (@ThisIsMissEm), Adarsh Arun (@Adarsh04Arun), Forest Savage (@forest-savage1234), Austin Ayers (@austin-ayers), and @mah1104ahm. And thank you to everyone in the community who filed an issue, reviewed a pull request, or otherwise contributed to this release!

Full changelog

1.1.0...1.2.0

Added

• Configurable execution timeouts for native async UDFs (#452 by @ayubun)
• HasLabel support in the asyncio worker (#453 by @ayubun)
• run-tests-local.sh to run tests with the local Python interpreter against an already-running test stack (#499 by @ThisIsMissEm)

Changed

• Pub/Sub publishing is now opt-in via OSPREY_PUBSUB_ENABLED (default off), so Osprey starts without GCP credentials instead of crashing (#388 by @julietshen)
• Immutable execution-plan scheduler for faster engine and coordinator execution (#454 by @cmttt)
• Replace MinIO with RustFS, an S3-compatible object store, in Docker Compose after MinIO's images were removed from Docker Hub (#513 by @reitblatt, #514 by @ThisIsMissEm)
• osprey_rpc now declares its runtime dependencies, requiring grpcio>=1.82.1 and protobuf>=7.35,<8 (#349 by @reitblatt)
• Upgrade grpcio from 1.49.1/1.53.x to 1.82.1, and typing-extensions from 4.6.3 to 4.12.2 (required by the grpcio upgrade) (#415 by @reitblatt)
• Upgrade protobuf from 4.25.8 to 7.36.2 (#317, #349 by @reitblatt); as a downstream consequence, also upgrade
  • grpcio-tools from 1.49.1/1.53.x to 1.82.1 (to match the grpcio version, and required by grpcio-tools for protobuf 7.x support)
  • google-cloud-pubsub from 2.15.2 to 2.41.0
  • tink from 1.9.0 to 1.15.0
  • google-api-core from 2.19.2 to 2.38.0 (older versions cap protobuf below 6.0)
  • googleapis-common-protos from 1.70.0 to 1.75.3 (older versions cap protobuf below 7.0)
  • types-protobuf from 4.24.0.1 to 7.35.1.20260906 to match the protobuf major version
  • google-cloud-kms from 3.5.1 to 3.15.0 (older versions cap protobuf below 7.0), keeping the exact pin
  • grpcio-status from 1.49.1/1.53.x to 1.82.1 (to match the grpcio version), collapsing the platform-specific pins into a single exact pin since 1.82.1 ships a universal wheel
  • grpcio-health-checking and grpcio-reflection remain pinned at 1.44.0, still compatible with the new grpcio/protobuf versions
  • regenerate all gRPC bindings

Fixed

• Cancel the parent execution and drain its owned tasks when a native async UDF is cancelled (#459 by @ayubun)
• Retry transient Snowflake errors in the coordinator (#448 by @ayubun)
• Narrow overly-broad [A-z] regular expressions in the string UDFs and UI (#473 by @cassidyjames)
• Running the test suite no longer deletes the development database; tests now use their own osprey_test database (#499 by @ThisIsMissEm)
• demo.sh waits for the worker before declaring the demo ready (#466 by @forest-savage1234)
• Shell scripts failing in containers when checked out with CRLF line endings on Windows (#451 by @Adarsh04Arun)
• Excessive CPU load from the test data generator, which started a new Kafka producer for every message (#455 by @reitblatt)

---

This discussion was created from the release 1.2.0.