fix: Defer feature-freshness thread to post-fork to avoid Gunicorn deadlock by casaar97 · Pull Request #6648 · feast-dev/feast

casaar97 · 2026-07-27T13:27:05Z

Summary

feast_metrics.start_metrics_server() starts a feature-freshness thread in the Gunicorn master process, before Gunicorn forks its worker(s). That thread's first action, with no delay, is update_feature_freshness() → store.list_feature_views(), which lazily builds the registry for the first time. For registry backends needing a lazy DBAPI import (e.g. the SQL registry importing pymysql via SQLAlchemy's create_engine()), this means a thread in the master can be mid-import, holding CPython's per-module import lock, at the exact moment Gunicorn forks a worker.

POSIX fork() only duplicates the calling thread into the child process; every other thread in the parent, including this one, simply ceases to exist in the worker. If the fork lands while that thread holds a module's import lock, the lock stays permanently held in the new worker, since there is no longer any thread that can finish the import and release it. The worker's own later attempt to build its registry then deadlocks forever with no error - the process just hangs at "Waiting for application startup.". This is intermittent by nature: it only manifests if the fork lands inside that narrow timing window (see #6647 for the full write-up, including a live py-spy stack trace confirming the exact mechanism).

Resource monitoring already avoids this correctly (start_resource_monitoring=not uses_gunicorn plus the post_worker_init hook calling init_worker_monitoring()), but the freshness thread wasn't given the same treatment. This PR applies the identical, already-established pattern:

start_metrics_server() gains a start_freshness_monitoring flag, deferred exactly like start_resource_monitoring already is.
A new init_worker_freshness_monitoring(store) mirrors init_worker_monitoring().
FeastServeApplication's post_worker_init hook now also calls init_worker_freshness_monitoring(store) after the fork (via a closure capturing store, since the freshness thread needs the store reference, unlike resource monitoring).

Test plan

Added TestInitWorkerFreshnessMonitoring covering that the new function starts/doesn't start the thread based on the freshness config flag, mirroring the existing init_worker_monitoring behavior.
Added test_freshness_monitoring_deferred_same_as_resource_monitoring to TestMetricsYamlConfig, asserting start_freshness_monitoring is always passed identically to start_resource_monitoring from start_server() - a regression guard so this can't silently drift out of sync again.
sdk/python/tests/unit/test_metrics.py (89 tests) and sdk/python/tests/unit/test_feature_server.py / test_feature_server_utils.py (92 tests) pass locally.
ruff check / ruff format --check pass on all changed files.

ntkathole · 2026-07-27T13:34:42Z

@jyejare ^

codecov-commenter · 2026-07-27T13:43:41Z

⚠️ Please install the $'codecov app svg image'$ to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 70.00000% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 45.93%. Comparing base (f9923bc) to head (5376751).

Files with missing lines	Patch %	Lines
sdk/python/feast/feature_server.py	60.00%	2 Missing ⚠️
sdk/python/feast/metrics.py	80.00%	1 Missing ⚠️
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

$Impacted file tree graph$

@@           Coverage Diff           @@
##           master    #6648   +/-   ##
=======================================
  Coverage   45.93%   45.93%           
=======================================
  Files         414      414           
  Lines       49999    50006    +7     
  Branches     7146     7147    +1     
=======================================
+ Hits        22965    22972    +7     
  Misses      25423    25423           
  Partials     1611     1611

Flag	Coverage Δ
go-feature-server	`30.58% <ø> (ø)`
python-unit	`47.20% <70.00%> (+<0.01%)`	⬆️

Files with missing lines	Coverage Δ
sdk/python/feast/metrics.py	`75.51% <80.00%> (+0.51%)`	⬆️
sdk/python/feast/feature_server.py	`61.85% <60.00%> (+0.18%)`	⬆️

Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f9923bc...5376751. Read the comment docs.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

…adlock feast_metrics.start_metrics_server() starts a feature-freshness thread in the Gunicorn master process, before Gunicorn forks its worker(s). That thread's first action, with no delay, is update_feature_freshness() -> store.list_feature_views(), which lazily builds the registry for the first time. For registry backends needing a lazy DBAPI import (e.g. the SQL registry importing pymysql via SQLAlchemy's create_engine()), this means a thread in the master can be mid-import, holding CPython's per-module import lock, at the exact moment Gunicorn forks a worker. POSIX fork() only duplicates the calling thread into the child process; every other thread in the parent, including this one, simply ceases to exist in the worker. If the fork lands while that thread holds a module's import lock, the lock stays permanently held in the new worker, since there is no longer any thread that can finish the import and release it. The worker's own later attempt to build its registry then deadlocks forever with no error - the process just hangs at "Waiting for application startup." This is intermittent by nature: it only manifests if the fork lands inside that narrow timing window. Resource monitoring already avoids this correctly (start_resource_monitoring= not uses_gunicorn plus the post_worker_init hook calling init_worker_monitoring()), but the freshness thread was not given the same treatment. This applies the identical pattern: start_metrics_server gains a start_freshness_monitoring flag (deferred exactly like resource monitoring's), and FeastServeApplication's post_worker_init hook now also calls the new init_worker_freshness_monitoring(store) after the fork, instead of feast_metrics.py starting it unconditionally beforehand. Fixes feast-dev#6647 Signed-off-by: Carlos Sánchez <carlos.sancheza@cabify.com>

Copilot

Pull request overview

This PR fixes an intermittent feast serve startup deadlock under Gunicorn by deferring the feature-freshness monitoring thread until after workers are forked, mirroring the existing “post-fork” pattern already used for resource monitoring.

Changes:

Add a start_freshness_monitoring flag to feast.metrics.start_metrics_server() to prevent starting the freshness thread in the Gunicorn master process.
Introduce init_worker_freshness_monitoring(store) and invoke it from Gunicorn’s post_worker_init hook (via a functools.partial capturing the FeatureStore).
Add unit/regression tests to ensure the worker-init freshness behavior is gated by config and stays in lockstep with resource monitoring deferral.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated no comments.

File	Description
sdk/python/feast/metrics.py	Adds post-fork freshness init helper and defers freshness thread startup via a new `start_freshness_monitoring` flag.
sdk/python/feast/feature_server.py	Updates Gunicorn `post_worker_init` hook wiring to start freshness monitoring per worker after fork.
sdk/python/tests/unit/test_metrics.py	Adds tests for `init_worker_freshness_monitoring` gating and ensures deferral flags remain synchronized.
sdk/python/tests/unit/test_feature_server.py	Adds a Gunicorn hook test asserting both resource and freshness monitoring are started post-fork.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

…#166) * feat: Multi-arch publish for feast operator image Signed-off-by: ntkathole <nikhilkathole2683@gmail.com> Co-authored-by: Cursor <cursoragent@cursor.com> * feat: Allow users to have protected project on shared registry Signed-off-by: ntkathole <nikhilkathole2683@gmail.com> * fix: Defer feature-freshness thread to post-fork to avoid Gunicorn deadlock (feast-dev#6648) feast_metrics.start_metrics_server() starts a feature-freshness thread in the Gunicorn master process, before Gunicorn forks its worker(s). That thread's first action, with no delay, is update_feature_freshness() -> store.list_feature_views(), which lazily builds the registry for the first time. For registry backends needing a lazy DBAPI import (e.g. the SQL registry importing pymysql via SQLAlchemy's create_engine()), this means a thread in the master can be mid-import, holding CPython's per-module import lock, at the exact moment Gunicorn forks a worker. POSIX fork() only duplicates the calling thread into the child process; every other thread in the parent, including this one, simply ceases to exist in the worker. If the fork lands while that thread holds a module's import lock, the lock stays permanently held in the new worker, since there is no longer any thread that can finish the import and release it. The worker's own later attempt to build its registry then deadlocks forever with no error - the process just hangs at "Waiting for application startup." This is intermittent by nature: it only manifests if the fork lands inside that narrow timing window. Resource monitoring already avoids this correctly (start_resource_monitoring= not uses_gunicorn plus the post_worker_init hook calling init_worker_monitoring()), but the freshness thread was not given the same treatment. This applies the identical pattern: start_metrics_server gains a start_freshness_monitoring flag (deferred exactly like resource monitoring's), and FeastServeApplication's post_worker_init hook now also calls the new init_worker_freshness_monitoring(store) after the fork, instead of feast_metrics.py starting it unconditionally beforehand. Fixes feast-dev#6647 Signed-off-by: Carlos Sánchez <carlos.sancheza@cabify.com> Co-authored-by: Carlos Sánchez <carlos.sancheza@cabify.com> --------- Signed-off-by: ntkathole <nikhilkathole2683@gmail.com> Signed-off-by: Carlos Sánchez <carlos.sancheza@cabify.com> Co-authored-by: ntkathole <nikhilkathole2683@gmail.com> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Carlos Sánchez <i52saarc@uco.es> Co-authored-by: Carlos Sánchez <carlos.sancheza@cabify.com>

…adlock (feast-dev#6648) feast_metrics.start_metrics_server() starts a feature-freshness thread in the Gunicorn master process, before Gunicorn forks its worker(s). That thread's first action, with no delay, is update_feature_freshness() -> store.list_feature_views(), which lazily builds the registry for the first time. For registry backends needing a lazy DBAPI import (e.g. the SQL registry importing pymysql via SQLAlchemy's create_engine()), this means a thread in the master can be mid-import, holding CPython's per-module import lock, at the exact moment Gunicorn forks a worker. POSIX fork() only duplicates the calling thread into the child process; every other thread in the parent, including this one, simply ceases to exist in the worker. If the fork lands while that thread holds a module's import lock, the lock stays permanently held in the new worker, since there is no longer any thread that can finish the import and release it. The worker's own later attempt to build its registry then deadlocks forever with no error - the process just hangs at "Waiting for application startup." This is intermittent by nature: it only manifests if the fork lands inside that narrow timing window. Resource monitoring already avoids this correctly (start_resource_monitoring= not uses_gunicorn plus the post_worker_init hook calling init_worker_monitoring()), but the freshness thread was not given the same treatment. This applies the identical pattern: start_metrics_server gains a start_freshness_monitoring flag (deferred exactly like resource monitoring's), and FeastServeApplication's post_worker_init hook now also calls the new init_worker_freshness_monitoring(store) after the fork, instead of feast_metrics.py starting it unconditionally beforehand. Fixes feast-dev#6647 Signed-off-by: Carlos Sánchez <carlos.sancheza@cabify.com> Co-authored-by: Carlos Sánchez <carlos.sancheza@cabify.com>

# [0.66.0](v0.65.0...v0.66.0) (2026-08-21) ### Bug Fixes * Add connection pre-warming for DynamoDB async client ([89240fa](89240fa)), closes [#6060](#6060) * Add remote registry client extra ([#6697](#6697)) ([b8dfcb0](b8dfcb0)) * Address review feedback on FIPS cipher suite configuration ([4a35fba](4a35fba)) * Allow remote-registry first apply for new projects ([39d408d](39d408d)) * Avoid importing feast.feature_store at mcp_server import time ([ddb2e9a](ddb2e9a)) * Bump pymssql to >=2.3.6 for macOS arm64 wheel support ([181eb35](181eb35)), closes [#5636](#5636) [#5193](#5193) [#5636](#5636) * Call ApplySavedDataset RPC instead of ApplyFeatureService in RemoteRegistry.apply_saved_dataset() ([934d341](934d341)) * Catch missing dbt parser dependency in dbt CLI commands ([#6534](#6534)) ([3c2ae3c](3c2ae3c)) * Default authentication to kubernetes auth ([6a4690a](6a4690a)) * Defer feature-freshness thread to post-fork to avoid Gunicorn deadlock ([#6648](#6648)) ([104ad10](104ad10)), closes [#6647](#6647) * Do not pass undeclared feature view columns to ODFV UDFs ([#6527](#6527)) ([75b9463](75b9463)) * downgrade mcp pin to 1.29.0 and fix CI lockfiles and unit tests ([98e5bca](98e5bca)), closes [#6706](#6706) * Feast apply silently ignoring ttl updates to None or timedelta(0) ([#6709](#6709)) ([97b0f25](97b0f25)), closes [#6703](#6703) * Fix mypy TorchTensor type alias error ([#6712](#6712)) ([34de6fa](34de6fa)), closes [#5563](#5563) * Fixed data source creation form gaps ([5d0f7d6](5d0f7d6)) * Handle parameterized and complex Trino types in type map ([326554d](326554d)) * Isolate default user permissions ([e37adbf](e37adbf)) * Isolate projection join key maps ([d1c709d](d1c709d)) * Map Postgres real to FLOAT instead of DOUBLE ([62db435](62db435)) * Merge shared ODFV source projections in feature resolution ([d269946](d269946)), closes [#6621](#6621) * More exhaustive athena types ([a9aaefc](a9aaefc)) * Normalize SQL registry read_path to the psycopg3 driver like path ([#6644](#6644)) ([996c6ea](996c6ea)), closes [#6643](#6643) * **operator:** add spec.services.onlineStore.disabled to opt out of the online store ([d81d4e3](d81d4e3)), closes [#6586](#6586) * Preinstall DuckDB delta extension for tests ([fd4d49d](fd4d49d)), closes [#6743](#6743) * Preserve event-time ordering within Redis online_write_batch ([40fb788](40fb788)), closes [#5163](#5163) * Prevent mutation of cached feature resolution results ([ea17419](ea17419)) * Remote feastRef FeatureStore fails first apply for a new feastProject ([9affee5](9affee5)) * Remove inert subjectaccessreviews and reorganize RBAC rules ([f771ea4](f771ea4)) * Report single-feature-view spark_application materialization success ([a9219d9](a9219d9)), closes [#6673](#6673) * Reset the global security manager after the permissions fixture ([7667215](7667215)) * Resolve kserve with pip --dry-run instead of installing it ([01da132](01da132)), closes [#6732](#6732) * Resolve write_to_offline_store feature view with a single registry lookup ([a42dc85](a42dc85)), closes [#4235](#4235) * Return False from __eq__ on cross-type comparison ([#6637](#6637)) ([0f149a9](0f149a9)), closes [#6636](#6636) * Reuse IdP-issued client tokens until near expiry ([602d752](602d752)) * Reuse the OIDC JWKS client across requests ([#6683](#6683)) ([a1e6fc2](a1e6fc2)) * Separate CronJob and feature-server ServiceAccounts ([398f643](398f643)) * Serialize UnixTimestamp proto values as raw int64 in remote online store transport ([1e7134f](1e7134f)) * Set FIPS cipher suites before pyarrow.flight import to prevent crash on IBM Power ([979b82a](979b82a)) * Support Entra ID (Azure AD) token claims in OIDC auth ([#6631](#6631)) ([f843c63](f843c63)) * UDF/ODFV source rehydrate (+ Postgres / online cache) ([#6655](#6655)) ([5fd7af7](5fd7af7)) * Updated projects-list.json in order to display newly added projects ([#6657](#6657)) ([3a6a103](3a6a103)) * Use correct image name in multi-arch imagetools push step ([faf85e0](faf85e0)) * Use join keys instead of entity names in ODFV materialization ([#6645](#6645)) ([abffebc](abffebc)), closes [#5965](#5965) * use matching proto class per feature view list in SqliteOnlineStore.plan() ([adb8c1c](adb8c1c)), closes [#6658](#6658) * Widen Athena integer type mapping for unsigned ints ([3425783](3425783)) ### Features * Add ConnectionRef to DataSource for pluggable external credential resolution ([28bde01](28bde01)) * Add Feature Service Create in UI ([0399380](0399380)) * Add hybrid to ValidOfflineStoreDBStorePersistenceTypes for HybridOfflineStore support ([#6707](#6707)) ([310ab51](310ab51)), closes [#6701](#6701) * Add MLflow integration support to Feast operator ([#6611](#6611)) ([52999f1](52999f1)) * Add opt-in filter_by_created_timestamp cutoff to get_historical_features ([#6617](#6617)) ([79b33ce](79b33ce)), closes [#6615](#6615) * Add optional OIDC token audience and issuer verification ([#6670](#6670)) ([ef307c6](ef307c6)) * Add packaged feature repository support to Feast Operator ([8112b1e](8112b1e)), closes [#6598](#6598) * add plan() support to DynamoDBOnlineStore ([51ce982](51ce982)), closes [#6658](#6658) [#6659](#6659) * Added optional namespace/colleciton to datasets ([165fcf2](165fcf2)) * Added SQL registry schema_mode and registry create command ([#6704](#6704)) ([037c4cd](037c4cd)) * Allow users to have protected project on shared registry ([f9923bc](f9923bc)) * Apply Intermediate TLS defaults on API fallback and handle transient errors ([#6587](#6587)) ([43ae993](43ae993)) * **cli:** Updated feast init demo by adding rag template ([#5946](#5946)) ([c8628eb](c8628eb)), closes [#5264](#5264) * Expose the OIDC JWKS tunables through the operator ([#6690](#6690)) ([fef4e78](fef4e78)), closes [#6683](#6683) * Making feast vector store with open ai search api compatible ([#6121](#6121)) ([54da19a](54da19a)) * Multi-arch publish for feast operator image ([b221036](b221036)) * OpenLineage lineage enhancements - full object coverage, richer UI, and API-level sync ([#6719](#6719)) ([120a868](120a868)) * **operator:** Add spec.services.initImage for init container image override ([#6598](#6598)) ([ca355cb](ca355cb)) * Pass optional OIDC audience and issuer through the operator ([#6677](#6677)) ([a13ed7b](a13ed7b)), closes [#6670](#6670) * **server:** Remote Materialization ([#6649](#6649)) ([b7ae488](b7ae488)), closes [#4526](#4526) * Support Lineage configs via operator ([bf1e54a](bf1e54a)) * Updated datasets UI to support grouping ([7ae64ec](7ae64ec))

$@casaar97$

casaar97 requested a review from a team as a code owner July 27, 2026 13:27

$@casaar97$

casaar97 force-pushed the fix/freshness-thread-postfork-deadlock branch from bb3deed to 597a845 Compare July 27, 2026 13:30

$@casaar97$

casaar97 force-pushed the fix/freshness-thread-postfork-deadlock branch from 597a845 to 5376751 Compare July 27, 2026 13:48

$@franciscojavierarceo$

franciscojavierarceo requested a review from Copilot July 27, 2026 14:03

Copilot started reviewing on behalf of franciscojavierarceo July 27, 2026 14:03 View session

Copilot AI reviewed Jul 27, 2026

View reviewed changes

$jyejare$

jyejare approved these changes Jul 27, 2026

View reviewed changes

$@jyejare$

jyejare merged commit 104ad10 into feast-dev:master Jul 27, 2026
27 checks passed

Navigation Menu

Uh oh!

fix: Defer feature-freshness thread to post-fork to avoid Gunicorn deadlock - #6648

fix: Defer feature-freshness thread to post-fork to avoid Gunicorn deadlock#6648
jyejare merged 1 commit into
feast-dev:masterfrom
casaar97:fix/freshness-thread-postfork-deadlock

$@casaar97$ casaar97 commented Jul 27, 2026

Uh oh!

ntkathole commented Jul 27, 2026

Uh oh!

codecov-commenter commented Jul 27, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Uh oh!

Conversation

casaar97 commented Jul 27, 2026

Summary

Test plan

Uh oh!

ntkathole commented Jul 27, 2026

Uh oh!

codecov-commenter commented Jul 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

$@casaar97$ casaar97 commented Jul 27, 2026

codecov-commenter commented Jul 27, 2026 •

edited

Loading