Skip to content

fix: Require signed tokens and a private secret for intra-server communication - #6801

Open
hiyufan wants to merge 2 commits into
feast-dev:masterfrom
hiyufan:fix/intra-comm-token-verification
Open

fix: Require signed tokens and a private secret for intra-server communication#6801
hiyufan wants to merge 2 commits into
feast-dev:masterfrom
hiyufan:fix/intra-comm-token-verification

Conversation

@hiyufan

@hiyufan hiyufan commented Aug 31, 2026

Copy link
Copy Markdown

What this PR does / why we need it:

Feast's RBAC layer grants a fully trusted intra-communication identity to any request whose
preferred_username (OIDC) or service-account name (Kubernetes) claim matches
INTRA_COMMUNICATION_BASE64. That claim was read from an unverified JWT decode performed
before any signature check or IdP round-trip, and the value it was compared against is hardcoded
in Feast's own Helm chart to the constant base64("intra-server-communication").

Any caller able to reach a Feast server could therefore mint

jwt.encode({"preferred_username": "aW50cmEtc2VydmVyLWNvbW11bmljYXRpb24="}, "", algorithm="none")

and have is_auth_necessary() skip every permission check, for every project on that server. The
advisory's PoC reads all permission policies, reads across projects, and creates and deletes
entities, using a token with an empty signature and no roles.

The fix

Both halves have to change, because either one on its own leaves the identity forgeable:

  1. The token is signed. IntraCommAuthClientManager signs with HS256 using the shared
    secret instead of algorithm="none", and both token parsers verify that signature before
    reading any claim. A caller that does not hold the secret cannot produce an acceptable token,
    whatever it writes in the payload.
  2. The chart no longer ships the secret. INTRA_COMMUNICATION_BASE64 now comes from a
    Kubernetes Secret named by the new intraCommunicationSecret value, and the variable is
    omitted entirely when that is not configured. Signing alone would not help while the key is a
    constant published in this repository.
  3. An unset or empty secret disables the path instead of being compared against.
    get_intra_comm_secret() normalizes "" to None, and is_auth_necessary() now requires a
    configured secret before it will skip permission checks, so a blank username cannot match a
    blank environment variable.

The shared logic lives in a new feast/permissions/auth/intra_comm.py, so the OIDC parser, the
Kubernetes parser and the client agree on how the token is minted and verified.

⚠️ Breaking change

Deployments that relied on the chart's default must now create a Secret and set
intraCommunicationSecret.name. Every Feast release that calls another one (a feature server and
the registry server it reads from, for example) has to point at the same Secret, and the value has
to be private to the installation. Until it is configured, intra-server communication is disabled
rather than silently trusting a well-known value: that fail-closed default is deliberate, since
the alternative is the vulnerability being fixed.

Which issue(s) this PR fixes:

Fixes #6785

Advisory: GHSA-h543-6vgr-fm36

Checks

  • I've made sure the tests are passing.
  • My commits are signed off (git commit -s)
  • My PR title follows conventional commits format

Testing Strategy

  • Unit tests
  • Integration tests
  • Manual tests

Behaviour, measured before and after

Token presented Before After
alg=none, preferred_username = the chart's constant trusted, all checks skipped rejected
HS256 signed with a different secret trusted (signature never read) rejected
HS256 signed with the configured secret trusted trusted
Any intra-comm token while the secret is unset or empty comparison still attempted rejected

Unit tests

test_oidc_inter_server_comm and test_k8s_inter_server_comm previously drove the intra-server
path with a placeholder token and a monkeypatched jwt.decode. Once the signature is checked they
no longer reach that path at all, so they now build a genuinely signed token and assert on it.
New cases cover what the old ones could not:

  • test_oidc_intra_comm_rejects_forged_token / test_k8s_intra_comm_rejects_forged_token
    unsigned, and signed with another secret
  • test_intra_comm_disabled_without_secret — unset and empty
  • test_intra_comm_token_verifies_only_with_its_own_secret
  • test_intra_comm_client_token_is_accepted_by_oidc_parser — the client and the parser still
    agree end to end
$ pytest sdk/python/tests/unit/permissions/ -q
337 passed

Helm chart

$ helm template t . --set feature_store_yaml_base64=abc | grep INTRA_COMMUNICATION
(no output - intra-server communication disabled)

$ helm template t . --set feature_store_yaml_base64=abc \
    --set intraCommunicationSecret.name=feast-intra-comm | grep -A 4 INTRA_COMMUNICATION
- name: INTRA_COMMUNICATION_BASE64
  valueFrom:
    secretKeyRef:
      name: feast-intra-comm
      key: INTRA_COMMUNICATION_BASE64

$ helm lint . --set feature_store_yaml_base64=abc
1 chart(s) linted, 0 chart(s) failed

ruff check, ruff format --check and mypy pass on every changed file.

Misc

Two design points worth a reviewer's opinion:

  • The secret still travels in the claim. The parsers keep comparing the claim to the secret,
    which keeps the change small and leaves is_auth_necessary() semantics untouched, but it means
    an observer of one internal request learns the secret. Moving the claim to a fixed non-secret
    marker and relying on the signature alone would fix that; it needs a different
    is_auth_necessary() check, so I left it out of a security fix that should stay reviewable.
    Happy to add it here or follow up separately.
  • Upgrade ordering. A signed token is not accepted by an older server and vice versa, so a
    mixed-version rolling upgrade will see intra-server calls fail until both sides are updated. If
    you would rather have a release that accepts both forms first, say so and I will add a
    transition path.

@hiyufan
hiyufan requested a review from a team as a code owner August 31, 2026 15:46
@codecov-commenter

codecov-commenter commented Sep 1, 2026

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 96.07843% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 47.12%. Comparing base (5ad5592) to head (93a26b2).

Files with missing lines Patch % Lines
.../feast/permissions/auth/kubernetes_token_parser.py 89.47% 1 Missing and 1 partial ⚠️
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #6801      +/-   ##
==========================================
+ Coverage   47.08%   47.12%   +0.03%     
==========================================
  Files         419      420       +1     
  Lines       51878    51912      +34     
  Branches     7525     7529       +4     
==========================================
+ Hits        24429    24463      +34     
- Misses      25700    25701       +1     
+ Partials     1749     1748       -1     
Flag Coverage Δ
go-feature-server 30.58% <ø> (ø)
python-unit 48.43% <96.07%> (+0.03%) ⬆️
Files with missing lines Coverage Δ
sdk/python/feast/permissions/auth/intra_comm.py 100.00% <100.00%> (ø)
...python/feast/permissions/auth/oidc_token_parser.py 75.18% <100.00%> (+1.30%) ⬆️
...client/intra_comm_authentication_client_manager.py 100.00% <100.00%> (ø)
sdk/python/feast/permissions/security_manager.py 93.90% <100.00%> (ø)
.../feast/permissions/auth/kubernetes_token_parser.py 41.66% <89.47%> (+3.57%) ⬆️

... and 1 file with indirect coverage changes


Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5ad5592...93a26b2. Read the comment docs.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

…unication

The RBAC layer granted a fully trusted intra-communication identity to any
request whose `preferred_username` (OIDC) or service-account name (Kubernetes)
claim matched `INTRA_COMMUNICATION_BASE64`. That claim was read from an
unverified JWT decode performed before any signature check, and the value it
was compared against is hardcoded in Feast's own Helm chart to the constant
`base64("intra-server-communication")`, so any caller able to reach a Feast
server could mint an unsigned token and skip every permission check on every
project that server serves.

Sign the intra-server token with HS256 using the shared secret and verify that
signature before reading any claim, so a caller without the secret cannot mint
an acceptable token. Take the secret from a Kubernetes Secret named by the new
`intraCommunicationSecret` chart value and omit the variable when it is not
configured, since signing does not help while the key is a constant published
in this repository. Treat an unset or empty secret as "disabled" rather than a
value to compare against.

This is a breaking change: deployments that relied on the chart default must
create a Secret and point every Feast release that talks to another one at it.

Fixes feast-dev#6785

Signed-off-by: Chen Yufan <yufan_ai@outlook.com>
@hiyufan
hiyufan force-pushed the fix/intra-comm-token-verification branch from 93a26b2 to 488a5e4 Compare September 1, 2026 04:06
@hiyufan

hiyufan commented Sep 1, 2026

Copy link
Copy Markdown
Author

Pushed a fix for the lint failure: detect-secrets flagged the two test fixture strings I added. Marked them with # pragma: allowlist secret, following the existing convention in sdk/python/tests/component/spark/test_spark_utils.py. A full pre-commit run over the changed files is clean locally now.

The unit-test-python (3.11, macos-14) failure looks unrelated to this change. It is test_offline_server.py::test_module_level_fips_sets_env_before_pyarrow_import hitting the 60s subprocess.run timeout rather than failing an assertion.

I did check whether the new import in security_manager.py could have made that import chain heavier, since it reaches jwt. It does not: feast.offline_server already imported jwt and cryptography before this PR, and the change adds exactly one module that imports nothing new. Measured on Linux, best of 3 cold subprocess imports:

base this branch
import feast.offline_server 2.51s 2.56s
jwt in sys.modules True True
cryptography in sys.modules True True
len(sys.modules) 1533 1534

A 2.5s import against a 60s timeout, so I read that job as runner contention. Happy to dig further if it reproduces.

The Kubernetes parser reads the service-account name out of a token whose
signature already verified, and rejects it when the subject is absent, is not a
string, does not split into four segments, or names a different account. None of
those branches had a test: the existing cases stop at the signature check, so the
subject handling below it was reached only on the happy path.

Add the rejected subjects, and the matching case for the OIDC parser's
`preferred_username`, so a holder of the shared secret cannot reach the internal
identity with a token whose subject does not name it.

Signed-off-by: Chen Yufan <yufan_ai@outlook.com>
@hiyufan

hiyufan commented Sep 1, 2026

Copy link
Copy Markdown
Author

Pushed a test-only commit for the patch-coverage gap Codecov flagged in kubernetes_token_parser.py.

The uncovered lines were the subject handling in _get_intra_comm_user, below the signature check. The existing cases all stop at the signature, so those branches were only ever reached on the happy path. The new cases exercise each rejected subject — absent, non-string, wrong segment count, and a different service account — plus the matching case for the OIDC parser's preferred_username.

Behaviour of the helper with the secret configured, one line per branch:

valid  :::<secret>        -> User(<secret>)
no-sub                    -> None
null-sub                  -> None
non-string-sub            -> None
too-few-segments          -> None
too-many-segments         -> None
another-account           -> None
signed with other secret  -> None
unsigned (alg=none)       -> None
secret unset/empty        -> None

pytest sdk/python/tests/unit/permissions/ is at 344 passed.

One note on the earlier Codecov comment: it was measured against 93a26b2, which the lint fix force-push replaced, so its line numbers no longer line up. The two branches it named are the ones covered here.

I could not produce a coverage percentage locally — coverage and this environment's numpy raise ImportError: cannot load module more than once per process on Python 3.14 as soon as feast is imported, so the table above is the direct behaviour instead. CI's number should be the one to go by.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Hardcoded intra-server-communication trust value in OidcTokenParser/KubernetesTokenParser allows full unauthenticated RBAC bypass

2 participants