feat: Refactor materialization engine - #5354
Conversation
Signed-off-by: HaoXuAI <sduxuhao@gmail.com>
Signed-off-by: HaoXuAI <sduxuhao@gmail.com>
|
@HaoXuAI will be helpful for review if you can add details in PR description about |
yeah for sure, it's still WIP and I'll update the PR once it's ready |
Signed-off-by: HaoXuAI <sduxuhao@gmail.com>
Signed-off-by: HaoXuAI <sduxuhao@gmail.com>
Signed-off-by: HaoXuAI <sduxuhao@gmail.com>
Signed-off-by: HaoXuAI <sduxuhao@gmail.com>
Signed-off-by: HaoXuAI <sduxuhao@gmail.com>
Signed-off-by: HaoXuAI <sduxuhao@gmail.com>
| tags: Optional[Dict[str, str]] = None, | ||
| online: bool = False, | ||
| offline: bool = True, | ||
| offline: bool = False, |
There was a problem hiding this comment.
To be backward compatible for testing
| A materialization engine abstracts over specific technologies or frameworks that are used to materialize data. It allows users to use a pure local serialized approach (which is the default LocalComputeEngine), or delegates the materialization to seperate components (e.g. AWS Lambda, as implemented by the the LambdaComputeEngine). | ||
|
|
||
| If the built-in engines are not sufficient, you can create your own custom materialization engine. Please see [this guide](../../how-to-guides/customizing-feast/creating-a-custom-materialization-engine.md) for more details. | ||
| If the built-in engines are not sufficient, you can create your own custom materialization engine. Please see [this guide](../../how-to-guides/customizing-feast/creating-a-custom-compute-engine.md) for more details. |
There was a problem hiding this comment.
There are other couple of reference to creating-a-custom-materialization-engine.md which needs to change as well
https://github.com/search?q=repo%3Afeast-dev%2Ffeast%20creating-a-custom-materialization-engine.md&type=code
There was a problem hiding this comment.
For sure, let me update in the next PR
franciscojavierarceo
left a comment
There was a problem hiding this comment.
lgtm
let's update docs in follow up 👍
What this PR does / why we need it:
Move old MaterializationEngine (Spark, Snowflake, Lambda, and Kubernetes engines) to ComputeEngine.
With this change all compute engines are unified, and can be used in the feature_store API. It also enables all compute engines with
aggregationandtransformationconfigs.And the materialization API is backward compatible.
Which issue(s) this PR fixes:
Misc