Training–serving skew caused rollbacks; Feast and MLflow with contract tests and a release gate aligned deployments

Overview

A fintech lending R&D team struggled to keep policy models from data science aligned with what engineering deployed. Feature definitions varied between training and serving, model artifacts moved by hand, and version drift led to unexpected outcomes and emergency rollbacks. Intelligex implemented a feature store with model versioning, automated contract tests in CI, and a controlled release gate tied to approvals and monitoring. Deployments aligned with approved models, rollback paths were clear, and product managers scheduled launches without last?minute surprises—while the team kept its existing data platform, orchestration, and planning tools.

Client Profile

• Industry: Fintech lending and risk decisioning
• Company size (range): Multi?product portfolio with shared underwriting and servicing platforms
• Stage: Established data pipelines and ML experimentation; handoffs between data science and engineering were manual
• Department owner: Product Management & R&D
• Other stakeholders: Data Science, Risk/Compliance, Platform Engineering, Data Engineering, QA, DevOps, Customer Support, Legal/Privacy

The Challenge

Policy models determined pricing, eligibility, and credit limits. Data science teams iterated in notebooks and jobs that produced training datasets and candidate models. Engineering owned online serving and business rules, integrating models into services that enforced policy. Feature definitions lived across notebooks, batch jobs, and service code. Encoders and transformations were re?implemented in different places. A change to a seemingly minor feature or threshold in research did not always reach serving, and online feature pipelines occasionally diverged from training logic. The result was unexpected shifts in decisions, friction during audits, and reactive hotfixes when dashboards spiked.

Versioning was inconsistent. Some teams registered models in a spreadsheet and shipped artifacts via storage buckets; others embedded models directly in services. Release notes focused on code changes, not on the policy definitions and features that drove outcomes. When a model underperformed or violated a guardrail, rollback meant redeploying an older service, then chasing which feature set went with which artifact. QA lacked a predictable way to validate that a model verified in research would behave the same in production because inputs and encoders could differ.

Risk and Compliance needed confidence that approved policy models were the ones actually being served. Reviews required traceability from feature definitions to model versions to runtime behavior. The organization wanted tighter alignment without stalling progress. The team chose to adopt a feature store for shared definitions and lineage, pair it with a model registry and contract tests, and gate releases with approvals and playbooks. For feature management, the approach aligned to Feast. For model registration and lineage, practices referenced the MLflow Model Registry.

Why It Was Happening

Root causes were fragmented feature computation and ungoverned model movement. Offline features were computed in batch with one set of transformations, while online services re?implemented those transformations with subtle differences. Ownership split: data science optimized training pipelines, engineering optimized latency and reliability, and neither had a single source of truth for feature definitions and encoders. Model artifacts moved outside of CI gates, and runtime systems accepted any model compatible with a basic interface, even if its feature contract no longer matched.

Vocabulary drift also played a role. Policy terms, cap thresholds, and eligibility signals were labeled differently by team or product line. Approval processes focused on business intent, but could not reliably bind an approval to the exact model and feature set deployed. Without a controlled registry, automated checks, and a release gate, version drift was a recurring risk.

The Solution

Intelligex implemented a governed MLOps layer centered on a feature store and model registry. Feature definitions and transformations were captured as code and served consistently to training and online inference. Models were versioned in a registry with lineage to feature views, training data snapshots, and policy notes. Contract tests validated input schemas, encoders, monotonicity and policy invariants, and basic performance expectations against curated fixtures. A controlled release gate required approvals from Product and Risk, ran smoke tests in staging, and synchronized routing rules so only approved models reached traffic. Human?in?the?loop review remained for exceptions and fairness considerations.

• Integrations: Feature store for shared definitions and online retrieval (for example, Feast); model registry for artifacts and lineage (for example, MLflow); data validation using patterns from Great Expectations; orchestration in existing tools such as Airflow or dbt; data warehouse and streams already in place; CI with GitHub Actions; planning and approvals in Jira and Confluence; optional flag?based routing for dark and canary traffic through the team’s gateway.
• Canonical feature views: Versioned feature definitions with transformation code, encoders, serving keys, freshness, and ownership. Training and serving pulled from the same definitions to prevent drift.
• Model registration and lineage: Registry entries linked artifacts to feature view versions, training datasets, policy notes, and approval records. Tags identified business domains and release readiness.
• Contract tests and policy checks: Automated tests asserted input schema, encoding parity, feature availability, and policy invariants such as monotonic relationships or cap behavior. Fixtures reflected representative cohorts.
• Release gate and approvals: CI jobs verified contracts, ran smoke tests in staging, and enforced approvals from Product and Risk before models were routed live. Exception paths captured reason codes and expirations.
• Routing and rollback: Deployment jobs updated model routing rules with dry?run previews. Rollback playbooks switched traffic to a known?good registry version tied to the same feature views. Status surfaced in dashboards and Jira badges.
• Monitoring and drift detection: Dashboards tracked feature freshness, distribution shifts, and policy guardrail metrics. Alerts signaled drift or contract failures and suggested mitigations.
• Governance and fairness: Documentation embedded with registry entries captured policy rationale and fairness considerations. Approvals aligned with internal standards and the NIST AI Risk Management Framework.
• Permissions and audit: Role?based access to feature and model repos; immutable logs of edits, approvals, tests, deployments, and rollbacks.

Implementation

• Discovery: Mapped current features used by policy models, training and serving pipelines, and release practices. Cataloged encoders, transformations, and known sources of drift. Gathered past incidents, rollback stories, and audit findings to shape guardrails.
• Design: Defined feature store schemas, ownership, and freshness rules; authored model registry metadata and tagging convention; specified contract tests for schemas, encoders, and policy invariants; designed the release gate with roles, approvals, and exception paths; aligned monitoring with guardrail metrics important to Product and Risk.
• Build: Implemented feature views and online retrieval; integrated training code with the same definitions; set up the model registry and lineage; built contract test suites and fixtures; wired CI with validation, staging smoke tests, and release gates; created routing and rollback playbooks; assembled dashboards and status badges in Jira and Confluence.
• Testing/QA: Ran in shadow mode: registered models and features while serving continued as?is. Replayed historic launches and incidents to validate contract tests and rollback mechanics. Tuned fixtures and invariants with Data Science, Engineering, and Risk. Included a human?in?the?loop board for fairness and exception reviews.
• Rollout: Enabled registry gating for selected policies first, with dark runs and limited canary exposure. Expanded across policy areas as teams grew comfortable with definitions, tests, and routing. Kept legacy paths as a controlled fallback during early cycles.
• Training/hand?off: Delivered short sessions for Product, Data Science, Engineering, QA, and Risk on feature views, registry use, contract tests, and release playbooks. Updated SOPs for model promotion, rollback, and monitoring. Transferred ownership of templates, tests, and gates to Product Ops and ML platform owners under change control.
• Human?in?the?loop review: Established a standing review for exceptions, fairness considerations, and policy updates that could not be fully automated. Decisions and rationale were captured with the registry entry.

Results

Deployments tracked to approved models and features rather than to informal handoffs. Feature definitions served both training and production, so inputs matched expectations. Contract tests caught encoding mismatches and policy invariant violations before promotion. When a model needed to be pulled back, routing switched to a known?good version tied to the same feature view, and the rollback path was clear and auditable.

Planning stabilized. Product managers saw registry status and gate outcomes in Jira, understood when a model was ready for canary and general release, and scheduled communications without hedging. Risk and Compliance reviewed a consistent package of lineage, tests, and monitoring plans, and post?release questions pointed to the same artifacts. The team kept its data platform, orchestration tools, and services; the difference was a governed layer that bound policy intent to deployed behavior.

What Changed for the Team

• Before: Features were re?implemented in training and serving. After: Feature views defined transformations once and served both paths.
• Before: Models moved by hand with unclear lineage. After: Registry entries linked artifacts to feature versions, datasets, and approvals.
• Before: QA relied on manual spot checks. After: Contract tests validated schemas, encoders, and policy invariants in CI.
• Before: Rollbacks meant redeploying services and hoping inputs matched. After: Routing switched to a known?good model tied to the same features with a documented playbook.
• Before: Launch timing slipped due to last?minute surprises. After: Release gates and dashboards made readiness and risk visible to PMs.
• Before: Compliance reviews reconstructed context. After: Lineage, approvals, and guardrail monitoring accompanied every promotion.

Key Takeaways

• Make features first?class; a feature store eliminates training?serving skew and simplifies audits.
• Treat models as governed artifacts; registration and lineage keep approvals and deployments aligned.
• Enforce contracts; schema and policy invariant tests catch issues earlier than performance metrics alone.
• Gate releases; approvals and staged routing reduce last?minute surprises and clarify accountability.
• Plan for rollback; documented routing and pairing of models to feature versions make reversions routine.
• Integrate, don’t replace; layer governance on existing data, CI, and planning tools rather than replatforming.

FAQ

What tools did this integrate with? The implementation used a feature store for shared definitions and online retrieval (for example, Feast), a model registry for artifacts and lineage (for example, MLflow), CI with GitHub Actions, and existing orchestration such as Airflow or dbt. Approvals and status lived in Jira and Confluence, and routing integrated with the team’s gateway or flag system for dark and canary traffic.

How did you handle quality control and governance? Feature views and model entries were versioned with owners and documentation. Contract tests asserted schema and encoder parity, feature availability, and policy invariants. Release gates required Product and Risk approvals, ran staging smoke tests, and logged all outcomes. Fairness and exception reviews remained human?in?the?loop, and governance aligned with the NIST AI Risk Management Framework.

How did you roll this out without disruption? The system ran in shadow mode while serving continued unchanged. Models and features registered in the background, contract tests ran in CI, and dark runs exercised the serving path. After teams tuned tests and playbooks, release gates were enabled for selected policies with canary exposure. Legacy paths remained a controlled fallback until confidence grew.

How were features and models versioned and linked? Feature views captured transformation code, encoders, and serving keys with explicit versions. Model registry entries referenced the feature view versions and training data snapshots they were built on. Approvals and notes were attached to the same entry, so promotion, monitoring, and rollback all pointed to a single source of truth.

How did rollback work in practice? Routing rules referenced registry versions. If a model needed to be reverted, the playbook switched traffic to a known?good version that shared the same feature views, preventing input mismatches. The change was logged with rationale and linked to monitoring and incident records in Jira.

How did you address data and performance drift? Dashboards tracked feature freshness, distribution shifts, and guardrail metrics. Alerts flagged drift beyond agreed bands and suggested mitigations, such as refreshing training data or narrowing exposure. Contract tests and policy checks prevented drift from silently breaking invariants during promotion.