Manual Foreign Object Checks Replaced with Integrated Vision and MES/QMS Workflow for Reliable HACCP Compliance in Food Manufacturing

Overview

A snack food line relied on manual visual checks to catch foreign objects, which varied by shift and often surfaced only after packaging. Intelligex integrated an inline vision system with the existing Manufacturing Execution System (MES) and Quality Management System (QMS) so anomalies triggered an automatic hold, routed a documented inspection task with images, and captured electronic approvals. Contamination risks were handled immediately at the source with clear evidence, fewer ad hoc stoppages, and a consistent path from detection to disposition—without replacing controls or reworking the line.

Client Profile

• Industry: Snack food manufacturing and packaging
• Company size (range): Multi-line plant with high-mix SKUs
• Stage: HACCP program in place; label/metal checks present; foreign material detection largely manual
• Department owner: Operations & Manufacturing
• Other stakeholders: FSQA/Quality, Packaging, Sanitation, Maintenance, Engineering, EHS, IT/OT Security, Supply Chain

The Challenge

Foreign object control depended on operator vigilance at transfer points and before bagging. Lighting, line speed, and product variation made consistent inspection difficult. When a suspicious event occurred, decisions were made on the fly: stop, clear, and restart with limited documentation. If a consumer complaint or internal discovery raised concern, teams retraced lots and shift notes to reconstruct what happened. Holds were applied late, rework was disruptive, and evidence for disposition was thin.

The plant had invested in metal detection and barcode checks, but non-metal contaminants and visual anomalies were still caught manually. Vision cameras existed for print verification on some packs, yet they were not designed for product-level anomaly detection and did not integrate with QMS. MES tracked orders and lots, but it did not pause the line or create a record when a potential foreign object appeared. Quality managed deviations in the QMS, but the trail back to a specific image and timestamp was inconsistent.

Compliance expectations were clear: document preventive controls, act on deviations, and maintain evidence suitable for inspection under current good manufacturing practices for human food; see 21 CFR Part 117. The site also operated under a hazard analysis and critical control points (HACCP) plan, where reliable detection and documented verification are core principles; reference: Codex HACCP. Any solution had to slot into these systems and practices.

Why It Was Happening

Root causes centered on fragmented detection and manual handoffs. Visual checks were not tied to a governed trigger or standard response. Potential contamination events lacked a consistent link to order and lot information, so holds were delayed or applied broadly. Images from ad hoc cameras, when captured at all, were stored on local PCs without a chain of custody. Operators did their best to keep the line moving, but they had no integrated way to act on anomalies with confidence.

Ownership was diffuse. Operations controlled the line, Quality controlled deviations and release, Maintenance owned reject mechanisms and lighting, and IT/OT protected PLCs and SCADA. Without a shared signal and workflow, each team optimized locally. The last mile lacked enforcement: an anomaly could be seen without a system hold, or a hold could be applied without the evidence needed to support disposition.

The Solution

Intelligex implemented a governed detection-to-disposition workflow. Inline camera systems identified anomalies in the product stream and sent events to an orchestration layer that matched them to the current job, line, and lot. The service tripped a hold in MES for the affected lot, opened a QMS record with captured images and parameters, and routed a documented inspection task to the appropriate role. Supervisors and Quality reviewed evidence on a mobile or station screen, approved actions with electronic signatures, and released or escalated based on standard criteria. The approach layered onto existing hardware and systems and emphasized human-in-the-loop control for critical decisions.

• Integrations: Connected vision platforms (e.g., Cognex, Keyence) through vendor APIs; read line state and lot context from MES (e.g., Siemens Opcenter, Rockwell FactoryTalk ProductionCentre, SAP ME); applied product holds and release gates in MES; opened deviations and corrective actions in the QMS (e.g., ETQ Reliance, MasterControl, TrackWise). Optional historian/SCADA reads captured correlated signals without modifying PLC logic.
• Detection pipeline: Configured regions of interest and lighting normalization. Combined rule-based checks with trained anomaly detection where appropriate. Captured image snippets and metadata (timestamp, camera, belt position) for each event.
• Lot and genealogy mapping: Mapped events to orders, lots, and shifts to scope holds precisely. Linked downstream packs or pallets to the event through existing MES genealogy.
• Automated holds and tasks: Triggered MES holds on the affected lot or zone and prevented start of new lots on the line until review completed. Created QMS records with pre-populated details and attachments; issued inspection and clearance tasks to operators and Quality.
• Human-in-the-loop review: Provided a fast review queue with zoomable images, side-by-side comparisons, and standard decision buttons (reject/clear/reinspect). Required e-signatures with reason codes for disposition and release.
• Exception handling: Integrated with reject mechanisms already present on the line; when a reject fired, the event and take-away bin status were logged. If a reject faulted, the workflow escalated and blocked continuation until resolved.
• Dashboards and alerts: Live views of detection rates, holds in effect, and inspection aging by line and product family. Notifications to supervisors and Quality in Microsoft Teams for new events and pending releases.
• Security and access: Enforced least-privilege read-only access to controls; all write actions occurred in MES/QMS via approved interfaces. Images and records were stored in governed repositories with immutable audit trails.
• Standards alignment: Evidence packages were designed to support HACCP verification and cGMP inspection expectations under 21 CFR Part 117 and guidance such as Codex HACCP.

Implementation

• Discovery: Walked the line to identify optimal camera placements, lighting, and reject points. Mapped product flow, changeover patterns, and where manual checks occurred. Reviewed QMS deviation types and MES hold/release paths.
• Design: Defined the event schema (image, metadata, threshold), lot mapping rules, and hold scope. Set review gates, approver roles, and e-signature requirements. Specified thresholds and model tuning procedures with Quality and Engineering.
• Build: Integrated camera APIs, built the event router and MES/QMS connectors, and configured image capture and storage. Implemented the review app with side-by-side comparison and decision logging. Stood up dashboards and Teams alerts.
• Testing/QA: Ran in shadow mode: the system captured events and images while operators continued manual checks. Compared detections to human observations, tuned thresholds to reduce nuisance alerts, and validated lot mapping and hold behavior. Included a human-in-the-loop review board to approve go-live criteria.
• Rollout: Enabled automated holds on one line and product family first. Kept manual checks as a controlled fallback. Expanded to additional lines after two cycles of clean operation under the new workflow. No PLC logic was changed; reject triggers and line stops used existing outputs.
• Training/hand-off: Delivered role-based training for operators, supervisors, and Quality on the review app, holds, and release. Updated HACCP plan references and SOPs. Transferred ownership of thresholds and model updates to Quality and Engineering under change control.

Results

Anomalies that previously depended on chance observation were flagged in real time with images, and scoped holds were applied immediately to the affected lot. Operators and Quality worked from a shared queue with clear criteria and supporting evidence. Decisions and actions were captured with e-signatures, and line restarts followed a predictable release path instead of ad hoc calls on the floor. Troubleshooting time at the start of shifts and after changeovers decreased because the workflow made root cause and actions explicit.

Audit readiness improved. Each event carried images, timestamps, equipment identifiers, and a link to the lot and order. Dispositions referenced standard criteria, and effectiveness checks were traceable. The plant maintained its HACCP plan and cGMP practices; the difference was that foreign object control moved from manual inspection to a governed, evidence-backed process integrated with MES and QMS.

What Changed for the Team

• Before: Visual checks varied by shift. After: Inline vision created a consistent trigger with documented review steps.
• Before: Holds were broad and late. After: Holds were applied automatically to the correct lot with scoped impact.
• Before: Evidence lived in notes and memory. After: Images and metadata attached to QMS records with e-signatures.
• Before: Restart decisions were ad hoc. After: Release followed defined criteria and approver roles in MES/QMS.
• Before: Rejects were unverified. After: Reject events were logged with bin status and tied to line genealogy.
• Before: Thresholds drifted informally. After: Thresholds and model changes followed change control with cross-functional approval.

Key Takeaways

• Make detection actionable; connect vision events to MES holds and QMS workflows so responses are immediate and documented.
• Preserve human judgment where it matters; a fast review with images and standard criteria improves decisions and trust.
• Scope holds precisely by mapping events to lots and orders; genealogy reduces unnecessary downtime and rework.
• Store evidence in governed systems; images, parameters, and signatures should live with the deviation and release record.
• Integrate with what you have; use vendor APIs and approved MES/QMS interfaces rather than modifying PLC/SCADA logic.
• Run in shadow mode first; tune thresholds to the product and lighting before enabling automatic holds.

FAQ

What tools did this integrate with? The solution connected to vision platforms such as Cognex or Keyence through vendor-supported APIs, applied holds and release gates in MES (for example, Siemens Opcenter, Rockwell FactoryTalk ProductionCentre, or SAP ME), and created deviations and corrective actions in the QMS (such as ETQ Reliance, MasterControl, or TrackWise). Optional historian or SCADA reads provided correlated signals, and alerts surfaced in Microsoft Teams.

How did you handle quality control and governance? Thresholds and model updates were owned by Quality and Engineering under change control. Each event generated a QMS record with images, metadata, and lot mapping. Dispositions and releases required e-signatures, and audit trails captured who approved what and when. The workflow supported cGMP and HACCP verification expectations; see 21 CFR Part 117 and Codex HACCP.

How did you roll this out without disruption? The system ran in shadow mode first, capturing events while operators continued manual checks. After tuning and a cross-functional review, automated holds were enabled on a single line and product family, then expanded. Existing rejectors and stop controls were used; no PLC logic changed. Manual inspection remained as a controlled fallback during early cycles.

How were false positives managed? Thresholds included hysteresis and region-of-interest logic, and models were tuned with product-specific samples. The review app provided fast triage with zoom and comparison tools. Events that did not meet criteria were cleared quickly, and learning from reviews informed threshold updates under change control.

What did recordkeeping look like for audits? Each anomaly record included the image or frame sequence, timestamp, camera ID, line and lot context, hold and release times, and e-signature approvals. Reports showed event rates, disposition outcomes, and effectiveness checks by line and product, making inspections and customer inquiries more predictable.