End-to-end perception: How can we make large-vehicle lane‑keeping corrections more precise?

01
System Overview and Technical Background

In port, Stack yard, and industrial‑site logistics settings, large vehicles typically travel long distances along fixed lanes. If a vehicle deviates laterally, it can lead to abnormal tire wear, lane‑occupying trajectories, collision risks, and reduced operational efficiency. Traditional manual correction relies on driver experience, resulting in delayed and inconsistent control actions. When vehicles are under heavy load, operating at low speeds, frequently starting and stopping, or working at night, lateral deviation accumulates more noticeably. Therefore, the system must continuously sense the vehicle’s position relative to the road or work lane with centimeter‑level accuracy and stably feed this deviation back to the control system.

Therefore, the core of a deviation‑correction control system is not merely “knowing where the vehicle is,” but rather, under complex operating conditions, reliably estimating the vehicle’s lateral deviation, heading error, and trend changes relative to the target trajectory, and translating this information into actionable, smooth, and low‑wear correction commands.

Technology Roadmap Evolution:

1.1 From “Hard Assist” to “Visual AI + GNSS Soft Fusion”

Early correction schemes relied more on GNSS-based correction. GNSS offers the advantages of relatively simple deployment and intuitive coordinate results, making it suitable for global positioning and path planning in open areas. However, in practical engineering, gantry cranes often operate near bridge cranes, factory buildings, stowages, containers, and metal structures, where satellite signals are easily affected by obstruction, multipath reflection, and transient drift. For scenarios involving the movement of large vehicles, relying solely on GNSS makes it difficult to maintain long-term stability.

To enhance local accuracy, the industry has also employed hard‑assist solutions such as sliding contact lines, magnetic markers, and underground cables. These approaches can provide a stable reference within specific areas, but they entail high installation costs, lengthy construction timelines, and ongoing maintenance that is susceptible to ground wear, settlement, and changes in routing due to renovations. For stack yards and port areas that require rapid deployment, minimal modifications, and replicability, these solutions lack sufficient engineering flexibility.

The value of visual correction lies in using the vehicle’s immediate surroundings as a direct reference for control. Cameras can detect local features such as lane markings, ground boundaries, and work‑area signage. , which can compensate for the shortcomings of GNSS in environments with signal blockage and drift.

1.2  HIRI The Core and Advantages of the Heavy Vehicle Steering Correction Control System

This is a high-precision automatic deviation correction system for large vehicles, designed for port, stack yard, and industrial transportation scenarios. Centered on visual AI perception, the system integrates GNSS positioning data, vehicle motion state information, and intelligent control algorithms to dynamically sense the spatial relationship between the vehicle and its target path. Continuously output key control parameters such as lateral deviation and heading deviation. and achieve automatic vehicle re‑tracking to the predefined trajectory via closed-loop control.

Unlike traditional solutions that rely on fixed infrastructure or a single positioning method, HIRI FUTURE adopts “Visual perception + GNSS assistance + Intelligent control” The collaborative architecture deeply integrates environmental perception capacity with vehicle control capacity, Enabling vehicles not only to perceive their own position, but also to understand the spatial relationships among road boundaries, travel lanes, and the intended route. , thereby achieving more precise and stable automatic deviation correction control.

HIRI’s practical advantages in deviation correction control systems:

Low-Cost Retrofit: It eliminates the need for sliding contact wires and magnetic markers, relying primarily on cameras, GNSS, and an onboard computing unit for deployment. This results in a short construction timeline and minimal disruption to existing facilities.

High precision: The visual model outputs more stable lane-line information, with a deviation of within 3 cm under both high-speed and low-speed conditions for large vehicles.

High reliability: Dual redundancy between vision and GNSS ensures that, when one sensor experiences a brief anomaly, the other can provide reliable support, enabling the fusion of data to deliver superior control performance.

Smoothness Control: A small torque adjustment is made every 200 ms to refine the direction, resulting in smoother voltage and current.

Comprehensive scenarios: It can support various operational scenarios within the Stack yard, such as Autonomous driving and fixed‑channel cruising, and provides a perception foundation for the future expansion of Autonomous driving functions.

02 Core Technical Approach: How to Achieve Visual Correction

2.1 Industry Challenges: Why Is It So Difficult for Traditional Solutions to Achieve 3 cm?

In the path‑following process of automated transport equipment, the core challenge in achieving centimeter‑level correction accuracy lies in how to continuously and stably acquire the relative positional relationship between the vehicle and the target route. Even when traditional GNSS‑based correction is combined with RTK augmentation, it remains susceptible in complex industrial environments to factors such as signal blockage, multipath reflections, antenna installation errors, and changes in vehicle attitude, leading to fluctuations in positioning results and making it difficult to maintain stable, high‑precision output over the long term.

Visual correction directly leverages lane markings, boundary lines, and ground‑level structural features in the road environment ahead of the vehicle for perception, establishing a relative spatial relationship between the vehicle and the road to achieve more stable path‑tracking capacity.

With the advancement of deep learning technologies, HIRI’s visual model has already demonstrated powerful feature extraction and environmental understanding capabilities, enabling it to accurately detect lane markings and road boundary information under varying lighting conditions, weather, and complex backgrounds. Even when confronted with conditions such as partially worn lane markings, shadow occlusions, ground stains, or reflective interference, the model can still rely on its learned global semantic features and, through joint modeling of road structure, boundary characteristics, and vehicle travel direction, Accurately calculate the vehicle’s lateral deviation and attitude deviation relative to the target trajectory. and feed this information back to the control system in real time.

Detection performance of the model under varying environmental conditions and illumination levels.

2.2  From YOLO segmentation to AI end-to-end model detection

In order to stably achieve [something] in complex industrial environments, With a correction accuracy at the 3 cm level, the system must not only precisely detect lane‑line positions but also continuously output smooth, stable offset estimates that are geometrically constrained. HIRI has adopted “AI end-to-end model detection” technology in its visual perception pipeline. 。

In the early stages of visual planning, the YOLO segmentation model is well-suited for quickly verifying feasibility. It can perform pixel-level segmentation of lane markings, channel boundaries, ground signs, and drivable areas, with fast inference speed and a mature engineering ecosystem, making it easy to deploy on in-vehicle computing units. Compared with traditional thresholding and edge detection, YOLO segmentation can extract road structure more directly; however, it has high requirements for the visibility of lane markings and ground boundaries: when lane markings are unclear, worn or broken, covered by mud or water, or when encountering rain, fog, nighttime glare, strong reflections, or shadow occlusions, the segmentation mask is prone to boundary jumps, local missing segments, and false segmentation. Subsequent processing steps such as perspective transformation, centerline fitting, lateral distance conversion, and temporal filtering are still needed to convert the segmentation results into deviation values usable by the controller.

▲ Lane marking wear—YOLO detection results (with partial occlusion)                  

HIRI’s AI end-to-end model detection has addressed these drawbacks.

This model no longer outputs intermediate segmentation masks; instead, it directly predicts the location and category (e.g., lane lines, boundary lines, crosswalk markings) of lane markings from the input image, along with their corresponding confidence scores. This design offers significant advantages:

Stronger environmental robustness:

During the training phase, the model has thoroughly learned from real-world samples of lane-marking wear, partial occlusion, rain and fog, glare, and standing water on the road surface. Even when local lane markings are unclear, it can still generate continuous, reasonable lane‑line results based on the global road structure features.

A more streamlined processing pipeline:

By eliminating the traditional, complex geometric post-processing steps, the model directly outputs lane‑line information with clear geometric meaning, thereby reducing error accumulation and system latency.

More stable bias input:

The end-to-end output exhibits minimal temporal jitter in lane‑line localization and high accuracy in category classification, providing a reliable perception foundation for subsequent geometric computation and control modules.

▲ Lane marking wear End-to-End AI Detection Performance (Complete)

2.3 Technical Breakdown: From “Seeing Bias” to “Controlling Stability”

The first step is visual acquisition. The system leverages the anti-collision cameras already installed on the main carriage door legs, thereby reducing production costs while capturing images of the on-site roadway and work access routes.
The second step is end-to-end model verification. The model not only determines whether there are lines or markings in the image, but also outputs more stable lane‑line positions, lane‑line categories, and confidence scores, providing a foundation for subsequent deviation calculations. During training, the model incorporates samples with varying lighting conditions, rain, fog, stains, wear, shadows, occlusions, and other factors to enhance its generalization capability under real-world operating conditions. The third step is the soft fusion of vision and GNSS. . GNSS provides global positioning and road‑segment identification, while vision enables high‑precision, short‑range deviation detection. These two modalities are not simply combined; instead, they are fused based on confidence levels and the current scene state. When visual perception is temporarily obstructed or degraded by adverse environmental conditions, the system leverages GNSS and historical trajectory data to maintain continuous control. The fourth step is closed-loop control. The controller computes the corrective steering input based on lateral deviation, heading error, vehicle speed, load condition, and historical trend data, while employing gain limiting, filtering, and micro‑torque adjustments to prevent lateral oscillation. The control objective is not to abruptly steer the vehicle back to the center in a single maneuver; rather, it aims to guide the vehicle smoothly back to the desired trajectory, all while ensuring safety and preserving mechanical durability.

03
Project Plan: How will the system be implemented?

3.1 Visual AI Implementation Pathway

To realize the aforementioned perception link, the HIRI visual correction system adopts the following three-tier design in engineering:

(1) Dual-channel industrial RGB camera + lane marking calibration

The system deploys onboard industrial equipment on the left and right sides of the trolley. The RGB camera selects either the forward‑facing or rear‑facing camera for computation, depending on the vehicle’s direction of travel. The camera’s field of view is focused on lane markings, roadway boundaries, pavement signage, and the vehicle’s relative position within the lane. This dual‑camera configuration not only provides primary correction cues in the forward direction but also preserves a rear‑view reference, enabling detection of any angular deviation in the vehicle’s overall orientation. After installation, lane‑line calibration is used to determine key parameters—such as reference points, lane width, and pixel‑to‑real‑world scale—to convert the spatial coordinates of lane markings in the image into actual deviations in the vehicle’s local coordinate system.

(2) Key ROI + End-to-End Lane Detection

The model tier establishes key visual regions around lane lines, channel boundaries, and drivable areas. The YOLO segmentation model primarily outputs lane‑line masks, which are then post‑processed to extract the centerlines; however, this approach is highly dependent on lane‑line clarity, lighting conditions, and weather. In contrast, HIRI’s AI‑driven end‑to‑end model directly learns both the structure and category information of lane lines, enabling it to deliver more continuous lane‑line results even under conditions such as worn markings, partial occlusion, rain or fog, glare, and standing water. This simplifies the traditional visual pipeline of lane‑line filtering and merging, allowing the model to rapidly output lane‑line data. Additionally, it can classify lane‑line types to help identify special areas like start/end zones and transition sections, preventing these critical scenarios from being mistakenly treated as ordinary deviations.

(3) Short-term sequence stability + deterministic control

Bias correction control does not rely on the results of a single-frame recognition; instead, it incorporates visual deviations, vehicle motion direction, speed status, and safety thresholds into a deterministic control loop. The system triggers deceleration or stop‑protection based on the calculated deviation magnitude and validates consecutive abnormal frames, thereby preventing a single-frame misrecognition from directly causing an unintended stop. In cases of visual line loss or low confidence, it can be combined with… GPS offset generates redundancy for protection. The final correction value is written to the PLC, enabling the vehicle to return to the target lane in small, incremental steps and ultimately maintaining the large vehicle within a 3 cm deviation tolerance.

3.2 Control Strategy

The end-to-end model enhances the Stability of lane‑line and category outputs, while the geometric computation module converts the lane‑line results into lateral deviation. The control module then generates correction commands based on the deviation, speed, driving direction, and the disparity between the front and rear cameras. Throughout the process, clear rule boundaries are maintained: which situations allow for normal correction, which require deceleration, and which mandate stopping are all determined by deterministic logic.

In terms of control performance, the system emphasizes “small steps with rapid adjustments” and continuous gain limiting. The controller updates the correction command at approximately 200-ms intervals, employing filtering and dynamic gain limiting to suppress oscillations and prevent abrupt, large‑magnitude adjustments that could induce vehicle sway or mechanical shocks. For scenarios such as heavy loads, low‑speed start‑stop operations, approaching the target position, or cases where front‑ and rear‑camera deviations differ, the system can further refine the attitude correction by integrating information from both cameras and the vehicle’s state, resulting in smoother, more controllable correction maneuvers.