Story Behind The EasyLPR

This is a two-year behind-the-scenes journey of how EasyLPR was initiated, underwent five iterations of redesign, and eventually evolved into a viable solution with help from experts, past research, and even AI Olympiad competitions.

Residential Safety in Alarming Trouble

In recent years, our residential communities in Jericho, New York have witnessed a severe increase in burglaries and break-ins. On one weekend alone, there were four incidents across all three villages in our area, including one occurrence across my house. Police went door to door asking for camera footage to look for identifiable features from the vehicles, especially clear license plate numbers. Unfortunately, there was no success.

It turned out the common pattern involved suspects driving repeatedly to “scout” neighborhoods, identify unoccupied homes (e.g., while residents are running errands or dining out), and enter quickly to steal valuables. Traditional alarm systems take upwards of 15 minutes to generate a police response, far too slow to prevent theft. Volunteer patrol groups help, but they are limited by lack of coverage. Many homes already employ surveillance cameras, but most systems are not designed to reliably capture license plates. Videos are unstructured, challenging to search through, and difficult to compare across homes. Residents may recall “a suspicious gray SUV,”, but without plate-level evidence, they can struggle to identify repeat visits or connect incidents.

Leaders in the local community association have held multiple meetings, gathering residents along with local law enforcement to discuss these safety issues during 2023 and 2024, and it was concluded that strong license plate reader (LPR) systems are necessary steps to tighten up security. The commercial offering, Flock Safety LPR, was proposed to be the de facto LPR system. However, due to its hefty upfront installation and future maintenance cost, only one of our three villages have successfully secured state funding to start installing 20 LPR systems starting in 2025.

Flock Safety Is Not Only Expensive but Lacks Prevention Measures

Although state funding was secured to install Flock Safety LPRs in one of the three villages, the other two villages, including the one I live in, didn’t follow this path as the cost and effectiveness of Flock Safety is questionable. The system is expensive with dual-camera LPR costs over $5000 plus thousands from an annual subscription fee per camera.

Furthermore, the Flock Safety LPR has to be installed on street poles connected to a power feed by a municipal system provider. For the 20 LPR systems installed in one of the villages, they are installed in ten major street ingress and egress points. This poses a significant limitation. If a suspect vehicle attempts to scout the areas it can only be observed at those points, but any activities through other neighboring areas won’t get tracked. And since the average homeowners won’t be in control of those LPR systems, they won’t get notified of any suspicious vehicle either. Hence, it leans more towards post-event analysis rather than providing real-time notifications as well as early warnings.

What is urgently needed by neighborhoods are LPR systems installed on homeowner’s premises that can be controlled and coordinated by residents by providing them with a structured vehicle identifier. If a community can reliably read plates at the neighborhood setting in a coordinated manner, it can: [1] detect vehicles that repeatedly scout the same streets, [2] correlate sightings across different cameras and houses, [3] provide accurate, time-stamped evidence to police, and [4] potentially deter crime with visible cameras and alert notifications.

Homeowner On-Premises LPR First Version – Power Supply Issue

In May 2024, I decided to take matters into my own hands by exploring the possibility of designing and installing my own cheap and on-premises homeowner LPR. I soon learned from YouTube that by connecting a Raspberry Pi computer board to a Raspberry Camera, I could form a basic video-capturing module. Due to the camera module being unable to capture clear pictures from far along with it being fairly small and light, I simply attached it to the back of my parent’s car parked on the driveway. Then, I moved the car as close to the street as possible so that it would be able to record clear plate images.

It turned out this setting worked well as the plate images were consistently identifiable. However, I soon observed that putting the Raspberry Pi in the car requires a portable battery to constantly provide power. With a Raspberry Pi consuming about 5 Watts in an idle state, I would have to regularly change and re-charge the portable battery, which is not feasible for practical use. Although one could argue for methods to work around that limitation, such as installing a portable solar panel to the car, these approaches all possess their own additional challenges and costs that may deter homeowners. 

Homeowner On-Premises LPR Second Version – YOLO Model for Plate Detection

In June of 2024, I designed a 3D CAD model to hold a Raspberry Pi while attached to a garage pillar, and then provide the power through the house power supply. As such, the battery issue is circumvented since the power supply can be constantly delivered through the house. However, it also means the original Raspberry Pi Camera module is not as fit for image capturing, being both further away from street (about 100-150 feet away) and with an oblique angle (30°  to 40°). Therefore, I had to replace the camera module with a low-cost telephoto lens that is capable of taking clear pictures from afar.

I also observed that in order to capture both the front and rear plates of a moving vehicle, I needed to install two cameras with each facing a different side of the street. Due to the previously mentioned awkward angles, they both have to capture live video at least about 10 to 15 frames per second. This is because of the limited time windows for vehicles to fall into the camera’s field of view, causing any frame rate below 10 to miss passing vehicles.

EasyLPR Placement

Another critical issue soon arose, though: my Python program for capturing the live video runs indiscriminately. This means I had to manually sift through video frames to identify vehicles as well as their license plate. Obviously, this is not how an LPR is supposed to work. Ideally, it should automatically identify the license plate of a passing vehicle. Sequentially, this is where license plate detection comes into play.

For the most optimal detection, I need to use the YOLOv8 deep learning model. With my parents’ help, I collected thousands of vehicle images with license plates that are clearly identifiable. I then manually labeled the boundary of the license plate for each of the images, then used YOLOv8 to train a detection model, which will outline the plate boundary. I loaded this trained detection model into my existing live camera program, and it is comfortably able to detect plate boundaries.

However, I ran into a big issue for the detection model. The detection inference is too slow, taking a few seconds per frame to identify, which is too slow for the 10 to 15 FPS requirement earlier mentioned.

Homeowner On-Premises LPR Third Version – Edge AI: A Must for Plate Detection

Several YouTube resources have pointed to Google’s Coral TPU as the de facto edge AI accelerator. It took me almost the entire next month to sift through tutorials on how to combine the Coral TPU with a Raspberry Pi. Thankfully, YOLOv8 supports the TPU and provides instructions for running AI accelerated models, which includes specific steps to convert the YOLOv8 trained model to a Coral TPU specific format called TensorFlow Lite.

Once I was able to get my existing Python video program to connect with the Coral TPU-built YOLOv8 plate detection model, the boost in inference speed was apparent: it only takes about 20 to 30 milliseconds, instead of a few seconds, to compile each frame, fast enough to accommodate 10 to 15 FPS for both cameras.

This is the first time I witnessed how much of a difference the Edge AI accelerator can make for machine learning and deep learning models.

With the license plate detection model and inference speed issue resolved, I can finally move onto the next step: license plate recognition.

Raspberry Pi 5 + Coral TPU + Pi Camera + 3D Mount

License Plate Recognition and License Plate Detection Are Completely Different

I initially confused detection and recognition, thinking they were the same until I used YOLOv8 to label and train the detection model. License plate detection is simply locating the exact position (and often the boundaries) of the license plate. The cropped plate image along with coordinates for the exact boundaries, usually a rectangular box, are then passed to a program for license plate recognition, which involves actually identifying the characters of the license plate. This process falls under the field of OCR – optical character recognition.

Knowing that license plate recognition is actually a subset of OCR, I moved on to researching existing OCR software, notably Tesseract, which is a very well known OCR software.

Unfortunately, it turned out Tesseract failed completely. It cannot correctly identify most of the characters, whether it be letters or digits, from the cropped plate image. The only way it could perform better for identification is if the camera was directly facing the street and much closer to the vehicle, making the plate extremely recognizable. This directly contradicts our conditions, which resulted in the plate image being small, blurred, and distorted.

License Plate Recognition with a Vision Large Language Model

I spent the entirety of August 2024 testing out all kinds of OCR software, but they all failed to consistently recognize the cropped plate images under my settings.

During this time, large language models such as ChatGPT had become very popular, and their vision model also began to launch and gain popularity. One day, I loaded a sample plate image into ChatGPT, and it somehow correctly identified all of the letters and digits.

This surprising discovery encouraged me to look for a vLLM based OCR model that can be integrated into my existing Python program of plate detection. Eventually I found MiniCPM-V, a vLLM model made exactly for that purpose. I got the MiniCPM-V to work with detection logic and passed the cropped plate image into it, and it demonstrated a modest performance at about 50% plate level accuracy. At the time, it was not bad at all, much better than existing OCR software.

However, there is a significant drawback for using a vLLM for plate recognition: it has to run on a desktop machine with a GPU in it. This restriction is due to the vLLM’s huge model size, meaning it cannot be loaded onto the Raspberry Pi/Coral TPU Edge AI device.

Not only that, the inference speed of the vLLM running in my GPU desktop, a gaming machine with NVIDIA’s RTX 4090, can only infer each cropped plate image at about 100 milliseconds, which is slower than the actual frame rate from camera, and that forced me to throttle the amount of the cropped plate images sent to the vLLM OCR model, that means, I had to design specific logics to select which cropped plates to send instead of all images.

Vision LLM (vLLM) OCR Works But Not Practical

Although the vLLM OCR “works”, it is not very practical for use by homeowners, particularly because of the need of additional desktop GPU. That GPU desktop, for the single purpose of running the vLLM, adds a few thousand bucks to the costs of the entire LPR system, making it much less attractive to general homeowners. Moreover, the OCR inference speed (100 milliseconds per frame) and plate recognition accuracy (50%) are not impressive enough to justify the expense.

Thus far, this version 3 of the LPR system, which contains a Raspberry Pi, HD telephoto lens, a Coral TPU for plate detection, and a GPU desktop for plate recognition, serves more as a proof of concept. The system works to a certain degree but demonstrates little practicality for community use.

Completing The End-To-End Solution

Even though the vLLM OCR proved to be not very practical, I decided to move on and complete the last puzzle of the entire LPR system, auto notification.

Throughout September to October 2024, I managed to augment a Discord notification mechanism into the LPR program such that once a cropped plate image is processed by the vLLM OCR, the resulting plate number (e.g. ABC1234) is then sent to my discord app automatically, allowing for real-time plate recognition and notification once a vehicle passes by my house.

EasyLPR End-to-End Solution with Separate Frontend (Pi5 + TPU) and Backend (Desktop GPU + LLM)

For the next few months from November 2024 to February 2025, I’ve been working to tweak my LPR programs, fixing bugs, and designing 3D enclosures that can either be attached to a garage pillar or installed on the front lawn area of a home property. The suggestion of adding a front lawn installation option is because many houses in our community simply lack a central garage pillar where the LPR could attach easily.

EasyLPR Installed on Front Lawn
EasyLPR Install on Garage Pillar

Another major field I’ve spent effort on was adding a correction database. The vLLM OCR has a major issue with recognizing plates with repeated numbers. For example, in the plate ABC1233, it often omits the last repeated number and returns ABC123. Additionally, the vLLM OCR sometimes confuses letters and digits that have similar shapes, such as O with 0, S with 5, etc. One solution is adding a correction database. By adding multiple entries to the database that converts the wrongly predicted plate to the ground-truth, it is able to recognize and correct the prediction for future instances. For example, if the ground-truth plate was LSK1234 but incorrectly predicted, I would add the entry L5K1234 to point to the correct plate. This correction database works well for plates that have already been recognized, but it won’t work for any unknown plates.

Getting Rid of the GPU Desktop

It was irritating to see the workable solution being held back by the additional GPU desktop essential for running the vLLM OCR, making the entire system impractical.

I decided to turn my attention to other Edge AI solutions that are not only capable of running the existing plate detection program as well as the Raspberry Pi + Coral TPU, but also have extra power to run the vLLM OCR for plate recognition to completely get rid of the extra GPU laptop.

Eventually, I found two possible replacement edge AI hardware: Hailo 8 and NVIDIA Jetson Orin. I decided to go with Jetson Orin because it is not only a standalone tiny computer but also equipped with a powerful edge GPU, which according to their specifications is over 16x faster than the Coral TPU (67 TOPS vs 4 TOPS). Hailo 8, on the other hand, still requires an extra tiny computer to work, such as connecting it with a Raspberry Pi board, and the combination of a Raspberry Pi ($60) and the Hailo 8 ($200) is not cheaper than the Jetson Orin ($249). Furthermore, the Hailo 8 itself can only provide about 26TOPS performance.

Homeowner On-Premises LPR Fourth Version – Standalone LPR with Jetson Orin

Finally, I have a cheap end-to-end LPR solution that can perform both plate detection and plate recognition on the same edge AI device w/o relying on external devices.

EasyLPR with Jetson Orin Nano and USB Camera

Except that I underestimated how much compute power the vLLM OCR requires. It turns out that on the Jetson Orin Nano which has about 8G memory, almost 6.4G is occupied by the vLLM OCR. This leaves little room for other programs such as for plate detection. To make matters worse, the 67TOPS performance of the Jetson Orin Nano board means it takes about 2 to 3 seconds for each OCR inference, far slower than what is needed for the 10 to 15 frames per second rate from the live video feed.

The slow inference speed defeats the purpose of a real-time Discord plate notification as the vLLM OCR program would fall behind the detection program.

Thus, this iteration, although satisfying the low cost and standalone conditions, possesses a major weakness in the OCR inference speed that is too slow for real-time performance.

Despite the months that had passed, I only had two sub-optimal solutions that each possessed a major limitation. The Raspberry Pi + Coral TPU + GPU desktop approach achieves a reasonably fast plate recognition speed but has mediocre plate level accuracy and a high cost due to the additional GPU desktop machine, whereas the other solution is low cost and standalone with the Jetson Orin Nano but is held back by its slow recognition speed modest recognition accuracy.

For the next six months from March 2025 to August 2025, I struggled to find breakthroughs for either approach, ultimately with no success.

USA AI Olympiad Come to Rescue

While stuck with these lackluster solutions, I participated in the first iteration of the USA AI Olympiad competition in March (Round 1) and April (Round 2), and I was awarded a bronze medal. The journey of preparing for the AI Olympiad opened my eyes to the mathematical theories behind machine learning and deep learning techniques, ranging from Linear Regression and CNNs to transformers. From this experience, I realized that there were other approaches outside of LLMs that held potential.

I began to reach out to professors for possible solutions to the OCR issue After discussing with a few of them, they immediately pointed out that a vLLM OCR for license plate recognition is overkill and suggested for me to study two papers that were tailored specifically for my goal.

The first paper is the MultiPath ViT OCR. It introduces a high-accuracy Vision Transformer (ViT) approach for license-plate recognition by modeling a plate as a fixed-length sequence of character “slots.” A ViT backbone extracts global features, while separate prediction heads are assigned to each character position, restricting each slot to its valid alphabet (letters or digits). This explicit structural modeling reduces many common errors. However, the approach relies on a heavy ViT backbone with weak locality bias, making it data-hungry and computationally expensive. Although the method achieves 77% accuracy with a 1.3M-image dataset, performance drops sharply to 63% when training data is reduced to 100K, highlighting the ViT’s dependence on large datasets. With only about 35K labeled plate images available, this approach is impractical for my setting.

The second paper is the MobileViT. It proposes a lightweight hybrid architecture that embeds small transformer blocks into a CNN backbone. Convolutions efficiently capture local features essential for character recognition from limited learning data, and compact transformer blocks are then used to provide global context at low latency and memory cost. Although not designed specifically for OCR, the MobileViT’s strong inductive bias, sample efficiency, and edge-device suitability make it compatible for short structured texts such as license plates.

MultiPath ViT OCR and MobileViT

Thanks to my preparation for the AI Olympiad, I was able to understand the underlying theory of the two papers and, from it, designed a novel OCR solution by merging the strengths of both papers: I designed a MultiPath MobileViT OCR for LPR, which uses a MobileViT backbone to provide local and global features efficiently, while adopting the multi-slot, template-aware prediction logic of MultiPath ViT. This architecture achieves strong character locality, global plate-level reasoning, and real-time performance on low-cost edge GPUs such as the Jetson Orin Nano.

Fifth Version of LPR With MultiPath MobileViT OCR

I borrowed the plate template concept from MultiPath ViT-OCR and applied it for New York State plate formats such as the standard NY7 plate (three letters followed by four digits: LLL-####) and the NY5D2L plate (five digits followed by two letters: #####-LL). From the template pattern, I generate per‑slot heads (e.g. 7 for NY7), each trained on the proper alphabet (letters or digits).

I use MobileViT as the model backbone. Input crops are standardized (e.g., 3×128×256), which aligns with my memory constraints and the plate’s aspect ratio. The MobileViT blocks first process locally with convolutions and then globally with a small transformer, allowing the network to learn strokes and spacing with few parameters. This hybrid bias is why MobileViT’s character accuracy rose drastically with only tens of thousands of images instead of millions.

Unlike the single plate format as mentioned in the MultiPath ViT-OCR paper, New York residential areas see various plate formats, including those from out-of-state (especially New Jersey). Consequently, I introduced a template router to predict which template is present (e.g., NY7 vs NY5D2L). For each template, I attach slot classifiers (one classifier per position). The architecture is shown in the figure below:

Finally, an End-To-End Solution with High Accuracy and Fast Inference Speed Of OCR

Throughout September and October of 2025, I conducted experiments for testing out the new MultiPath MobileViT OCR solution running on a Jetson Orin Nano.

EasyLPR End-to-End Solution with Jetson Orin and MobileViT

A total of 35,232 plate images (3 x 128 x 256, RGB) were used. The dimensions of the raw images are roughly 128 pixels x 256 pixels. Each image is cropped from a 1920×1080 vehicle image shot by a camera positioned 100 to 150 feet away at a 30–40° angle relative to the street. All the plate crops were collected from three residential locations, with each location equipped with two cameras.

There are a total of 5,148 unique plates among the 35,232 images. Many images of the same plates are selected because of varying conditions (lights, blurry, tilt, etc.).

Datasets are split for a training set, validation set, and test set. Datasets are grouped by plate identity to prevent leakage across splits. Both the training set and validation set were collected from a single residential location, with no data from the other two sites. The test set images were collected from the other two locations. The training set has 3,208 unique plates, and the validation and test set have 356 and 1,584 unique plates, respectively.

Although the model supports multiple plate templates, our experiments are conducted only with the NY7 format (LLL‑####). Experiments on additional templates (e.g., NY5D2L, out‑of‑state) are planned for future studies. I evaluated on the Jetson Orin Nano (8 GB, 67 TOPS) across three model settings. Unless noted otherwise, accuracy is measured as plate‑level top‑1 accuracy on the same test set, and latency is per‑image inference time (median over multiple runs) for a 128×256 crop. Memory usage is peak resident memory during inference. Three models are compared: The MultiPath MobileViT OCR (~35 K dataset size), a CNN OCR (lightweight CNN backbone; ~35 K dataset size), and a pre‑trained LLM vision OCR. Test results are shown in the table below:

ModelTest Acc (%)DatasetGPU Mem (GB)Inference Speed (ms)
MP MobileViT OCR88.5935K0.1339.0
CNN OCR Baseline73.0835K0.8257.6
LLM Vision OCR47.9—6.42619.5

With ~35K images, MultiPath MobileViT substantially outperforms the CNN OCR and the LLM OCR on the same test split, is orders of magnitude faster than the LLM OCR, comfortably achieve real‑time inference on the Orin Nano, and retains peak GPU memory near 0.13 GB, leaving sufficient resources for detection and the streaming pipeline.

Initial Pilot Deployment Impressive

With a viable low-cost and practical solution in hand, starting in November 2025 I began contacting a few friends and they agreed to pilot test the design in their homes. All of the installations of the LPR devices performed as expected and sent real-time notifications with the 10 to 15 FPS live video feed.

We have deployed this LPR system within our community. During routine monitoring, the system successfully identified several cases of potential irregularities: (1) vehicles appearing with two distinct license plates and (2) vehicles whose front and rear plates were inconsistent. These incidents were reported to the authorities, and detectives from the local police department expressed strong interest and were very impressed by the system’s utility during follow‑up discussions.

Future Work and Improvements

For future work, I plan to increase the sample training data to further bump plate recognition from 88% to over 95%, which is very feasible because of the inherent features of vision transformers that are capable of reaching greater accuracies with much larger training datasets incorporated. We conducted a preliminary data scaling test demonstrating that plate-level accuracy exceeded 95% with the number of training samples increased from 22.8K to 28K. I plan to continue increasing the training dataset size from 30K to 50K or above.

Another possible future addition is to design an algorithm to identify suspicious vehicles that are potentially scouting our communities, hence providing early warnings to residents and to deter possible crimes. This algorithm requires high accuracy plate recognition across multiple LPR devices installed in our communities and operating in a coordinated and intelligent way while simultaneously protecting local homeowners’ privacy.

My Reflection of This Two-Year Journey

This is a two-year long journey of designing an Edge AI and Vision Transformer assisted low-cost home-mounted LPR device.

I started the journey with a very simple Raspberry Pi + Camera module solution, a very cheap and seemingly promising setup, only to find out that an edge AI is a necessity for making real-time plate detection work.

Afterwards, I didn’t expect I would have to go through another four major iterations, each facing significant hurdles in attempting to make plate recognition both cost effective and highly accurate.

What interested me the most throughout the journey is that although LLMs dominate today’s AI conversations, they are not necessarily the right tool for resource-limited Edge-AI tasks such as the LPR I have been working on. My vLLM solution demonstrated this firsthand.

My breakthrough came unexpectedly after I built up a solid foundation in basic machine learning and deep learning mathematics from preparing and participating in the USAAIO competitions, and only when I sought assistance from experts did I finally realize that simple transformer technology aligned much better with my task.

With the innovation of applying strengths from the MultiPath transformer and the MobileViT, the OCR program finally completed a beautiful end-to-end LPR solution with dramatically higher accuracy, faster inference, tiny GPU usage, and minimal training data.

As a side product of this journey: I was awarded Camper status for the 2026 USAAIO competition, and my research on the MultiPath MobileViT OCR for License Plate Recognition was published by the 4th International Conference on Software Engineering and Machine Learning. A patent related to this research has also been filed. My sister assisted me in participating in the Y-Combinator funding application (Spring 2025 Cohort), and although we didn’t make it to the interview round, our application was ranked in the top 10% by the YC committee.

Acknowledgment

I could not successfully complete this whole journey without assistance from the following people:

  • Dr. Rita Ahamadi’s valuable insights in license plate detection algorithms as well as topics on optimization and transformers
  • Prof. Jens Rittscher and Prof. Kangning Cui’s valuable advice and guidance on computer vision and light-weight transformer techniques
  • My math teacher Dr. Matthew De Marinis’s encouragement in participating in USAAIO competition and valuable insights in helping me understand statistics in ML and AI
  • My friends’ assistance on collecting training samples and conducting pilot tests
  • My sister’s advice and guidance on NVIDIA’s Jetson Edge AI computer
  • My parents’ many rounds of driving around to help test out the effectiveness of the vehicle image capture and real-time notifications