Blob CV & Neural Networks

AprilTags are great for figuring out where your robot is on the field, but what if you need to find a game piece scattered on the floor? Game pieces don't have AprilTags taped to them!

To find game pieces, we have two primary methods on the Limelight: Color Blob Tracking (Traditional CV) and Neural Networks (AI Object Detection).

1. Color Blob Tracking (Traditional CV)

The simplest way to find a game piece is to track its color. If the game piece is bright yellow, you can tell the Limelight to ignore everything in the image that isn't yellow.

This is done entirely in the Limelight Web UI (no Java code required to set it up!):

1. Open the Limelight Web Interface (http://limelight.local:5801).
2. Create a new Pipeline and set its type to Color Tracking.
3. Use the crosshairs on the camera feed to click on the yellow game piece. The Limelight will automatically isolate the Hue, Saturation, and Value (HSV) of that color.
4. Adjust the threshold sliders until only the game piece is highlighted in green on the screen.

In your Java code, you switch to that pipeline (limelight.pipelineSwitch(1)) and ask the Limelight where the target is. It will return an LLResult containing tx (how far left/right the target is from the center of the camera) and ty (how far up/down it is). You can feed tx directly into a PID controller to automatically turn the robot toward the game piece!

[!WARNING] The problem with Color Tracking: If there is a bright yellow banner in the audience, or if the stadium lights change the shade of the game piece, the Limelight will get confused and track the wrong thing. Color tracking is highly sensitive to lighting changes!

2. Neural Networks (AI Object Detection)

To solve the lighting problem, modern teams use Neural Networks. Instead of looking for "the color yellow," we train an AI model to look for the shape and texture of the game piece. It works in the dark, under bright spotlights, and ignores background noise.

To do this, we use a free tool called Roboflow.

Training Your Own AI Model

Uploading to the Limelight

Once you have your .tflite file from Roboflow:

1. Open the Limelight Web UI.
2. Create a new Pipeline and set its type to Detector (Neural Net).
3. In the "Neural Net" tab, click "Upload Model" and select your .tflite file.
4. Upload the associated labels.txt file (which tells the Limelight the names of the objects it found).

Reading the AI Data in Java

Now, just like with Color Tracking, the Limelight will draw a bounding box around any game pieces it sees.

In your FTCLib Command, you can grab the results:

public void execute() {
    LLResult result = limelight.getLatestResult();
    
    if (result != null && result.isValid()) {
        // Get a list of all objects the AI found
        List<LLResultTypes.DetectorResult> detections = result.getDetectorResults();
        
        for (LLResultTypes.DetectorResult d : detections) {
            if (d.getClassName().equals("YellowSample")) {
                // We found a yellow sample!
                double targetXOffset = d.getTargetXDegrees(); // "tx"
                
                // Pass targetXOffset into a PID controller to turn the drivetrain!
                driveSubsystem.turnToAlign(targetXOffset);
                break;
            }
        }
    }
}

Check Your Understanding