FaceAnalyzer DetectLandmarks completely crashes PI 5

[Scattercatt]

I am new to OpenFace, and am trying to get started writing a simple program that captures an image from a webcam every half a second, and output "BLINK!" when it detects a blink.

Describe the bug
(Source code on this issue below)
This was tested on two systems.

The system where the bug is happening:

Raspberry PI 5
OS: Ubuntu 23.10 aarch64
Kernel: 6.5.0-1017-raspi
CPU: BCM2835 (4) @ 2.400GHz
Memory: 169MiB / 7943MiB

The system where the bug does not happen (Dell poweredge ubuntu server)

PowerEdge R710
OS: Ubuntu 22.04.3 LTS x86_64
Kernel: 5.15.0-107-generic
CPU: Intel Xeon X5550 (16) @ 1.974GHz
GPU: 08:03.0 Matrox Electronics Systems Ltd. MGA G200eW WPCM450
Memory: 654MiB / 24054Mi

I am running this program I created that uses the following libraries:

#include <opencv2/opencv.hpp>
#include <FaceAnalyser.h>
#include <FaceAnalyserParameters.h>
#include <GazeEstimation.h>
#include <LandmarkCoreIncludes.h>

I have a for loop that constantly uses a webcam to capture a frame, converts it to grayscale with openCV using cv::cvtColor(frame, grayscale, cv::COLOR_BGR2GRAY);. Once I do that, I use face_model.DetectLandmarks(grayscale, det_params) to try and detect landmarks, so that I can look up the "codes" (I forgot what they're officially called).

When running on the raspberry pi, the program will loop through an arbitrary amount of times (from as little as 3 to as much as 20 loops of this) before the entire pi just shuts down completely. No error message, just have to unplug and plug it back in.

On my dell poweredge ubuntu server, it will loop over this forever without crashing.

It is important to note however, that on both setups, the DetectLandmarks function always returns FALSE, indicating that it was unsuccessful, but I figured that would be a bridge I crossed at a different point.

Even worse, the program always crashes between COUT lines.

Things I have ruled out:

• It is not a memory issue. Memory peaks at maybe half usage whilst it is running.
• It is not the CPU getting overloaded. CPU peaks at 25% usage
• It is not temperature. Temperature peaks at 60-65C
• It is not a memory leak. I ran valgrind, and found nothing of note
• It is not a power issue. Originally I thought maybe the webcam was drawing too much power. However, if I comment out the DetectLandmarks, it will run just fine forever. I have confirmed the webcam is working correctly, and the device can handle constant images being captured.

SOURCE CODE:

#include <iostream>
#include <csignal>
#include <opencv2/opencv.hpp>
#include <FaceAnalyser.h>
#include <FaceAnalyserParameters.h>
#include <GazeEstimation.h>
#include <LandmarkCoreIncludes.h>

int DELAY = 500;
cv::VideoCapture cap;
bool keep_running = true;

// Signal handler for SIGINT
void handle_sigint(int sig) {
    std::cout << "Interrupt signal (" << sig << ") received. Cleaning up..." << std::endl;
    if (cap.isOpened()) {
        cap.release();
    }
    cv::destroyAllWindows();
    keep_running = false;
}

int main() {
    std::signal(SIGINT, handle_sigint);

    std::cout << "Starting webcam..." << std::endl;

    // Open the default webcam
    cap.open(0);
    if (!cap.isOpened()) {
        std::cerr << "Error: Could not open webcam" << std::endl;
        return -1;
    }

    cap.set(cv::CAP_PROP_FRAME_WIDTH, 320);
    cap.set(cv::CAP_PROP_FRAME_HEIGHT, 240);

    // Load face model
    LandmarkDetector::FaceModelParameters det_params;
    det_params.model_location = "/usr/local/etc/OpenFace/model/main_ceclm_general.txt";  // Ensure the correct model path
    LandmarkDetector::CLNF face_model(det_params.model_location);

    if (!face_model.loaded_successfully) {
        std::cerr << "Error: Could not load face model from " << det_params.model_location << std::endl;
        return -1;
    }

    // Initialize FaceAnalyser with FaceAnalyserParameters
    FaceAnalysis::FaceAnalyserParameters params;
    FaceAnalysis::FaceAnalyser face_analyser(params);

    std::cout << "Libraries initialized successfully." << std::endl;

    while (keep_running) {
        cv::Mat frame;
        cap >> frame;

        if (frame.empty()) {
            std::cerr << "Error: Could not capture frame" << std::endl;
            break;
        }

        //Convert to grayscale
        cv::Mat grayscale;
        std::cout << "Before grayscale convert | ";
        cv::cvtColor(frame, grayscale, cv::COLOR_BGR2GRAY);
        std::cout << "After grayscale convert | " << std::endl;
      
        ///////////////////////////////////////
        //THIS IS WHERE IT ALWAYS DIES//
        ///////////////////////////////////////


        // Detect landmarks
        std::cout << "Before detect landmarks | ";
        bool detection_success;
        try {
            detection_success = false;
            //detection_success = face_model.DetectLandmarks(grayscale, det_params);
        } catch (const std::exception &e) {
            std::cerr << "Exception during DetectLandmarks: " << e.what() << std::endl;
            break;
        } catch (...) {
            std::cerr << "Unknown exception during DetectLandmarks." << std::endl;
            break;
        }
        std::cout << "After detect landmarks" << std::endl;

        if (detection_success) {
            std::cout << "DETECT SUCCESS" << std::endl;
            face_analyser.AddNextFrame(frame, face_model.detected_landmarks, detection_success, 0.0, false);

            // Check for blinks
            std::vector<std::pair<std::string, double>> aus = face_analyser.GetCurrentAUsReg();
            for (const auto& au : aus) {
                if (au.first == "AU45" && au.second > 0.1) {
                    std::cout << "BLINK!" << std::endl;
                }
            }
        } else {
            std::cout << "DETECT FAILURE" << std::endl;
        }

        cv::waitKey(DELAY);
        std::cout << std::endl;
    }

    cap.release();
    std::cout << "Program terminated successfully." << std::endl;

    return 0;

To Reproduce
I have no idea how you would reproduce this besides cloning my project and attempting to run it. If someone would like to try to do that, let me know and I can post it.

Expected behavior
Program outputs when it detects a blink.

Screenshots
Probably not applicable

Desktop (please complete the following information):
Device specifications above

Additional context
Compiling OpenFace on my raspberry PI was a chore. I had to compile the following dependencies from source:

• ffmpeg using ./configure --enable-gpl --enable-version3 --enable-nonfree --enable-small --enable-libx264 --enable-libx265 --enable-libvpx --enable-libmp3lame --enable-libopus --extra-cflags="-fPIC" --extra-ldflags="-fPIC" --disable-libfdk-aac --enable-shared --disable-static
• openCV after changing --param=ipcp-unit-growth=1000000 to --param=ipa-cp-unit-growth=1000000 in some of the CMakeLists, and then using cmake -D CMAKE_CXX_COMPILER=g++-9 -D CMAKE_C_COMPILER=gcc-9 -D CMAKE_BUILD_TYPE=RELEASE ..

The rest of the dependencies were installed with apt-get

Then for OpenFace itself, I had to use cmake -D CMAKE_CXX_COMPILER=g++-9 -D CMAKE_C_COMPILER=gcc-9 -D CMAKE_BUILD_TYPE=RELEASE -D OpenBLAS_INCLUDE_DIR=/usr/include/aarch64-linux-gnu -D OpenBLAS_LIB=/usr/lib/aarch64-linux-gnu/libopenblas.so ..

I also had to remove -msse flags from CMakeLists due to architecture reasons (ARM vs x86)

What I believe the problem to be

My guess is that the CPU architecture is the cause of this issue. It was already a pain enough to get this to compile on ARM, as a had to do some special steps to get it to happen (described under additional context). I've just been banging my head against this for weeks now going through stack overflow and chatGPT. Please help, I would appreciate it so much.