I’m designing a web application. One of the main functions annotates frames from a webcam stream with the landmarks of different parts of the body (using Mediapipe). It then calculates the angle between the different parts of the arm (depending on how sure the Mediapipe AI is of having correctly located the landmark) and then yields a list that contains whether or not it could succesfully calculate angles, image data, and possibly the angles and rotation of the different parts of the arm to the central Flask function (views.py). There the list is processed and passed on to a html file that displays the image data. All this works fine. There is however also a button for Ajax requests. These are also being handled well by views.py, However the annotating function stops as soon as an Ajax request is made.

One of the main ways I tried to solve the problem was by using multithreading. This works (I’ve tested it with a different function which works well) and has made the video stream a bit more fluent. It has however not solved the problem. This is the code for the video stream annotation process, it is quite standard code for an application that combines Mediapipe with video streams:

def webcamcaptureannotated():
    import cv2
    import mediapipe as mp
    import numpy as np

    # Definitions of the functions for extracting koordinates from Mediapipe results, checking Mediapipe's confidence
    # in the points and calculating angles/rotation


    mp_drawing = mp.solutions.drawing_utils
    mp_drawing_styles = mp.solutions.drawing_styles
    mp_holistic = mp.solutions.holistic

    
    

    with mp_holistic.Holistic(min_detection_confidence=0.5, min_tracking_confidence=0.5) as holistic:

        # I added the ininite loop as another precaution that the webcam never closes for some reason during an 
        # Ajax request. That however didn't change anything
        while True:
            cap = cv2.VideoCapture(0)
            while cap.isOpened():
                success, image = cap.read()
                if not success:
                    print("Ignoring empty camera frame.")
                    continue
                
                image = cv2.cvtColor(cv2.flip(image, 1), cv2.COLOR_BGR2RGB)
                image.flags.writeable = False
                results = holistic.process(image)
                image.flags.writeable = True
                image = cv2.cvtColor(image, cv2.COLOR_RGB2BGR)

                mp_drawing.draw_landmarks(image, results.left_hand_landmarks, mp_holistic.HAND_CONNECTIONS)
                mp_drawing.draw_landmarks(image, results.right_hand_landmarks, mp_holistic.HAND_CONNECTIONS)
                mp_drawing.draw_landmarks(image, results.pose_landmarks, mp_holistic.POSE_CONNECTIONS)

                landmarks=results.pose_landmarks


                # Calculating angles with pre-defined functions (landmarks as input)


                # Geänderte Frame werden gestreamt
                ret, buffer = cv2.imencode('.jpg', image)
                image = buffer.tobytes()
                if proceed[0]==False:

                    # proceed[0] says whether angles have been calculated
                    yield [proceed[0],(b'--framern' b'Content-Type: image/jpegrnrn' + image + b'rn')]
                else:

                    # winkel and rotation are the results of the calculation of angles and rotations
                    yield [proceed[0], (b'--framern' b'Content-Type: image/jpegrnrn' + image + b'rn'),winkel,rotation]
            
                # Debug statements
                print("caprunning")
            print("running")

The following code is from views.py. There are a lot of other branches but I’ve isolated the relevant ones:

from flask import Blueprint, render_template, request, session, Response
from threading import Thread
from .JF_Webcam_Annotated import webcamcaptureannotated


# Ajax Request Handling. Currently it is only printing a String
@views.route("/toggleschicken", methods=["POST"])
def updateStatus():
    
    # lastvs is a somewhat bodged variable that just checks whether the last page the user was on is 
    # related to the Webcam-Annotation function
    global lastvs
    lastvs=1

    print("performedAjaxbool")

    # is_webcam_in_use used psutils to check whether the webcam is still on
    state=is_webcam_in_use
    if state == False:
        print("Webacam not on")
    else:
        print("Webcam is on")
    return '', 204


def generate():
        print("Starting the generate function")
        gen = webcamcaptureannotated()
        print("Started...")
        running=True
        while running:
            
            yieldinf = next(gen)

            #Checks if any angles/rotations have been calculated
            if yieldinf[0] == True:
                angles.clear()
                angles.append(yieldinf[2])
                rotation.clear()
                rotation.append(yieldinf[3])
            
            print(angles)
            angles.clear()
            print(rotation)
            rotation.clear()

            # Since I'm using multithreading, I'm checking if the user has swapped to an unrelated branch
            if lastvs == 0:
                print("Stopping generate function")
                break
            
            print("Currently running the generate function")

            # Yields the frame picture information
            yield yieldinf[1]

@views.route("/annotated_video_feed")
def annotated_video_feed():
    print("Changing to Webcam with annotations")
    global lastvs
    lastvs=1

    thread = threading.Thread(target=generate)
    print("Starting the tread for webcam annotation")
    thread.start()

    # Test thread to check if multithreading works (it does)
    thread = threading.Thread(target=testhi)
    thread.start()

    return Response(generate(), mimetype='multipart/x-mixed-replace; boundary=frame')

@views.route("/annotatedwebcam")
def annotatedwebcam():
    global lastvs
    lastvs=1
    session["last_branch"] = "JF_annotatedWebcam.html"
    return render_template("JF_annotatedWebcam.html")

And finally the html file.

<nav class="navbar fixed-bottom navbar-dark bg-dark">
      <form action="/toggleschicken" method="POST" id="isTrue">
        
          <button type="submit" name="button" value="links" class="btn btn-info"><</button>
          <span class="text-light">6.Achse</span>
          <button type="submit" name="button" value="rechts" class="btn btn-info mr-4" >></button>
          {% block navbar_button %}
          {% endblock %}
      </form>
    </nav>
    <div class="container">
      {% block content %}
      {% endblock %}
    </div>

    <script src="https://ajax.googleapis.com/ajax/libs/jquery/3.5.1/jquery.min.js"></script>


    <script>
      $(document).ready(function() {
          // Initial status
          var isTrue = false;
          updateStatus(isTrue);

          $('#schicke_winkel').click(function() {
              // Toggle the boolean value
              isTrue = !isTrue;
              // Call the function to update the status
              updateStatus(isTrue);
          });

          function updateStatus(isTrue) {
              // Send Ajax request to Flask route to update status
              console.log("Value of isTrue:", isTrue);
              $.ajax({
                  type: "POST",
                  url: "/toggleschicken",
                  data: {isTrue: isTrue},
                  // success: function(response) {
                  //     $('#status').text(response.status);
                  // },
                  // error: function(xhr, status, error) {
                  //     console.error(xhr.responseText);
                  // }
              });
          }
      });
    </script>

The actual button that triggers the Ajax request is not in this html file. Instead it is located in a different one that extends the above html file:

{% extends "kibase.html" %}{% block title %}Roboterarm - Webcam{% endblock %}
{% block navbar_button %}

<button type="submit" name="button" value="stopp" id="stopp" class="btn btn-danger mr-4">Stopp</button>
<button type="submit" id="schicke_winkel" class="btn btn-info">Winkel schicken</button>

{% endblock %}
{% block content %}
</br>
</br>

<a href="/webcam">Webcam Stream ohne Landmarks</a>

<div class="row">
    <div>
        
        <img src="{{ url_for('views.annotated_video_feed') }}" width="100%">
    </div>
</div>

</br>
</br>
</br>

{% endblock %}

I’d be very thankful for any help. I’ve been struggling with this for quite a while now.