I want to create a piece of code that can detect cursor’s live position and update what action was taken in the video recording.
Basically,

1. first : it takes screenshot of the video every 0.5 or less seconds here’s the code for it.

2. second : screenshot array or folder goes to a yolo pipeline where the cursor position is detected and a new array is created.
   CursorArray = [‘xaxis’,’yaxis’, CursorType, CursorAction]
   CursorType = [‘Hand Gesture’, ‘Pointer cursor’, ‘writing cursor’]

3. third : what action was taken by cursor
   CursorAction
   if CursorType == ‘Hand Gesture’ : CursorAction = ‘Clicked on a URL’
   if CursorType == ‘Pointer cursor’ : CursorAction = ‘LEFT CLICK, HOVER’
   (NOTE : on pointer cursor It will check the difference in next image so I’ll write the code for left click or Hover was happend etc)
   if CursorType == ‘writing cursor’ : CursorAction = ‘COPYING A TEXT’

4. fourth : I also want to know how it will detect writing bar if someone is writing ?? :confused:

For the first part I figured it out and 3 part is as simple as I wrote in the above text. I need help with 2nd and 4th part Please…. and I also looked up for template matching in opencv but I will need atleast 1600 images for it but it will take forever to loop over every image like 1600 times in 32 px of a area so If I get a screen of 1080 P it will atleast take 10 mins I think to get the result on 1 image.

Thanks in Advance and Please help me thanks alot everyone!!!

#import pytesseract
import cv2
import os
import numpy as np
import pyautogui, sys
import os
import time
from time import sleep
import re
from collections import defaultdict
from subprocess import call

#taskName = 
taskName = 'OpenNotepad'

def takingScreenshot():
    counterExample = 0
    counter = 0
    screenshotArray = []
    os.mkdir('D:\pyautogui\'+ str(taskName))
    while counterExample < 50:
        counterExample = counterExample + 1
        screenshot = pyautogui.screenshot()    
        time.sleep(0.5)
        counter = counter + 1
        pathNew = ('D:\pyautogui\'+ str(taskName) + r'screenshot' + str(counter) + '.png')
        print(type(screenshotArray) , len(screenshotArray))
        screenshotArray.append(screenshot)
    print(type(screenshotArray) , len(screenshotArray))
   
    #Converting the Screenshots into actuall images and saving in folder.
        screenshot.save(pathNew)

takingScreenshot()
#THIS FUNCTION IS WORKING PROPERLY AND SAVING IMAGES AS WELL