I am trying to detect slightly oval-shaped squares on the 15×15 game board area and the player’s hand area in a screenshot from a game similar to scrabble. Below is the original screenshot:

screenshot

Here is the code i have got but it is missing some empty squares and most of the squares containing letters.

import cv2
import numpy as np

image = cv2.imread(r'image_path')

gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)

blurred = cv2.GaussianBlur(gray, (5, 5), 0)

edges = cv2.Canny(blurred, 50, 150)

contours, _ = cv2.findContours(edges, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)

output_image = image.copy()

for contour in contours:
    epsilon = 0.02 * cv2.arcLength(contour, True)
    approx = cv2.approxPolyDP(contour, epsilon, True)
    
    if len(approx) == 4 and cv2.isContourConvex(approx):
        (x, y, w, h) = cv2.boundingRect(approx)
        
        aspect_ratio = float(w) / h
        if aspect_ratio > 0.8 and aspect_ratio < 1.2:
            cv2.drawContours(output_image, [approx], -1, (0, 255, 0), 3)

cv2.imshow('Detected Squares', output_image)
cv2.imwrite('processed_cleaned_image.png', output_image)
cv2.waitKey(0)
cv2.destroyAllWindows()

Here is the output image:
output

Two letters in the player’s hand and most of the squares on the board are missing.

Given the constraint that there will always be 15×15 squares on the board, and that there will be 7 squares in the player’s hand, except for a few of the last rounds.

Also i tried the Square detection in image solution but it got worse.