I have an application made in Laravel for selling courses. For each course, I store a title, description, and categories.

I need to implement an intelligent search. For example, if a user types “Software developing” in the search bar and there are two courses named “Learn Python in 3 Weeks” and “JavaScript from Scratch”, the search should return those two results, even though none of the keywords from the user’s query appear in the course titles.

After a week of investigation, the best solution I’ve found is to implement a search with a TF-IDF tokenizer. I would create an API in Python with two endpoints: one to save the course when it is created and store the vectors of the title, description, and categories with its ID, and another endpoint to perform the search. This last one would receive the search query, convert it to vectors, and return an array of course IDs ordered by cosine similarity.

My idea is to store the vectors serialized with pickle because the database I’m using is MySQL, and it doesn’t support vectors and searches.

The main issue with this solution is that it doesn’t filter the results effectively. For example, if the user searches for “software developing,” it would return a search with two software development courses at the top and then other courses that have nothing to do with the user’s interest.

To filter results, I guess the only options would be to set a minimum similarity threshold or use an LLM model to filter the results.

Additionally, I would have to depend on an API call to the Python API each time the user searches because I don’t think there are similar solutions in PHP, right?

Is there any other solution?

I’ve tried TF-IDF tokenizer searches