Building a categorization system and ran into a problem which I am trying to find a solution. My system has tags used for categorizing with the caveat that more tags and categories will be added over time.

Rather than use a word vectorizer and MultinomialNB, I choose to build a translation table where tags equate to categories and tags/categories can be added over time.

This theoretically solved two of my problems as 1: I do not have training data (literally the goal is to start with nothing and have categorizing as a feature as data is added), only new entries with tags associated and 2: it is scalable.

The question is how to weight a system like this? the tags added cannot be even like I currently have it set as just because a tag is part of a category, it doesn’t 100% represents that category. I could set it manually but it can be extensively time consuming.

And the tags to category is added when new data is entered. It is just how it is. The only thing I do have on my side is that the classification do have descriptions.

here is the code below

unique_categories_with_tags = {"FPS" : {"First Person": 1,
                                        "shooting": 1,
                                        "weapons": 1,}
                              "JRPG": {"Japanese": 1,
                                       "Role Playing": 1,
                                       "Anime style": 1,
                                       "Stat building": 1,
                                       "weapons": 1,
                                       "exploration": 1},
                              "adventure": {"exploration": 1,
                                            "platforming": 1,
                                            "weapons": 1,
                                            "open world: 1}

descriptions = {"FPS": """FPS stands for first person shooter where players in the first
                        person perspective run around set environments and shoot at 
                        opponents. Weapons can range from guns, bows, and other distance 
                        style weapons""",
                "JRPG": """JRPG stands for Japanese Roleplaying Game, and is essentially, an 
                         RPG designed and produced in Japan. JRPGs are typically identified
                         as having leveling systems, anime aesthetics, and have themes about
                         killing god with the power of friendship."""}

Here is the code for building the prediction system. Currently I have the weights being based on frequency within the dictionary. The more times a tag appears the weaker the tag.

class PredSystem:
    weights: pd.DataFrame
    
    def build_weights(self, weight_dictionary: Dict) -> None:
        keys = list(data.keys())
        
        unique_tags = []
        for cat in data:
            for entry in data[cat]:
                unique_tags.append(entry)
        unique_tags = list(set(unique_tags))
        
        translation_table = np.zeros([len(unique_tags),len(keys)])
        
        translation_df = pd.DataFrame(data=translation_table,index=unique_tags,columns=keys)
        
        for data_weights in data:
            weight_keys = list(data[data_weights].keys())
            weight_df = pd.DataFrame(list(data[data_weights].values()), index=weight_keys,
                                     columns=[data_weights])
            translation_df.update(weight_df)
            
        translation_df_weighted = translation_df.div(translation_df.sum(axis=1),axis=0)        
        self.weights = translation_df_weighted
    
    def predict_classification(self, keywordTags: List) -> str:
        predictions = self.weights.loc[keywordTags,:].sum().sort_values(ascending=False)
        return predictions.keys()[0]

to run the code

test_tags = ["Japanese","weapons","exploration","shooting"]

PS = PredSystem()
PS.build_weights(unique_categories_with_tags )
PS.predict_classification(test_tags)