Im working on a project where we store millions of endpoints for users (like user bookmarking a page, user likes a link and we store it, user saves fav end point etc etc ). We have more than 10-15 million endpoints stored. We estimate 10-20% of those end points are non functional/got removed/moved/ dead end points. How do I remove those URLs from data base ?
Sometimes the website may be down for temporary time and i dont want to remove it if scripts gives me 404, and so many cases like these.
What should be my approach ? Thank you