I have been working on a project, and a part of it involves receiving a link from the user end. Currently, I am using a very rudimentary mechanism in terms of verifying the user input (all it does is check whether the input is a url or not). In an attempt to improve the security, I have been googling around to find solutions. Some of the solutions I came across involved checking the characters in the link, checking the links length, etc. I also came across some websites that have auto updating databases with known malicious websites. However, going through and comparing thousands of links would be very inefficient, and the first solution felt a bit spotty.

After a little more searching, I came across Google’s Lookup API:

https://cloud.google.com/web-risk/docs/lookup-api#python

The gist of it is that it compares inputted URLs with various Web Risk lists.

But, I also came across the following thread:

Automating the Detection of Malicious URLs
byu/jlteja incybersecurity

The thread suggests to use a paid intelligence service over anything free, or self made.

So, my questions are as follows:

1. Is there a reason to use the paid services over google’s? (I realize that Google limits the number of requests but for now assume that I am not dealing with a huge amount of requests)

2. Can I myself make something myself that checks URLs with updated databases in a quick and easy way? (The impression I got from the thread above was that this is not possible)

3. What services should I use if I opt against google (I saw some listed in the thread above, however, those services would check more for sites that already have a fair bit of traffic. My project deals with some users who have sites with little to no traffic, which could result in a false malicious tag)

The thread posted above mirrors my situation quite similarly, in that I have no cybersecurity experience and I too am accessing a lot of the sites metadata.

Any advice would be much appreciated!