I tried to make this simple crawler to crawl down the entire ICD-11 database (https://icd.who.int/browse/2024-01/foundation/en#455013390) and collect all the titles and descriptions of all diseases, but it keeps stopping at the very top level of the link tree and does not go down the tree into the various branches and their sub-branches and so forth. Where is this going wrong?

I am trying to crawl a webpage “https://en.wikipedia.org/wiki/Glossary_of_artificial_intelligence”. My objective is to dump the data in an excel file where the column 1 will contain the heading for example “abductive logic programming” and column 2 will contain the senteces under it.

Even if I try using Selenium, it doesn’t work and neither dynamic nor static crawling works. Is there something wrong with my code?