I am trying to build a proof of concept for web scraping using Selenium in Foundry. The requirement is to scrape potentially hundreds-thousands of websites using Foundry. I realize this is probably not ideal, but I was given this requirement and I intend to find a way to make it work unless someone from Palantir can tell me why it won’t work and that this is a terrible idea. Beautiful soup works fine in regular transforms, but we will need something to scrape dynamically generated content. Due to the need to run a browser, I am not sure how to make Selenium work other than containers and sidecar transforms – if there is a better way please let me know.