I grasp that it can fetch internet resources, but I’m not really fully appreciating what that means. I was doing the challenges at pythonchallenge.com, and I got past the one that requires urllib to fetch the page and read the source, but I don’t see how this is useful quite yet. And examples are kind of lacking in diversity.
What else can it do, or how can I use this to my advantage?
(Besides building a web browser…)
you seem to be lacking use-cases: some examples:
- access public APIs: google, twitter, etc.
- write a custom web-crawler
or in the context of a web-application:
- use http-based services (e.g. uploading to AWS)
- test your own site / webapp
- authenticate your webapp users with a third party service
… and many more, you can access any service exposed on the web.
Screen Scraping is one possible application
It is a technique that is commonly used to automate tasks that would normally involve a person opening a browser, browsing to a page and manually analysing a page.
Other more common uses might be to access and use web services, or any sort of communication where the server is accessible via a URL
1
The internet and the world wide web is much, much bigger than just a bunch of web pages. There is a lot of information held behind web apis. For example, you can get stock quotes, weather, the price of tea in china, and so on.
Urllib, and libraries built on top of it, make it remarkably easy to gain access to this data. Much of it is in the form of JSON, which can easily be converted to native python data structures such as dictionaries and lists.