I have been working on a python web crawler for a while now and I’m now wanting to add logging into it so that I can leave it running for a long time and come back to see how it’s doing and analyze its behaviors.

My question is, what is the best practice for implementing this? Currently I have my 1 main logger instance which is getting passed around to each of the classes (and sub-classes) of the project to avoid having to create a whole new logger for each instance of each class that is created while running, but I’m starting to think that passing this one instance around all over the place may not be the best way and would like to hear others input. Below is a simple example/illustration of what I’m talking about:

class AppManager:

   def __init__(self):
      self.my_logger = logging.getLogger(__name__)
      #... logger setup
      self.my_scraper = WebCrawler(self.my_logger) <- pass in member variable into constructor of this class so that I can use the same logger instance

   def some_function(self):
      #...doing something, something goes wrong
      self.my_logger.warning("Warning message") <- log warnings in this class if a method runs into issues

   def get_wp_contenets(self, url):
      contents = self.my_scraper.get_wp_contents(url)

#In another file
class WebCrawler:

   def__init__(self, logger):
      self.my_logger = logger <- store the logger instance as a member of this class
      #...other constructor code

   def get_wp_contents(self, url):
      r = requests.get(url, headers=self._my_header)
      if r.status_code != 200:
         self.my_logger.critical("Bad status code") <- use logger to log to same location as other classes
         #...error handling
         
      return(r.content)

This same behavior is what I have in a small section of my code and works as intended, but, in some cases the WebCrawler would need to create other classes and those sub-classes are being passed the same logger object as the one in the AppManager class. It makes the code ugly and now ever single little class I have in my project all now has a my_logger member and it is all the same as the one defined in the AppManager class. It seems like there should be a better way.