My setup right now is that I have multiple databricks files running in a pipeline in Azure Data Factory. Each step of the pipeline is a databricks notebook and in each notebook, I have boilerplate code to init my logging setup.

Here’s my logging code:

import logging
from azure.monitor.opentelemetry import configure_azure_monitor

configure_azure_monitor(connection_string="my conn string")

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

logger.info("This is a test log")

I know this is an issue with it being initialized over and over again, but how do I check if it has already been initialized?

I could only set it once at the beginning of the pipeline, but if the pipeline is ran multiple times back to back, we’d still get this error.

Also, here is other errors I have been getting from this problem:

Attempting to instrument while already instrumented

Overriding of current MeterProvider is not allowed

Overriding of current LoggerProvider is not allowed

Overriding of current TracerProvider is not allowed