I am new to prefect for workflow orchestration. I was exploring it’s databricks integration and used it’s tutorial example here . I am able to connect to datarbricks instance and I can print the jobs outside the flow i.e. in main with something like this :
if __name__ == "__main__":
jobs = example_execute_endpoint_flow()
print(jobs)
but if i try to print this inside the flow itself
@task
def get_job_list():
databricks_credentials = DatabricksCredentials.load("tnaiv-databricks")
jobs = jobs_list(
databricks_credentials,
limit=5
)
print(jobs)
it’s not prinitng it , it is sending a coroutine. I have tried async await, I have also tried doing this inside a task and return the jobs from there and priniting it in flow. Nothing works. Any suggestions or solutions !
Prefect employee here.
Can you try adding log_prints=True to your task decorator? So that it would look like this:
@task(log_prints=True)
def get_job_list():
databricks_credentials = DatabricksCredentials.load("tnaiv-databricks")
jobs = jobs_list(
databricks_credentials,
limit=5
)
print(jobs)
That should include the print of the jobs to the Prefect UI.
olearycrew is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.