I am trying to run snowflake notebook paralley using python concurrent process- ThreadPoolExecutor
However the execution failing
from concurrent.futures import ThreadPoolExecutor, as_completed
from snowflake.snowpark.context import get_active_session
from snowflake.snowpark.exceptions import SnowparkSQLException
session = get_active_session()
def execute_copy_command_with_session(cmd):
try:
result = session.sql(cmd).collect()
print("result is", result)
return {"success": True, "result": result}
except SnowparkSQLException as e:
print(f"An error occurred during the copy operation: {e}")
return {"success": False, "error": str(e)}
copy_list2 = ["""select count(*) from table1""",
"""select count(*) from table2""",
"""select count(*) from table3"""]
# Use ThreadPoolExecutor to execute copy commands in parallel
with ThreadPoolExecutor(max_workers=len(copy_list)) as executor:
futures = [executor.submit(execute_copy_command_with_session, cmd) for cmd in copy_list2]
for future in as_completed(futures):
result = future.result()
if result["success"]:
print("Task succeeded:", result["result"])
else:
print("Task failed with error:", result["error"])
below is the error i am getting
No sessioncontext
is this the right approach to achieve parallel processing with in notebook?
New contributor
Umeh Narayanan is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.