I’m trying to adapt the following code (from this answer:
from sqlalchemy import create_engine, select
conn = create_engine("DB URL...").connect()
q = select([huge_table])
proxy = conn.execution_options(stream_results=True).execute(q)
while 'batch not empty': # equivalent of 'while True', but clearer
batch = proxy.fetchmany(100000) # 100,000 rows at a time
if not batch:
break
for row in batch:
# Do your stuff here...
proxy.close()
to use Session instead of the connection directly. I basically create sessions like this:
def get_db():
db = SessionLocal()
try:
yield db
finally:
db.close()
where
engine = create_engine(get_db_url(), **SQLALCHEMY_ENGINE_OPTIONS)
create_database(engine.url)
SessionLocal = sessionmaker(autocommit=False, autoflush=False, bind=engine)
Any idea how to use the execution options and fetchmany equivalent but with sessions?