Today, while programming, I found myself managing a resource (ssh connection) inside a generator function, something similar to the following:
#I decided to use a generator to avoid looping through the list of fnames 2 times
def _yield_fname(host_address, usr, pwd, datapath):
with paramiko.SSHClient() as ssh_client:
ssh_client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ssh_client.connect(host, username=usr, password=pwd)
stdin, stdout, stderr = ssh_client.exec_command(
'find . -type d -maxdepth 1 -printf "%fn"'
)
for line in stdout:
yield fname
def get_foos():
foos= []
for fname in self._yield_fname(self.FOO_PATH):
foos.append(Foo.from_fname(fname))
return foos
def get_bars():
bars= []
for fname in self._yield_fname(self.FOO_PATH):
bars.append(Bar.from_fname(fname))
return bars
In my case, I plan to always consume yield_fname‘s generator from start to finish, so that the connection is always closed.
However, my concern is that someone (e.g., future me) might use the generator without fully consuming it thus leaving the connection open.
Is there a way to make it evident that the generator should be closed after usage, even if not fully consumed?
5
is it bad practice to manage a resource inside a generator function in this way?
IMO no, that’s pretty normal usage
Should I worry about someone (e.g. future me) using the generator without fully consuming it and leaving the connection open?
Probably not, but it’s a possible concern. I’m fairly certain that (as written) the context manager will “live” so long as the generator hasn’t been exhausted or garbage collected.
I generally wouldn’t worry about it though, it’s kind of a performance/resource edge case, and exacting performance and resource management isn’t a high priority in large majority of Python use-cases.
is there a way to prevent that?
You can create the context outside of the generator, passing it in. Even if the generator isn’t exhausted, you can still exit the context.
gen = None # even if we keep generator from being garbage collected
found = None
with get_resource() as res:
gen = build_generator(res)
for item in gen:
if desired(item):
found = item
break # incomplete iteration
print(item)