Total no: of filters possible is M . User can select m filters where m <= M . A typical example is files from a folder , he could say modified between so and so date , start with so and so and so on.
meets = []
for each in something:
if user_filter1:
if user_filter1_condition_met:
meets.append(True)
#hardcode all other filters here
if all(meets):
do_something(each)
else:
meets = True
for each in something:
do_something(each)
Here I am repeating the entire code again which is against DRY principles.
for each in something:
if user_filter:
if not filter_condition_met:
continue
do_something(each)
Here I am doing the filter check for all items in the loop, while essentially it needs to be done just once.
Can I use higher order functions to check the filter condition once and generate a code that does the job? It looks like a filter and then transform operation. Can someone give some ideas?
An addendum to the question with a similar essence:
def function(inner,outer):
for each in outer:
for each in inner:
print "Something"
#Border condition inner doesn't exist I need a code object that goes like this:
def function(inner,outer):
for each in outer:
print "Something"
#Inner doesn't exist , so I want to to not go a for each in inner operation i.
#I mean it won't change the complexity of the problem at hand
#But suppose there are billions elements , I save billion for each in [].
6
Conceptually, each filter is a predicate – a function that given an input, returns either true or false. The python function filter(function, iterable) accepts a predicate and an iterable, and returns only items that the predicate is true for. So what you want is a way to combine all the filters a user selects into one filter, that only matches if all of them match. The all function can be used here, all(iterable) returns true only if every element of iterable is true. Using this, we can combine predicates:
def andPredicate(predicates):
def pred(v):
return all((p(v) for p in predicates))
return pred
The andPredicates function accepts a list of predicates and gives you a new one that only returns true for an item if all the predicates return true for that item. For example, if you have predicates isNumber and isLessThanZero, you can define isNegativeNumber like this:
isNegativeNumber = andPredicate([isNumber, isLessThanZero])
print(isNegativeNumber("foo")) # False
print(isNegativeNumber(8)) # False
print(isNegativeNumber(-5)) # True
So this lets you combine filters. If you have a list of filters, where each filter is a predicate, then you can just do this:
filteredItems = filter(andPredicate(userFilters), items)
Done and done. The iterable is only iterated once, and the filters are checked one at a time on an item until one fails. If one fails, the rest of the filters are skipped and the next item is taken, if they all succeed, then the item is included in the final iterable. Note that filter produces a list in Python 2.x and an iterator in Python 3.x.
Additionally, the definition of andPredicates can be further shortened with an anonymous function, but that may be unclear to readers of your code if they’re not familiar with functional programming patterns:
def andPredicate(predicates):
return lambda v: all((p(v) for p in predicates))
For completeness, here are definitions of the aforementioned isNumber and isLessThanZero predicates:
from numbers import Number
def isNumber(v):
return isinstance(v, Number)
def isLessThanZero(x):
return x < 0
In the second part of your question, you want to either do something once for each item in the cross product of two lists, or you want to do something once for each item in the second list of the first list is empty. The itertools module contains a function product that basically does the same thing as nested for loops. For instance instead of this:
for a in as:
for b in bs:
print("Something!")
You can write this:
from itertools import product
for a, b in product(as, bs):
print("Something!")
Both pieces of code are equivalent. When one of the iterables is empty however, the whole product is empty, just like in nested for loops. We can write our own version of product that instead of being empty when one of the iterables is empty, is only empty when both are empty and if just one is empty, then it just returns the values in the other iterable and uses None as the values for the first iterable. This can be defined like so:
from itertools import product
def productWithNones(as, bs):
if as and bs:
return product(as, bs)
if as:
return product(as, [None])
if bs:
return product([None], bs)
return []
Now this:
for a, b in productWithNones([1, 2, 3], []):
print((a, b))
Prints this:
(1, None)
(2, None)
(3, None)
Instead of simply being empty. So your original loop can be written as:
for eachOuter, eachInner in productWithNones(outer, inner):
print("Something!")
6
Here I am doing the filter check for all items in the loop, while essentially it needs to be done just once.
Why is that a problem?
Write code in the cleanest way possible. If it makes sense to structure the code with the boolean test made multiple times, then write it that way.
Avoid unneeded optimizations, to quote Knuth, “premature optimization is the root of all evil”.
1
Something like this is what I wanted: (Applying a sample use case I could come up with)
predicates = []
if filenamme:
predicates.append(lambda x,filename: x.name == filename)
if from_dt and to_dt:
predicates.append(lambda x, from_dt, to_date: from_dt <= x.mod_dt <= to_dt)
for file in files:
if all(predicate(file) for each in predicates):
do_something(file)
I wanted to generate my predicates outside.
1