Good question.

Here is a C#-ish code that lets you have it both ways. The trick here is lazy evaluation (yield).

// Lazy producer that will auto-close the file.
// The OS and the disk do the caching for you.
public static IEnumerable<FooBar> GetFoobars(string fileName, long maxNumber = 1000000000000)
{
    using(File file = open(fileName)) // pseudocode
    {
        FooBar nextFooBar;
        do
        {
            nextFoobar = new Foobar(file.ReadByte()); // or something like that.
            yield return nextFoobar;
        }
        while(nextFooBar != null || /* reach max */); // pseudocode
    }
}

// Consumer ... stupid example. Oh well ...
public static int Consume()
{
    int result = (from foo in GetFoobars("foo.txt").AsParallel()
                 where foo.Depth % 10 == 0).Count();
    return result;
}

The GetFoobars(...) function know which file to open, how many thingies to read and how to close the stream. You just pull on the results as you need them. Concerns are separated. What seems to be the problem? You could probably do the whole thing as a one-liner … maybe not, but close.

P.S. I recommend going through SICP video lectures.

Is it possible to incrementally process data without violating layers?

Yes, it is. You have to define unit of work which will solve your confusion. I think processing set of records (100 or 1000 depend on processing time of an average record) or a record at a time will have more reliable and manageable results.

In addition, i would make sure that data manipulation process (business processing) is in a transaction, all error processing’s are recorded and rolled back for investigation. You may always have some exceptional situations which must be taken into account.

Define an abstract interface that presents a foobar (or whatever set of foobars is required), leaving it up to the implementation of the abstraction to determine how the foobars are acquired and/or modified.

Naturally, you need to define the elements of this interface around that actual conditions. If the only way to count the foobars is to stream past all trillion, don’t include “get the number of foobars” in your interface. But probably your interface will contain some elements like “start” “stop” and “abort” in addition to “get”

The way I see it is this:

• You have a data layer which provides functionality to read individual foobars, one at a time.

• This functionality has an API. That API has a contract. Anyone wishing to use the API must adhere to the contract. The contract might specify, for example, that you must call NextFoobar() until you get null or something, and then you must stop calling it, but of course it could specify something more complex and it would simply be the contract to use.

• The business layer doesn’t care how you implemented the API, it only needs to use this API to do what it needs to do. Using the API includes that it must adhere to the contract.

Now, if you look at the C#-based answer posted by @Job, it follows all of these principles:

• The data layer provides an IEnumerable<Foobar> — an object that allows anyone to retrieve foobars from it, one at a time, without having to care where the foobars are coming from.

• The API on IEnumerable<T> is well-defined. You can either use it the hard way (call GetEnumerator(), and then on the enumerator keep calling MoveNext() and Current until MoveNext() returns false, and then call Dispose()) or use a language feature or method that uses the API automatically (e.g. foreach or, in his case, .Count()).

• The business layer correctly uses this API. It does need to call Dispose() (which it does indirectly by calling .Count()) but it doesn’t need to care what Dispose() actually does, it only calls it to fulfill the contract.

(Incidentally, if you enumerate all foobars, the file is not actually closed by the call to Dispose() but by the last call to MoveNext()… but Dispose() is still part of the API so that it can (and does) close the file if you decide to stop enumerating in the middle. The beauty of separation-of-concerns is that the business layer doesn’t need to care about any of that.)