When Java was first designed it was considered appropriate to leave out anonymous functions. I can think of two reasons (but they might be different from the official ones):

1. Java was designed as an object-oriented language without functions, so it was not very natural to have anonymous functions in a language without functions. Or at least, this would have influenced the design of the language a lot.
2. Anonymous functions were not popular in the programmer communities that Java was meant to attract (C, C++, Pascal?). Even now, many Java programmers seem to consider these features quite exotic (but this will probably change very quickly with Java 8).

In the following years, as Robert Harvey has explained, the policy of Sun was always to keep Java backward compatible and very stable.

On the other hand, other competing languages have emerged (the most important being C#, which was born as a Java clone and then took its own development direction).

Competing languages have put Java under pressure for two reasons:

Expressive power

New features can make certain programming idioms easier to write, making the language more attractive for programmers. Normally the set of features provided by a language is a compromise between expressive power, language complexity, design coherence: adding more features makes a language more expressive but also more complex and difficult to master.

Anyway, in the last few years Java’s competitors added lots of new features that Java did not have, and this can be considered an advantage.

Hype

Yes, unfortunately this is a factor in technology choice, at least from what I can see in my daily experience as a programmer: a tool must have a certain feature, even if most members of the team don’t know how to use it and those who would be able to use it don’t need it most of the times.

Hype can be even more important for non technical people like managers, who may be the ones who decide the platform for a certain project. Managers sometimes only remember some keywords like lambda, parallelism, multicore, functional programming, cloud computing, … If our technology of choice has a green mark
on each item of the list, then we are up to date.

So IMO for some time Java has been caught between

• the original policy of language stability and design simplicity, a huge code base and developer community on one hand, and
• the pressure of competing languages that could attract Java programmers, C# at first, and then Scala, Clojure, F# (I name the ones I am aware of, there may be others).

Eventually Oracle decided to upgrade Java to make it more competitive.
In my opinion, the new features address especially Java programmers that might be tempted to switch to C# and who see other languages like Scala and Clojure as too different from Java.
On the other hand, developers who have some experience with functional programming and still want to use the JVM have probably already switched to
Scala, Clojure, or another language.

So the new Java 8 features will make Java more powerful as a language and the declared focus is concurrent and parallel programming, but the upgrade seems to address also the marketing aspects (Mark Reinhold, chief architect for Java at Oracle, said: “Some would say adding Lambda expressions is just to keep up with the cool kids, and there’s some truth in that, but the real reason is multicore processors; the best way to handle them is with Lambda”, see this article).

So, yes, many (all) Java 8 features were well known already, but why and when a feature is added to a language depends on many factors: target audience,
existing community, existing code base, competitors, marketing, etc.

EDIT

A short note regarding “… I had read about streams in SIC (1996).”: do you mean that you need Java 8 lambdas to implement streams? Actually you can implement them using anonymous inner classes.

Java has changed focus with time. At first it was designed as a simple powerful language, as a reaction to “powerful complex” C++. Some features that were in C++ were intentionally left out, like operator overloading, templates, enums, that were deemed too complicated or relics of the C era, and OOP being at the peak of its popularity, everything was made an Object in a single-paradigm worldview. Lambdas at this time were considered simply “not needed” since introduction of anonymous/inner classes in Java 1.1. The fact that the syntax was far more verbose was almost considered a feature.

Java having found its public, there were no incentive to change until Microsoft introduction of C#, which learned from the lessons of Java design mistakes, and launched a series of new language features. They were not constrained by backwards compatiblity. I think that Java conceptors realized the danger of C# competition and release Java 5 with generics, enums, etc.

Inclusion of lambdas in Java is discussed since that time and it was only exacerbated by the current trend for functional programming. But things like this are slow to make right, and it has to be right the first time. In my opinion Java botched generics with type erasure because backwards compatibility was considered a reason to implement it as no more than syntactic sugar. Closures has been thought up more thoroughly, it appears, and it will be more than just syntactic sugar.

In conclusion, what is Java 8 main topic? I don’t think a language version has a topic. As C++11, the Java 8’s raison d’etre is to keep up with the competition by introducing in the language the things that more and more programmers are taking for granted now. Lisp may have lambda since 1958, its popularity has plateaued for decades now and only recently functional programming has been considered seriously for “mainstream” programming (for lack of a better word).

Evidently none of these features are new in the programming world and
that made wonder why are getting all these things in Java until now.

Because Java has to go through an approval process that involves several high-visibility stakeholders in a process akin to “design by committee”, and that process takes time, that’s all.

Contrast that with other languages and you’ll find either a benevolent dictator, or a small committee of language designers who work closely together and are not tied to corporate interests.

Combine that with an established code base of millions of lines of Java code that has to stay backwards compatible, and you have all the ingredients for change at a glacial pace.

I would say that the most important purpose of a programming language is to be used; currently C and Java don’t have lambda expressions and they are the most used languages (according to TIOBE for example).

And to answer the question I believe that Java is addressed to the enterprise; in this domain things need to be very stable and reliable; for example Java 7 has appeared for almost 2 years yet I don’t know directly any project in Java 7. Also another important thing is backwards compatibility which is very important for the enterprise.