Then we first need to refactor many of things because the real needs are a little bit different

You are creating a false dilemma.

Real needs are almost always different than what users think/can tell you. Agile attempts to find these differences closer to when the code is written instead of at the end of a full project.

Agile is NOT “we will just start coding and see what happens.” If you don’t even know what features you are going to be working on for a sprint or they consistently change, have additions, or significant modifications all during the sprint, you are not really doing Agile development.

You need enough planning to be able to do a sprint without revisiting the requirements/user stories constantly. If this isn’t happening, you aren’t really using Agile appropriately.

This related question describes approximate planning times. You might find this useful. Additionally, this answer directly discusses planning for Agile.

Agile != no planning (contrary to many perspectives…).

Define “planning”

By “planning” I think you mean understanding the needs and figuring out how to do it, generally with a group of stakeholders (esp. users) and developers (et al). This involves a lot of conversations, thought, learning-time, etc.

Traditional methods try to plan too much to start, almost always with incomplete/unstable information.

Agile recognizes that information is incomplete (at any point in the project, but especially in the beginning), and so favors frequent, incremental conversations (and design/plan adaptations).

Assume that a certain volume V of communication is necessary to gain the mutual understanding necessary to properly plan and execute a project. How much V should be consumed up-front as opposed to in-process is subjective.

Personally, I like to consume just enough V up-front to get started; once everyone gets the basic idea/metaphor/goal and is excited about the project, jump into something with tangible value and show benefits – but with as many additional (but more focused) conversations as necessary.

Agile doesn’t mean no planning, or even less planning; but it can mean no planning phase in the project.

In a true Agile project, the team is always planning, but in smaller chunks. Every new piece of knowledge gained nudges ‘the plan’ in one way or another. The plan itself is fluid and accepting of change.

The problem is that there is a tendency to want to sit down and document the plan. The moment you do that, the plan becomes a little bit out of touch with reality, which is constantly changing. The more time you spend documenting the plan, the more out of touch it can become.

Agile recognizes this counter-intuitive fact, and tries to minimize planning when you least know what your users ultimately need (i.e. at the beginning of the project) and maximize the planning closer to when you know more (i.e. later in the project, as you’re developing and getting feedback). Agile also recognizes that in many cases, documentation of the plan is less useful/important than execution of the plan.

As such, Agile asks that you only spend just enough time and effort planning (and documenting the plan) to allow the project to move forward responsibly. Any more than that, and it’s a waste.

I would argue Agile (in my case agile scrum) is actually deferred planning compared to waterfall. The difference is agile planning I personally feel is your large scale plan is fuzzy / not detailed yet, but your short term plan is planned to extreme detail. As apposed to Waterfall where traditionally you planned the entire project up front (to the extent you could realistically do so)

Agile planning

Agile itself can be broken up into tons of sub methodologies, but at the highest level it’s having detailed plans for short time spans 1 week to 1 month, past that is really pushing it. Your longer term plans more than 1 month down the road typically are not planned into much detail (yet) and live as list of features, bugs, etc to be done some day in your backlog.

The planning of your short term “sprint” is very detailed. You plan out the entire sprint, who will do what, technical details, etc. The precision of these plans vary team to team, but generally speaking successful shops tend to air on the side of more planning than less. The end of your sprint is your deadline, all work listed should be done by then (exceptions apply)

Typically you list all your want / need to haves in a backlog, each sprint you start at the most important items divvy them out you your team until you’ve assigned to the point they’ll be kept busy, while still realistically finishing on time.

You repeat this process every sprint. This makes planning progressive so as the situation of the project changes the project can adapt.

Traditionally waterfall you planned everything up front and spent 3,6, or 12 months pounding it all out, Then going back and adjusting things at the end.

one way to think of agile is it’s still waterfall, only your releases are weekly not annually. (someone will not be happy I said that, my point is you still plan things through just the scale of the plan is WAY smaller, but more often.)

BA talks to key users, collects data, which are pretty much just a
sketch what they really want.

Do you feel the BA should spend more time, ask better questions, collect more data so you can plan out the entire strategy with little to no refactoring? Why don’t the developers ask for more details before starting to code? Seems like there is a history here.

we realize they need a bigger system we expected, following that we
need to make bigger refactor and generalize everything afterwards.

Do some sort of retrospective and determine if these things could have been identified sooner rather than later. You may find out that no one expected this at all.

Is this the way it should be?

I don’t know about “should be” but this is usually the way it is and agile is a way to handle it better. How would you feel if you spent more time gathering incomplete specs, spent months writing code, and then having to rewrite all of that? This sounds even worse.

Given by the information are we doing it in the good way? And if the
answer yes, then does the low planning costs match with the higher
refactoring and rewriting costs? So to say: profitable?

You need to understand why “real” requirements aren’t getting recognized until after some development has taken place (It’s not that uncommon.). Maybe the BA can gather more or maybe you need to ask more questions and have him go back to the clients for details.

Sometimes you have to build something and get it in front of the clients so they can then determine what they really want. Few can do it in the abstract. Don’t expect to double your upfront planning and cut your development in half. You’ll probably find yourself making more detailed plans that just get thrown away because many places don’t take the time to rewrite them due to falling behind.

I’ve been doing agile for years now – the one any only answer to your question is : “yes”.

Agile development results in somethinh between little and next to no planning at all – it has its focus on flexible, customer-bound solutions – actually planning the solution strips the flexibility / agility away nigh-completely because the customer is bound to his own specifications and stories.

It is a somewhat “chaotic” process – although not really; solutions are implemented in a strictly pre-set environment which bends according to rules of flexibility / agility, scrum (for example) has no planning at all (if you dont count sprint planning), but the process of change can only happen via teamwork, strategic and tactial planning and large amounts of user-feedback. In the result, the team will have implemented a highly functional, highly user-compliant solution in even less time than other processes / frameworks facilitate… if carried out correctly (you will need a highly competent team) the lack of planning does next to no harm, only minor drawbacks are to be expected which are very manageable and can be eliminated over time with little effort.