For as ill-defined a problem as this, I prefer to write an actual application first, without particular reference to what the interface will be. Of course, you should think about what the interface will be as you go, but there’s a single client so you’re free to reorganize as necessary.

When you start the second application that would use the interface, start abstracting methods you need for the second application from those provided by the first.
Iterate. The second application will inevitably force the first one to be changed to support a common interface. The third will force the second and first to change. Etc.

When you get to the point where adding a new application no longer requires the interfaces to change, then you are ready to publish your interface.

My first idea was to write an interface […] but then it came to my mind that if I add any other functionality later all code will break for those who are using the old interface.

Use an abstract class instead:

public interface Hexagon<T extends HexagonData>{
    T getData();
    void setData(T data);
}

public abstract class HexagonData{
    public boolean isTransparent(){ return false; }
    public boolean isPassable(){ return true; }
    public int getPassageCost(){ return 0; }
}

Don’t use annotations unless you actually need metadata about the member that has been annotated.

I propose faceted approach similar to the one used in C++ standard library.

interface Hexagon {
  void setData(int id, T data); //these should only be used from Facet class
  Object getData(int id);

  sealed class IFacet<T> {
    private static int count = 0;
    private int _id = count++;
    public T getData(Hexagon hexagon) {
       return (T) hexagon.getData(_id);
    }
    public void setData(Hexagon hexagon, T data) {
       hexagon.setData(_id, data);
    }
  }
}

//Usages:

class CostManager {
  public readonly Facet<int> PASSAGE_COST = new Facet<int>();

  static void incrementCost(Hexagon hex) {
    PASSAGE_COST.setData(hex, PASSAGE_COST.getData(hex)+1);
  }
}

class TransparencyManager {
  public readonly Facet<boolean> TRANSPARENT = new Facet<boolean>();
  static void disable(Hexagon hex) {
    TRANSPARENT.setData(hex, false);
  }

}

Each Facet addresses exactly one property of Hexagon (as they use unique _id). If Hexagon.setData() and Hexagon.getData() are always used through a Facet class, the data stored will always have the type of corresponding Facet. This will prevent user from putting trying to write data of wrong type to any given id.

To illustrate the kind of errors this approach protects from consider an example based on @k3b answer:

  static void configure(Hexagon hex) {
     hex.setData(DataType.TRANSPARENCY, "very transparent"); // compiler accepts this just fine, strings are objects, aren't they?
  }

  static void processHexagon(Hexagon hex) {
    boolean data = (Boolean)hex.getData(DataType.TRANSPARENCY); // compiler accepts this, Boolean are objects too
  }

  static void doSomeJob(Hexagon hex) {
     configure(hex);
     // working
     // more work
     // ...
     processHexagon(hex); //oops, you've got runtime error
  }

Now consider the same code with Facets:

  static void configure(Hexagon hex) {
     TRANSPARENT.setData(hex, "very transparent"); // compiler slaps you
  }

  static void processHexagon(Hexagon hex) {
    boolean data = TRANSPARENT.getData(hex); // no downcast!
  }

  static void doSomeJob(Hexagon hex) {
     configure(hex);
     // working
     // more work
     // ...
     processHexagon(hex); //no runtime errors, just compile-time ones
  }

As you can see, the code is even less verbose, as there is no need for downcasts now.

If you want to add dataelements/properties/variables to your Hexagon without changing the other HexagonHandlers like HexagonalGridCalculator you can add dynamic properties like this:

public class Hexagon implements Serializable {

    // ... other methods not important in this context

    void setProperty(int id, object data);

    object getProperty(int id, object notFoundValue);
} 

and define constants for the dataelements/properties/variables

const int SatelliteData = 1;

If you later want to have additialal data just define new constants

const int Transparent = 2;
const int Passable = 3;
const int tPassageCost = 4;

which does not affect the other handlers.

I would build it like a DTO. Have interfaces with getters and setters for the data you need and do your processing in external systems. Maybe provide a default Hex they could extend if they wanted to. The result would be more entity component system than object oriented but that is kind of the point of component systems the flexibility to extend in ways you didn’t anticipate up front.

As far as I can tell, you want to provide an API allowing users of your library to build complex objects (Hexagons), having a set of attributes that potentially can be extended in the future (transparent, passable, passage cost etc).

Design issues like that are targeted by Builder pattern:

the builder pattern uses another object, a builder, that receives each initialization parameter step by step and then returns the resulting constructed object at once… Builders are good candidates for a fluent interface…

For your case, implementation could look as follows (assuming newAttribute being added in later version library):

public class HexagonBuilder {

    public Hexagon build() { /*...*/ }

    public HexagonBuilder transparent(boolean isTransparent) { /*...*/; return this }

    public HexagonBuilder passable(boolean isPassable) { /*...*/; return this }

    public HexagonBuilder passageCost(int cost) { /*...*/; return this }

    public HexagonBuilder newAttribute(Object attribute) { /*...*/; return this }
}

As you can see, old code would compile just fine with newer version of your library, because it’s only difference is compatible – it just doesn’t invoke newAttribute method.

Note that for old code not only compile, but also work smoothly with newer version library, you will need to provide an appropriate default behavior for client code that doesn’t set new attribute.

_{For the sake of completeness, consider taking a look at other creational patterns – Factory, Prototype etc, although per details provided in the question so far, Builder looks like the best fit.}