Why does build time increase when using docker-compose with “args”?

I have a compose file that has the following:

build:
 context: /docker/container-setups/user
 dockerfile: user_webserver.dockerFile
 args:
 - db_hostname=${USER_ID}_database

and the dockerFile:

FROM php:apache
ARG db_hostname

Without docker-compose args, the container is started 20 seconds faster.

If instead of compose I use docker build –build-arg, it goes just as fast as if I had excluded the args.

So can someone please explain why the build time slows down when I include args with docker-compose and if I can do something different to make it work faster

building with –build-arg which is faster, but I need it to work with compose