I have a C# app, using HttpClient to connect to a remote server. The server owner has asked for Client Certificate authentication. I sent through the public key certificate to them, and loaded the .pfx certificate (which includes the private key) when I make the request.
I receive the following error:

The credentials supplied to the package were not recognized

I now load the server owners public key certificate into my Azure Certificate store, in the “Public key certificates” area, and it works.

Why does loading their certificate make any difference? I already needed to ignore server side validation, as we are connecting via an IP address, as opposed to a URL. This is the configuration:

        var handler = new HttpClientHandler();
        handler.ServerCertificateCustomValidationCallback =
            (httpRequestMessage, cert, cetChain, policyErrors) =>
            {
                return true;
            };
        handler.SslProtocols = SslProtocols.Tls12;
        handler.ClientCertificateOptions = ClientCertificateOption.Manual;
        var certificate = new X509Certificate2(certificateData, password);
        handler.ClientCertificates.Add(certificate);

        using (var client = new HttpClient(handler))
        {
            ...
        }

So, how is adding the server certificate doing anything? What have I missed?