This is similar to this question, but I am running on bare metal.
Below are my agent config and install config. I am simulating a fully offline install where I use a custom repository to build everything.
I have started the offline install by generating an iso with openshift-install agent create image
and then am monitoring the progress with openshift-install --dir=./ wait-for bootstrap-complete --log-level=debug
I noticed that if I try to add agent openshift-install --dir=./ agent wait-for bootstrap-complete --log-level=debug
, that command fails with:
<code>DEBUG Loading Agent Hosts...
DEBUG failed to load Install Config: failed to load asset: failed to load asset "Install Config": failed to create install config: invalid "install-config.yaml" file: [platform.baremetal.hosts: Invalid value: []*baremetal.Host(nil): bare metal hosts are missing, platform.baremetal.Hosts: Required value: not enough hosts found (0) to support all the configured ControlPlane replicas (3)]
FATAL failed to load AgentConfig, NMStateConfig, InstallConfig, or AgentHosts
<code>DEBUG Loading Agent Hosts...
DEBUG failed to load Install Config: failed to load asset: failed to load asset "Install Config": failed to create install config: invalid "install-config.yaml" file: [platform.baremetal.hosts: Invalid value: []*baremetal.Host(nil): bare metal hosts are missing, platform.baremetal.Hosts: Required value: not enough hosts found (0) to support all the configured ControlPlane replicas (3)]
FATAL failed to load AgentConfig, NMStateConfig, InstallConfig, or AgentHosts
</code>
DEBUG Loading Agent Hosts...
DEBUG failed to load Install Config: failed to load asset: failed to load asset "Install Config": failed to create install config: invalid "install-config.yaml" file: [platform.baremetal.hosts: Invalid value: []*baremetal.Host(nil): bare metal hosts are missing, platform.baremetal.Hosts: Required value: not enough hosts found (0) to support all the configured ControlPlane replicas (3)]
FATAL failed to load AgentConfig, NMStateConfig, InstallConfig, or AgentHosts
while the non-agent version says:
<code>[grant@rockydesktop openshift_install]$ openshift-install --dir=./ wait-for bootstrap-complete --log-level=debug
DEBUG OpenShift Installer 4.16.3
DEBUG Built from commit e1f9f057ce87c1a4a5f3c268812fa4c9dc003cb7
INFO Waiting up to 20m0s (until 12:35PM EDT) for the Kubernetes API at https://api.vmware-cluster.openshift.lan:6443...
DEBUG Loading Agent Config...
DEBUG Loading Agent Config from both state file and target directory
DEBUG Using Agent Config loaded from target directory
WARNING An agent configuration was detected but this command is not the agent wait-for command
DEBUG Still waiting for the Kubernetes API: Get "https://api.vmware-cluster.openshift.lan:6443/version": dial tcp 10.10.25.179:6443: connect: no route to host
<code>[grant@rockydesktop openshift_install]$ openshift-install --dir=./ wait-for bootstrap-complete --log-level=debug
DEBUG OpenShift Installer 4.16.3
DEBUG Built from commit e1f9f057ce87c1a4a5f3c268812fa4c9dc003cb7
INFO Waiting up to 20m0s (until 12:35PM EDT) for the Kubernetes API at https://api.vmware-cluster.openshift.lan:6443...
DEBUG Loading Agent Config...
DEBUG Loading Agent Config from both state file and target directory
DEBUG Using Agent Config loaded from target directory
WARNING An agent configuration was detected but this command is not the agent wait-for command
DEBUG Still waiting for the Kubernetes API: Get "https://api.vmware-cluster.openshift.lan:6443/version": dial tcp 10.10.25.179:6443: connect: no route to host
</code>
[grant@rockydesktop openshift_install]$ openshift-install --dir=./ wait-for bootstrap-complete --log-level=debug
DEBUG OpenShift Installer 4.16.3
DEBUG Built from commit e1f9f057ce87c1a4a5f3c268812fa4c9dc003cb7
INFO Waiting up to 20m0s (until 12:35PM EDT) for the Kubernetes API at https://api.vmware-cluster.openshift.lan:6443...
DEBUG Loading Agent Config...
DEBUG Loading Agent Config from both state file and target directory
DEBUG Using Agent Config loaded from target directory
WARNING An agent configuration was detected but this command is not the agent wait-for command
DEBUG Still waiting for the Kubernetes API: Get "https://api.vmware-cluster.openshift.lan:6443/version": dial tcp 10.10.25.179:6443: connect: no route to host
so it correctly detects I’m not using the agent-based version of the command, but then does actually tell me the install status.
Agent Config
<code>apiVersion: v1alpha1
rendezvousIP: 10.10.25.170 # For a SNO cluster this is the same IP as the server. For a non-SNO install, pick an IP of one of the control plane servers
- name: eno1 # This doesn't need to be the actual interface name, it has to match the names below
macAddress: 00:50:56:8a:39:a0
- name: eno1 # Match the name here
mac-address: 00:50:56:8a:39:a0
next-hop-address: 10.10.25.1
next-hop-interface: eno1 # Match the name here
- name: eno1 # This doesn't need to be the actual interface name, it has to match the names below
macAddress: 00:50:56:8a:f2:5d
- name: eno1 # Match the name here
mac-address: 00:50:56:8a:f2:5d
next-hop-address: 10.10.25.1
next-hop-interface: eno1 # Match the name here
- name: eno1 # This doesn't need to be the actual interface name, it has to match the names below
macAddress: 00:50:56:8a:5c:ec
- name: eno1 # Match the name here
mac-address: 00:50:56:8a:5c:ec
next-hop-address: 10.10.25.1
next-hop-interface: eno1 # Match the name here
- name: eno1 # This doesn't need to be the actual interface name, it has to match the names below
macAddress: 00:50:56:8a:e1:e7
- name: eno1 # Match the name here
mac-address: 00:50:56:8a:e1:e7
next-hop-address: 10.10.25.1
next-hop-interface: eno1 # Match the name here
- name: eno1 # This doesn't need to be the actual interface name, it has to match the names below
macAddress: 00:50:56:8a:44:52
- name: eno1 # Match the name here
mac-address: 00:50:56:8a:44:52
next-hop-address: 10.10.25.1
next-hop-interface: eno1 # Match the name here
- name: eno1 # This doesn't need to be the actual interface name, it has to match the names below
macAddress: 00:50:56:8a:fb:ea
- name: eno1 # Match the name here
mac-address: 00:50:56:8a:fb:ea
next-hop-address: 10.10.25.1
next-hop-interface: eno1 # Match the name here
<code>apiVersion: v1alpha1
kind: AgentConfig
metadata:
name: agent-config
rendezvousIP: 10.10.25.170 # For a SNO cluster this is the same IP as the server. For a non-SNO install, pick an IP of one of the control plane servers
hosts:
- hostname: master-0
interfaces:
- name: eno1 # This doesn't need to be the actual interface name, it has to match the names below
macAddress: 00:50:56:8a:39:a0
networkConfig:
interfaces:
- name: eno1 # Match the name here
type: ethernet
state: up
mac-address: 00:50:56:8a:39:a0
ipv4:
enabled: true
address:
- ip: 10.10.25.170
prefix-length: 24
dhcp: false
mtu: 9000
dns-resolver:
config:
server:
- 10.10.25.120
routes:
config:
- destination: 0.0.0.0/0
next-hop-address: 10.10.25.1
next-hop-interface: eno1 # Match the name here
table-id: 254
#####
- hostname: master-1
interfaces:
- name: eno1 # This doesn't need to be the actual interface name, it has to match the names below
macAddress: 00:50:56:8a:f2:5d
networkConfig:
interfaces:
- name: eno1 # Match the name here
type: ethernet
state: up
mac-address: 00:50:56:8a:f2:5d
ipv4:
enabled: true
address:
- ip: 10.10.25.171
prefix-length: 24
dhcp: false
mtu: 9000
dns-resolver:
config:
server:
- 10.10.25.120
routes:
config:
- destination: 0.0.0.0/0
next-hop-address: 10.10.25.1
next-hop-interface: eno1 # Match the name here
table-id: 254
#####
- hostname: master-2
interfaces:
- name: eno1 # This doesn't need to be the actual interface name, it has to match the names below
macAddress: 00:50:56:8a:5c:ec
networkConfig:
interfaces:
- name: eno1 # Match the name here
type: ethernet
state: up
mac-address: 00:50:56:8a:5c:ec
ipv4:
enabled: true
address:
- ip: 10.10.25.172
prefix-length: 24
dhcp: false
mtu: 9000
dns-resolver:
config:
server:
- 10.10.25.120
routes:
config:
- destination: 0.0.0.0/0
next-hop-address: 10.10.25.1
next-hop-interface: eno1 # Match the name here
table-id: 254
#####
- hostname: worker-0
interfaces:
- name: eno1 # This doesn't need to be the actual interface name, it has to match the names below
macAddress: 00:50:56:8a:e1:e7
networkConfig:
interfaces:
- name: eno1 # Match the name here
type: ethernet
state: up
mac-address: 00:50:56:8a:e1:e7
ipv4:
enabled: true
address:
- ip: 10.10.25.173
prefix-length: 24
dhcp: false
mtu: 9000
dns-resolver:
config:
server:
- 10.10.25.120
routes:
config:
- destination: 0.0.0.0/0
next-hop-address: 10.10.25.1
next-hop-interface: eno1 # Match the name here
table-id: 254
#####
- hostname: worker-1
interfaces:
- name: eno1 # This doesn't need to be the actual interface name, it has to match the names below
macAddress: 00:50:56:8a:44:52
networkConfig:
interfaces:
- name: eno1 # Match the name here
type: ethernet
state: up
mac-address: 00:50:56:8a:44:52
ipv4:
enabled: true
address:
- ip: 10.10.25.174
prefix-length: 24
dhcp: false
mtu: 9000
dns-resolver:
config:
server:
- 10.10.25.120
routes:
config:
- destination: 0.0.0.0/0
next-hop-address: 10.10.25.1
next-hop-interface: eno1 # Match the name here
table-id: 254
#####
- hostname: worker-2
interfaces:
- name: eno1 # This doesn't need to be the actual interface name, it has to match the names below
macAddress: 00:50:56:8a:fb:ea
networkConfig:
interfaces:
- name: eno1 # Match the name here
type: ethernet
state: up
mac-address: 00:50:56:8a:fb:ea
ipv4:
enabled: true
address:
- ip: 10.10.25.175
prefix-length: 24
dhcp: false
mtu: 9000
dns-resolver:
config:
server:
- 10.10.25.120
routes:
config:
- destination: 0.0.0.0/0
next-hop-address: 10.10.25.1
next-hop-interface: eno1 # Match the name here
table-id: 254
</code>
apiVersion: v1alpha1
kind: AgentConfig
metadata:
name: agent-config
rendezvousIP: 10.10.25.170 # For a SNO cluster this is the same IP as the server. For a non-SNO install, pick an IP of one of the control plane servers
hosts:
- hostname: master-0
interfaces:
- name: eno1 # This doesn't need to be the actual interface name, it has to match the names below
macAddress: 00:50:56:8a:39:a0
networkConfig:
interfaces:
- name: eno1 # Match the name here
type: ethernet
state: up
mac-address: 00:50:56:8a:39:a0
ipv4:
enabled: true
address:
- ip: 10.10.25.170
prefix-length: 24
dhcp: false
mtu: 9000
dns-resolver:
config:
server:
- 10.10.25.120
routes:
config:
- destination: 0.0.0.0/0
next-hop-address: 10.10.25.1
next-hop-interface: eno1 # Match the name here
table-id: 254
#####
- hostname: master-1
interfaces:
- name: eno1 # This doesn't need to be the actual interface name, it has to match the names below
macAddress: 00:50:56:8a:f2:5d
networkConfig:
interfaces:
- name: eno1 # Match the name here
type: ethernet
state: up
mac-address: 00:50:56:8a:f2:5d
ipv4:
enabled: true
address:
- ip: 10.10.25.171
prefix-length: 24
dhcp: false
mtu: 9000
dns-resolver:
config:
server:
- 10.10.25.120
routes:
config:
- destination: 0.0.0.0/0
next-hop-address: 10.10.25.1
next-hop-interface: eno1 # Match the name here
table-id: 254
#####
- hostname: master-2
interfaces:
- name: eno1 # This doesn't need to be the actual interface name, it has to match the names below
macAddress: 00:50:56:8a:5c:ec
networkConfig:
interfaces:
- name: eno1 # Match the name here
type: ethernet
state: up
mac-address: 00:50:56:8a:5c:ec
ipv4:
enabled: true
address:
- ip: 10.10.25.172
prefix-length: 24
dhcp: false
mtu: 9000
dns-resolver:
config:
server:
- 10.10.25.120
routes:
config:
- destination: 0.0.0.0/0
next-hop-address: 10.10.25.1
next-hop-interface: eno1 # Match the name here
table-id: 254
#####
- hostname: worker-0
interfaces:
- name: eno1 # This doesn't need to be the actual interface name, it has to match the names below
macAddress: 00:50:56:8a:e1:e7
networkConfig:
interfaces:
- name: eno1 # Match the name here
type: ethernet
state: up
mac-address: 00:50:56:8a:e1:e7
ipv4:
enabled: true
address:
- ip: 10.10.25.173
prefix-length: 24
dhcp: false
mtu: 9000
dns-resolver:
config:
server:
- 10.10.25.120
routes:
config:
- destination: 0.0.0.0/0
next-hop-address: 10.10.25.1
next-hop-interface: eno1 # Match the name here
table-id: 254
#####
- hostname: worker-1
interfaces:
- name: eno1 # This doesn't need to be the actual interface name, it has to match the names below
macAddress: 00:50:56:8a:44:52
networkConfig:
interfaces:
- name: eno1 # Match the name here
type: ethernet
state: up
mac-address: 00:50:56:8a:44:52
ipv4:
enabled: true
address:
- ip: 10.10.25.174
prefix-length: 24
dhcp: false
mtu: 9000
dns-resolver:
config:
server:
- 10.10.25.120
routes:
config:
- destination: 0.0.0.0/0
next-hop-address: 10.10.25.1
next-hop-interface: eno1 # Match the name here
table-id: 254
#####
- hostname: worker-2
interfaces:
- name: eno1 # This doesn't need to be the actual interface name, it has to match the names below
macAddress: 00:50:56:8a:fb:ea
networkConfig:
interfaces:
- name: eno1 # Match the name here
type: ethernet
state: up
mac-address: 00:50:56:8a:fb:ea
ipv4:
enabled: true
address:
- ip: 10.10.25.175
prefix-length: 24
dhcp: false
mtu: 9000
dns-resolver:
config:
server:
- 10.10.25.120
routes:
config:
- destination: 0.0.0.0/0
next-hop-address: 10.10.25.1
next-hop-interface: eno1 # Match the name here
table-id: 254
Install Config
additionalTrustBundlePolicy: Always
baseDomain: openshift.lan
replicas: 3 # Set to 0 for a 3 node cluster. Minimum 2 for a standard cluster
networkType: OVNKubernetes
- cidr: 10.10.25.0/24 # CIDR block of IPs for the physical servers
apiVIPs: # IP for the API. This is not the same IP of any of the bare metal servers but in the same CIDR block as the machine network
ingressVIPs: # IP for the Ingress. This is not the same IP of any of the bare metal servers but in the same CIDR block as the machine network
# Make sure the pull secret has creds for your private repo
# Make sure the additional trust bundle includes the CA of your private registry
#additionalTrustBundle: |
# -----BEGIN CERTIFICATE-----
# -----END CERTIFICATE-----
# -----BEGIN CERTIFICATE-----
# -----END CERTIFICATE-----
# Note: oc mirror will create a file called imageContentSourcePolicy.yaml after pushing images to your private registry
# The mappings below can be found in that file and will already have the right registry name and location
# these are default examples for a command like
# oc-mirror --from <archive_dir> bastion.localdomain:5000
-----BEGIN CERTIFICATE-----
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
-----END CERTIFICATE-----
- grant-staging.openshift.lan:8443/migration-toolkit-virtualization
source: registry.redhat.io/migration-toolkit-virtualization
- grant-staging.openshift.lan:8443/rhel9
source: registry.redhat.io/rhel9
- grant-staging.openshift.lan:8443/openshift4
source: registry.redhat.io/openshift4
- grant-staging.openshift.lan:8443/openshift-serverless-1
source: registry.redhat.io/openshift-serverless-1
- grant-staging.openshift.lan:8443/container-native-virtualization
source: registry.redhat.io/container-native-virtualization
- grant-staging.openshift.lan:8443/ubi8
source: registry.redhat.io/ubi8
- grant-staging.openshift.lan:8443/source-to-image
source: registry.redhat.io/source-to-image
- grant-staging.openshift.lan:8443/rhel8
source: registry.redhat.io/rhel8
- grant-staging.openshift.lan:8443/openshift-serverless-1-tech-preview
source: registry.redhat.io/openshift-serverless-1-tech-preview
- grant-staging.openshift.lan:8443/odf4
source: registry.redhat.io/odf4
- grant-staging.openshift.lan:8443/openshift-service-mesh
source: registry.redhat.io/openshift-service-mesh
- grant-staging.openshift.lan:8443/rhceph
source: registry.redhat.io/rhceph
<code>apiVersion: v1
additionalTrustBundlePolicy: Always
baseDomain: openshift.lan
compute:
- architecture: amd64
hyperthreading: Enabled
name: worker
replicas: 3 # Set to 0 for a 3 node cluster. Minimum 2 for a standard cluster
controlPlane:
architecture: amd64
hyperthreading: Enabled
name: master
replicas: 3
metadata:
name: vmware-cluster
creationTimestamp: null
networking:
clusterNetwork:
- cidr: 10.128.0.0/14
hostPrefix: 23
networkType: OVNKubernetes
serviceNetwork:
- 172.30.0.0/16
machineNetwork:
- cidr: 10.10.25.0/24 # CIDR block of IPs for the physical servers
platform:
baremetal:
apiVIPs: # IP for the API. This is not the same IP of any of the bare metal servers but in the same CIDR block as the machine network
- 10.10.25.179
ingressVIPs: # IP for the Ingress. This is not the same IP of any of the bare metal servers but in the same CIDR block as the machine network
- 10.10.25.178
publish: External
fips: false
# Make sure the pull secret has creds for your private repo
pullSecret: ''
sshKey: ''
# Make sure the additional trust bundle includes the CA of your private registry
#additionalTrustBundle: |
# -----BEGIN CERTIFICATE-----
# -----END CERTIFICATE-----
# -----BEGIN CERTIFICATE-----
# -----END CERTIFICATE-----
# Note: oc mirror will create a file called imageContentSourcePolicy.yaml after pushing images to your private registry
# The mappings below can be found in that file and will already have the right registry name and location
# these are default examples for a command like
# oc-mirror --from <archive_dir> bastion.localdomain:5000
additionalTrustBundle: |
-----BEGIN CERTIFICATE-----
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
HypseWyvTZ2KIiI=
-----END CERTIFICATE-----
imageContentSources:
- mirrors:
- grant-staging.openshift.lan:8443/migration-toolkit-virtualization
source: registry.redhat.io/migration-toolkit-virtualization
- mirrors:
- grant-staging.openshift.lan:8443/rhel9
source: registry.redhat.io/rhel9
- mirrors:
- grant-staging.openshift.lan:8443/openshift4
source: registry.redhat.io/openshift4
- mirrors:
- grant-staging.openshift.lan:8443/openshift-serverless-1
source: registry.redhat.io/openshift-serverless-1
- mirrors:
- grant-staging.openshift.lan:8443/container-native-virtualization
source: registry.redhat.io/container-native-virtualization
- mirrors:
- grant-staging.openshift.lan:8443/ubi8
source: registry.redhat.io/ubi8
- mirrors:
- grant-staging.openshift.lan:8443/source-to-image
source: registry.redhat.io/source-to-image
- mirrors:
- grant-staging.openshift.lan:8443/rhel8
source: registry.redhat.io/rhel8
- mirrors:
- grant-staging.openshift.lan:8443/openshift-serverless-1-tech-preview
source: registry.redhat.io/openshift-serverless-1-tech-preview
- mirrors:
- grant-staging.openshift.lan:8443/odf4
source: registry.redhat.io/odf4
- mirrors:
- grant-staging.openshift.lan:8443/openshift-service-mesh
source: registry.redhat.io/openshift-service-mesh
- mirrors:
- grant-staging.openshift.lan:8443/rhceph
source: registry.redhat.io/rhceph
</code>
apiVersion: v1
additionalTrustBundlePolicy: Always
baseDomain: openshift.lan
compute:
- architecture: amd64
hyperthreading: Enabled
name: worker
replicas: 3 # Set to 0 for a 3 node cluster. Minimum 2 for a standard cluster
controlPlane:
architecture: amd64
hyperthreading: Enabled
name: master
replicas: 3
metadata:
name: vmware-cluster
creationTimestamp: null
networking:
clusterNetwork:
- cidr: 10.128.0.0/14
hostPrefix: 23
networkType: OVNKubernetes
serviceNetwork:
- 172.30.0.0/16
machineNetwork:
- cidr: 10.10.25.0/24 # CIDR block of IPs for the physical servers
platform:
baremetal:
apiVIPs: # IP for the API. This is not the same IP of any of the bare metal servers but in the same CIDR block as the machine network
- 10.10.25.179
ingressVIPs: # IP for the Ingress. This is not the same IP of any of the bare metal servers but in the same CIDR block as the machine network
- 10.10.25.178
publish: External
fips: false
# Make sure the pull secret has creds for your private repo
pullSecret: ''
sshKey: ''
# Make sure the additional trust bundle includes the CA of your private registry
#additionalTrustBundle: |
# -----BEGIN CERTIFICATE-----
# -----END CERTIFICATE-----
# -----BEGIN CERTIFICATE-----
# -----END CERTIFICATE-----
# Note: oc mirror will create a file called imageContentSourcePolicy.yaml after pushing images to your private registry
# The mappings below can be found in that file and will already have the right registry name and location
# these are default examples for a command like
# oc-mirror --from <archive_dir> bastion.localdomain:5000
additionalTrustBundle: |
-----BEGIN CERTIFICATE-----
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
HypseWyvTZ2KIiI=
-----END CERTIFICATE-----
imageContentSources:
- mirrors:
- grant-staging.openshift.lan:8443/migration-toolkit-virtualization
source: registry.redhat.io/migration-toolkit-virtualization
- mirrors:
- grant-staging.openshift.lan:8443/rhel9
source: registry.redhat.io/rhel9
- mirrors:
- grant-staging.openshift.lan:8443/openshift4
source: registry.redhat.io/openshift4
- mirrors:
- grant-staging.openshift.lan:8443/openshift-serverless-1
source: registry.redhat.io/openshift-serverless-1
- mirrors:
- grant-staging.openshift.lan:8443/container-native-virtualization
source: registry.redhat.io/container-native-virtualization
- mirrors:
- grant-staging.openshift.lan:8443/ubi8
source: registry.redhat.io/ubi8
- mirrors:
- grant-staging.openshift.lan:8443/source-to-image
source: registry.redhat.io/source-to-image
- mirrors:
- grant-staging.openshift.lan:8443/rhel8
source: registry.redhat.io/rhel8
- mirrors:
- grant-staging.openshift.lan:8443/openshift-serverless-1-tech-preview
source: registry.redhat.io/openshift-serverless-1-tech-preview
- mirrors:
- grant-staging.openshift.lan:8443/odf4
source: registry.redhat.io/odf4
- mirrors:
- grant-staging.openshift.lan:8443/openshift-service-mesh
source: registry.redhat.io/openshift-service-mesh
- mirrors:
- grant-staging.openshift.lan:8443/rhceph
source: registry.redhat.io/rhceph
I am simulating a bare metal install. I am using VMWare, but doing it as if it were bare metal. I have successfully built a cluster on these VMs using the assisted installer so I know the problem has something to do with my agent-based installer config.
I’m at a bit of a loss as to what it wants as far as troubleshooting. I know all of these IPs are reachable among each other and the docs say the API IP should be something other than the hosts so I have set it to something completely separate; it is the same IP I used in the assisted installer.