I’ve been setting up SeaweedFS on a cluster of three nodes and encountered issues when configuring the S3 gateway. The S3 gateway tries to connect to the incorrect gRPC port 28888 instead of the configured 18888. Below are the steps I have followed and the commands I’ve used:

Commands Used:

1. Initial Setup of SeaweedFS:

• Start Master Servers on Each Node:

nohup ./weed master -mdir=./data/seaweedfs/master -peers=192.168.2.225:9333,192.168.2.224:9333,192.168.2.222:9333 -ip=192.168.2.222 > /var/log/seaweedfs_master.log 2>&1 &
nohup ./weed master -mdir=./data/seaweedfs/master -peers=192.168.2.225:9333,192.168.2.224:9333,192.168.2.222:9333 -ip=192.168.2.224 > /var/log/seaweedfs_master.log 2>&1 &
nohup ./weed master -mdir=./data/seaweedfs/master -peers=192.168.2.225:9333,192.168.2.224:9333,192.168.2.222:9333 -ip=192.168.2.225 > /var/log/seaweedfs_master.log 2>&1 &

• Check Master Logs:

tail -f /var/log/seaweedfs_master.log

2. Start Volume Servers on Each Node:

mkdir ./data/seaweedfs/volume

nohup ./weed volume -mserver=192.168.2.225:9333,192.168.2.224:9333,192.168.2.222:9333 -dataCenter=pve -rack=rack1 -dir=./data/seaweedfs/volume -ip=192.168.2.222 -max=0 > /var/log/seaweedfs_volume.log 2>&1 &
nohup ./weed volume -mserver=192.168.2.225:9333,192.168.2.224:9333,192.168.2.222:9333 -dataCenter=pve -rack=rack1 -dir=./data/seaweedfs/volume -ip=192.168.2.224 -max=0 > /var/log/seaweedfs_volume.log 2>&1 &
nohup ./weed volume -mserver=192.168.2.225:9333,192.168.2.224:9333,192.168.2.222:9333 -dataCenter=pve -rack=rack1 -dir=./data/seaweedfs/volume -ip=192.168.2.225 -max=0 > /var/log/seaweedfs_volume.log 2>&1 &

• Check Volume Logs:

tail -f /var/log/seaweedfs_volume.log

3. Start Filer on Each Node:

nohup ./weed filer -port=8888 -port.grpc=18888 -defaultStoreDir=./data/seaweedfs/filer -master=192.168.2.225:9333,192.168.2.224:9333,192.168.2.222:9333 > /var/log/seaweedfs_filer.log 2>&1 &

• Check Filer Logs:

tail -f /var/log/seaweedfs_filer.log

4. Start S3 Gateway:

nohup ./weed s3 -filer=192.168.2.222:18888,192.168.2.224:18888,192.168.2.225:18888 -port=8333 -domainName=s3.pve.local -config=/etc/seaweedfs/s3config.json > /var/log/seaweedfs_s3.log 2>&1 &

• Check S3 Gateway Logs:

tail -f /var/log/seaweedfs_s3.log

Issues Encountered:

1. S3 Gateway Logs:

Despite specifying gRPC port 18888, the S3 gateway attempts to connect to 28888:

I0609 20:17:43.350912 s3.go:206 wait to connect to filer 192.168.2.222:18888,192.168.2.224:18888,192.168.2.225:18888 grpc address [192.168.2.222:18888,192.168.2.224:18888,192.168.2.225]:28888
...

2. Verification Steps:

• Checked Filers:
  All filers are listening on the correct ports 8888 and 18888 as verified by netstat.

  netstat -tuln | grep 8888
  netstat -tuln | grep 18888
  

• Verified Connectivity:
  Connectivity to gRPC ports was confirmed using telnet.

  telnet 192.168.2.222 18888
  telnet 192.168.2.224 18888
  telnet 192.168.2.225 18888
  

Questions:

1. Why is the S3 gateway trying to connect to the wrong gRPC port (28888) instead of the specified port (18888)?

2. Are there any additional configurations or flags required to ensure the S3 gateway uses the correct gRPC port?

3. How can I troubleshoot and fix this issue to make the S3 gateway connect properly to the filers?

Additional Information:

• s3config.json:

{
  "identities": [
    {
      "name": "admin",
      "credentials": [
        {
          "accessKey": "admin",
          "secretKey": "admin"
        }
      ],
      "actions": ["*"],
      "allowed_buckets": ["*"]
    }
  ],
  "bucket_auto_create": true
}

Any insights or suggestions would be greatly appreciated!