Background:
Currently I have an existing GCP VPC-SC Service Perimeter which was created and is managed using Terraform via a Cloud Build Pipeline. In it’s current state, the Terraform Provider version being defined is 4.28, which you can see below.

Where I’m currently facing an issue is in updating that provider version from 4.28 to 5.40.0.

Current Provider:

terraform {
  required_version = "1.2.2"

  required_providers {
    google = {
      source  = "hashicorp/google"
      version = "~> 4.28" #Attempting to update this value to "~> 5.40.0"
    }
  }
}

Steps to Update:

1. Create a Pull Request to update the provider version to 5.40.0. This is the only change being made in the PR.
2. Step 1 kicks off a Cloud Build Pipeline. Within the Pipeline, the following Terraform steps take place.
   • terraform init -input=false -no-color
   • terraform validate -no-color
   • terraform plan -input=false -out="${temp_plan_dir}" -detailed-exitcode -no-color
     • Also tried out -parallelism=5 flag for the Terraform plan

So Terraform init successfully fetches the provider hashicorp/google versions matching "~> 5.40.0". And then Terraform Validate is also successful. It’s only when it gets to the 3rd step where it runs Terraform Plan that issues are faced.

Error: So the issue begins to occur for the google_access_context_manager_service_perimeter resource when it seems to be reading the state for the resource. When it reached this point, the pipeline seems to hang with no actual error being outputted. Instead, it seems to continue with retrying the dag/walk step until inevitably the pipeline hits it’s 20 minute timeout with no error being outputted by Terraform.

Keep in mind, normal updates through the pipeline only take around 5 minutes of run time before the pipeline is complete.

[TRACE] vertex "module.vpcsc.google_access_context_manager_service_perimeter.service_perimeter": entering dynamic subgraph
[TRACE] vertex "module.vpcsc.google_access_context_manager_service_perimeter.service_perimeter["prod_vpcsc_perimeter"]": starting visit (*terraform.NodePlannableResourceInstance)
[TRACE] readResourceInstanceState: reading state for module.vpcsc.google_access_context_manager_service_perimeter.service_perimeter["prod_vpcsc_perimeter"]
[TRACE] upgradeResourceState: schema version of module.vpcsc.google_access_context_manager_service_perimeter.service_perimeter["prod_vpcsc_perimeter"] is still 0; calling provider "google" for any other minor fixups
[TRACE] GRPCProvider: UpgradeResourceState
[TRACE] provider.terraform-provider-google_v5.40.0_x5: Received request: tf_provider_addr=registry.terraform.io/hashicorp/google tf_req_id=12a22123-a11b-3344-c555-defr12ab345 tf_rpc=UpgradeResourceState @caller=github.com/hashicorp/[email protected]/tfprotov5/tf5server/server.go:743 @module=sdk.proto tf_proto_version=5.6 tf_resource_type=google_access_context_manager_service_perimeter timestamp=
[TRACE] provider.terraform-provider-google_v5.40.0_x5: Sending request downstream: tf_rpc=UpgradeResourceState @module=sdk.proto tf_provider_addr=registry.terraform.io/hashicorp/google tf_req_id=12a22123-a11b-3344-c555-defr12ab345 tf_resource_type=google_access_context_manager_service_perimeter @caller=github.com/hashicorp/[email protected]/tfprotov5/internal/tf5serverlogging/downstream_request.go:22 tf_proto_version=5.6 timestamp=
[TRACE] provider.terraform-provider-google_v5.40.0_x5: calling downstream server: @module=sdk.mux tf_mux_provider=*schema.GRPCProviderServer tf_rpc=UpgradeResourceState @caller=github.com/hashicorp/[email protected]/internal/logging/mux.go:19 timestamp=
[TRACE] provider.terraform-provider-google_v5.40.0_x5: Upgrading JSON state: tf_mux_provider=*schema.GRPCProviderServer tf_req_id=12a22123-a11b-3344-c555-defr12ab345 @module=sdk.helper_schema tf_provider_addr=registry.terraform.io/hashicorp/google tf_resource_type=google_access_context_manager_service_perimeter tf_rpc=UpgradeResourceState @caller=github.com/hashicorp/terraform-plugin-sdk/[email protected]/helper/schema/grpc_provider.go:365 timestamp=
[TRACE] provider.terraform-provider-google_v5.40.0_x5: Received downstream response: @module=sdk.proto tf_proto_version=5.6 tf_provider_addr=registry.terraform.io/hashicorp/google tf_req_id=12a22123-a11b-3344-c555-defr12ab345 tf_rpc=UpgradeResourceState diagnostic_error_count=0 diagnostic_warning_count=0 tf_req_duration_ms=402 tf_resource_type=google_access_context_manager_service_perimeter @caller=github.com/hashicorp/[email protected]/tfprotov5/internal/tf5serverlogging/downstream_request.go:42 timestamp=
[TRACE] provider.terraform-provider-google_v5.40.0_x5: Served request: tf_req_id=12a22123-a11b-3344-c555-defr12ab345 tf_rpc=UpgradeResourceState @module=sdk.proto tf_proto_version=5.6 tf_provider_addr=registry.terraform.io/hashicorp/google tf_resource_type=google_access_context_manager_service_perimeter @caller=github.com/hashicorp/[email protected]/tfprotov5/tf5server/server.go:762 timestamp=
[TRACE] NodeAbstractResouceInstance.writeResourceInstanceState to prevRunState for module.vpcsc.google_access_context_manager_service_perimeter.service_perimeter["prod_vpcsc_perimeter"]
[TRACE] NodeAbstractResouceInstance.writeResourceInstanceState: writing state object for module.vpcsc.google_access_context_manager_service_perimeter.service_perimeter["prod_vpcsc_perimeter"]
[TRACE] NodeAbstractResouceInstance.writeResourceInstanceState to refreshState for module.vpcsc.google_access_context_manager_service_perimeter.service_perimeter["prod_vpcsc_perimeter"]
[TRACE] NodeAbstractResouceInstance.writeResourceInstanceState: writing state object for module.vpcsc.google_access_context_manager_service_perimeter.service_perimeter["prod_vpcsc_perimeter"]
module.vpcsc.google_access_context_manager_service_perimeter.service_perimeter["prod_vpcsc_perimeter"]: Refreshing state... [id=accessPolicies/123456543212/servicePerimeters/prod_vpcsc_perimeter]
[TRACE] NodeAbstractResourceInstance.refresh for module.vpcsc.google_access_context_manager_service_perimeter.service_perimeter["prod_vpcsc_perimeter"]
[TRACE] GRPCProvider: ReadResource
[TRACE] provider.terraform-provider-google_v5.40.0_x5: Received request: @caller=github.com/hashicorp/[email protected]/tfprotov5/tf5server/server.go:771 @module=sdk.proto tf_proto_version=5.6 tf_provider_addr=registry.terraform.io/hashicorp/google tf_req_id=34b33123-c11f-4345-g656-fgfe23bc456 tf_resource_type=google_access_context_manager_service_perimeter tf_rpc=ReadResource timestamp=
[TRACE] provider.terraform-provider-google_v5.40.0_x5: No announced client capabilities: tf_proto_version=5.6 tf_rpc=ReadResource @module=sdk.proto tf_provider_addr=registry.terraform.io/hashicorp/google tf_req_id=34b33123-c11f-4345-g656-fgfe23bc456 tf_resource_type=google_access_context_manager_service_perimeter @caller=github.com/hashicorp/[email protected]/tfprotov5/internal/tf5serverlogging/client_capabilities.go:44 timestamp=
[TRACE] provider.terraform-provider-google_v5.40.0_x5: Sending request downstream: tf_rpc=ReadResource @caller=github.com/hashicorp/[email protected]/tfprotov5/internal/tf5serverlogging/downstream_request.go:22 @module=sdk.proto tf_provider_addr=registry.terraform.io/hashicorp/google tf_resource_type=google_access_context_manager_service_perimeter tf_proto_version=5.6 tf_req_id=34b33123-c11f-4345-g656-fgfe23bc456 timestamp=
[TRACE] provider.terraform-provider-google_v5.40.0_x5: calling downstream server: @module=sdk.mux tf_mux_provider=*schema.GRPCProviderServer tf_rpc=ReadResource @caller=github.com/hashicorp/[email protected]/internal/logging/mux.go:19 timestamp=
[TRACE] dag/walk: vertex "provider["registry.terraform.io/hashicorp/google"] (close)" is waiting for "module.vpcsc.google_access_context_manager_service_perimeter.service_perimeter (expand)"
[TRACE] provider.terraform-provider-google_v5.40.0_x5: Calling downstream: tf_rpc=ReadResource @caller=github.com/hashicorp/terraform-plugin-sdk/[email protected]/helper/schema/resource.go:1088 @module=sdk.helper_schema tf_mux_provider=*schema.GRPCProviderServer tf_req_id=34b33123-c11f-4345-g656-fgfe23bc456 tf_provider_addr=registry.terraform.io/hashicorp/google tf_resource_type=google_access_context_manager_service_perimeter timestamp=
[DEBUG] provider.terraform-provider-google_v5.40.0_x5: 2024/09/05 20:07:09 [DEBUG] Waiting for state to become: [success]
[DEBUG] provider.terraform-provider-google_v5.40.0_x5: 2024/09/05 20:07:09 [DEBUG] Retry Transport: starting RoundTrip retry loop
[DEBUG] provider.terraform-provider-google_v5.40.0_x5: 2024/09/05 20:07:09 [DEBUG] Retry Transport: request attempt 0
[DEBUG] provider.terraform-provider-google_v5.40.0_x5: 2024/09/05 20:07:09 [DEBUG] Google API Request Details:
[DEBUG] provider.terraform-provider-google_v5.40.0_x5: ---[ REQUEST ]---------------------------------------
[DEBUG] provider.terraform-provider-google_v5.40.0_x5: GET /v1/accessPolicies/123456543212/servicePerimeters/prod_vpcsc_perimeter?alt=json HTTP/1.1
[DEBUG] provider.terraform-provider-google_v5.40.0_x5: Host: accesscontextmanager.googleapis.com
[DEBUG] provider.terraform-provider-google_v5.40.0_x5: User-Agent: Terraform/1.2.2 (+https://www.terraform.io) Terraform-Plugin-SDK/2.33.0 terraform-provider-google/5.40.0
[DEBUG] provider.terraform-provider-google_v5.40.0_x5: Content-Type: application/json
[DEBUG] provider.terraform-provider-google_v5.40.0_x5: Accept-Encoding: gzip
[DEBUG] provider.terraform-provider-google_v5.40.0_x5: 
[DEBUG] provider.terraform-provider-google_v5.40.0_x5: 
[DEBUG] provider.terraform-provider-google_v5.40.0_x5: -----------------------------------------------------

[TRACE] dag/walk: vertex "module.vpcsc (close)" is waiting for "module.vpcsc.google_access_context_manager_service_perimeter.service_perimeter (expand)"
[TRACE] dag/walk: vertex "root" is waiting for "module.vpcsc (close)"
[DEBUG] provider.terraform-provider-google_v5.40.0_x5: 2024/09/05 20:07:09 [DEBUG] Google API Response Details:
[DEBUG] provider.terraform-provider-google_v5.40.0_x5: ---[ RESPONSE ]--------------------------------------
[DEBUG] provider.terraform-provider-google_v5.40.0_x5: HTTP/2.0 200 OK
[DEBUG] provider.terraform-provider-google_v5.40.0_x5: Cache-Control: private
[DEBUG] provider.terraform-provider-google_v5.40.0_x5: Content-Type: application/json; charset=UTF-8
[DEBUG] provider.terraform-provider-google_v5.40.0_x5: Date: Thu, 05 Sep 2024 20:07:09 GMT
[DEBUG] provider.terraform-provider-google_v5.40.0_x5: Server: ESF
[DEBUG] provider.terraform-provider-google_v5.40.0_x5: Vary: Origin
[DEBUG] provider.terraform-provider-google_v5.40.0_x5: Vary: X-Origin
[DEBUG] provider.terraform-provider-google_v5.40.0_x5: Vary: Referer
[DEBUG] provider.terraform-provider-google_v5.40.0_x5: X-Content-Type-Options: nosniff
[DEBUG] provider.terraform-provider-google_v5.40.0_x5: X-Frame-Options: SAMEORIGIN
[DEBUG] provider.terraform-provider-google_v5.40.0_x5: X-Xss-Protection: 0
[DEBUG] provider.terraform-provider-google_v5.40.0_x5: 
[DEBUG] provider.terraform-provider-google_v5.40.0_x5: {
[DEBUG] provider.terraform-provider-google_v5.40.0_x5:   "name": "accessPolicies/123456543212/servicePerimeters/prod_vpcsc_perimeter",
[DEBUG] provider.terraform-provider-google_v5.40.0_x5:   "title": "prod_vpcsc_perimeter",
[DEBUG] provider.terraform-provider-google_v5.40.0_x5:   "status": {
### Continues and displays entire perimeter                     #THE ENTIRE PERIMETER OUTPUTS HERE
[DEBUG] provider.terraform-provider-google_v5.40.0_x5:   "useExplicitDryRunSpec": true
[DEBUG] provider.terraform-provider-google_v5.40.0_x5: }
[DEBUG] provider.terraform-provider-google_v5.40.0_x5: 
[DEBUG] provider.terraform-provider-google_v5.40.0_x5: -----------------------------------------------------
[DEBUG] provider.terraform-provider-google_v5.40.0_x5: 2024/09/05 20:07:09 [DEBUG] Retry Transport: Stopping retries, last request was successful
[DEBUG] provider.terraform-provider-google_v5.40.0_x5: 2024/09/05 20:07:09 [DEBUG] Retry Transport: Returning after 1 attempts
[TRACE] provider.terraform-provider-google_v5.40.0_x5: Called downstream: @caller=github.com/hashicorp/terraform-plugin-sdk/[email protected]/helper/schema/resource.go:1090 @module=sdk.helper_schema tf_mux_provider=*schema.GRPCProviderServer tf_provider_addr=registry.terraform.io/hashicorp/google tf_resource_type=google_access_context_manager_service_perimeter tf_rpc=ReadResource tf_req_id=34b33123-c11f-4345-g656-fgfe23bc456 timestamp=
[TRACE] provider.terraform-provider-google_v5.40.0_x5: Received downstream response: diagnostic_error_count=0 tf_provider_addr=registry.terraform.io/hashicorp/google @caller=github.com/hashicorp/[email protected]/tfprotov5/internal/tf5serverlogging/downstream_request.go:42 diagnostic_warning_count=0 tf_proto_version=5.6 tf_req_duration_ms=3468 tf_req_id=34b33123-c11f-4345-g656-fgfe23bc456 tf_resource_type=google_access_context_manager_service_perimeter tf_rpc=ReadResource @module=sdk.proto timestamp=
[TRACE] provider.terraform-provider-google_v5.40.0_x5: Served request: tf_resource_type=google_access_context_manager_service_perimeter tf_rpc=ReadResource @module=sdk.proto tf_proto_version=5.6 tf_provider_addr=registry.terraform.io/hashicorp/google tf_req_id=34b33123-c11f-4345-g656-fgfe23bc456 @caller=github.com/hashicorp/[email protected]/tfprotov5/tf5server/server.go:802 timestamp=
[TRACE] dag/walk: vertex "provider["registry.terraform.io/hashicorp/google"] (close)" is waiting for "module.vpcsc.google_access_context_manager_service_perimeter.service_perimeter (expand)"
[TRACE] dag/walk: vertex "module.vpcsc (close)" is waiting for "module.vpcsc.google_access_context_manager_service_perimeter.service_perimeter (expand)"
[TRACE] dag/walk: vertex "root" is waiting for "module.vpcsc (close)"
[TRACE] dag/walk: vertex "provider["registry.terraform.io/hashicorp/google"] (close)" is waiting for "module.vpcsc.google_access_context_manager_service_perimeter.service_perimeter (expand)"
[TRACE] dag/walk: vertex "module.vpcsc (close)" is waiting for "module.vpcsc.google_access_context_manager_service_perimeter.service_perimeter (expand)"
[TRACE] dag/walk: vertex "root" is waiting for "module.vpcsc (close)"
[TRACE] dag/walk: vertex "provider["registry.terraform.io/hashicorp/google"] (close)" is waiting for "module.vpcsc.google_access_context_manager_service_perimeter.service_perimeter (expand)"
[TRACE] dag/walk: vertex "module.vpcsc (close)" is waiting for "module.vpcsc.google_access_context_manager_service_perimeter.service_perimeter (expand)"
[TRACE] dag/walk: vertex "root" is waiting for "module.vpcsc (close)"

The dag/walk items above continue to be outputted over and over again until the pipeline ends up timing out.

Additional Details:

The weird thing with this is that locally, as well as within a separate environments pipeline, we face no issues updating the provider. It only seems to occur with this specific resource. The main difference though between those environments and this one is that the Service Perimeter within this environment which is facing issues is pretty large with a large number of Ingress/Egress Policies as well as projects contained within the perimeter.

Also, I’ve tested updating the provider first by only updating the provider. And a second time by updating my Terraform Resource/Module to include new variables released in the newer version in addition to bumping the Provider Version. Both faced the same issue.

I already have one perimeter that is created using the updated Provider 5.40.0 and it faced no issues. So I know it’s possible to use that provider. Just happens to be the case where attempting to update it for an existing resource it isn’t working.

Question: Has anyone by chance faced an error similar to this? And if so, what was the resolution to resolving the issue? Would this be an issue with the provider itself? Could this be an issue with the pipeline itself such as DNS issue? Could the pipeline be running out of resources?

Resource References:

• Terraform Service Perimeter Resource
• Hashicorp Google Terraform Provider