1

I am trying to connect to a PostgreSQL database instance in Google Cloud (with a private IP connection) using the SQL Connector library in Go. However, I am encountering the following error:

ERROR Failed to dial instance {"error": "failed to get instance: Refresh error: failed to get instance metadata (connection name = \"<connection-string>\"): Get \"https://sqladmin.googleapis.com/sql/v1beta4/projects/\/instances/postgres/connectSettings?alt=json&prettyPrint=false\": compute: Received 403 `Unable to generate access token; IAM returned 403 Forbidden: Permission 'iam.serviceAccounts.getAccessToken' denied on resource (or it may not exist).
This error could be caused by a missing IAM policy binding on the target IAM service account.
For more information, refer to the Workload Identity documentation:

Environment Details:

Google Cloud: PostgreSQL instance with private IP connection enabled

Service Account Permissions:

  • Cloud SQL Admin
  • Cloud SQL Client
  • Compute Network Admin
  • Project IAM Admin
  • Service Account Admin
  • Service Account Token Creator
  • Storage Admin
  • Workload Identity User

Go Environment: Running the code in a VM instance that is part of the GKE cluster in the same VPC.

Here’s the relevant portion of my Go code:

func connectDB() error {
    user := "postgres"
    password := "password"
    connectionName := "connectionName"
    sslmode := "require"
    dbName := "databaseName"

    config, err := pgx.ParseConfig(connStr)
    if err != nil {
        return err
    }

    opts := append([]cloudsqlconn.Option{},
        cloudsqlconn.WithDefaultDialOptions(cloudsqlconn.WithPrivateIP()),
    )

    d, err := cloudsqlconn.NewDialer(context.Background(), opts...)
    if err != nil {
        return err
    }

    config.DialFunc = func(ctx context.Context, network, instance string) (net.Conn, error) {
        log.FromContext(ctx).Info(fmt.Sprintf("Dialing instance: %s", instance))
        conn, err := d.Dial(ctx, connectionName)
        if err != nil {
            log.FromContext(ctx).Error(err, "Failed to dial instance")
        }
        return conn, err
    }

    dbURI := stdlib.RegisterConnConfig(config)
    dbPool, err := sql.Open("pgx", dbURI)
    if err != nil {
        return err
    }   

    err = dbPool.Ping()
    if err != nil {
        return err
    }   
    return nil
}

Troubleshooting Attempts:

Verified that the database instance has a private IP and is reachable within the same VPC. Checked and ensured that the service account has the following permissions: Cloud SQL Admin Cloud SQL Client Service Account Token Creator Workload Identity User, among others. Ensured that the Go application is running inside a VM that has access to the private VPC.

5
  • Are you using workload identity? Recommended way is workload identity cloud.google.com/kubernetes-engine/docs/how-to/… Commented Nov 22, 2024 at 14:39
  • Also are you following this documentation? cloud.google.com/sql/docs/postgres/… Commented Nov 22, 2024 at 14:48
  • @danyL yes I am using the workload identity and using the documentation only. Commented Nov 23, 2024 at 8:18
  • I wonder if you need to gran the VM service account the Service Account Token Creator role instead of the Workload Identity service account. Commented Nov 28, 2024 at 15:41
  • This may also be useful: googlecloudcommunity.com/gc/Developer-Tools/… Commented Nov 28, 2024 at 15:42

1 Answer 1

0

Notice this part of the error you linked: "Permission 'iam.serviceAccounts.getAccessToken' denied on resource (or it may not exist)". The only two roles that you mentioned that have that particular permissions are

  • Service Account Token Creator
  • Workload Identity User

In the documentation, only workload identity user is mentioned, so I would to try to troubleshoot where that role is mentioned:

  1. Follow this best practice if you did not
  2. Make sure you complete all the steps as per the doc. Tips: make sure you complete all the required steps, 1 to 6 ( from namespace creation to policy to impersonation to annotation). For step 5, do you get

The member name must include the namespace and Kubernetes ServiceAccount name. For example, serviceAccount:example-project.svc.id.goog[example-namespace/example-serviceaccount].

Other troubleshooting steps you can take for Workload Identity Federation for GKE

Check step 5 (Check that the IAM service account is configured correctly)

gcloud iam service-accounts get-iam-policy \
GSA_NAME@GSA_PROJECT.iam.gserviceaccount.com

Expected:

  - members:
  - serviceAccount:PROJECT_ID.svc.id.goog[NAMESPACE/KSA_NAME]
  role: roles/iam.workloadIdentityUser
Sign up to request clarification or add additional context in comments.

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.