Overview
pkg.go.dev - cloud.google.com/go/spanner Notes on issues encountered while using ReadOnlyTransaction.
What happened?
I was writing batch processing code to handle tens of thousands of data records split across multiple requests. The processing using ReadOnlyTransaction was written as follows:
At first glance, the code seemed fine, so I ran the batch process. However, the process stopped after exceeding a certain number of records.
Cause
The Go Spanner client has a session management mechanism. Due to missing transaction termination handling, the session pool was exhausted, causing requests to time out. Internally, it seems the session management mechanism was blocking the execution of ReadOnlyTransaction.
Solution
Modify the code to call the transaction termination process:
Without transaction termination handling, a new session is created for each transaction execution, exhausting the session pool. The process consistently stopped at the same number of records due to hitting the SPANNER_SESSION_POOL_MAX_OPEND limit. Calculations confirmed this.
Countermeasures
Besides thoroughly reading the documentation, tools can help resolve this issue. github.com - gcpug/zagane
Additionally, monitoring the session count of Cloud Spanner via GCP monitoring might be a viable option.