Overview
I made some notes about the issues I encountered while using ReadOnlyTransaction from pkg.go.dev - cloud.google.com/go/spanner.
What Happened?
I was writing batch processing code that handles tens of thousands of records split across multiple requests. I wrote the processing using ReadOnlyTransaction as follows:
for {
// 〜略〜
// c is *spanner.Client
iter := c.ReadOnlyTransaction().Query(ctx, stmt)
defer iter.Stop()
// 〜略〜
}
At first glance, it seemed fine, so I ran the batch process, but a problem occurred where processing would stop after exceeding a certain number of records.
Cause
The Spanner Go client has a session management mechanism, but due to a missing transaction termination process, the session pool was exhausted, causing requests to time out. Internally, it seemed that the session management mechanism was blocking the execution of ReadOnlyTransaction.
Solution
I changed it to call the transaction termination process:
for {
// 〜略〜
// c is *spanner.Client
tx := c.ReadOnlyTransaction()
defer tx.Close()
iter := tx.Query(ctx, stmt)
defer iter.Stop()
// 〜略〜
}
Without the transaction termination process, a new session is generated every time a transaction is executed, leading to exhaustion of the session pool. The reason processing stopped at the same record count each time was likely due to hitting the limit of SPANNER_SESSION_POOL_MAX_OPEND. When calculated, it adds up.
Countermeasures
Aside from reading the documentation properly, there are also tool-based solutions available. github.com - gcpug/zagane
Additionally, monitoring the cloudspanner session count in GCP might also be a good idea.
