Overview

Notes on insights gathered about Cloud Spanner. These are rough notes and not categorized.

Notes

• Guaranteed up to 99.999% availability with no downtime for planned maintenance or schema changes
  • Cloud SQL involves planned maintenance with downtime
• Cloud Spanner architecture
  • Client
    • Servers or containers where R/W code is executed
    • Using client libraries allows communication via gRPC or REST API
  • Node
    • Compute resources executing R/W operations
    • Nodes do not have attached local storage (disks)
    • Data is stored in a distributed file system (internally named Colossus)
      • Colossus is a cluster-level file system, successor to Google File System
        • Used as a storage foundation for various services operated by Google
  • Split
    • Unit of data in Cloud Spanner
    • Managed by specific nodes
    • Capacity limit is 4GB; exceeding this generates a new Split, dividing the data
• Replication
  • Read-write replicas
  • Read-only replicas
  • Witness replicas
    • Replicas that do not hold a complete copy of the database, only participate in voting
• For read-only sections, using Read-Only Transactions is recommended. They can only read but do not lock, so they do not block other transactions. Ensure idempotency within Read-Only Transactions to handle retries.
• Even if a hotspot occurs, performance equivalent to 1 Node is achieved. Thus, when running on 1 Node in development/testing environments, hotspots may go unnoticed. Be cautious during load testing.
• Performance is achieved not only by increasing/decreasing Nodes but also by generating Splits that distribute data.
  • Splits are generated by load-based split (increased load from writes/reads) and size-based split (increased data volume)
  • Keep this in mind when wanting to warm up
• Indexes are stored in tables like regular tables
  • Indexes can also be interleaved
• Index creation can take time depending on the number of nodes and data volume, so be cautious (most efficient during table creation)
• If data is expected to concentrate on specific key values, it might be better not to create an index from the start (potential for hotspots)
• When adding an index to an existing table with many records, be mindful of the number of indexes added per day (documents suggest about 3 is good)
  • Adding too many indexes can increase Spanner's system load, potentially degrading performance
• Using the STORING clause allows additional columns to be stored in the index
  • Particularly effective when there are many references. Be cautious as high write volume increases write costs
• Whether to use FORCE_INDEX or a covering index should be determined by referring to the query execution plan
• The order of index columns should match the desired retrieval order to avoid unnecessary disk seeks
• Interleaving is possible up to 6 levels
• Interleaving of child tables that are interleaved cannot be removed
• Counting mutations
  • Number of Columns*rows included in a transaction
    • Mutations per commit (including indexes) : 20000
  • Be cautious of mutation limits
  • Google Cloud Spanner Deep Dive for reference

Impressions

If the design is wrong, Spanner's scalability cannot be utilized, and sufficient performance cannot be achieved. The lack of downtime for planned maintenance or schema changes is a significant operational advantage. To maximize performance during operation, be mindful of data distribution with the mindset of a Split. Since I've been mostly looking at Japanese articles, I should probably check more overseas articles too.

References

• Facing various Anomalies with Cloud Spanner
• Impressions after using Spanner in production
• github.com - gcpug/nouhau
• GCPUG Summary "Key Points When Using Secondary Indexes in Cloud Spanner"
• Options for Large Schema Updates
• Overview and Key Design Points of Cloud Spanner
• High-Level Architecture Explanation of Cloud Spanner
• Behind the Scenes of Colossus: Google's Scalable Storage System
• Explaining Spanner Turned into a Lecture
• Current State of NewSQL as of 2020
• Understanding Cloud Spanner's Multi-Region Configuration
• Days of Battle with Spanner at Merpay
• Gained Insights from Adding an Index to a Production Spanner Without Downtime Over 57 Hours
• Google Cloud Spanner Deep Dive
• Practical Cloud Spanner Design Course