baseplate.clients.redis_cluster

Redis is an in-memory data structure store used where speed is necessary but complexity is beyond simple key-value operations. (If you’re just doing caching, prefer memcached). Redis-py-cluster is a Python client library that supports interacting with Redis when operating in cluster mode.

New in version 2.1.

Example

To integrate redis-py-cluster with your application, add the appropriate client declaration to your context configuration:

baseplate.configure_context(
   app_config,
   {
      ...
      "foo": ClusterRedisClient(),
      ...
   }
)

configure it in your application’s configuration file:

[app:main]

...


# required: what redis instance to connect to
foo.url = redis://localhost:6379/0

# optional: the maximum size of the connection pool
foo.max_connections = 99

# optional: how long to wait for a connection to establish
foo.timeout = 3 seconds

# optional: Whether read requests should be directed to replicas
# as well instead of just the primary
foo.read_from_replicas = true
...

and then use the attached Redis-like object in request:

def my_method(request):
    request.foo.ping()

Configuration

class baseplate.clients.redis_cluster.ClusterRedisClient(**kwargs)[source]

Configure a clustered Redis client.

This is meant to be used with baseplate.Baseplate.configure_context(). See cluster_pool_from_config() for available configuration settings.

baseplate.clients.redis_cluster.cluster_pool_from_config(app_config, prefix='rediscluster.', **kwargs)[source]

Make a ClusterConnectionPool from a configuration dictionary.

The keys useful to cluster_pool_from_config() should be prefixed, e.g. rediscluster.url, rediscluster.max_connections, etc. The prefix argument specifies the prefix used to filter keys. Each key is mapped to a corresponding keyword argument on the rediscluster.ClusterConnectionPool constructor.

Supported keys:

  • url (required): a URL like redis://localhost/0.

  • max_connections: an integer maximum number of connections in the pool

  • max_connections_per_node: Boolean, whether max_connections should be applied

    globally (False) or per node (True).

  • skip_full_coverage_check: Skips the check of cluster-require-full-coverage config, useful for clusters without the CONFIG command (like aws)

  • nodemanager_follow_cluster: Tell the node manager to reuse the last set of nodes it was operating on when intializing.

  • read_from_replicas: (Boolean) Whether the client should send all read queries to

    replicas instead of just the primary

  • timeout: . e.g. 200 milliseconds (Timespan()).

    How long to wait for a connection to become available. Additionally, will set socket_connect_timeout and socket_timeout if they’re not set explicitly.

  • socket_connect_timeout: e.g. 200 milliseconds (Timespan())

    How long to wait for sockets to connect.

  • socket_timeout: e.g. 200 milliseconds (Timespan())

    How long to wait for socket operations.

  • track_key_reads_sample_rate: If greater than zero, which percentage of requests will

    be inspected to keep track of hot key usage within Redis when reading. Every command inspected will result in a write to a sorted set (baseplate-hot-key-tracker-reads) for tracking.

  • track_key_writes_sample_rate: If greater than zero, which percentage of requests will

    be inspected to keep track of hot key usage within Redis when writing. Every command inspected will result in a write to a sorted set (baseplate-hot-key-tracker-reads) for tracking.

Return type

ClusterConnectionPool

Classes

class baseplate.clients.redis_cluster.ClusterRedisContextFactory(connection_pool)[source]

Cluster Redis client context factory.

This factory will attach a MonitoredClusterRedisConnection to an attribute on the RequestContext. When Redis commands are executed via this connection object, they will use connections from the provided rediscluster.ClusterConnectionPool and automatically record diagnostic information. :type connection_pool: ClusterConnectionPool :param connection_pool: A connection pool.

report_runtime_metrics(batch)[source]

Report runtime metrics to the stats system.

Parameters

batch (Client) – A metrics client to report statistics to.

Return type

None

make_object_for_context(name, span)[source]

Return an object that can be added to the context object.

Parameters
  • name (str) – The name assigned to this object on the context.

  • span (Span) – The current span this object is being made for.

Return type

MonitoredClusterRedisConnection

class baseplate.clients.redis_cluster.MonitoredClusterRedisConnection(context_name, server_span, connection_pool, track_key_reads_sample_rate=0, track_key_writes_sample_rate=0)[source]

Cluster Redis connection that collects diagnostic information.

This connection acts like rediscluster.Redis except that all operations are automatically wrapped with diagnostic collection. The interface is the same as that class except for the pipeline() method.

execute_command(*args, **kwargs)[source]

Wrapper for CLUSTERDOWN error handling.

If the cluster reports it is down it is assumed that:
  • connection_pool was disconnected

  • connection_pool was reseted

  • refereh_table_asap set to True

It will try the number of times specified by the config option “self.cluster_down_retry_attempts” which defaults to 3 unless manually configured.

If it reaches the number of times, the command will raises ClusterDownException.

Return type

Any

pipeline(name)[source]

Create a pipeline.

This returns an object on which you can call the standard Redis commands. Execution will be deferred until execute is called. This is useful for saving round trips even in a clustered environment . :type name: str :param name: The name to attach to diagnostics for this pipeline.

Return type

MonitoredClusterRedisPipeline

transaction(*args, **kwargs)[source]

Not currently implemented.

Return type

Any

Runtime Metrics

In addition to request-level metrics reported through spans, this wrapper reports connection pool statistics periodically via the Process-level metrics system. All metrics are tagged with client, the name given to configure_context() when registering this context factory.

The following metrics are reported:

runtime.pool.size

The size limit for the connection pool.

runtime.pool.in_use

How many connections have been established and are currently checked out and being used.

Hot Key Tracking

Optionally, the client can help track key usage across the Redis cluster to help you identify if you have “hot” keys (keys that are read from or written to much more frequently than other keys). This is particularly useful in clusters with noeviction set as the eviction policy, since Redis lacks a built-in mechanism to help you track hot keys in this case.

Since tracking every single key used is expensive, the tracker works by tracking a small percentage or reads and/or writes, which can be configured on your client:

[app:main]

...
# Note that by default the sample rate will be zero for both reads and writes

# optional: Sample keys for 1% of read operations
foo.track_key_reads_sample_rate = 0.01

# optional: Sample keys for 10% of write operations
foo.track_key_writes_sample_rate = 0.01

...

The keys tracked will be written to a sorted set in the Redis cluster itself, which we can query at any time to see what keys are read from or written to more often than others. Keys used for writes will be stored in baseplate-hot-key-tracker-writes and keys used for reads will be stored in baseplate-hot-key-tracker-reads. Here’s an example of how you can query the top 10 keys on each categories with their associated scores:

> ZREVRANGEBYSCORE baseplate-hot-key-tracker-reads +inf 0 WITHSCORES LIMIT 0 10

> ZREVRANGEBYSCORE baseplate-hot-key-tracker-writes +inf 0 WITHSCORES LIMIT 0 10

Note that due to how the sampling works the scores are only meaningful in a relative sense (by comparing one key’s access frequency to others in the list) but can’t be used to make any inferences about key access rate or anything like that.

Both tracker sets have a default TTL of 24 hours, so once they stop being written to (for instance, if key tracking is disabled) they will clean up after themselves in 24 hours, allowing us to start fresh the next time we want to enable key tracking.