baseplate.clients.redis_cluster
¶
Redis is an in-memory data structure store used where speed is necessary but complexity is beyond simple key-value operations. (If you’re just doing caching, prefer memcached). Redis-py-cluster is a Python client library that supports interacting with Redis when operating in cluster mode.
New in version 2.1.
Example¶
To integrate redis-py-cluster with your application, add the appropriate client declaration to your context configuration:
baseplate.configure_context(
app_config,
{
...
"foo": ClusterRedisClient(),
...
}
)
configure it in your application’s configuration file:
[app:main]
...
# required: what redis instance to connect to
foo.url = redis://localhost:6379/0
# optional: the maximum size of the connection pool
foo.max_connections = 99
# optional: how long to wait for a connection to establish
foo.timeout = 3 seconds
# optional: Whether read requests should be directed to replicas
# as well instead of just the primary
foo.read_from_replicas = true
...
and then use the attached Redis
-like object in
request:
def my_method(request):
request.foo.ping()
Configuration¶
-
class
baseplate.clients.redis_cluster.
ClusterRedisClient
(**kwargs)[source]¶ Configure a clustered Redis client.
This is meant to be used with
baseplate.Baseplate.configure_context()
. Seecluster_pool_from_config()
for available configuration settings.
-
baseplate.clients.redis_cluster.
cluster_pool_from_config
(app_config, prefix='rediscluster.', **kwargs)[source]¶ Make a ClusterConnectionPool from a configuration dictionary.
The keys useful to
cluster_pool_from_config()
should be prefixed, e.g.rediscluster.url
,rediscluster.max_connections
, etc. Theprefix
argument specifies the prefix used to filter keys. Each key is mapped to a corresponding keyword argument on therediscluster.ClusterConnectionPool
constructor. Supported keys: *url
(required): a URL likeredis://localhost/0
. *max_connections
: an integer maximum number of connections in the pool *max_connections_per_node
: Boolean, whether max_connections should be appliedglobally (False) or per node (True).
skip_full_coverage_check
: Skips the check of cluster-require-full-coverage config, useful for clusters without the CONFIG command (like aws)nodemanager_follow_cluster
: Tell the node manager to reuse the last set of nodes it was operating on when intializing.read_from_replicas
: (Boolean) Whether the client should send all read queries toreplicas instead of just the primary
timeout
: how long to wait for sockets to connect. e.g.200 milliseconds
(Timespan()
)
track_key_reads_sample_rate
: If greater than zero, which percentage of requests willbe inspected to keep track of hot key usage within Redis when reading. Every command inspected will result in a write to a sorted set (baseplate-hot-key-tracker-reads) for tracking.
track_key_writes_sample_rate
: If greater than zero, which percentage of requests willbe inspected to keep track of hot key usage within Redis when writing. Every command inspected will result in a write to a sorted set (baseplate-hot-key-tracker-reads) for tracking.
- Return type
ClusterConnectionPool
Classes¶
-
class
baseplate.clients.redis_cluster.
ClusterRedisContextFactory
(connection_pool)[source]¶ Cluster Redis client context factory.
This factory will attach a
MonitoredClusterRedisConnection
to an attribute on theRequestContext
. When Redis commands are executed via this connection object, they will use connections from the providedrediscluster.ClusterConnectionPool
and automatically record diagnostic information. :type connection_pool:ClusterConnectionPool
:param connection_pool: A connection pool.
-
class
baseplate.clients.redis_cluster.
MonitoredClusterRedisConnection
(context_name, server_span, connection_pool, track_key_reads_sample_rate=0, track_key_writes_sample_rate=0)[source]¶ Cluster Redis connection that collects diagnostic information.
This connection acts like
rediscluster.Redis
except that all operations are automatically wrapped with diagnostic collection. The interface is the same as that class except for thepipeline()
method.-
execute_command
(*args, **kwargs)[source]¶ Wrapper for CLUSTERDOWN error handling.
- If the cluster reports it is down it is assumed that:
connection_pool was disconnected
connection_pool was reseted
refereh_table_asap set to True
It will try the number of times specified by the config option “self.cluster_down_retry_attempts” which defaults to 3 unless manually configured.
If it reaches the number of times, the command will raises ClusterDownException.
- Return type
-
pipeline
(name)[source]¶ Create a pipeline.
This returns an object on which you can call the standard Redis commands. Execution will be deferred until
execute
is called. This is useful for saving round trips even in a clustered environment . :type name:str
:param name: The name to attach to diagnostics for this pipeline.- Return type
MonitoredClusterRedisPipeline
-
Runtime Metrics¶
In addition to request-level metrics reported through spans, this wrapper
reports connection pool statistics periodically via the Process-level metrics
system. All metrics are tagged with client
, the name given to
configure_context()
when registering this context
factory.
The following metrics are reported:
runtime.pool.size
The size limit for the connection pool.
runtime.pool.in_use
How many connections have been established and are currently checked out and being used.
Hot Key Tracking¶
Optionally, the client can help track key usage across the Redis cluster to
help you identify if you have “hot” keys (keys that are read from or
written to much more frequently than other keys). This is particularly useful
in clusters with noeviction
set as the eviction policy, since Redis
lacks a built-in mechanism to help you track hot keys in this case.
Since tracking every single key used is expensive, the tracker works by tracking a small percentage or reads and/or writes, which can be configured on your client:
[app:main]
...
# Note that by default the sample rate will be zero for both reads and writes
# optional: Sample keys for 1% of read operations
foo.track_key_reads_sample_rate = 0.01
# optional: Sample keys for 10% of write operations
foo.track_key_writes_sample_rate = 0.01
...
The keys tracked will be written to a sorted set in the Redis cluster itself, which we can query at any time to see what keys are read from or written to more often than others. Keys used for writes will be stored in baseplate-hot-key-tracker-writes and keys used for reads will be stored in baseplate-hot-key-tracker-reads. Here’s an example of how you can query the top 10 keys on each categories with their associated scores:
> ZREVRANGEBYSCORE baseplate-hot-key-tracker-reads +inf 0 WITHSCORES LIMIT 0 10
> ZREVRANGEBYSCORE baseplate-hot-key-tracker-writes +inf 0 WITHSCORES LIMIT 0 10
Note that due to how the sampling works the scores are only meaningful in a relative sense (by comparing one key’s access frequency to others in the list) but can’t be used to make any inferences about key access rate or anything like that.
Both tracker sets have a default TTL of 24 hours, so once they stop being written to (for instance, if key tracking is disabled) they will clean up after themselves in 24 hours, allowing us to start fresh the next time we want to enable key tracking.