baseplate-serve
¶
Baseplate comes with a simple Gevent-based server for both Thrift and WSGI
applications called baseplate-serve
.
Configuration¶
There is one required parameter on the command line, the path to an INI-format
configuration file. There should be two sections in the file: the server
section and the app
section. The section headers look like server:main
or app:main
where the part before the :
is the type of section and the
part after is the “name”. Baseplate looks for sections named main
by
default but can be overridden with the --server-name
and --app-name
options.
Shell-like environment variable references in configuration values will be
expanded at server startup. For example, foo = $MY_PATH
or foo =
${MY_PATH}
will result in the application seeing the value from the
$MY_PATH
environment variable as the value of the foo
setting as if it
had been written in the config file directly.
The Server¶
Here’s an example of a server
section:
[server:main]
factory = baseplate.server.thrift
stop_timeout = 30
The factory
tells baseplate what code to use to run the server. Baseplate
comes with two servers built in:
baseplate.server.thrift
A Gevent Thrift server.
baseplate.server.wsgi
A Gevent WSGI server.
Both take two configuration values as well:
max_concurrency
The maximum number of simultaneous clients the server will handle. Note that this is how many connections will be accepted, but some of those connections may be idle at any given time.
stop_timeout
(Optional) How long, in seconds, to wait for active connections to finish up gracefully when shutting down. By default, the server will shut down immediately.
The WSGI server takes an additional optional parameter:
handler
A full name of a class which subclasses
gevent.pywsgi.WSGIHandler
for extra functionality.
There are some additional configuration settings in this section that start
with a monitoring
prefix. For more information on those, see Process-level
metrics.
The Application¶
And now the real bread and butter, your app
section:
[app:main]
factory = my_app.processor:make_processor
foo = 3
bar = 22
noodles.blah = one, two, three
The app
section also takes a factory
. This should be the name of a
callable in your code which builds and returns your application. The part
before the :
is a Python module. The part after the :
is the name of a
callable object within that module.
The rest of the options in the app
section of the configuration file get
passed as a dictionary to your application callable. You can parse these
options with baseplate.lib.config
.
The application factory should return an appropriate object for your server:
- Thrift
A
TProcessor
.- WSGI
A WSGI callable.
Logging¶
The baseplate server provides a default configuration for the Python standard
logging
system. The root logger will print to stdout
with a format that
includes trace information. The default log level is INFO
or DEBUG
if
the --debug
flag is passed to baseplate-serve
.
If more complex logging configuration is necessary, the configuration file will override the default setup. The Configuration file format is documented in the standard library.
Automatic reload on source changes¶
In development, it’s useful for the server to restart itself when you change
code. You can do this by passing the --reload
flag to baseplate-serve
.
This should not be used in production environments.
Einhorn¶
baseplate-serve
can run as a worker in Stripe’s Einhorn socket manager.
This allows Einhorn to handle binding the socket, worker management, rolling
restarts, and worker health checks.
Baseplate supports Einhorn’s “manual ACK” protocol. Once the application is loaded and ready to serve, Baseplate notifies the Einhorn master process via its command socket.
An example command line:
einhorn -m manual -n 4 --bind localhost:9190 \
baseplate-serve myapp.ini
Graceful shutdown¶
The flow of graceful shutdown while handling live traffic looks like this:
The server receives a
SIGTERM
from the infrastructure.The server sets
baseplate.server.SERVER_STATE.shutting_down
toTrue
.If the
drain_time
setting is set in the server configuration, the server will wait the specified amount of time before continuing to the next step. This gives your application a chance to use theshutting_down
flag in healthcheck responses.The server begins graceful shutdown. No new connections will be accepted. The server will continue processing the existing in-flight requests until they are all done or
stop_timeout
time has elapsed.The server exits and lets the infrastructure clean up.
During the period between receiving the SIGTERM
and the server exiting, the
application may still be routed new requests. To ensure requests aren’t lost
during the graceful shutdown (where they won’t be listened for) your
application should set an appropriate drain_time
and use the
SERVER_STATE.shutting_down
flag to fail READINESS
healthchecks.
For example:
def is_healthy(self, context, healthcheck_request):
if healthcheck_request.probe == IsHealthyProbe.LIVENESS:
return True
elif healthcheck_request.probe == IsHealthyProbe.READINESS:
if SERVER_STATE.shutting_down:
return False
return True
return True
Debug Signal¶
Applications running under baseplate-serve
will respond to SIGUSR1
by
printing a stack trace to the logger. This can be useful for debugging
deadlocks and other issues.
Note that Einhorn will exit if you send it a SIGUSR1
. You can instead open up
einhornsh
and instruct the master to send the signal to all workers:
$ einhornsh
> signal SIGUSR1
Successfully sent USR1s to 4 processes: [...]
Prometheus Exporter¶
When enabled, baseplate-serve
will start up a Prometheus exporter on HTTP
port 6060. The exporter is designed for multiprocess use, like if running under
Einhorn. Each worker process writes its metrics to a file on disk and runs its
own copy of the exporter, all listening on the same port. Any given worker can
serve the metrics for all workers from the same pod. It doesn’t matter which
one is hit by the Prometheus collector.
To enable the exporter, install the prometheus-client
package from PyPI.
$ pip install prometheus-client
The PROMETHEUS_MULTIPROC_DIR
environment variable must be set to the path
to an extant writeable directory. This is handled automatically in the official
base Docker images.
As long as the exporter is enabled, you can create Prometheus metrics in your
application using the objects in prometheus-client
and they will get
exported without any extra configuration required.
Note that for non-server jobs, like baseplate-script
scripts, you will need
to export metrics manually. See exporting to a push gateway.
New in version 2.3.
Process-level metrics¶
If your application has registered a metrics client with
configure_observers()
, baseplate-serve
will automatically send process-level metrics every 10 seconds. Which metrics
are sent depends on your server configuration, for example:
[server:main]
factory = baseplate.server.thrift
monitoring.blocked_hub = 100 milliseconds
monitoring.concurrency = true
will enable the blocked_hub
reporter (configuring it to trigger at a 100ms
threshold) and the concurrency
reporter (which has no special
configuration).
The following reporters are available:
monitoring.blocked_hub
Enabled if a valid
Timespan()
is set, defaults to disabled.This will turn on Gevent’s monitoring thread and report events indicating that Gevent detects the main event loop was blocked by a greenlet for longer than the given time span. This can indicate excessive CPU usage causing event loop starvation or the use of non-patched blocking IO calls. More detailed information, including stack traces, is also printed to the logging system.
Each instance of the hub being blocked will be reported as a
Timer
measuring the duration of the blockage.Note: the performance impact of this reporter is not currently understood. Watch your metrics closely if you turn this on.
monitoring.concurrency
Enabled if
true
, disabled iffalse
. Defaults to enabled.This will track the number of in-flight requests being processed concurrently by this server process.
At each report interval, this will update two
Gauge
metrics with the current number of open connections (open_connections
) and current number of in-flight requests being processed concurrently (active_requests
).monitoring.connection_pool
Enabled if
true
, disabled iffalse
. Defaults to disabled.This will track the usage of connection pools for various clients in the application. The metrics generated will depend on which clients are used.
monitoring.gc.stats
Enabled if
true
, disabled iffalse
. Defaults to enabled.This will report the Python garbage collector’s statistics to the metrics system.
At each report interval, this will update gauges with the current values returned by
gc.get_stats()
.monitoring.gc.timing
Enabled if
true
, disabled iffalse
. Defaults to disabled.This will track the duration of time taken by Python’s garbage collector doing a collection sweep.
The duration of each pass of the garbage collector will be reported as a timer.
Note: the performance impact of this reporter is not currently understood. Watch your metrics closely if you turn this on.
monitoring.gc.refcycle
Enabled if a path to a writable directory is set, defaults to disabled.
This should only be used in debugging, it will certainly have a negative performance impact.
This will turn off automatic garbage collection and instead run a sweep every reporting interval. Any objects found by the collector will be graphed using objgraph to help find reference cycles. The resulting graphs images will be written to the directory specified.
The
objgraph
library andgraphviz
package must be installed for this to work properly.
All metrics generated by baseplate-serve
are prefixed with runtime
and
are tagged with hostname
and PID
.
Changed in version 2.0: The hostname
and PID
fields of the metric name were moved to tags.