Allow for settings based configuration

This commit is contained in:
Gavin M. Roy 2021-01-08 14:55:25 -05:00
parent 6c2e26ef0f
commit ca362e2bc5
7 changed files with 129 additions and 78 deletions

View file

@ -26,7 +26,7 @@ jobs:
POSTGRES_HOST_AUTH_METHOD: trust POSTGRES_HOST_AUTH_METHOD: trust
strategy: strategy:
matrix: matrix:
python: [3.7, 3.8] python: [3.7, 3.8, 3.9]
container: container:
image: python:${{ matrix.python }}-alpine image: python:${{ matrix.python }}-alpine
steps: steps:

View file

@ -19,39 +19,48 @@ Documentation is available at `sprockets-postgres.readthedocs.io <https://sprock
Configuration Configuration
------------- -------------
The following table details the environment variable configuration options: Configuration of sprockets-postgres is done by using of environment variables or
`tornado.web.Application.settings <https://www.tornadoweb.org/en/stable/web.html#tornado.web.Application.settings>`_
dictionary. The `sprockets_postgres.ApplicationMixin <https://sprockets-postgres.readthedocs.io/en/stable/application.html>`_
will use configuration as applied to the settings dictionary, falling back to the
environment variable if the value is not set in the dictionary. Keys in the
settings dictionary are lowercase, and if provided as environment variables,
are uppercase.
+---------------------------------+--------------------------------------------------+-----------+ The following table details the available configuration options:
| Variable | Definition | Default |
+=================================+==================================================+===========+
| ``POSTGRES_URL`` | The PostgreSQL URL to connect to | |
+---------------------------------+--------------------------------------------------+-----------+
| ``POSTGRES_MAX_POOL_SIZE`` | Maximum connection count to Postgres per backend | ``10`` |
+---------------------------------+--------------------------------------------------+-----------+
| ``POSTGRES_MIN_POOL_SIZE`` | Minimum or starting pool size. | ``1`` |
+---------------------------------+--------------------------------------------------+-----------+
| ``POSTGRES_CONNECTION_TIMEOUT`` | The maximum time in seconds to spend attempting | ``10`` |
| | to create a new connection. | |
+---------------------------------+--------------------------------------------------+-----------+
| ``POSTGRES_CONNECTION_TTL`` | Time-to-life in seconds for a pooled connection. | ``300`` |
+---------------------------------+--------------------------------------------------+-----------+
| ``POSTGRES_QUERY_TIMEOUT`` | Maximum execution time for a query in seconds. | ``60`` |
+---------------------------------+--------------------------------------------------+-----------+
| ``POSTGRES_HSTORE`` | Enable HSTORE support in the client. | ``FALSE`` |
+---------------------------------+--------------------------------------------------+-----------+
| ``POSTGRES_JSON`` | Enable JSON support in the client. | ``FALSE`` |
+---------------------------------+--------------------------------------------------+-----------+
| ``POSTGRES_UUID`` | Enable UUID support in the client. | ``TRUE`` |
+---------------------------------+--------------------------------------------------+-----------+
If ``POSTGRES_URL`` uses a scheme of ``postgresql+srv``, a SRV DNS lookup will be +---------------------------------+--------------------------------------------------+------+-----------+
| Variable | Definition | Type | Default |
+=================================+==================================================+======+===========+
| ``postgres_url`` | The PostgreSQL URL to connect to | str | |
+---------------------------------+--------------------------------------------------+------+-----------+
| ``postgres_max_pool_size`` | Maximum connection count to Postgres per backend | int | ``10`` |
+---------------------------------+--------------------------------------------------+------+-----------+
| ``postgres_min_pool_size`` | Minimum or starting pool size. | int | ``1`` |
+---------------------------------+--------------------------------------------------+------+-----------+
| ``postgres_connection_timeout`` | The maximum time in seconds to spend attempting | int | ``10`` |
| | to create a new connection. | | |
+---------------------------------+--------------------------------------------------+------+-----------+
| ``postgres_connection_ttl`` | Time-to-life in seconds for a pooled connection. | int | ``300`` |
+---------------------------------+--------------------------------------------------+------+-----------+
| ``postgres_query_timeout`` | Maximum execution time for a query in seconds. | int | ``60`` |
+---------------------------------+--------------------------------------------------+------+-----------+
| ``postgres_hstore`` | Enable HSTORE support in the client. | bool | ``FALSE`` |
+---------------------------------+--------------------------------------------------+------+-----------+
| ``postgres_json`` | Enable JSON support in the client. | bool | ``FALSE`` |
+---------------------------------+--------------------------------------------------+------+-----------+
| ``postgres_uuid`` | Enable UUID support in the client. | bool | ``TRUE`` |
+---------------------------------+--------------------------------------------------+------+-----------+
If ``postgres_url`` uses a scheme of ``postgresql+srv``, a SRV DNS lookup will be
performed and the lowest priority record with the highest weight will be selected performed and the lowest priority record with the highest weight will be selected
for connecting to Postgres. for connecting to Postgres.
AWS's ECS service discovery does not follow the SRV standard, but creates SRV AWS's ECS service discovery does not follow the SRV standard, but creates SRV
records. If ``POSTGRES_URL`` uses a scheme of ``aws+srv``, a SRV DNS lookup will be records. If ``postgres_url`` uses a scheme of ``aws+srv``, a SRV DNS lookup will be
performed using the correct format for ECS service discovery. The lowest priority performed and the URL will be constructed containing all host and port combinations
record with the highest weight will be selected for connecting to Postgres. in priority and weighted order, utilizing `libpq's supoprt <https://www.postgresql.org/docs/12/libpq-connect.html>`_
for multiple hosts in a URL.
Requirements Requirements
------------ ------------

View file

@ -1 +1 @@
1.5.0 1.6.0

View file

@ -1,36 +1,61 @@
Configuration Configuration
============= =============
:py:mod:`sprockets-postgres <sprockets_postgres>` is configured via environment variables. The following table Configuration of :py:mod:`sprockets-postgres <sprockets_postgres>` is done by
details the configuration options and their defaults. using of environment variables or :py:attr:`tornado.web.Application.settings`
dictionary. The :py:class:`sprockets_postgres.ApplicationMixin` will use configuration
as applied to the settings dictionary, falling back to the environment variable
if the value is not set in the dictionary. Keys in the settings dictionary are
lowercase, and if provided as environment variables, are uppercase. For example
to set the Postgres URL in a :py:attr:`tornado.web.Application.settings`,
you'd do the following:
+---------------------------------+--------------------------------------------------+-----------+ .. code-block::
| Variable | Definition | Default |
+=================================+==================================================+===========+
| ``POSTGRES_URL`` | The PostgreSQL URL to connect to | |
+---------------------------------+--------------------------------------------------+-----------+
| ``POSTGRES_MAX_POOL_SIZE`` | Maximum connection count to Postgres per backend | ``10`` |
+---------------------------------+--------------------------------------------------+-----------+
| ``POSTGRES_MIN_POOL_SIZE`` | Minimum or starting pool size. | ``1`` |
+---------------------------------+--------------------------------------------------+-----------+
| ``POSTGRES_CONNECTION_TIMEOUT`` | The maximum time in seconds to spend attempting | ``10`` |
| | to create a new connection. | |
+---------------------------------+--------------------------------------------------+-----------+
| ``POSTGRES_CONNECTION_TTL`` | Time-to-life in seconds for a pooled connection. | ``300`` |
+---------------------------------+--------------------------------------------------+-----------+
| ``POSTGRES_QUERY_TIMEOUT`` | Maximum execution time for a query in seconds. | ``60`` |
+---------------------------------+--------------------------------------------------+-----------+
| ``POSTGRES_HSTORE`` | Enable HSTORE support in the client. | ``FALSE`` |
+---------------------------------+--------------------------------------------------+-----------+
| ``POSTGRES_JSON`` | Enable JSON support in the client. | ``FALSE`` |
+---------------------------------+--------------------------------------------------+-----------+
| ``POSTGRES_UUID`` | Enable UUID support in the client. | ``TRUE`` |
+---------------------------------+--------------------------------------------------+-----------+
If ``POSTGRES_URL`` uses a scheme of ``postgresql+srv``, a SRV DNS lookup will be settings = {'postgres_url': 'postgresql://postgres@localhost:5432/postgres'}
app = web.Application([routes], settings=settings)
and as an environment variable:
.. code-block::
POSTGRES_URL=postgresql://postgres@localhost:5432/postgres
Available Settings
------------------
The following table details the available configuration options:
+---------------------------------+--------------------------------------------------+------+-----------+
| Variable | Definition | Type | Default |
+=================================+==================================================+======+===========+
| ``postgres_url`` | The PostgreSQL URL to connect to | str | |
+---------------------------------+--------------------------------------------------+------+-----------+
| ``postgres_max_pool_size`` | Maximum connection count to Postgres per backend | int | ``10`` |
+---------------------------------+--------------------------------------------------+------+-----------+
| ``postgres_min_pool_size`` | Minimum or starting pool size. | int | ``1`` |
+---------------------------------+--------------------------------------------------+------+-----------+
| ``postgres_connection_timeout`` | The maximum time in seconds to spend attempting | int | ``10`` |
| | to create a new connection. | | |
+---------------------------------+--------------------------------------------------+------+-----------+
| ``postgres_connection_ttl`` | Time-to-life in seconds for a pooled connection. | int | ``300`` |
+---------------------------------+--------------------------------------------------+------+-----------+
| ``postgres_query_timeout`` | Maximum execution time for a query in seconds. | int | ``60`` |
+---------------------------------+--------------------------------------------------+------+-----------+
| ``postgres_hstore`` | Enable HSTORE support in the client. | bool | ``FALSE`` |
+---------------------------------+--------------------------------------------------+------+-----------+
| ``postgres_json`` | Enable JSON support in the client. | bool | ``FALSE`` |
+---------------------------------+--------------------------------------------------+------+-----------+
| ``postgres_uuid`` | Enable UUID support in the client. | bool | ``TRUE`` |
+---------------------------------+--------------------------------------------------+------+-----------+
If ``postgres_url`` uses a scheme of ``postgresql+srv``, a SRV DNS lookup will be
performed and the lowest priority record with the highest weight will be selected
for connecting to Postgres.
AWS's ECS service discovery does not follow the SRV standard, but creates SRV
records. If ``postgres_url`` uses a scheme of ``aws+srv``, a SRV DNS lookup will be
performed and the URL will be constructed containing all host and port combinations performed and the URL will be constructed containing all host and port combinations
in priority and weighted order, utilizing `libpq's supoprt <https://www.postgresql.org/docs/12/libpq-connect.html>`_ in priority and weighted order, utilizing `libpq's supoprt <https://www.postgresql.org/docs/12/libpq-connect.html>`_
for multiple hosts in a URL. for multiple hosts in a URL.
AWS's ECS service discovery does not follow the SRV standard, but creates SRV
records. If ``POSTGRES_URL`` uses a scheme of ``aws+srv``, a SRV DNS lookup will be
performed using the correct format for ECS service discovery.

View file

@ -12,7 +12,7 @@ project_urls =
Documentation = https://sprockets-postgres.readthedocs.io Documentation = https://sprockets-postgres.readthedocs.io
Source Code = https://github.com/sprockets/sprockets-postgres/ Source Code = https://github.com/sprockets/sprockets-postgres/
classifiers = classifiers =
Development Status :: 3 - Alpha Development Status :: 4 - Beta
Intended Audience :: Developers Intended Audience :: Developers
License :: OSI Approved :: BSD License License :: OSI Approved :: BSD License
Natural Language :: English Natural Language :: English

View file

@ -459,31 +459,43 @@ class ApplicationMixin:
try: try:
self._postgres_pool = await pool.Pool.from_pool_fill( self._postgres_pool = await pool.Pool.from_pool_fill(
url, url,
maxsize=int( maxsize=self.settings.get(
os.environ.get( 'postgres_max_pool_size',
int(os.environ.get(
'POSTGRES_MAX_POOL_SIZE', 'POSTGRES_MAX_POOL_SIZE',
DEFAULT_POSTGRES_MAX_POOL_SIZE)), DEFAULT_POSTGRES_MAX_POOL_SIZE))),
minsize=int( minsize=self.settings.get(
os.environ.get( 'postgres_min_pool_size',
int(os.environ.get(
'POSTGRES_MIN_POOL_SIZE', 'POSTGRES_MIN_POOL_SIZE',
DEFAULT_POSTGRES_MIN_POOL_SIZE)), DEFAULT_POSTGRES_MIN_POOL_SIZE))),
timeout=int( timeout=self.settings.get(
os.environ.get( 'postgres_connect_timeout',
int(os.environ.get(
'POSTGRES_CONNECT_TIMEOUT', 'POSTGRES_CONNECT_TIMEOUT',
DEFAULT_POSTGRES_CONNECTION_TIMEOUT)), DEFAULT_POSTGRES_CONNECTION_TIMEOUT))),
enable_hstore=util.strtobool( enable_hstore=self.settings.get(
'postgres_hstore',
util.strtobool(
os.environ.get( os.environ.get(
'POSTGRES_HSTORE', DEFAULT_POSTGRES_HSTORE)), 'POSTGRES_HSTORE', DEFAULT_POSTGRES_HSTORE))),
enable_json=util.strtobool( enable_json=self.settings.get(
os.environ.get('POSTGRES_JSON', DEFAULT_POSTGRES_JSON)), 'enable_json',
enable_uuid=util.strtobool( util.strtobool(
os.environ.get('POSTGRES_UUID', DEFAULT_POSTGRES_UUID)), os.environ.get(
'POSTGRES_JSON', DEFAULT_POSTGRES_JSON))),
enable_uuid=self.settings.get(
'postgres_uuid',
util.strtobool(
os.environ.get(
'POSTGRES_UUID', DEFAULT_POSTGRES_UUID))),
echo=False, echo=False,
on_connect=None, on_connect=None,
pool_recycle=int( pool_recycle=self.settings.get(
os.environ.get( 'postgres_connection_ttl',
int(os.environ.get(
'POSTGRES_CONNECTION_TTL', 'POSTGRES_CONNECTION_TTL',
DEFAULT_POSTGRES_CONNECTION_TTL))) DEFAULT_POSTGRES_CONNECTION_TTL))))
except (psycopg2.OperationalError, except (psycopg2.OperationalError,
psycopg2.Error) as error: # pragma: nocover psycopg2.Error) as error: # pragma: nocover
LOGGER.warning( LOGGER.warning(

View file

@ -319,6 +319,11 @@ class TestCase(testing.SprocketsHttpTestCase):
name, _, value = line.strip().partition('=') name, _, value = line.strip().partition('=')
os.environ[name] = value os.environ[name] = value
def tearDown(self):
if self.app._postgres_pool is not None:
self.app._postgres_pool.terminate()
super().tearDown()
def get_app(self): def get_app(self):
self.app = Application(handlers=[ self.app = Application(handlers=[
web.url('/callproc', CallprocRequestHandler), web.url('/callproc', CallprocRequestHandler),