github
/
sprockets-postgres
mirror of https://github.com/sprockets/sprockets-postgres.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Allow for settings based configuration

This commit is contained in:
Gavin M. Roy 2021-01-08 14:55:25 -05:00
parent 6c2e26ef0f
commit ca362e2bc5
7 changed files with 129 additions and 78 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
.github/workflows/testing.yaml vendored
View File

@ -26,7 +26,7 @@ jobs:
POSTGRES_HOST_AUTH_METHOD: trust
strategy:
matrix:
python: [3.7, 3.8]
python: [3.7, 3.8, 3.9]
container:
image: python:${{ matrix.python }}-alpine
steps:

63
README.rst
View File

@ -19,39 +19,48 @@ Documentation is available at `sprockets-postgres.readthedocs.io <https://sprock
Configuration
-------------
The following table details the environment variable configuration options:
Configuration of sprockets-postgres is done by using of environment variables or
`tornado.web.Application.settings <https://www.tornadoweb.org/en/stable/web.html#tornado.web.Application.settings>`_
dictionary. The `sprockets_postgres.ApplicationMixin <https://sprockets-postgres.readthedocs.io/en/stable/application.html>`_
will use configuration as applied to the settings dictionary, falling back to the
environment variable if the value is not set in the dictionary. Keys in the
settings dictionary are lowercase, and if provided as environment variables,
are uppercase.
+---------------------------------+--------------------------------------------------+-----------+
| Variable | Definition | Default |
+=================================+==================================================+===========+
| ``POSTGRES_URL`` | The PostgreSQL URL to connect to | |
+---------------------------------+--------------------------------------------------+-----------+
| ``POSTGRES_MAX_POOL_SIZE`` | Maximum connection count to Postgres per backend | ``10`` |
+---------------------------------+--------------------------------------------------+-----------+
| ``POSTGRES_MIN_POOL_SIZE`` | Minimum or starting pool size. | ``1`` |
+---------------------------------+--------------------------------------------------+-----------+
| ``POSTGRES_CONNECTION_TIMEOUT`` | The maximum time in seconds to spend attempting | ``10`` |
| | to create a new connection. | |
+---------------------------------+--------------------------------------------------+-----------+
| ``POSTGRES_CONNECTION_TTL`` | Time-to-life in seconds for a pooled connection. | ``300`` |
+---------------------------------+--------------------------------------------------+-----------+
| ``POSTGRES_QUERY_TIMEOUT`` | Maximum execution time for a query in seconds. | ``60`` |
+---------------------------------+--------------------------------------------------+-----------+
| ``POSTGRES_HSTORE`` | Enable HSTORE support in the client. | ``FALSE`` |
+---------------------------------+--------------------------------------------------+-----------+
| ``POSTGRES_JSON`` | Enable JSON support in the client. | ``FALSE`` |
+---------------------------------+--------------------------------------------------+-----------+
| ``POSTGRES_UUID`` | Enable UUID support in the client. | ``TRUE`` |
+---------------------------------+--------------------------------------------------+-----------+
The following table details the available configuration options:
If ``POSTGRES_URL`` uses a scheme of ``postgresql+srv``, a SRV DNS lookup will be
+---------------------------------+--------------------------------------------------+------+-----------+
| Variable | Definition | Type | Default |
+=================================+==================================================+======+===========+
| ``postgres_url`` | The PostgreSQL URL to connect to | str | |
+---------------------------------+--------------------------------------------------+------+-----------+
| ``postgres_max_pool_size`` | Maximum connection count to Postgres per backend | int | ``10`` |
+---------------------------------+--------------------------------------------------+------+-----------+
| ``postgres_min_pool_size`` | Minimum or starting pool size. | int | ``1`` |
+---------------------------------+--------------------------------------------------+------+-----------+
| ``postgres_connection_timeout`` | The maximum time in seconds to spend attempting | int | ``10`` |
| | to create a new connection. | | |
+---------------------------------+--------------------------------------------------+------+-----------+
| ``postgres_connection_ttl`` | Time-to-life in seconds for a pooled connection. | int | ``300`` |
+---------------------------------+--------------------------------------------------+------+-----------+
| ``postgres_query_timeout`` | Maximum execution time for a query in seconds. | int | ``60`` |
+---------------------------------+--------------------------------------------------+------+-----------+
| ``postgres_hstore`` | Enable HSTORE support in the client. | bool | ``FALSE`` |
+---------------------------------+--------------------------------------------------+------+-----------+
| ``postgres_json`` | Enable JSON support in the client. | bool | ``FALSE`` |
+---------------------------------+--------------------------------------------------+------+-----------+
| ``postgres_uuid`` | Enable UUID support in the client. | bool | ``TRUE`` |
+---------------------------------+--------------------------------------------------+------+-----------+
If ``postgres_url`` uses a scheme of ``postgresql+srv``, a SRV DNS lookup will be
performed and the lowest priority record with the highest weight will be selected
for connecting to Postgres.
AWS's ECS service discovery does not follow the SRV standard, but creates SRV
records. If ``POSTGRES_URL`` uses a scheme of ``aws+srv``, a SRV DNS lookup will be
performed using the correct format for ECS service discovery. The lowest priority
record with the highest weight will be selected for connecting to Postgres.
records. If ``postgres_url`` uses a scheme of ``aws+srv``, a SRV DNS lookup will be
performed and the URL will be constructed containing all host and port combinations
in priority and weighted order, utilizing `libpq's supoprt <https://www.postgresql.org/docs/12/libpq-connect.html>`_
for multiple hosts in a URL.
Requirements
------------

2
VERSION
View File

@ -1 +1 @@
1.5.0
1.6.0

81
docs/configuration.rst
View File

@ -1,36 +1,61 @@
Configuration
=============
:py:mod:`sprockets-postgres <sprockets_postgres>` is configured via environment variables. The following table
details the configuration options and their defaults.
Configuration of :py:mod:`sprockets-postgres <sprockets_postgres>` is done by
using of environment variables or :py:attr:`tornado.web.Application.settings`
dictionary. The :py:class:`sprockets_postgres.ApplicationMixin` will use configuration
as applied to the settings dictionary, falling back to the environment variable
if the value is not set in the dictionary. Keys in the settings dictionary are
lowercase, and if provided as environment variables, are uppercase. For example
to set the Postgres URL in a :py:attr:`tornado.web.Application.settings`,
you'd do the following:
+---------------------------------+--------------------------------------------------+-----------+
| Variable | Definition | Default |
+=================================+==================================================+===========+
| ``POSTGRES_URL`` | The PostgreSQL URL to connect to | |
+---------------------------------+--------------------------------------------------+-----------+
| ``POSTGRES_MAX_POOL_SIZE`` | Maximum connection count to Postgres per backend | ``10`` |
+---------------------------------+--------------------------------------------------+-----------+
| ``POSTGRES_MIN_POOL_SIZE`` | Minimum or starting pool size. | ``1`` |
+---------------------------------+--------------------------------------------------+-----------+
| ``POSTGRES_CONNECTION_TIMEOUT`` | The maximum time in seconds to spend attempting | ``10`` |
| | to create a new connection. | |
+---------------------------------+--------------------------------------------------+-----------+
| ``POSTGRES_CONNECTION_TTL`` | Time-to-life in seconds for a pooled connection. | ``300`` |
+---------------------------------+--------------------------------------------------+-----------+
| ``POSTGRES_QUERY_TIMEOUT`` | Maximum execution time for a query in seconds. | ``60`` |
+---------------------------------+--------------------------------------------------+-----------+
| ``POSTGRES_HSTORE`` | Enable HSTORE support in the client. | ``FALSE`` |
+---------------------------------+--------------------------------------------------+-----------+
| ``POSTGRES_JSON`` | Enable JSON support in the client. | ``FALSE`` |
+---------------------------------+--------------------------------------------------+-----------+
| ``POSTGRES_UUID`` | Enable UUID support in the client. | ``TRUE`` |
+---------------------------------+--------------------------------------------------+-----------+
.. code-block::
If ``POSTGRES_URL`` uses a scheme of ``postgresql+srv``, a SRV DNS lookup will be
settings = {'postgres_url': 'postgresql://postgres@localhost:5432/postgres'}
app = web.Application([routes], settings=settings)
and as an environment variable:
.. code-block::
POSTGRES_URL=postgresql://postgres@localhost:5432/postgres
Available Settings
------------------
The following table details the available configuration options:
+---------------------------------+--------------------------------------------------+------+-----------+
| Variable | Definition | Type | Default |
+=================================+==================================================+======+===========+
| ``postgres_url`` | The PostgreSQL URL to connect to | str | |
+---------------------------------+--------------------------------------------------+------+-----------+
| ``postgres_max_pool_size`` | Maximum connection count to Postgres per backend | int | ``10`` |
+---------------------------------+--------------------------------------------------+------+-----------+
| ``postgres_min_pool_size`` | Minimum or starting pool size. | int | ``1`` |
+---------------------------------+--------------------------------------------------+------+-----------+
| ``postgres_connection_timeout`` | The maximum time in seconds to spend attempting | int | ``10`` |
| | to create a new connection. | | |
+---------------------------------+--------------------------------------------------+------+-----------+
| ``postgres_connection_ttl`` | Time-to-life in seconds for a pooled connection. | int | ``300`` |
+---------------------------------+--------------------------------------------------+------+-----------+
| ``postgres_query_timeout`` | Maximum execution time for a query in seconds. | int | ``60`` |
+---------------------------------+--------------------------------------------------+------+-----------+
| ``postgres_hstore`` | Enable HSTORE support in the client. | bool | ``FALSE`` |
+---------------------------------+--------------------------------------------------+------+-----------+
| ``postgres_json`` | Enable JSON support in the client. | bool | ``FALSE`` |
+---------------------------------+--------------------------------------------------+------+-----------+
| ``postgres_uuid`` | Enable UUID support in the client. | bool | ``TRUE`` |
+---------------------------------+--------------------------------------------------+------+-----------+
If ``postgres_url`` uses a scheme of ``postgresql+srv``, a SRV DNS lookup will be
performed and the lowest priority record with the highest weight will be selected
for connecting to Postgres.
AWS's ECS service discovery does not follow the SRV standard, but creates SRV
records. If ``postgres_url`` uses a scheme of ``aws+srv``, a SRV DNS lookup will be
performed and the URL will be constructed containing all host and port combinations
in priority and weighted order, utilizing `libpq's supoprt <https://www.postgresql.org/docs/12/libpq-connect.html>`_
for multiple hosts in a URL.
AWS's ECS service discovery does not follow the SRV standard, but creates SRV
records. If ``POSTGRES_URL`` uses a scheme of ``aws+srv``, a SRV DNS lookup will be
performed using the correct format for ECS service discovery.

2
setup.cfg
View File

@ -12,7 +12,7 @@ project_urls =
Documentation = https://sprockets-postgres.readthedocs.io
Source Code = https://github.com/sprockets/sprockets-postgres/
classifiers =
Development Status :: 3 - Alpha
Development Status :: 4 - Beta
Intended Audience :: Developers
License :: OSI Approved :: BSD License
Natural Language :: English

52
sprockets_postgres.py
View File

@ -459,31 +459,43 @@ class ApplicationMixin:
try:
self._postgres_pool = await pool.Pool.from_pool_fill(
url,
maxsize=int(
os.environ.get(
maxsize=self.settings.get(
'postgres_max_pool_size',
int(os.environ.get(
'POSTGRES_MAX_POOL_SIZE',
DEFAULT_POSTGRES_MAX_POOL_SIZE)),
minsize=int(
os.environ.get(
DEFAULT_POSTGRES_MAX_POOL_SIZE))),
minsize=self.settings.get(
'postgres_min_pool_size',
int(os.environ.get(
'POSTGRES_MIN_POOL_SIZE',
DEFAULT_POSTGRES_MIN_POOL_SIZE)),
timeout=int(
os.environ.get(
'POSTGRES_CONNECT_TIMEOUT',
DEFAULT_POSTGRES_CONNECTION_TIMEOUT)),
enable_hstore=util.strtobool(
os.environ.get(
'POSTGRES_HSTORE', DEFAULT_POSTGRES_HSTORE)),
enable_json=util.strtobool(
os.environ.get('POSTGRES_JSON', DEFAULT_POSTGRES_JSON)),
enable_uuid=util.strtobool(
os.environ.get('POSTGRES_UUID', DEFAULT_POSTGRES_UUID)),
DEFAULT_POSTGRES_MIN_POOL_SIZE))),
timeout=self.settings.get(
'postgres_connect_timeout',
int(os.environ.get(
'POSTGRES_CONNECT_TIMEOUT',
DEFAULT_POSTGRES_CONNECTION_TIMEOUT))),
enable_hstore=self.settings.get(
'postgres_hstore',
util.strtobool(
os.environ.get(
'POSTGRES_HSTORE', DEFAULT_POSTGRES_HSTORE))),
enable_json=self.settings.get(
'enable_json',
util.strtobool(
os.environ.get(
'POSTGRES_JSON', DEFAULT_POSTGRES_JSON))),
enable_uuid=self.settings.get(
'postgres_uuid',
util.strtobool(
os.environ.get(
'POSTGRES_UUID', DEFAULT_POSTGRES_UUID))),
echo=False,
on_connect=None,
pool_recycle=int(
os.environ.get(
pool_recycle=self.settings.get(
'postgres_connection_ttl',
int(os.environ.get(
'POSTGRES_CONNECTION_TTL',
DEFAULT_POSTGRES_CONNECTION_TTL)))
DEFAULT_POSTGRES_CONNECTION_TTL))))
except (psycopg2.OperationalError,
psycopg2.Error) as error: # pragma: nocover
LOGGER.warning(

5
tests.py
View File

@ -319,6 +319,11 @@ class TestCase(testing.SprocketsHttpTestCase):
name, _, value = line.strip().partition('=')
os.environ[name] = value
def tearDown(self):
if self.app._postgres_pool is not None:
self.app._postgres_pool.terminate()
super().tearDown()
def get_app(self):
self.app = Application(handlers=[
web.url('/callproc', CallprocRequestHandler),