Commit graph

35 commits

Author SHA1 Message Date
Gavin M. Roy
661a5986c7 Attempt to fix the reconnection race condition
1. Add new property function in the ApplicationMixin to indicate if Postgres is connected
2. Add a new guard in the RequestHandlerMixin to check that Postgres is connected prior to executing a query, raising a 503 if it is not
3. Catch OperationalError in RequestHandlerMixin and return 503 for it
4. Timeout when waiting on the connection when attempting to reconnect
5. Log when we're creating a new pool
6. Add debug logging to trace when connectios open
7. Add tests that ensure reconnect logic works as expected
2021-04-14 15:59:13 -04:00
Andrew Rabert
afb6ea99a5 Fix postgres_status before first connection attempt
It's possible to call postgres_status before the first connection is
established. Before this fix, an exception was raised due to
`self._postgres_connected` being `None`.

```
  File "/home/ar/code/github/sprockets/sprockets-postgres/sprockets_postgres.py", line 411, in postgres_status
    if not self._postgres_connected.is_set():
AttributeError: 'NoneType' object has no attribute 'is_set'
```
2021-01-20 16:04:21 -05:00
Gavin M. Roy
16b12cf470 Explicitly raise a FK error 2021-01-11 22:31:54 -05:00
Gavin M. Roy
230983e803 Melt a few flake8 flakes 2021-01-08 18:19:46 -05:00
Gavin M. Roy
3d6345a882 Missed some abstractions from envvars in 1.6 2021-01-08 18:14:39 -05:00
Gavin M. Roy
ca362e2bc5 Allow for settings based configuration 2021-01-08 14:55:25 -05:00
Gavin M. Roy
499972a7e8 Remove detail due to potential info disclosure issues 2020-09-17 12:05:18 -04:00
Gavin M. Roy
839fb82034 Fix a missing edge case for no error handler, add tests
The error handler behavior descended down through the connection management part of the code and that was missed, this covers that.
Add test coverage for the new branches
2020-09-17 11:40:34 -04:00
Gavin M. Roy
e2e80eea2f Add tornado-problem-details support 2020-09-16 19:57:01 -04:00
Gavin M. Roy
b7ddd137b6
Merge pull request #5 from nvllsvm/on_error
Make on_error optional for PostgresConnector
2020-09-16 19:28:52 -04:00
Andrew Rabert
fbb7b4dd9a Add on_postgres_error and on_postgres_timing to public API of RequestHandlerMixin
The methods are intended to be overrideable, but the leading underscore
convention conflicts with Python practice.

See https://docs.python.org/3.8/tutorial/classes.html#private-variables
2020-09-16 17:28:22 -04:00
Andrew Rabert
e57646df00 Make on_error optional for PostgresConnector 2020-09-16 17:20:50 -04:00
Brian Korty
893308e340 Obscure password in debug logs.
Obscure password in the postgres connection DSN when logging in debug mode.
Will also not print the DSN with obscured password when failing to connect
at start up.
2020-09-01 10:55:06 -04:00
Gavin M. Roy
e880017fc8 Add a little context of why this code exists 2020-08-11 19:40:56 -04:00
Gavin M. Roy
9a49d769df Include a StatusRequestHandler to make things easy 2020-08-11 19:40:14 -04:00
Gavin M. Roy
507704a6e7 Merge branch 'master' of github.com:sprockets/sprockets-postgres 2020-08-11 19:31:56 -04:00
Gavin M. Roy
664bedbb24 Don't blindly swallow ProgrammingError
ProgrammingError is raised when you try and fetch data from a cursor and there is no data to fetch. When this happens ProgrammingError.pgcode is None.

It is also raised when your query has errors in it. Now if that's the case, they will be caught on L249 and not inside the function at L259.

This new branch in the code will ensure that should we unexpectedly encounter a "real" programming error from Postgres, it is not blindly swallowed and a warning is issued. This should NEVER happen based upon my understanding of the psycopg2 internals.

Unfortunately I couldn't come up with a good test case using mocks to make it happen, as ProgrammingError() takes no keyword arguments and pgcode is a read-only attribute on a ProgrammingError instance. I also couldn't figure out a way to raise ProgrammingError from psycopg2.errors.lookup/1.

Thus, the # pragma: nocover
2020-08-11 19:25:34 -04:00
Gavin M. Roy
54a92667d1 Don't catch every psycopg2 exception in connector
This was swallowing exceptions downstream in the RequestHandler, like psycopg2.DataError
2020-08-11 18:30:30 -04:00
Kevin Vance
c53cf4f94b Fix argument order for record_timing() 2020-07-30 15:00:09 -04:00
Gavin M. Roy
440def6f3c Dont create objects if we do not need to 2020-07-08 14:34:41 -04:00
Gavin M. Roy
4f60c7bf0a Delay the initialization of asyncio objects to on_start 2020-07-08 14:33:04 -04:00
Gavin M. Roy
70cf5c005b Show the actual URL that is being connected to 2020-07-07 18:37:06 -04:00
Gavin M. Roy
aaf0548bdf Fix indentation, duplicate test name 2020-07-07 17:31:16 -04:00
Gavin M. Roy
a1736bccae Actually wait while the first in reconnects 2020-07-07 16:21:01 -04:00
Gavin M. Roy
ecc070e974 Add reconfiguration for SRV based connections
- Attempt to add graceful reconfiguration for SRV based connections. On connection failure, the pool will be closed and reopened after fetching to get new SRV records.
- When using SRV, use all return hosts in the PostgreSQL URL
- If multiple requests hit a disconnect error, the reconnect logic will allow the first one in to reconnect and the others to wait
2020-07-07 16:17:56 -04:00
Gavin M. Roy
3e36210c91 Add support for SRV based configuration
Supports both postgresql+srv and aws+srv where aws+srv supports the ECS service discovery Route53 SRV record lookup behavior.
2020-06-02 11:26:07 -04:00
Gavin M. Roy
10b98cba10 Change sprockets_postgres.QueryResult to a class
After use in a couple APIs, I found having to check against QueryResult.row and QueryResult.rows too difficult, so this change will always have content in QueryResult.rows, even if it's one row.

In addition, it turns the object into an iterator and adds __repr__ and __len__ magic methods to make interacting with it easier
2020-04-27 18:16:18 -04:00
Gavin M. Roy
86b6c022f0 Update DEFAULT_POSTGRES_MAX_POOL_SIZE
Even though ``0`` is documented as the max pool size in the aiopg docs, setting it to 0 causes an exception in `aiopg.pool.Pool._fill_free_pool`:

```
  File "/Users/gavinr/Source/PSE/anabroker/env/lib/python3.7/site-packages/aiopg/pool.py", line 208, in _fill_free_pool
    if override_min and self.size < self.maxsize:
TypeError: '<' not supported between instances of 'int' and 'NoneType'
```

Set to the same default as aiopg, @ 10.
2020-04-13 14:20:09 -04:00
Gavin M. Roy
4e70836ddd More documentaiton updates 2020-04-10 16:37:23 -04:00
Gavin M. Roy
d057b140ae Add docstrings, fix edge cases 2020-04-08 18:43:51 -04:00
Gavin M. Roy
603eb4d6dd Testing update and fixes found in testing 2020-04-07 16:59:06 -04:00
Gavin M. Roy
611dfd1ec7 WIP refactor 2020-04-07 13:42:02 -04:00
Gavin M. Roy
780a428f9d Specify the max size of the lrucache for Python 3.7 2020-04-06 18:23:53 -04:00
Gavin M. Roy
b620e1048c Address flake8 errors 2020-04-06 18:20:34 -04:00
Gavin M. Roy
ca5dc26de3 Move from a package to a module 2020-04-06 17:47:50 -04:00
Renamed from sprockets_postgres/__init__.py (Browse further)