ttl was -1 for this host, thus the test failed. Not sure what the
intention of this test was; ttl is never used by
sprockets-postgres.
---
aiodns was returning the following records for this lookup
```
[<ares_query_srv_result> host=alt2.xmpp-server.l.google.com, port=5269, priority=20, weight=0, ttl=-1,
<ares_query_srv_result> host=xmpp-server.l.google.com, port=5269, priority=5, weight=0, ttl=-1,
<ares_query_srv_result> host=alt4.xmpp-server.l.google.com, port=5269, priority=20, weight=0, ttl=-1,
<ares_query_srv_result> host=alt1.xmpp-server.l.google.com, port=5269, priority=20, weight=0, ttl=-1,
<ares_query_srv_result> host=alt3.xmpp-server.l.google.com, port=5269, priority=20, weight=0, ttl=-1]
```
Also noticed a ttl of -1 with `_xmpp-server._tcp.jabber.org`
1. Add new property function in the ApplicationMixin to indicate if Postgres is connected
2. Add a new guard in the RequestHandlerMixin to check that Postgres is connected prior to executing a query, raising a 503 if it is not
3. Catch OperationalError in RequestHandlerMixin and return 503 for it
4. Timeout when waiting on the connection when attempting to reconnect
5. Log when we're creating a new pool
6. Add debug logging to trace when connectios open
7. Add tests that ensure reconnect logic works as expected
It's possible to call postgres_status before the first connection is
established. Before this fix, an exception was raised due to
`self._postgres_connected` being `None`.
```
File "/home/ar/code/github/sprockets/sprockets-postgres/sprockets_postgres.py", line 411, in postgres_status
if not self._postgres_connected.is_set():
AttributeError: 'NoneType' object has no attribute 'is_set'
```
The error handler behavior descended down through the connection management part of the code and that was missed, this covers that.
Add test coverage for the new branches
Obscure password in the postgres connection DSN when logging in debug mode.
Will also not print the DSN with obscured password when failing to connect
at start up.
ProgrammingError is raised when you try and fetch data from a cursor and there is no data to fetch. When this happens ProgrammingError.pgcode is None.
It is also raised when your query has errors in it. Now if that's the case, they will be caught on L249 and not inside the function at L259.
This new branch in the code will ensure that should we unexpectedly encounter a "real" programming error from Postgres, it is not blindly swallowed and a warning is issued. This should NEVER happen based upon my understanding of the psycopg2 internals.
Unfortunately I couldn't come up with a good test case using mocks to make it happen, as ProgrammingError() takes no keyword arguments and pgcode is a read-only attribute on a ProgrammingError instance. I also couldn't figure out a way to raise ProgrammingError from psycopg2.errors.lookup/1.
Thus, the # pragma: nocover
- Attempt to add graceful reconfiguration for SRV based connections. On connection failure, the pool will be closed and reopened after fetching to get new SRV records.
- When using SRV, use all return hosts in the PostgreSQL URL
- If multiple requests hit a disconnect error, the reconnect logic will allow the first one in to reconnect and the others to wait
