13 KiB
13 KiB
2020-07-16
Import evaluation issue
Got an alert in Pagerduty about Import Evaluation message age.
Saw some of this earlier, where it was getting null for its compromised
addresses database URL. The configuration is correct in consul, so maybe this
was a weird intermittent thing.
{"name":"rejected.process","levelname":"ERROR","asctime":"2020-07-16T09:49:46.103741+0000","processName":"subscriber-import-evaluation-1","message":"Error creating the consumer \"subscriber_import_evaluation.consumer.Consumer\": pgsql_compromised_addresses: null is invalid - expected format: ^postgres(ql)?://.*","module":"process","correlation_id":"c20e024c-160c-45a1-9ea3-e3bb04d26413","exc_info":["Traceback (most recent call last):\n"," File \"/usr/local/lib/python3.7/site-packages/subscriber_import_evaluation/consumer.py\", line 338, in validate_settings\n jsonschema.validate(settings, self.SETTINGS_SCHEMA)\n"," File \"/usr/local/lib/python3.7/site-packages/jsonschema/validators.py\", line 899, in validate\n raise error\n","jsonschema.exceptions.ValidationError: 'null' does not match '^postgres(ql)?://.*'\n\nFailed validating 'pattern' in schema['properties']['pgsql_compromised_addresses']:\n {'description': 'The RDS compromised addresses connection URI',\n 'pattern': '^postgres(ql)?://.*',\n 'type': 'string'}\n\nOn instance['pgsql_compromised_addresses']:\n 'null'\n","\nDuring handling of the above exception, another exception occurred:\n\n","Traceback (most recent call last):\n"," File \"/usr/local/lib/python3.7/site-packages/rejected/process.py\", line 455, in get_consumer\n return handle(**kwargs)\n"," File \"/usr/local/lib/python3.7/site-packages/avroconsumer.py\", line 25, in __init__\n super(Consumer, self).__init__(*args, **kwargs)\n"," File \"/usr/local/lib/python3.7/site-packages/rejected/consumer.py\", line 208, in __init__\n self.initialize()\n"," File \"/usr/local/lib/python3.7/site-packages/subscriber_import_evaluation/consumer.py\", line 291, in initialize\n self.validate_settings(self.settings.dict())\n"," File \"/usr/local/lib/python3.7/site-packages/subscriber_import_evaluation/consumer.py\", line 351, in validate_settings\n raise Exception(message)\n","Exception: pgsql_compromised_addresses: null is invalid - expected format: ^postgres(ql)?://.*\n"],"log":"{\"name\":\"rejected.process\",\"levelname\":\"ERROR\",\"asctime\":\"2020-07-16T09:49:46.103741+0000\",\"processName\":\"subscriber-import-evaluation-1\",\"message\":\"Error creating the consumer \\\"subscriber_import_evaluation.consumer.Consumer\\\": pgsql_compromised_addresses: null is invalid - expected format: ^postgres(ql)?://.*\",\"module\":\"process\",\"correlation_id\":\"c20e024c-160c-45a1-9ea3-e3bb04d26413\",\"exc_info\":[\"Traceback (most recent call last):\\n\",\" File \\\"/usr/local/lib/python3.7/site-packages/subscriber_import_evaluation/consumer.py\\\", line 338, in validate_settings\\n jsonschema.validate(settings, self.SETTINGS_SCHEMA)\\n\",\" File \\\"/usr/local/lib/python3.7/site-packages/jsonschema/validators.py\\\", line 899, in validate\\n raise error\\n\",\"jsonschema.exceptions.ValidationError: 'null' does not match '^postgres(ql)?://.*'\\n\\nFailed validating 'pattern' in schema['properties']['pgsql_compromised_addresses']:\\n {'description': 'The RDS compromised addresses connection URI',\\n 'pattern': '^postgres(ql)?://.*',\\n 'type': 'string'}\\n\\nOn instance['pgsql_compromised_addresses']:\\n 'null'\\n\",\"\\nDuring handling of the above exception, another exception occurred:\\n\\n\",\"Traceback (most recent call last):\\n\",\" File \\\"/usr/local/lib/python3.7/site-packages/rejected/process.py\\\", line 455, in get_consumer\\n return handle(**kwargs)\\n\",\" File \\\"/usr/local/lib/python3.7/site-packages/avroconsumer.py\\\", line 25, in __init__\\n super(Consumer, self).__init__(*args, **kwargs)\\n\",\" File \\\"/usr/local/lib/python3.7/site-packages/rejected/consumer.py\\\", line 208, in __init__\\n self.initialize()\\n\",\" File \\\"/usr/local/lib/python3.7/site-packages/subscriber_import_evaluation/consumer.py\\\", line 291, in initialize\\n self.validate_settings(self.settings.dict())\\n\",\" File \\\"/usr/local/lib/python3.7/site-packages/subscriber_import_evaluation/consumer.py\\\", line 351, in validate_settings\\n raise Exception(message)\\n\",\"Exception: pgsql_compromised_addresses: null is invalid - expected format: ^postgres(ql)?://.*\\n\"]}\n","stream":"stdout","docker":{"container_id":"8cee26b83b012277ad0ee740e11b260e25baed5aff912264ae195b30ace5ab21"},"kubernetes":{"container_name":"subscriber-import-evaluation","namespace_name":"cp","pod_name":"subscriber-import-evaluation-6bdd96bdcd-jx4w9","pod_id":"4e098e40-bae8-11ea-9f30-6a3b5e3f247e","labels":{"app":"subscriber-import-evaluation","component":"application","pod-template-hash":"2688526878","version":"e142756"},"host":"kube01-w02.testing.aweberint.com","master_url":"https://10.50.0.1:443/api"},"timestamp":"2020-07-16T09:49:46+00:00"}
Here, though, we've got a message repeatedly being requeued due to a list lookup failure.
{"name":"subscriber_import_evaluation.consumer","levelname":"DEBUG","asctime":"2020-07-16T12:54:18.250984+0000","processName":"subscriber-import-evaluation-1","message":"Returning {'account': '62949269-3df5-46d9-b883-74e7771504f2', 'import_uuid': 'f4cd8ddc-95c3-445d-afbe-386e93a29854', 'legacy_account_id': 1326226, 'legacy_list_id': 5202426, 'list': 'ce641169-a75d-45bc-bc48-ef8a8bda31c8', 'timestamp': '2020-07-16T09:36:37.989035+00:00', 'acquisition_method': 'Acquisition method: \"other\"\\nProvider: Did not bring from another provider\\nDescribe how you got this list of subscribers\\nIam exporting and re-importing and merging my lists.', 'automation_enabled': True, 'coi_enabled': False, 'column_mapping': {'email': 0, 'name': 1, 'note': 8, 'tags': 19}, 'existing_subscriber_action': 'Ignore', 'remote_ip': '110.144.41.250', 'tags': [], 'message_number': None}","module":"avroconsumer","correlation_id":"43b4305c-02fa-4e5a-aba1-e7db9c4bb926","exc_info":null}
{"name":"subscriber_import_evaluation.consumer","levelname":"WARNING","asctime":"2020-07-16T12:54:18.268914+0000","processName":"subscriber-import-evaluation-1","message":"get_list: GET request to http://list.service.production.consul/v1/accounts/1326226/lists/5202426 failed with message: HTTP 404: ListNotFound","module":"services","correlation_id":"43b4305c-02fa-4e5a-aba1-e7db9c4bb926","exc_info":null}
{"name":"subscriber_import_evaluation.consumer","levelname":"ERROR","asctime":"2020-07-16T12:54:18.269250+0000","processName":"subscriber-import-evaluation-1","message":"Exception processing delivery 1: HTTP 404: ListNotFound","module":"consumer","correlation_id":"43b4305c-02fa-4e5a-aba1-e7db9c4bb926","exc_info":null}
{"name":"subscriber_import_evaluation.consumer","levelname":"ERROR","asctime":"2020-07-16T12:54:18.269401+0000","processName":"subscriber-import-evaluation-1","message":"Processor handled HTTPClientError: HTTP 404: ListNotFound","module":"consumer","correlation_id":"43b4305c-02fa-4e5a-aba1-e7db9c4bb926","exc_info":["Traceback (most recent call last):\n"," File \"/usr/local/lib/python3.7/site-packages/rejected/consumer.py\", line 908, in execute\n yield result\n"," File \"/usr/local/lib/python3.7/site-packages/tornado/gen.py\", line 735, in run\n value = future.result()\n"," File \"/usr/local/lib/python3.7/site-packages/tornado/gen.py\", line 742, in run\n yielded = self.gen.throw(*exc_info) # type: ignore\n"," File \"/usr/local/lib/python3.7/site-packages/subscriber_import_evaluation/consumer.py\", line 734, in process\n fields=[\"coi_bitmap\"],\n"," File \"/usr/local/lib/python3.7/site-packages/tornado/gen.py\", line 735, in run\n value = future.result()\n"," File \"/usr/local/lib/python3.7/site-packages/tornado/gen.py\", line 742, in run\n yielded = self.gen.throw(*exc_info) # type: ignore\n"," File \"/usr/local/lib/python3.7/site-packages/subscriber_import_evaluation/services.py\", line 113, in get_list\n error_reason='Unable to fetch list')\n"," File \"/usr/local/lib/python3.7/site-packages/tornado/gen.py\", line 735, in run\n value = future.result()\n"," File \"/usr/local/lib/python3.7/site-packages/tornado/gen.py\", line 742, in run\n yielded = self.gen.throw(*exc_info) # type: ignore\n"," File \"/usr/local/lib/python3.7/site-packages/subscriber_import_evaluation/services.py\", line 65, in _http_fetch\n request_timeout=request_timeout, **kwargs)\n"," File \"/usr/local/lib/python3.7/site-packages/tornado/gen.py\", line 735, in run\n value = future.result()\n","tornado.httpclient.HTTPClientError: HTTP 404: ListNotFound\n"]}
{"name":"rejected.process","levelname":"WARNING","asctime":"2020-07-16T12:54:18.281276+0000","processName":"subscriber-import-evaluation-1","message":"Rejecting message 1 with requeue","module":"process","correlation_id":"7a3a8390-10de-40ef-96be-100e8fa98596","exc_info":null}
{"name":"rejected.process","levelname":"INFO","asctime":"2020-07-16T12:54:18.281544+0000","processName":"subscriber-import-evaluation-1","message":"Resetting failure window, 1594904058 seconds since last","module":"process","correlation_id":"01ab189d-9c3e-4fb1-830b-eec0049dc434","exc_info":null}
The lookup fails with the account in the path, but succeeds without it?
GET http://list.service.production.consul/v1/accounts/1326226/lists/5202426{
"error": {
"status_code": 404,
"exception": "ListNotFound",
"message": "List with given id not found"
}
}
GET http://list.service.production.consul/v1/lists/5202426{
"friendly_list_name": "15 minute manifestation.",
"address_id": 779131,
"coi_confirmation_text": "",
"sender_name": "Geoff Stevens",
"coi_subject_text": "Response Required: Please confirm your request for information.\n",
"coi_button_text": "Confirm my subscription",
"contact_address": "Palmer Rd\nPortland vic 3305\nAUSTRALIA",
"list_name": "awlist5202426",
"custom_fields": {},
"coi_closing": "",
"city": "Portland",
"country_id": 13,
"zipcode": "3305",
"locale": "en-US",
"state": "vic",
"list_description": "0000000",
"coi_subject_id": 6,
"company_name": "Geoff Stevenson",
"contact_emails": [
{
"reply": 1,
"confirmation": 0,
"email": "gst1965@bigpond.com",
"name": "Geoff Stevens"
}
],
"list_id": 5202426,
"country_name": "AUSTRALIA",
"account_id": 1326226,
"address1": "Palmer Rd",
"address2": null,
"company_website": "",
"coi_bitmap": 10,
"confirmation_button_text_id": 1,
"sender_email": "gst1965@bigpond.com",
"signature": null
}
Popped the offending message off the queue: Properties:
app_id: subscriberimportapi/3.0.11 type: import_requested.v1 timestamp: 1594892197 message_id: e4b0e88c-d90c-4d1f-bf22-863ca6a140a7 correlation_id: 43b4305c-02fa-4e5a-aba1-e7db9c4bb926 headers: account_id: 62949269-3df5-46d9-b883-74e7771504f2 x-shovelled: dest-exchange: events dest-uri: amqp://rabbitmq.aweberprod.com:5672/%2f shovel-name: rabbitmq-events shovel-type: dynamic shovel-vhost: / shovelled-by: production src-queue: rabbitmq-events src-uri: amqp:// content_type: application/vnd.apache.avro.datum
Payload:
SDYyOTQ5MjY5LTNkZjUtNDZkOS1iODgzLTc0ZTc3NzE1MDRmMkhmNGNkOGRkYy05NWMzLTQ0NWQtYWZiZS0zODZlOTNhMjk4NTSk8qEB9If7BEhjZTY0MTE2 OS1hNzVkLTQ1YmMtYmM0OC1lZjhhOGJkYTMxYzhAMjAyMC0wNy0xNlQwOTozNjozNy45ODkwMzUrMDA6MDDYAkFjcXVpc2l0aW9uIG1ldGhvZDogIm90aGVy IgpQcm92aWRlcjogRGlkIG5vdCBicmluZyBmcm9tIGFub3RoZXIgcHJvdmlkZXIKRGVzY3JpYmUgaG93IHlvdSBnb3QgdGhpcyBsaXN0IG9mIHN1YnNjcmli ZXJzCklhbSBleHBvcnRpbmcgYW5kIHJlLWltcG9ydGluZyBhbmQgbWVyZ2luZyBteSBsaXN0cy4BAAgKZW1haWwACG5hbWUCCG5vdGUQCHRhZ3MmAAAcMTEw LjE0NC40MS4yNTAAAA==
There seem to be more messages failing the same way, it'd probably be faster to just make a code change to drop messages on 4XX errors.
Login and Landing pages
# Send logged in customers to /users
RewriteCond %{HTTP:Cookie} (^|;\ *)loggedIn=1 [NC]
RewriteRule ^/(landing|login).htm$ /users [L,R=302]
loginAjax only appears to be hit from the landing/login pages:
root@web-controlpanel1:/home/aweber/logs# grep loginAjax httpd/access_log |cut -d ' ' -f 14|cut -d '?' -f 1|sed 's/"//g'|sort|uniq -c
9 -
2244 https://www.aweber.com/landing.htm
500 https://www.aweber.com/login.htm