updates

aweber/20220630170620-addlead_rewrite.org

@@ -0,0 +1,10 @@
+:PROPERTIES:
+:ID:       9adaef5d-9cfa-424f-966b-64fd558a5122
+:END:
+#+title: Addlead Rewrite
+
+A [[id:207560cc-7700-4d06-918d-cc01ae530146][Project]] to rewrite [[id:03e00c18-99c0-477c-b7fb-95ddc538755e][Addlead]] in Python as a modern, highly available service.
+
+- [[id:d30d8d19-f9bd-4c98-827a-5895e3902688][Addlead Rewrite ACP]]
+
+* New Product Requirements

aweber/20220706110652-addlead_rewrite_acp.org

@@ -0,0 +1,38 @@
+:PROPERTIES:
+:ID:       d30d8d19-f9bd-4c98-827a-5895e3902688
+:END:
+#+title: Addlead Rewrite ACP
+
+* Purpose
+The purpose of this ACP is to replace the existing subscriber web form endpoint
+with a modern implementation using our current language, library, and deployment
+standards.
+* Problem
+Addlead is currently written in Perl, relying on old libraries, and is running
+on deprecated VM hardware deployed with Chef.
+* Current State
+Addlead is far behind our current technology stack and standard practices:
+- It is written in Perl, which is not one of our supported languages.
+  Increasingly fewer engineers are familiar enough with it to make modifications
+  to it.
+- Its libraries are no longer being actively maintained.
+- It is running on old hardware which would be difficult to reprovision if lost.
+- It is deployed using Chef, which we have deprecated and are actively removing
+  from our stack.
+* Requirements
+* Terminology
+* Proposed Solution
+** Addlead HTTP Endpoint
+- Accept web form POST
+- Validate request
+  - If validation services are unavailable, proceed as though validation
+    succeeded. Validation will be applied when the event is processed.
+- Submit subscriber event to RabbitMQ
+  - If RabbitMQ is unavailable, store the event in a failover queue (S3, shared
+    volume?)
+** Failover Queue Consumer
+This will be a job that runs periodically to take any events in the failover
+queue and deliver them to RabbitMQ, removing them once they've been accepted by
+RabbitMQ.
+** Addlead Consumer
+Listens for subscription requests and processes them using the subscriber API.

daily/2022-06-24.org

@@ -0,0 +1,143 @@
+:PROPERTIES:
+:ID:       48fbbbdd-3532-4567-ab46-fb1c8a166c1a
+:END:
+#+title: 2022-06-24
+* Broadcasts report
+:PROPERTIES:
+:header-args:sql: :engine postgresql :dbhost app.service.staging.consul :database app :dbport 6000
+:header-args: :var account="26b49bc2-207e-4a33-b267-5b9e64f2702e"
+:END:
+
+#+begin_src sql :exports both :eval no-export
+  WITH broadcasts AS (
+     SELECT d.a_id,
+            a.a_serv_id,
+            a.friendly_list_name,
+            CASE WHEN b.service_id = 7 THEN 'premium' else 'traditional' END AS analytics_type,
+            e.broadcast_id,
+            e.track_click_rate,
+            e.assassin_pts,
+            e.mesg_type,
+            e.mesg_encoding,
+            e.created_date,
+            e.sent_date,
+            e.subject,
+            e.status,
+            COALESCE(e.num_emailed, 0) AS num_emailed,
+            COALESCE(e.num_undeliv, 0) AS num_undeliv,
+            COALESCE(e.num_opened, 0) AS num_opened,
+            COALESCE(e.num_attachments, 0) AS num_attachments,
+            COALESCE(e.num_complaints, 0) AS num_complaints,
+            CASE WHEN e.num_emailed > 0
+                 THEN e.num_emailed - e.num_undeliv
+                 ELSE 0
+                 END AS num_sent,
+            CASE WHEN e.num_emailed > 0
+                 THEN (e.num_opened / (e.num_emailed - e.num_undeliv)) * 100
+                 ELSE 0
+                 END AS pct_opened,
+            CASE WHEN e.num_emailed > 0
+                 THEN (e.num_undeliv / num_emailed) * 100
+                 ELSE 0
+                 END AS pct_undeliv
+       FROM public.autoresponders as a
+  LEFT JOIN public.account_services as b
+         ON b.a_serv_id = a.a_serv_id
+  LEFT JOIN public.account_packages as c
+         ON c.a_pkg_id = b.a_pkg_id
+  LEFT JOIN public.accounts as d
+         ON d.a_id = c.a_id
+  LEFT JOIN cust_mail_list as e
+         ON e.unit_id = a.a_serv_id
+      WHERE a.a_serv_id IS NOT NULL
+        AND a.deactivate_date IS NULL
+        AND account='$account'
+        AND e.send_date IS NOT NULL
+        AND e.sent_date IS NOT NULL
+   ORDER BY e.sent_date DESC
+      LIMIT 10
+  ),
+  extra_lists AS (
+  SELECT cust_mail_list_extra.unit_id,
+         cust_mail_list_extra.created_date,
+         COALESCE(autoresponders.friendly_list_name, autoresponders.unit) AS friendly_list_name
+    FROM autoresponders
+    JOIN cust_mail_list_extra ON (cust_mail_list_extra.extra_unit_id = autoresponders.a_serv_id)
+  ),
+  excluded_lists AS (
+  SELECT cust_mail_list_exclude.unit_id,
+         cust_mail_list_exclude.created_date,
+         COALESCE(autoresponders.friendly_list_name, autoresponders.unit) AS friendly_list_name
+    FROM autoresponders
+    JOIN cust_mail_list_exclude ON (cust_mail_list_exclude.excluded_unit_id = autoresponders.a_serv_id)
+  )
+  SELECT broadcasts.a_id,
+         broadcasts.a_serv_id,
+         broadcasts.friendly_list_name,
+         broadcasts.analytics_type,
+         broadcasts.broadcast_id,
+         broadcasts.track_click_rate,
+         broadcasts.assassin_pts,
+         broadcasts.mesg_type,
+         broadcasts.mesg_encoding,
+         broadcasts.created_date,
+         broadcasts.sent_date,
+         broadcasts.subject,
+         broadcasts.status,
+         broadcasts.num_emailed,
+         broadcasts.num_undeliv,
+         broadcasts.num_opened,
+         broadcasts.num_attachments,
+         broadcasts.num_complaints,
+         broadcasts.num_sent,
+         broadcasts.pct_opened,
+         broadcasts.pct_undeliv,
+         CASE WHEN broadcasts.num_complaints > 0
+              THEN (broadcasts.num_complaints / broadcasts.num_sent) * 100
+              ELSE 0
+              END AS pct_complaints,
+         COALESCE(SUM(click_tracking.clicks), 0) AS total_clicks,
+         COALESCE(SUM(click_tracking.clicks), 0) / broadcasts.num_sent * 100 AS pct_clicks,
+         ARRAY_AGG(extra_lists.friendly_list_name) AS extra_lists,
+         ARRAY_AGG(excluded_lists.friendly_list_name) AS excluded_lists
+  FROM broadcasts
+  LEFT OUTER JOIN click_tracking ON (click_tracking.a_serv_id = broadcasts.a_serv_id AND click_tracking.message_id LIKE broadcasts.created_date || '%')
+  LEFT OUTER JOIN extra_lists ON (extra_lists.unit_id = broadcasts.a_serv_id AND extra_lists.created_date = broadcasts.created_date)
+  LEFT OUTER JOIN excluded_lists ON (excluded_lists.unit_id = broadcasts.a_serv_id AND excluded_lists.created_date = broadcasts.created_date)
+  GROUP BY broadcasts.a_id,
+         broadcasts.a_serv_id,
+         broadcasts.friendly_list_name,
+         broadcasts.analytics_type,
+         broadcasts.broadcast_id,
+         broadcasts.track_click_rate,
+         broadcasts.assassin_pts,
+         broadcasts.mesg_type,
+         broadcasts.mesg_encoding,
+         broadcasts.created_date,
+         broadcasts.sent_date,
+         broadcasts.subject,
+         broadcasts.status,
+         broadcasts.num_emailed,
+         broadcasts.num_undeliv,
+         broadcasts.num_opened,
+         broadcasts.num_attachments,
+         broadcasts.num_complaints,
+         broadcasts.num_sent,
+         broadcasts.pct_opened,
+         broadcasts.pct_undeliv,
+         pct_complaints
+#+end_src
+
+#+RESULTS:
+| a_id | a_serv_id | friendly_list_name | analytics_type | broadcast_id | track_click_rate | assassin_pts | mesg_type | mesg_encoding | created_date                  | sent_date                     | subject                                              | status        | num_emailed | num_undeliv | num_opened | num_attachments | num_complaints | num_sent | pct_opened | pct_undeliv | pct_complaints | total_clicks | pct_clicks | extra_lists | excluded_lists |
+|------+-----------+--------------------+----------------+--------------+------------------+--------------+-----------+---------------+-------------------------------+-------------------------------+------------------------------------------------------+---------------+-------------+-------------+------------+-----------------+----------------+----------+------------+-------------+----------------+--------------+------------+-------------+----------------|
+|  778 |      3854 | Fluff Cafe         | traditional    |      2311911 |                1 |            0 | Text      | utf-8         | 2022-03-23 15:04:04.487716-04 | 2022-03-23 15:06:59.745136-04 | Tag is any normal                                    | Sent Composer |           4 |           0 |          0 |               0 |              0 |        4 |          0 |           0 |              0 |            0 |          0 | {NULL}      | {NULL}         |
+|  778 |      3854 | Fluff Cafe         | traditional    |      2311913 |                1 |            0 | Text      | utf-8         | 2022-03-23 15:08:08.081967-04 | 2022-03-23 15:11:00.814339-04 | Tag is normal and abnormal                           | Sent Composer |           4 |           0 |          0 |               0 |              0 |        4 |          0 |           0 |              0 |            0 |          0 | {NULL}      | {NULL}         |
+|  778 |      3854 | Fluff Cafe         | traditional    |      2311362 |                1 |            0 | Text      | utf-8         | 2022-03-23 10:46:53.288753-04 | 2022-03-23 10:49:26.144702-04 | Tag filter broadcast                                 | Sent Composer |          10 |           0 |          0 |               0 |              0 |       10 |          0 |           0 |              0 |            0 |          0 | {NULL}      | {NULL}         |
+|  778 |      3854 | Fluff Cafe         | traditional    |      2311371 |                1 |            0 | Text      | utf-8         | 2022-03-23 11:07:49.799343-04 | 2022-03-23 11:10:30.257927-04 | Tag filter broadcast is not normalized               | Sent Composer |          27 |           0 |          0 |               0 |              0 |       27 |          0 |           0 |              0 |            0 |          0 | {NULL}      | {NULL}         |
+|  778 |      3854 | Fluff Cafe         | traditional    |      2311370 |                1 |            0 | Text      | utf-8         | 2022-03-23 11:07:22.617798-04 | 2022-03-23 11:09:29.523583-04 | Tag filter broadcast is not repeated embedded spaces | Sent Composer |          27 |           0 |          0 |               0 |              0 |       27 |          0 |           0 |              0 |            0 |          0 | {NULL}      | {NULL}         |
+|  778 |      3854 | Fluff Cafe         | traditional    |      2311372 |                1 |            0 | Text      | utf-8         | 2022-03-23 11:08:47.329714-04 | 2022-03-23 11:11:30.586211-04 | Tag filter broadcast is not quoted tag               | Sent Composer |          31 |           0 |          0 |               0 |              0 |       31 |          0 |           0 |              0 |            0 |          0 | {NULL}      | {NULL}         |
+|  778 |      3854 | Fluff Cafe         | traditional    |      2311367 |                1 |            0 | Text      | utf-8         | 2022-03-23 11:03:26.973335-04 | 2022-03-23 11:05:28.547155-04 | Tag filter broadcast repeated embedded space         | Sent Composer |          14 |           0 |          0 |               0 |              0 |       14 |          0 |           0 |              0 |            0 |          0 | {NULL}      | {NULL}         |
+|  778 |      3854 | Fluff Cafe         | traditional    |      2311366 |                1 |            0 | Text      | utf-8         | 2022-03-23 11:02:27.559842-04 | 2022-03-23 11:04:28.553726-04 | Tag filter broadcast                                 | Sent Composer |          14 |           0 |          0 |               0 |              0 |       14 |          0 |           0 |              0 |            0 |          0 | {NULL}      | {NULL}         |
+|  778 |      3854 | Fluff Cafe         | traditional    |      2311909 |                1 |            0 | Text      | utf-8         | 2022-03-23 15:02:06.632764-04 | 2022-03-23 15:04:58.947163-04 | Tag is normalized                                    | Sent Composer |          13 |           0 |          0 |               0 |              0 |       13 |          0 |           0 |              0 |            0 |          0 | {NULL}      | {NULL}         |
+|  778 |      3854 | Fluff Cafe         | traditional    |      1977364 |                1 |            0 | HTML      | utf-8         | 2021-11-18 14:02:16.970285-05 | 2021-11-18 14:05:00.465057-05 | Message test                                         | Sent Composer |          33 |           0 |          0 |               0 |              0 |       33 |          0 |           0 |              0 |            0 |          0 | {NULL}      | {NULL}         |

daily/2022-06-30.org

@@ -0,0 +1,27 @@
+:PROPERTIES:
+:ID:       6400a3e6-75ac-4280-937b-4701bec0f5f9
+:END:
+#+title: 2022-06-30
+* Chatting with Gavin
+** [[id:03e00c18-99c0-477c-b7fb-95ddc538755e][Addlead]] storage
+Discussing queuing options for the [[id:9adaef5d-9cfa-424f-966b-64fd558a5122][Addlead Rewrite]].
+
+- Needs to be HIGHLY available
+- Could it function Lambda?
+- Similar availablity constraints as Analytics ingestion, which must function if
+  - AppDB is down
+  - RabbitMQ is down
+  - Could use redis as a persistent cache for url codes (on a standalone VM?)
+- Disk becomes more complicated in K8s (shared storage, locking, etc.)
+  - Issues with the CEPH filesystem may impede our ability to write (would affect CEPH-backed VMs as well)
+- Examining KeyDB as a highly-available alternative to Redis, still a lot of unknowns
+- Considering multiple layers of storage options (rabbitmq, fail over to SQS, ...)
+- Kafka could be an alternative to RabbitMQ for do-it-later queues
+- Consider the simplicity in disaster recovery of the current implementation
+  (take files on the server, put them in the right spot)
+- Amazon EFS file systems with ECS?
+- Note that signup forms / thank you pages are included in a [[https://confluence.aweber.io/pages/viewpage.action?spaceKey=DelDoc&title=Optin.com+Domain+URL+Layout][future state URL
+  layout]]
+  - Routing is handled by a PSE-maintained Kong instance running in Kubernetes
+    (so it would have to route to a service addressable in K8s)
+- RabbitMQ Streams https://wrobell.dcmod.org/rbfly/index.html?

daily/2022-07-01.org

@@ -0,0 +1,10 @@
+:PROPERTIES:
+:ID:       671ec1c5-90a9-4447-8570-359f5a2d2dc1
+:END:
+#+title: 2022-07-01
+* Continuing discussion on high-availability queuing
+Continuing the discussion from [[id:6400a3e6-75ac-4280-937b-4701bec0f5f9][2022-06-30]] on queuing options for the [[id:9adaef5d-9cfa-424f-966b-64fd558a5122][Addlead
+Rewrite]] and analytics ingestion, this time with both Gavin and Brian Jones.
+
+- Spike routing for optin to services running in AWS including Lambda
+- Plan for a layered approach with rabbitmq / ... (s3?)