From d13b06a454be35a9e22ad80575a6765cec690898 Mon Sep 17 00:00:00 2001 From: Correl Roush Date: Thu, 9 Jun 2022 15:46:25 -0400 Subject: [PATCH] updates --- ...hs_1reports_1new_subscribers_daily_get.org | 0 aweber/20210813142844-projects.org | 4 ++- .../20210915133101-analytics_view_service.org | 33 +++++++++++++++++++ aweber/20220520154427-recipient_service.org | 1 + aweber/20220609145347-retire_redcache.org | 6 ++++ ...truth_for_subscriber_data_in_recipient.org | 33 +++++++++++++++++++ ...0220609151255-subscriber_sync_consumer.org | 5 +++ 7 files changed, 81 insertions(+), 1 deletion(-) create mode 100644 20220609135721-http_analytics_view_service_production_consul_static_index_html_tag_reports_paths_1reports_1new_subscribers_daily_get.org create mode 100644 aweber/20220609145347-retire_redcache.org create mode 100644 aweber/20220609150836-use_appdb_as_the_source_of_truth_for_subscriber_data_in_recipient.org create mode 100644 aweber/20220609151255-subscriber_sync_consumer.org diff --git a/20220609135721-http_analytics_view_service_production_consul_static_index_html_tag_reports_paths_1reports_1new_subscribers_daily_get.org b/20220609135721-http_analytics_view_service_production_consul_static_index_html_tag_reports_paths_1reports_1new_subscribers_daily_get.org new file mode 100644 index 0000000..e69de29 diff --git a/aweber/20210813142844-projects.org b/aweber/20210813142844-projects.org index eb28788..e7ca6bf 100644 --- a/aweber/20210813142844-projects.org +++ b/aweber/20210813142844-projects.org @@ -54,7 +54,7 @@ :LOGBOOK: - State "BACKLOG" from [2021-10-26 Tue 14:02] :END: -** BACKLOG Deploying Recipient Service to Kubernetes +** DONE Deploying Recipient Service to Kubernetes :LOGBOOK: - State "BACKLOG" from "TODO" [2021-10-20 Wed 15:53] - State "TODO" from [2021-10-13 Wed 16:26] @@ -75,6 +75,7 @@ :LOGBOOK: - State "TODO" from [2021-10-20 Wed 15:57] :END: +* [[id:ee5b8d5f-e3d4-45c2-9ce6-bcd8c7a63376][Retire Redcache]] * [[id:4df15f2f-d2e1-40f4-8acd-dbfb78fe304f][Deploy CoreAPI to Kubernetes]] * Replacing CAPI Services ** TODO [[id:619b6c78-7be9-4ee4-a0b7-9d1a4d7536e2][Migrating services to use the new List service]] @@ -90,6 +91,7 @@ endpoints be in the new service as well, deprecating public api lists? - [2021-10-18 Mon] :: The expectation is set to be migrated to the new list service exclusively by the end of Q2 2022 ** Subscribers API +*** [[id:2c1a7b1d-8726-4b88-9534-2f5abfec35f0][Use AppDB as the source of truth for subscriber data in Recipient]] * Frontend Client Upgrades ** Upgrade Dashboard to React *** BACKLOG Create an API for broadcasts and sent messages across lists diff --git a/aweber/20210915133101-analytics_view_service.org b/aweber/20210915133101-analytics_view_service.org index 2cb3129..9d047fe 100644 --- a/aweber/20210915133101-analytics_view_service.org +++ b/aweber/20210915133101-analytics_view_service.org @@ -2,6 +2,7 @@ :ID: c45881de-46f2-4f76-9579-063626c5956c :END: #+title: Analytics View Service +#+TODO: WAITING(w) READY(r) | DONE(d) The Analytics View Service provides a collection of report endpoints. These endpoints handle querying the analytics databases or, in the longer term, @@ -702,3 +703,35 @@ Uses the =public.leads_stats_day= table in AppDB. #+end_src ** Migrate reports to the Analytics View Service +*** READY Use the new analytics view endpoints for the opens over time reports +Update the opens over time report for [[https://www.aweber.com/users/report/opens_all][all lists]] and the [[https://www.aweber.com/users/report/opens][current list]] to use the +new [[http://analytics-view.service.production.consul/static/index.html#tag/Reports/paths/~1reports~1opens/get][daily opens endpoint]] in the Analytics View service. +*** READY Use the new analytics view endpoints for the clicks over time reports +Update the clicks over time report for [[https://www.aweber.com/users/report/clicks_all][all lists]] and the [[https://www.aweber.com/users/report/clicks][current list]] to use the +new [[http://analytics-view.service.production.consul/static/index.html#tag/Reports/paths/~1reports~1clicks/get][daily clicks endpoint]] in the Analytics View service. +*** WAITING Use the new analytics view endpoints for the sales over time report +Update the [[https://www.aweber.com/users/report/sales_tracked_all][sales over time report]] to use the new [[http://analytics-view.service.production.consul/static/index.html#tag/Reports/paths/~1reports~1sales-by-day/get][daily sales endpoint]] for graph +summary data and the sale events endpoint for the sales table. + +#+begin_notes +The sale events endpoint present, but is not yet documented! +#+end_notes +*** DONE New subscribers +Update the new subscribers report to use the new [[http://analytics-view.service.production.consul/static/index.html#tag/Reports/paths/~1reports~1new-subscribers-daily/get][daily]], [[http://analytics-view.service.production.consul/static/index.html#tag/Reports/paths/~1reports~1new-subscribers-weekly/get][weekly]], and [[http://analytics-view.service.production.consul/static/index.html#tag/Reports/paths/~1reports~1new-subscribers-monthly/get][monthly]] new +subscriber endpoints in the Analytics View service. +*** DONE Subscriber totals +Update the subscriber totals report to use the new [[http://analytics-view.service.production.consul/static/index.html#tag/Reports/paths/~1reports~1total-subscribers-daily/get][daily]], [[http://analytics-view.service.production.consul/static/index.html#tag/Reports/paths/~1reports~1total-subscribers-weekly/get][weekly]], and [[http://analytics-view.service.production.consul/static/index.html#tag/Reports/paths/~1reports~1total-subscribers-monthly/get][monthly]] +total subscriber endpoints in the Analytics View service. +*** WAITING Use the new analytics view endpoints for the broadcast totals report +Update the [[https://www.aweber.com/users/report/broadcast_totals][broadcast totals report]] to use the new completed broadcasts endpoint +in the Analytics View service. + +#+begin_notes +The completed broadcasts endpoint is [[https://jira.aweber.io/browse/CCPANEL-11788][not yet complete]]. +#+end_notes +*** Use the new analytics view endpoints for the follow-up totals report +Update the [[https://www.aweber.com/users/report/followup_totals][followup totals report]] to use the new [[http://analytics-view.service.production.consul/static/index.html#tag/Reports/paths/~1reports~1followup-totals/get][followup totals]] endpoint in the +Analytics View service. +*** Use the new analytics view endpoints for the location totals report +Update the [[https://www.aweber.com/users/report/subscribers_by_location][location totals report]] to use the new [[http://analytics-view.service.production.consul/static/index.html#tag/Reports/paths/~1reports~1subscribers-by-location/get][subscribers by location]] +endpoint in the Analytics View service. diff --git a/aweber/20220520154427-recipient_service.org b/aweber/20220520154427-recipient_service.org index 39f10cf..a40733e 100644 --- a/aweber/20220520154427-recipient_service.org +++ b/aweber/20220520154427-recipient_service.org @@ -1,4 +1,5 @@ :PROPERTIES: :ID: 1ff6586e-2dba-41a2-a887-753cc5ac27c9 +:ROAM_REFS: https://gitlab.aweber.io/CP/Services/recipient :END: #+title: Recipient Service diff --git a/aweber/20220609145347-retire_redcache.org b/aweber/20220609145347-retire_redcache.org new file mode 100644 index 0000000..6b1b6ac --- /dev/null +++ b/aweber/20220609145347-retire_redcache.org @@ -0,0 +1,6 @@ +:PROPERTIES: +:ID: ee5b8d5f-e3d4-45c2-9ce6-bcd8c7a63376 +:END: +#+title: Retire Redcache + +https://jira.aweber.io/browse/AWEB-378 diff --git a/aweber/20220609150836-use_appdb_as_the_source_of_truth_for_subscriber_data_in_recipient.org b/aweber/20220609150836-use_appdb_as_the_source_of_truth_for_subscriber_data_in_recipient.org new file mode 100644 index 0000000..b9e8a1c --- /dev/null +++ b/aweber/20220609150836-use_appdb_as_the_source_of_truth_for_subscriber_data_in_recipient.org @@ -0,0 +1,33 @@ +:PROPERTIES: +:ID: 2c1a7b1d-8726-4b88-9534-2f5abfec35f0 +:END: +#+title: Use AppDB as the source of truth for subscriber data in Recipient + +The [[id:1ff6586e-2dba-41a2-a887-753cc5ac27c9][Recipient Service]] shall replace [[id:aa9b1fdc-d766-41bb-ab7b-11c35bca54fa][AWSubscribers]] as the API for mailing list +subscribers. Currently, [[id:aa9b1fdc-d766-41bb-ab7b-11c35bca54fa][AWSubscribers]] is responsible for all subscriber creation +and updating and uses [[id:dd113e53-6144-4cb2-a4aa-da3dc2e3e6ea][AppDB]] as its data store, which is the current source of +truth for subscriber data. The [[id:1ff6586e-2dba-41a2-a887-753cc5ac27c9][Recipient Service]] maintains its own copy of +subscriber data in DynamoDB along with its additional recipient data, which is +kept up to date via the [[id:b285adee-2dab-48ed-b2d8-2df5594f9d30][Subscriber Sync Consumer]]. Having both services sharing +the same data store will allow us to migrate calls towards the [[id:1ff6586e-2dba-41a2-a887-753cc5ac27c9][Recipient Service]] +without risking data becoming desynchronized between them. + + + +* Read subscriber data from AppDB by default +Campaigns may fail as they race to retrieve information from the [[id:1ff6586e-2dba-41a2-a887-753cc5ac27c9][Recipient +Service]] for a newly created subscriber, which may not have synchronized over +from [[id:dd113e53-6144-4cb2-a4aa-da3dc2e3e6ea][AppDB]]. Reads are also the least impactful change to make, as it will only +affect the [[id:b285adee-2dab-48ed-b2d8-2df5594f9d30][Subscriber Sync Consumer]] as it checks whether to create or update a +record. This makes this change the ideal starting point for this project. + +The current DynamoDB behavior will be kept intact so that the [[id:b285adee-2dab-48ed-b2d8-2df5594f9d30][Subscriber Sync +Consumer]] can continue to keep the data stored in DynamoDB up to date. +** DONE Add support for reading subscriber data from AppDB to Recipient +** DONE Update subscriber-sync to use flag to read from DynamoDB + +** TODO Switch recipient to read subscriber data from AppDB by default + +* Write subscriber data to AppDB + +* Retire subscriber data in DynamoDB diff --git a/aweber/20220609151255-subscriber_sync_consumer.org b/aweber/20220609151255-subscriber_sync_consumer.org new file mode 100644 index 0000000..b32af32 --- /dev/null +++ b/aweber/20220609151255-subscriber_sync_consumer.org @@ -0,0 +1,5 @@ +:PROPERTIES: +:ID: b285adee-2dab-48ed-b2d8-2df5594f9d30 +:ROAM_REFS: https://gitlab.aweber.io/CP/Consumers/subscriber-sync +:END: +#+title: Subscriber Sync Consumer