:PROPERTIES: :ID: c45881de-46f2-4f76-9579-063626c5956c :END: #+title: Analytics View Service #+TODO: WAITING(w) READY(r) | DONE(d) The Analytics View Service provides a collection of report endpoints. These endpoints handle querying the analytics databases or, in the longer term, exposing efficient materialized data views. * Plan - Parent ticket :: [[https://jira.aweber.io/browse/CCPANEL-11781][CCPANEL-11781]] ** Create the analytics view service - New project using cookie cutter - Deployed to kubernetes - Grafana dashboard created ** Create the analytics view service playbook ** Plan API structure - Pathing (=/reports/*=)? - Report versioning? (=/reports/$NAME.v$VERSION=)? ** Create endpoints for existing reports - Based on the endpoints provided in the [[id:3ddc4e32-932f-4748-bfe9-7025d4d6b352][Report API Controller]] - Are all of these report endpoints in use? *** Opens over time #+attr_confluence: :as-table t - Name :: daily-opens - Parameters :: - List (default: all lists) - Date range (default: last 30 days) - Report API controller endpoints :: - opens_all_range - opens_list_range #+caption: Sample response #+begin_src json { "2021-11-02T00:00:00Z": { "broadcasts": 2499, "followups": 2547, "unique": 2923, "total": 5046 }, "2021-11-03T00:00:00Z": { "broadcasts": 25808, "followups": 2430, "unique": 24876, "total": 28238 }, "2021-11-04T00:00:00Z": { "broadcasts": 16733, "followups": 1437, "unique": 14780, "total": 18170 } } #+end_src *** Clicks over time #+attr_confluence: :as-table t - Name :: daily-clicks - Parameters :: - List (default: all lists) - Date range (default: last 30 days) - Report API controller endpoints :: - clicks_all_range - clicks_list_range #+caption: Sample response #+begin_src json { "2021-11-02T00:00:00Z": { "broadcasts": 105, "followups": 137, "unique": 130, "total": 242 }, "2021-11-03T00:00:00Z": { "broadcasts": 636, "followups": 185, "unique": 622, "total": 821 }, "2021-11-04T00:00:00Z": { "broadcasts": 480, "followups": 109, "unique": 426, "total": 589 } } #+end_src *** Sales over time (events) #+attr_confluence: :as-table t - Name :: sale-events - Parameters :: - Date range (default: last 30 days) - Currency (default: USD) - Report API controller endpoints :: - sales_tracked_events #+caption: Sample response #+begin_src json [ { "time": "2021-11-02 09:37:36-04", "type": "followup", "currency": "USD", "revenue": "19.00", "note": "", "description": "Upgraded to Pro", "source_url": "https://www.aweber.com/users/#upgraded", "email": "team@harmoniamedia.com" }, { "time": "2021-11-02 09:37:37-04", "type": "followup", "currency": "USD", "revenue": "19.00", "note": "", "description": "Upgraded to Pro", "source_url": "https://www.aweber.com/users/#upgraded", "email": "team@harmoniamedia.com" }, { "time": "2021-11-02 12:01:17-04", "type": "followup", "currency": "USD", "revenue": "19.00", "note": "", "description": "Upgraded to Pro", "source_url": "https://www.aweber.com/users/#upgraded", "email": "giuliagiardino12@gmail.com" }, { "time": "2021-11-02 12:01:19-04", "type": "followup", "currency": "USD", "revenue": "19.00", "note": "", "description": "Upgraded to Pro", "source_url": "https://www.aweber.com/users/#upgraded", "email": "giuliagiardino12@gmail.com" }, { "time": "2021-11-04 05:21:35-04", "type": "broadcast", "currency": "USD", "revenue": "19.00", "note": "", "description": "Upgraded to Pro", "source_url": "https://www.aweber.com/users/#upgraded", "email": "jeremy@jeremy-quick.com" }, { "time": "2021-11-04 05:21:36-04", "type": "broadcast", "currency": "USD", "revenue": "19.00", "note": "", "description": "Upgraded to Pro", "source_url": "https://www.aweber.com/users/#upgraded", "email": "jeremy@jeremy-quick.com" } ] #+end_src *** Sales over time (summary) #+attr_confluence: :as-table t - Name :: daily-sales - Parameters :: - Date range (default: last 60 days) - Currency (default: USD) #+caption: Sample response #+begin_src json { "2021-11-02T00:00:00Z": { "broadcast": 0, "followup": 76, "pageview": 76, "ecommerce": 0, "total": 76 }, "2021-11-03T00:00:00Z": { "broadcast": 0, "followup": 0, "pageview": 0, "ecommerce": 0, "total": 0 }, "2021-11-04T00:00:00Z": { "broadcast": 38, "followup": 0, "pageview": 38, "ecommerce": 0, "total": 38 } } #+end_src *** Lifetime Sale Totals #+attr_confluence: :as-table t - Name :: sale-totals - Parameters :: - Currency (default: USD) - Report API controller endpoints :: - sales_tracked_total #+caption: Sample response #+begin_src json { "count": 94924, "pagehits": 1820122.79, "revenue": 2181498.96 } #+end_src *** Sale Currencies #+attr_confluence: :as-table t - Name :: sale-currencies - Parameters :: - Date range (default: last 60 days) - Report API controller endpoints :: - sales_tracked_currencies #+caption: Sample response #+begin_src json ["USD", "CAD"] #+end_src *** Pending Broadcasts #+attr_confluence: :as-table t - Name :: pending-broadcasts - Parameters :: - List (default: all lists) - Report API controller endpoints :: - broadcasts_pending #+caption: Sample response #+begin_src json [ { "unit_id": "3854", "send_date": "01/01/25 12:00am", "broadcast_id": "1800243", "subject": "Scheduled Broadcast Test", "status": "Queue", "percent_done": "0", "mesg_encoding": "utf-8", "campaign_id": "1802217", "unit": "awlist3854", "orig_send_date": "2025-01-01 00:00:00-05", "friendly_list_name": "Fluff Cafe", "id": "1802217", "list_id": "3854", "account_id": "778", "campaign_type_id": "b", "uses_block_editor": "t" } ] #+end_src *** Completed Broadcasts #+attr_confluence: :as-table t - Name :: completed-broadcasts - Parameters :: - List (default: all lists) - Report API controller endpoints :: - broadcasts_completed - broadcasts_completed_all #+caption: Sample response #+begin_src json [ { "a_id": "778", "broadcast_id": "1258855", "for_sent_date": "08/14/20 03:41 PM", "track_click_rate": "1", "unit_id": "3854", "assassin_pts": "0", "mesg_type": "HTML", "mesg_encoding": "utf-8", "sent_date": "2020-08-14 15:41:05.24086-04", "num_emailed": "1", "num_undeliv": 0, "num_opened": 0, "num_attachments": 0, "num_complaints": 0, "subject": "Testing a bad segment", "created_date": "2020-08-14 15:38:57.59072-04", "status": "Sent Composer", "lead_view_id": "42564", "unit": "awlist3854", "pct_opened": "0", "show_opens_warning": true, "pct_undeliv": "0", "total_clicks": 0, "clicks_analytics_type": "premium", "pct_click": "0", "pct_complaints": "0", "segment_id": "42564", "extra_lists": "", "excluded_lists": "", "extra_lists_count": 0, "excluded_lists_count": 0, "friendly_list_name": "Fluff Cafe" }, { "a_id": "778", "broadcast_id": "1243650", "for_sent_date": "08/05/20 06:33 PM", "track_click_rate": "1", "unit_id": "3854", "assassin_pts": "0", "mesg_type": "Text/HTML", "mesg_encoding": "utf-8", "sent_date": "2020-08-05 18:33:48.638875-04", "num_emailed": "27", "num_undeliv": 0, "num_opened": 0, "num_attachments": 0, "num_complaints": 0, "subject": "Buggssss 🐛🐛🐛🐛🐛🐛🐛", "created_date": "2020-08-05 18:31:10.567418-04", "status": "Sent Composer", "lead_view_id": "8", "unit": "awlist3854", "pct_opened": "0", "show_opens_warning": true, "pct_undeliv": "0", "total_clicks": 0, "clicks_analytics_type": "premium", "pct_click": "0", "pct_complaints": "0", "segment_id": "8", "extra_lists": "", "excluded_lists": "", "extra_lists_count": 0, "excluded_lists_count": 0, "friendly_list_name": "Fluff Cafe" } ] #+end_src *** Cities, States, and Countries #+attr_confluence: :as-table t - Name :: subscribers-by-location - Parameters :: - List (default: all lists) - Report API controller endpoints :: - city_state_countries #+caption: Sample response #+begin_src json [ { "country": null, "state": null, "city": null, "pending": 0, "unsubscribed": 457262, "subscribed": 276245, "total": 733507 }, { "country": "US", "state": "PA", "city": "Newtown", "pending": 0, "unsubscribed": 279304, "subscribed": 3236, "total": 282540 }, { "country": "US", "state": null, "city": null, "pending": 0, "unsubscribed": 109074, "subscribed": 2456, "total": 111530 }, { "country": "US", "state": "PA", "city": "Philadelphia", "pending": 0, "unsubscribed": 48485, "subscribed": 657, "total": 49142 }, { "country": "US", "state": "PA", "city": "Huntingdon Valley", "pending": 0, "unsubscribed": 4839, "subscribed": 90, "total": 4929 }, { "country": "VN", "state": "64", "city": "Hanoi", "pending": 0, "unsubscribed": 204, "subscribed": 2, "total": 206 }, { "country": "VN", "state": "65", "city": "Ho Chi Minh City", "pending": 0, "unsubscribed": 176, "subscribed": 1, "total": 177 } ] #+end_src *** Campaign Statistics Add the campaign start endpoint into analytics-view including - DynamoDB fixtures - Dynamo dbhelpers (currently there is no dynamo connectivity in analytics-view) - CampaignStarted Handler https://gitlab.aweber.io/CP/Services/campaignstats/-/blob/master/campaignstats/handlers.py - Grafana dashboard updated - Alerts configured - Confluence docs updated *** Followups #+attr_confluence: :as-table t - Name :: followup-totals - Parameters :: - List (default: all lists) - Report API controller endpoints :: - followups Uses the =freq_mesg= and =freq_mesg_stats= tables in AppDB combined with data from the =messages= table in Analytics. =editor_path= is hard-coded in the current report endpoint. #+caption: Sample Response (Production AID 91) #+begin_src json [ { "followup_message": "1", "num_emailed": 83024, "num_opened": 89950, "open_percentage": 23.659423781075, "clicks": "13198", "message_id": "28613652", "click_percentage": 0, "unique_opens": "19643", "unique_clicks": "1584", "clicks_percentage": 23.659423781075, "editor_path": "messages#/active" }, { "followup_message": "2", "num_emailed": 73714, "num_opened": 49979, "open_percentage": 13.675828200884, "clicks": "6169", "message_id": "28613655", "click_percentage": 0, "unique_opens": "10081", "unique_clicks": "649", "clicks_percentage": 13.675828200884, "editor_path": "messages#/active" }, { "followup_message": "3", "num_emailed": 70880, "num_opened": 38472, "open_percentage": 10.272291196388, "clicks": "4322", "message_id": "28613659", "click_percentage": 0, "unique_opens": "7281", "unique_clicks": "409", "clicks_percentage": 10.272291196388, "editor_path": "messages#/active" }, { "followup_message": "4", "num_emailed": 69375, "num_opened": 37496, "open_percentage": 10.105945945946, "clicks": "4419", "message_id": "28613661", "click_percentage": 0, "unique_opens": "7011", "unique_clicks": "447", "clicks_percentage": 10.105945945946, "editor_path": "messages#/active" }, { "followup_message": "5", "num_emailed": 67478, "num_opened": 30996, "open_percentage": 8.2752897240582, "clicks": "3236", "message_id": "28613662", "click_percentage": 0, "unique_opens": "5584", "unique_clicks": "257", "clicks_percentage": 8.2752897240582, "editor_path": "messages#/active" }, { "followup_message": "6", "num_emailed": 65414, "num_opened": 32483, "open_percentage": 9.2334974164552, "clicks": "3127", "message_id": "28613664", "click_percentage": 0, "unique_opens": "6040", "unique_clicks": "323", "clicks_percentage": 9.2334974164552, "editor_path": "messages#/active" }, { "followup_message": "7", "num_emailed": 63871, "num_opened": 28756, "open_percentage": 8.1883797028385, "clicks": "1964", "message_id": "28613665", "click_percentage": 0, "unique_opens": "5230", "unique_clicks": "166", "clicks_percentage": 8.1883797028385, "editor_path": "messages#/active" } ] #+end_src *** New Subscribers Daily #+attr_confluence: :as-table t - Name :: daily-new-subscribers - Parameters :: - List (default: all lists) - Report API controller endpoints :: - new_subscribers_daily Uses the =public.leads_stats_day= table in AppDB. #+caption: Sample Response (Production AID 91) #+begin_src json { "2022-01-09T00:00:00Z": { "subscribed": 324, "unsubscribed": 83 }, "2022-01-10T00:00:00Z": { "subscribed": 417, "unsubscribed": 80 }, "2022-01-11T00:00:00Z": { "subscribed": 433, "unsubscribed": 92 }, ... } #+end_src *** New Subscribers Weekly #+attr_confluence: :as-table t - Name :: weekly-new-subscribers - Parameters :: - List (default: all lists) - Report API controller endpoints :: - new_subscribers_weekly Uses the =public.leads_stats_day= table in AppDB. #+caption: Sample Response (Production AID 91) #+begin_src json { "2021-03-26T00:00:00Z": { "subscribed": 3182, "unsubscribed": 1249 }, "2021-04-02T00:00:00Z": { "subscribed": 3423, "unsubscribed": 1497 }, "2021-04-09T00:00:00Z": { "subscribed": 3052, "unsubscribed": 1217 }, ... } #+end_src *** New Subscribers Monthly #+attr_confluence: :as-table t - Name :: monthly-new-subscribers - Parameters :: - List (default: all lists) - Report API controller endpoints :: - new_subscribers_monthly Uses the =public.leads_stats_day= table in AppDB. #+caption: Sample Response (Production AID 91) #+begin_src json { "2021-03-01T00:00:00Z": { "subscribed": 14972, "unsubscribed": 4770 }, "2021-04-01T00:00:00Z": { "subscribed": 14973, "unsubscribed": 7181 }, "2021-05-01T00:00:00Z": { "subscribed": 12652, "unsubscribed": 5243 }, ... } #+end_src *** Subscriber Totals Daily #+attr_confluence: :as-table t - Name :: subscribers-by-location - Parameters :: - List (default: all lists) - Report API controller endpoints :: - subscriber_totals_daily Uses the =public.leads_stats_day= table in AppDB. #+caption: Sample Response (Production AID 91) #+begin_src json { "2022-01-09T00:00:00Z": { "subscribed": 289192, "unsubscribed": 920812 }, "2022-01-10T00:00:00Z": { "subscribed": 289609, "unsubscribed": 920892 }, "2022-01-11T00:00:00Z": { "subscribed": 290042, "unsubscribed": 920984 }, ... } #+end_src *** Subscriber Totals Weekly #+attr_confluence: :as-table t - Name :: subscribers-by-location - Parameters :: - List (default: all lists) - Report API controller endpoints :: - subscriber_totals_weekly Uses the =public.leads_stats_day= table in AppDB. #+caption: Sample Response (Production AID 91) #+begin_src json { "2021-02-08T00:00:00Z": { "subscribed": 191745, "unsubscribed": 887713 }, "2021-02-15T00:00:00Z": { "subscribed": 193972, "unsubscribed": 888733 }, "2021-02-22T00:00:00Z": { "subscribed": 196200, "unsubscribed": 889717 }, ... } #+end_src *** Subscriber Totals Monthly #+attr_confluence: :as-table t - Name :: subscribers-by-location - Parameters :: - List (default: all lists) - Report API controller endpoints :: - subscriber_totals_monthly Uses the =public.leads_stats_day= table in AppDB. #+caption: Sample Response (Production AID 91) #+begin_src json { "2021-02-01T00:00:00Z": { "subscribed": 196200, "unsubscribed": 889717 }, "2021-03-01T00:00:00Z": { "subscribed": 206569, "unsubscribed": 894298 }, "2021-04-01T00:00:00Z": { "subscribed": 215813, "unsubscribed": 900020 }, ... } #+end_src ** Migrate reports to the Analytics View Service *** READY Use the new analytics view endpoints for the opens over time reports Update the opens over time report for [[https://www.aweber.com/users/report/opens_all][all lists]] and the [[https://www.aweber.com/users/report/opens][current list]] to use the new [[http://analytics-view.service.production.consul/static/index.html#tag/Reports/paths/~1reports~1opens/get][daily opens endpoint]] in the Analytics View service. *** READY Use the new analytics view endpoints for the clicks over time reports Update the clicks over time report for [[https://www.aweber.com/users/report/clicks_all][all lists]] and the [[https://www.aweber.com/users/report/clicks][current list]] to use the new [[http://analytics-view.service.production.consul/static/index.html#tag/Reports/paths/~1reports~1clicks/get][daily clicks endpoint]] in the Analytics View service. *** WAITING Use the new analytics view endpoints for the sales over time report Update the [[https://www.aweber.com/users/report/sales_tracked_all][sales over time report]] to use the new [[http://analytics-view.service.production.consul/static/index.html#tag/Reports/paths/~1reports~1sales-by-day/get][daily sales endpoint]] for graph summary data and the sale events endpoint for the sales table. #+begin_notes The sale events endpoint present, but is not yet documented! #+end_notes *** DONE New subscribers Update the new subscribers report to use the new [[http://analytics-view.service.production.consul/static/index.html#tag/Reports/paths/~1reports~1new-subscribers-daily/get][daily]], [[http://analytics-view.service.production.consul/static/index.html#tag/Reports/paths/~1reports~1new-subscribers-weekly/get][weekly]], and [[http://analytics-view.service.production.consul/static/index.html#tag/Reports/paths/~1reports~1new-subscribers-monthly/get][monthly]] new subscriber endpoints in the Analytics View service. *** DONE Subscriber totals Update the subscriber totals report to use the new [[http://analytics-view.service.production.consul/static/index.html#tag/Reports/paths/~1reports~1total-subscribers-daily/get][daily]], [[http://analytics-view.service.production.consul/static/index.html#tag/Reports/paths/~1reports~1total-subscribers-weekly/get][weekly]], and [[http://analytics-view.service.production.consul/static/index.html#tag/Reports/paths/~1reports~1total-subscribers-monthly/get][monthly]] total subscriber endpoints in the Analytics View service. *** WAITING Use the new analytics view endpoints for the broadcast totals report Update the [[https://www.aweber.com/users/report/broadcast_totals][broadcast totals report]] to use the new completed broadcasts endpoint in the Analytics View service. #+begin_notes The completed broadcasts endpoint is [[https://jira.aweber.io/browse/CCPANEL-11788][not yet complete]]. #+end_notes *** Use the new analytics view endpoints for the follow-up totals report Update the [[https://www.aweber.com/users/report/followup_totals][followup totals report]] to use the new [[http://analytics-view.service.production.consul/static/index.html#tag/Reports/paths/~1reports~1followup-totals/get][followup totals]] endpoint in the Analytics View service. *** Use the new analytics view endpoints for the location totals report Update the [[https://www.aweber.com/users/report/subscribers_by_location][location totals report]] to use the new [[http://analytics-view.service.production.consul/static/index.html#tag/Reports/paths/~1reports~1subscribers-by-location/get][subscribers by location]] endpoint in the Analytics View service.