:PROPERTIES:
:ID:       07210cf2-cca0-401e-8492-fb9bd4b0eba1
:END:
#+title: 2021-12-09
* [[id:bdea0611-e377-4378-a118-aef6d4a70bdf][CREASE]]
Session with Steve Eisner.

Reviewing the [[https://confluence.aweber.io/pages/viewpage.action?pageId=155890548.][notes]] on what we wanted to achieve in our [[id:29e51b04-ce89-4934-b17f-1f64bffc2069][last session]], which was
cut short.

Steve's current queue is a bit sparse, so we're looking through the Top ASE
Customer Requests and Upcoming Product Team Backlogs - CSLead SLAs sections of
the [[https://jira.aweber.io/secure/StructureBoard.jspa?s=156#][ASE structure board]].

** Etsy imports
Requires less babysitting now. Better than before, but not quite all the way
there.
** [[https://jira.aweber.io/browse/ASE-8301][Campaign messages]]
- Possibly related to [[https://jira.aweber.io/browse/CC-1759][CC-1759]]

Messages attached to a campaign and subsequently removed aren't fully untethered
from campaigns, causing them to not show up in drafts. Steve has Jupyter
notebooks for correcting the issue.
** [[https://jira.aweber.io/browse/CCPANEL-11147][Export broadcast quickstats in bulk]]
Should be much less relevant now that clicks and opens are in the account
export.
** [[https://jira.aweber.io/browse/CCPANEL-8608][Create a way for customers to automatically merge lists]]
Steve would discard some data, including custom fields, to simplify the manual
process.
** [[https://jira.aweber.io/browse/CCPANEL-10695][Ability to bulk tag over 50,000 subscribers]]
Steve wonders whether this could be worked around by using update-on-import
(does that have a row limit? 20k?).
** [[https://jira.aweber.io/browse/CCPANEL-8844][View duplicate subscribers across lists]]
Steve wonders if the customer is aiming to lower their bill?
** [[https://jira.aweber.io/browse/CCPANEL-11140][Remove and edit previous contact addresses from account]]
I suggested that it is probably best to add a flag to retire it / keep it from
being displayed to the user.
** [[https://jira.aweber.io/browse/PSE-4468][Bulk delete inactive subscribers across many lists]]
Steve believes this makes sense to keep this in ASE, warns that in the cases of
mistakes things have to be restored, which can be difficult and miss things
([[id:4bf81e33-8020-40f2-b6ed-ce4e1eae2234][Admin restore deleted controller]]). The subscriber ID /should/ be restored now,
but are the mappings?
** [[https://jira.aweber.io/browse/CCPANEL-11219][Delete duplicate subscribers across lists]]
** [[https://jira.aweber.io/browse/CCPANEL-11864][Customer export file size limited to 100MB (now 200MB)]]
Due to a cap in file upload sizes in Tornado.
** [[https://jira.aweber.io/browse/CCPANEL-11345][As an AWeber customer, I would like a report of tags applied on a given day]]
Steve managed this once using Athena queries, doesn't recall requests from any
other customers.
** [[https://jira.aweber.io/browse/CCPANEL-11680][Broadcast Automations are not Applying/Removing Tags]]
Steve looked into this and was able to replicate the issue on one of his own
subscribers.

[[id:4bf81e33-8020-40f2-b6ed-ce4e1eae2234][Restoring a subscriber]] doesn't restore the recipient record (remains in a
deleted state).
* Average Response Time alert - Subscriber Proxy
[[id:ebea379a-8fa6-4e22-9275-a9fc98c02804][Pagerduty]] alert.

- [[https://grafana.aweber.io/d/CTPUDEjMk/subscriber-proxy-kubernetes-service?tab=alert&viewPanel=4&orgId=1&from=1639058400000&to=1639069200000&var-summarize=1m&var-environment=production][Subscriber proxy graph]]
- [[https://grafana.aweber.io/d/000000031/awsubscribers-dashboard?orgId=1&viewPanel=3&from=1639058400000&to=1639069200000&var-summarize=1m][AWSubscribers graph]]

Seems related to bulk tag job retrieval (=JobHandler.GET=).

This aligns with requests in the bulk tagging logs.
#+begin_src json
  {
    "asctime": "2021-12-09T18:20:45.538764+0000",
    "correlation_id": "-",
    "levelname": "INFO",
    "message": "10.51.23.1 - - [09/Dec/2021:18:20:44 +0000] \"GET /jobs?account_id=1044937 HTTP/1.1\" 200 - \"-\" \"subscriberproxy/1.2.5\" (secs:1.281)",
    "module": "helpers",
    "name": "tornado.access",
    "service": "bulk-tagging",
    "exc_info": null
  }
#+end_src

This account in particular is a test account with a large number of jobs, and it
appears to be regularly requested. It looks like bulk-tagging isn't successfully
applying limits and ends up taking a while to collect a large number of jobs.

#+begin_src http :cache yes :pretty :select "length" :exports both
  GET http://bulk-tagging.service.production.consul/jobs?account=1044937&limit=3
#+end_src

#+RESULTS[0b736d0b171d5cb83cafebbfdfc9ebaea9ee7eae]:
: 4530