correlr/roam
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Remove AWeber org files

Browse source
This commit is contained in:
Correl Roush 2022-07-06 17:15:12 -04:00
parent 74f9662c68
commit ae2156ae09
220 changed files with 0 additions and 18175 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

14
aweber/20200713131302-migration_to_common_rabbitmq.org
View file

@ -1,14 +0,0 @@
:PROPERTIES:
:ID: ac416861-ce45-49ac-8b60-f8ea39362135
:END:
#+title: Migration to common RabbitMQ
All services and consumers pointed at the legacy RabbitMQ cluster in
Conshohocken should be migrated to the new common-rabbitmq cluster as a [[id:db322997-ff5e-416a-8dc8-f29e6a4928c8][Technical Initiative]].
The new servers are available at
=common-rabbitmq.service.${ENVIRONMENT}.consul=.
* Legacy hostnames
- =rabbitmq.service.${ENVIRONMENT}.consul=
- =rabbit{1-3}.int.{stg,prd}.csh=

4
aweber/20200713131512-coreapi.org
View file

@ -1,4 +0,0 @@
:PROPERTIES:
:ID: e4d00c11-da8a-4c91-8f38-ce939846e5cb
:END:
#+title: CoreAPI

10
aweber/20200713162108-puppet.org
View file

@ -1,10 +0,0 @@
:PROPERTIES:
:ID: ddeea682-c8f0-4607-8e2b-0f8ee4fd6191
:END:
#+title: Puppet
- Repository :: [[https://gitlab.aweber.io/PSE/config-management/puppet/]]
* Applying changes on a node
- Ensure changes are merged and tagged in the upstream repository.
- As root, run =puppetd --test=.

4
aweber/20200714100836-pgbouncer_port_migration.org
View file

@ -1,4 +0,0 @@
:PROPERTIES:
:ID: 592aa825-154c-4659-8193-75b0ce1f2e5c
:END:
#+title: PGBouncer port migration

7
aweber/20200714125028-pagerduty.org
View file

@ -1,7 +0,0 @@
:PROPERTIES:
:ID: ebea379a-8fa6-4e22-9275-a9fc98c02804
:END:
#+title: Pagerduty
https://aweber.pagerduty.com/

8
aweber/20200714130151-mail_relay.org
View file

@ -1,8 +0,0 @@
:PROPERTIES:
:ID: 24578fe5-6ca0-4000-a7cd-201e952e4c76
:END:
#+title: Mail Relay
A postfix instance running in Kubernetes for the express purpose of supporting
legacy emails in the [[id:57ee2f00-9bcd-4e0f-8a77-ae1f2d4cda89][Control Panel]] until they are all migrated to use [[id:32c66bc8-a397-4f50-96cd-2aec70dd14c5][Corporate
Notifications]].

4
aweber/20200714141221-momentum.org
View file

@ -1,4 +0,0 @@
:PROPERTIES:
:ID: e1b95d0e-366e-4ecf-b867-409b6b6c6ee8
:END:
#+title: Momentum

4
aweber/20200714141314-control_panel.org
View file

@ -1,4 +0,0 @@
:PROPERTIES:
:ID: 57ee2f00-9bcd-4e0f-8a77-ae1f2d4cda89
:END:
#+title: Control Panel

4
aweber/20200714141333-corporate_notifications.org
View file

@ -1,4 +0,0 @@
:PROPERTIES:
:ID: 32c66bc8-a397-4f50-96cd-2aec70dd14c5
:END:
#+title: Corporate Notifications

11
aweber/20200714143255-refunding_an_order.org
View file

@ -1,11 +0,0 @@
:PROPERTIES:
:ID: 7a362881-875f-4f74-8053-55f63826da63
:END:
#+title: Refunding an Order
Refund options (via Admin)
1. Use a balance item
1. Insert a manual balance item, set Term at 1, a negative amount, and optionally a description.
2. Add an invoice and pay it.
2. Close the current package
1. Open the current package, select cancellation in date closed, with a cancel reason

46
aweber/20200714212153-login_throttling.org
View file

@ -1,46 +0,0 @@
:PROPERTIES:
:ID: d17e934b-b340-4246-88f0-9b36527100c0
:END:
#+title: Login Throttling
* CAPTCHA Throttling
We have login captcha throttling in place for the following:
| Tracked behavior | CAPTCHA threshold | Time Interval |
|-------------------------------------------------------+-------------------+---------------|
| Repeated unsuccessful attempts with the same username | 3 attempts | 10 minutes |
| Repeated attempts from the same IP address | 3 attempts | 12 hours |
| Repeated attempts using the same Sift ID | 3 attempts | 30 minutes |
| Invalid or missing CSRF token | Immediate | N/A |
| Missing customer cookie | Immediate | N/A |
When a user meets one of the thresholds above, they will be presented with a
CAPTCHA challenge. This does not necessarily mean a puzzle will have to be
solved, only that the CAPTCHA script will attempt to determine if the user is a
bot. Even if the user has correctly entered their credentials on the subsequent
attempt, the CAPTCHA challenge will still occur.
All of the above thresholds are checked concurrently for each login attempt.
When a throttled user logs in successfully, the following occurs, the *username*
threshold is reset. No other thresholds are cleared. This means that even after
a user is able to successfully log in to an account, it is still possible for
them to be throttled after failing to log in again because they are now being
throttled by IP address.
* Sift ID Blocking
During previous login attacks, we've documented a set of Sift IDs that have been
used repeatedly during those attempts. Those IDs are blocked with CAPTCHA
*immediately*, with a 20% chance that we will present them a faked successful
response. This is done to throw off attackers using these IDs.
* Code
All the captcha / throttling logic that’s currently in place lives in
[[https://gitlab.aweber.io/CP/applications/sites/-/blob/master/aweber_app/controllers/account_controller.php][aweber_app/controllers/account_controller.php]], mainly in the =loginAjax= and
=isThrottled= methods. The repeated actions are tracked using
[[https://gitlab.aweber.io/CP/applications/sites/-/blob/master/php5-vendors/vendors/throttler.php][php5-vendors/vendors/throttler.php]], which uses counters in Redis with a TTL
attached.
* Graphs
Login attempts and throttling are graphed in Grafana on the [[https://grafana.aweber.io/d/000000530/account-logins][Account Logins
dashboard]].

70
aweber/20201022141542-python_services.org
View file

@ -1,70 +0,0 @@
:PROPERTIES:
:ID: a81b2ff0-5ede-44b3-8f82-960357f15428
:END:
#+title: Python Services
#+OPTIONS: ^:nil
#+PROPERTY: header-args :exports code
* Platform
- Python 3.9[fn:programming-languages]
- Tornado 6[fn:frameworks]
* Code Style
Code style must be enforced using flake8.[fn:python-lint-checking]
#+begin_example
[flake8]
application-import-names = PACKAGE_NAME,tests
exclude = build,env
import-order-style = pycharm
#+end_example
* Requirements
** Uncaught errors are logged and alerted via Sentry
#+NAME: packages
- =sentry-sdk=
#+begin_src python
from sentry_sdk import init
init(SENTRY_DSN)
#+end_src
** A status endpoint is exposed
The endpoint should be provide the following fields:
- application :: The name of the service
- environment :: The operating environment the instance of the service is
running in (i.e. "development", "testing", "staging" or "production")
- status :: Current service status (e.g.: "ok", "starting")
- version :: Packaged version of the service instance
- python_version :: The python version running the service instance
(=platform.python_version()=)
The endpoint should return =200= when the service is healthy, and =503= if the
service is not ready to serve requests.
** The service self-hosts its API documentation
An OpenAPI specification is hosted using ReDoc at the root service URL.[fn:backend-services]
** Test coverage reports are available in SonarQube
** Structured logging
json-scribe
** Provides consistent error responses
json-problem
** The service represents itself using its service name and version
- The service must include its name and version in its =Server= response header.[fn:response-headers]
- The service must include its name and version in the =User-Agent= header for all its HTTP requests.[fn:user-agent]
Both of these should be presented as =${service-name}/${version}=, e.g.:
=user-management/1.0.0=.
* References
- [[https://confluence.aweber.io/display/STD/Back+End+Services]]
* Footnotes
[fn:user-agent] https://confluence.aweber.io/display/STD/RESTful+APIs#heading-User-AgentRequestHeader
[fn:response-headers] https://confluence.aweber.io/display/STD/RESTful+APIs#heading-ResponseHeaders
[fn:backend-services] https://confluence.aweber.io/display/STD/Back+End+Services
[fn:frameworks] https://confluence.aweber.io/display/STD/Development+Frameworks
[fn:python-lint-checking] https://confluence.aweber.io/display/STD/Python+Lint+Checking
[fn:programming-languages] https://confluence.aweber.io/display/STD/Programming+Languages

58
aweber/20201028130030-easy_commerce_mvp_brainstorm_notes.org
View file

@ -1,58 +0,0 @@
:PROPERTIES:
:ID: dcb2f0ad-72e8-41ff-84f5-07caf8c7fe8e
:END:
#+title: Easy Commerce MVP Brainstorm Notes
#+TODO: ASK(a) FOLLOW-UP(f) | ANSWERED(d)
#+DATE: <2020-10-29 Thu>
* ANSWERED How is a product identified and tracked?
- The customer configures a product name and price on the landing page.
- The product name is the "goal description" (page description or note in the
DB) in our current sales tracking.
- We'd like to include the product name, price, and URL in the subscriber's
activity.
- Full, separate product tracking could be something we build later if there is
sufficient interest.
* ANSWERED How will the Stripe integration be configured?
Will it be configured at the account level, and if so, could it leverage the
existing system for linking things like FB & Twitter?
----------------------------------------------------------------------
- Stripe links via OAuth.
- We will have only one Stripe account per AWeber account.
- We will be logging charges to Stripe, not full orders (with product tracking
information).
- Investigation on whether the existing system will work is pending.
* ANSWERED How does this interact with service limits?
- Availability of the easy commerce feature
- Subscriber limits?
----------------------------------------------------------------------
- Handle errors with free-tier subscriber limits.
- Consider transaction service fee changes based on service limits.
* ANSWERED When should the buyer be added as a subscriber?
Immediately upon payment submission, or upon asynchronous payment confirmation?
----------------------------------------------------------------------
- The async web hook confirms the payment was made successfully.
+ We're avoiding putting logic here only to limit the scope of adding storage
for subscriber and purchase information to react to the event with.
- If the subscriber already exists, we'll update them with any new information.
* ANSWERED How will reports be handled?
Is there a dependency on the upcoming Analytics View service?
----------------------------------------------------------------------
- Only add All Lists: Products Sold in MVP
+ This seems like it'd just be a filter on event type in our current sales
over time report.
- The Analytics View service will provide report data via the authenticated
public API interface.
+ May not need to be coupled to the project, as we may be able to filter the
existing report to distinguish between current sales tracking and landing
page sales.
+ Anything /new/ should be built using the new Analytics View service.

53
aweber/20210112104838-stripe_payments_service.org
View file

@ -1,53 +0,0 @@
:PROPERTIES:
:ID: 9b3ed74d-41d4-4784-89e3-6a9183903b9e
:END:
#+title: Stripe Payments Service
* Overview
The Stripe Payments service mediates purchases made by buyers from AWeber
customers through the [[id:7d940785-68b9-4da7-bad1-4771d496168c][Stripe payment platform]] and delivers their goods via list
subscription.
* How It Works
** Ordering
The order endpoint prepares a [[https://stripe.com/docs/api/payment_intents][Payment Intent]] with Stripe that the buyer will pay
directly using Stripe's public API. The payment intent captures the amount to be
paid, any fees that AWeber will collect, and account and list metadata used to
fulfill the order.
*** Email validation
Emails are checked against the [[https://gitlab.aweber.io/CP/Services/validation][CP Email Validation Service]] to ensure they can be
added as subscribers successfully. The following steps are performed by the service:
1. Validates that the format of the submitted email address matches AWeber's
internal email formatting rules.
2. Applies address normalization to remove any ISP specific markup.
3. Extracts the domain-part from the normalized address and validates that there
are MX records for the domain-part via DNS lookup.
4. Checks if the both the submitted and normalized version of the email address
would be blocked by the blocklist.
Should the validation service fail unexpectedly or be unavailable at the time an
order is placed, a warning will be logged and only the email format check will
be performed.
*** CAPTCHA
**** Domain validation
***** Allowed domains
The following root domains provided by AWeber for [[https://confluence.aweber.io/pages/viewpage.action?pageId=118885193][Web Content (a.k.a Landing
Pages)]] always pass CAPTCHA domain validation:
- =aweberpages.com=
- =aweb.page=
***** Custom domains
All other domains are checked against the [[https://confluence.aweber.io/display/AR/Custom+Domain+Service][Custom Domain Service]] to ensure they
are owned and managed by the AWeber account from which the purchase is being
made.
*** Fees
Fees are collected for each sale as a percentage of the sale value configured in
the [[https://confluence.aweber.io/display/CT/Service+Limits+API][Service Limits API]] for the AWeber account, rounded to the nearest cent with
a minimum fee of $0.01.
** Fulfillment
*** Adding the subscriber
Subscribers are added to the list configured for the payment if they are not
already subscribed. If the subscriber is currently on the list in an unconfirmed
state, they will be marked as subscribed. Any tags configured for the sale will
be added to the subscriber.
*** Purchase Tracking
A [[https://confluence.aweber.io/display/AR/Pageview][Pageview Event]] is emitted with details of the order so that the purchase is
tracked in the subscriber's activity.

448
aweber/20210127141844-recurring_and_split_stripe_payments.org
View file

@ -1,448 +0,0 @@
:PROPERTIES:
:ID: a9835afc-e0be-4436-8274-c3898fdf119c
:END:
#+title: Recurring and split Stripe payments
#+options: prop:t
To be implemented in the [[id:9b3ed74d-41d4-4784-89e3-6a9183903b9e][Stripe Payments Service]].
Recurring and split payments are implementable in Stripe as [[https://stripe.com/docs/billing/subscriptions/overview#subscription-lifecycle][subscriptions]]. The
key difference is that split payments have a limited number of payment
iterations (e.g. 5 monthly installments) whereas recurring payments are ongoing.
* Stripe objects
** Products
Represents something being sold.
** Prices
Amount and frequency to be charged for a product. Many prices may be available
for a single product.
** Customers
Represents a buyer (subscriber?). Needs to be configured with a payment method.
** Subscriptions
Represents a [[*Products][Product]] being offered to a [[* Customers][Customer]] at a [[* Prices][Price]].
** Invoices
Generated at billing times.
** Payment Intents
Attempts to pay an [[* Invoices][Invoice]].
* Easy Commerce Flow
** Setup
[[*Products][Products]] and [[* Prices][Prices]] should be set up in the customer's stripe account and
referenced in their landing page.
#+begin_src plantuml :file "ecommerce-products.svg"
actor "AWeber Customer" as customer
participant "Landing Page Editor" as lp
participant "Stripe Payments (Authenticated)" as sp
participant "Stripe" as stripe
== Load product information ==
customer -> lp : Edit landing page
lp -> sp : GET /stripe-authenticated/products
sp -> stripe : Get products with prices
sp -> lp : Return list of products with pricing
== Save product information ==
customer -> lp : Save landing page
alt create a new product & price
lp -> sp : POST /stripe-authenticated/products
else update an existing product & price
lp -> sp : PATCH /stripe-authenticated/products/{UUID}
end
sp -> stripe : Store product and price
sp -> lp : 200 Return product and price IDs
#+end_src
#+RESULTS:
[[file:ecommerce-products.svg]]
** Purchase
#+begin_src plantuml :file "ecommerce-subscribing.svg"
actor "Buyer" as buyer
participant "Landing Page" as lp
participant "Stripe Payments (Unauthenticated)" as sp
box "Internal"
participant "Core API" as capi
end box
participant "Stripe" as stripe
buyer -> lp : Place order
lp -> sp : POST /stripe/subscription with product and price IDs
sp -> stripe : Create customer
sp -> stripe : Create subscription
sp -> stripe : Create initial invoice
sp -> lp : Return payment intent client secret
lp -> stripe : Confirm payment
stripe -> lp : Return payment intent
lp -> sp : POST /stripe/fulfillment
sp -> stripe : Fetch payment intent and check status
sp -> capi : Add subscriber with tags
sp -> lp : 200 OK
#+end_src
#+RESULTS:
[[file:ecommerce-subscribing.svg]]
*** Subscribing
Unlike ordering, which creates a single payment intent, we'll instead need to
take the payment information provided by the buyer and create a [[* Customers][Customer]] object
in Stripe, as well as a [[* Subscriptions][Subscription]] to a [[*Products][Product]] at a specific [[* Prices][Price]].
*** Fulfillment
After a successful payment, the [[* Subscriptions][Subscription]] is marked as =active=, and the
buyer should be subscribed to start receiving their content.
** Automatic payments
#+begin_src plantuml :file "ecommerce-payment-webhooks.svg"
participant "Stripe Payments (Unauthenticated)" as sp
box "Internal"
participant "RabbitMQ" as amqp
end box
participant "Stripe" as stripe
stripe -> sp : POST /stripe/webhooks (payment_intent.succeeded)
sp -> amqp : Sales tracking event (pageview.v4)
sp -> amqp : Payment succeeded event (stripe_payment_succeeded.v1)
sp -> stripe : 200 OK
#+end_src
#+RESULTS:
[[file:ecommerce-payment-webhooks.svg]]
** Payment failure / Cancellation
#+begin_src plantuml :file "ecommerce-cancellation-webhooks.svg"
participant "Stripe Payments (Unauthenticated)" as sp
box "Internal"
participant "Core API" as capi
end box
participant "Stripe" as stripe
stripe -> sp : POST /stripe/webhooks (customer.subscription.updated)
sp -> stripe : Fetch product metadata
sp -> capi : Remove tags from subscriber or unsubscribe
sp -> stripe : 200 OK
#+end_src
#+RESULTS:
[[file:ecommerce-cancellation-webhooks.svg]]
* Questions
- Should stripe-payments or the integrations service handle price and product management?
+ stripe-payments would need authenticated endpoints exposed
* Project :taskjuggler_project:
:PROPERTIES:
:COLUMNS: %50ITEM(Task) %Effort %allocate %task_id %blocker
:start: 2021-02-22
:END:
#+begin: columnview
| Task | Effort | allocate | task_id | blocker |
|-------------------------------------------------------------------+--------+-------------------+-------------------------+-------------------------------------------------------------------------------|
| Project | | | | |
| Tasks | | | | |
| Backend: Product Management | | | | |
| Define API for managing products and prices | 1d | backend | api | |
| Endpoint: Create products and prices | 3d | backend | create | api |
| Endpoint: Update products and prices | 5d | backend | update | api |
| Endpoint: Fetch products and prices | 2d | backend | fetch | api |
| Endpoint: Update /order | 2d | backend | order_both | api |
| Migrate products from existing landing pages into Stripe on save | | frontend | migrate | order_both |
| Backfill products from existing landing pages into Stripe | 10d | backend, frontend | backfill | order_both |
| Backend: Subscriptions | | | | |
| Endpoint: Update /order to create customers for product purchases | 10d | backend | order_products | milestone_management |
| Endpoint: Add subscription support to /order | 10d | backend | order_subscriptions | milestone_management |
| Endpoint: Update /fulfill & /webhooks (iteration 1) | 3d | backend | webhooks_1 | milestone_management |
| Endpoint: Update /webhooks to process subscriptions (iteration 2) | 8d | backend | webhooks_2 | webhooks_1, order_subscriptions |
| Database: Add database for event tracking | 3d | backend | database | |
| Endpoint: Add event tracking to /webhooks | 5d | backend | tracking | database, webhooks_2 |
| Backend: Resiliency | | | | |
| Poller: Create poller to check for unprocessed events | 10d | backend | poller | milestone_subscriptions |
| Backend: Cleanup | | | | |
| Backfill products from existing landing pages into Stripe | 10d | backend, frontend | backfill2 | milestone_subscriptions |
| Endpoint: Update /order remove /fulfill | 1d | backend | cleanup | backfill2 |
| Frontend | | | | |
| Frontend: Plugin | 2d | frontend | plugin | |
| Frontend: Templates | 1d | frontend | templates | plugin |
| Frontend: Builder (Product Management) | 12d | frontend | builder_products | plugin, templates |
| Frontend: Builder (Recurring Payments) | 17d | frontend | builder_recurring | milestone_management |
| Milestones | | | | |
| Milestone 1: Product Management | 5d | release | milestone_management | api, create, update, fetch, order_both, migrate, backfill, builder_products |
| Milestone 2: Support Subscriptions | 5d | release | milestone_subscriptions | order_products, webhooks_1, webhooks_2, database, tracking, builder_recurring |
| Milestone 3: Add Resiliency | | | milestone_resiliency | poller |
| Milestone 4: Deprecate Product Name/Price | | | milestone_cleanup | milestone_subscriptions, cleanup |
#+end:
#+begin_export html
<iframe src="reports/recurring-and-split-stripe-payments/Plan.html" width="100%" height="1000"></iframe>
#+end_export
** Tasks
*** Backend: Product Management
**** DONE Define API for managing products and prices
:PROPERTIES:
:EFFORT: 1d
:ALLOCATE: backend
:task_id: api
:END:
Probably makes sense to treat price + recurrence as product attributes, e.g.
- name :: String
- Price :: Object
+ amount :: Integer
+ currency :: Enum["usd"]
+ recurrence :: Optional[Object]
- interval :: Enum["weekly", "monthly", "yearly"]
- times :: Union["unlimited", Integer]
**** DONE Endpoint: Create products and prices
:PROPERTIES:
:EFFORT: 3d
:ALLOCATE: backend
:blocker: api
:TASK_ID: create
:END:
- Define metadata schema for purchase and cancellation actions (add tags, unsubscribe)
**** DONE Endpoint: Update products and prices
:PROPERTIES:
:EFFORT: 5d
:ALLOCATE: backend
:blocker: api
:TASK_ID: update
:END:
- Add new price object if the price or recurrence has changed, leaving the old one
**** DONE Endpoint: Fetch products and prices
:PROPERTIES:
:EFFORT: 2d
:ALLOCATE: backend
:blocker: api
:TASK_ID: fetch
:END:
**** DONE Endpoint: Update /order
:PROPERTIES:
:EFFORT: 2d
:ALLOCATE: backend
:blocker: api
:task_id: order_both
:END:
**** DONE Migrate products from existing landing pages into Stripe on save
:PROPERTIES:
:ALLOCATE: frontend
:blocker: order_both
:TASK_ID: migrate
:END:
**** CANCELLED Backfill products from existing landing pages into Stripe
:PROPERTIES:
:EFFORT: 10d
:ALLOCATE: backend, frontend
:blocker: order_both
:task_id: backfill
:END:
**** DONE Separate Products and Prices
*** Backend: Subscriptions
**** TODO Endpoint: Create /purchase to create customers for product purchases
:PROPERTIES:
:ALLOCATE: backend
:blocker: milestone_management
:start: 2021-04-22
:Effort: 12d
:TASK_ID: order_products
:END:
If the product being ordered is specified by id:
- Create the customer w/ the provided payment method
- Create an invoice
- Pay the invoice
- Document the /order endpoint as deprecated
- Create acceptance tests
**** TODO Endpoint: Update /fulfill and /webhooks (iteration 1)
:PROPERTIES:
:Effort: 5d
:ALLOCATE: backend
:BLOCKER: milestone_management
:task_id: webhooks_1
:start: 2021-04-22
:END:
- move add subscriber and sales tracking event emission to the webhook endpoint
- no-op /fulfill
- Document /fulfill endpoint as deprecated
- Send =AWeber-Options: update-unconfirmed= header
- Update acceptance tests to account for out-of-band fulfillment
**** TODO Endpoint: Add subscription support to /purchase
:PROPERTIES:
:ALLOCATE: backend
:blocker: milestone_management
:start: 2021-04-22
:Effort: 12d
:TASK_ID: order_subscriptions
:END:
If the product being ordered has a recurring price type:
- Create the customer w/ the provided payment method
- Create a subscription
- Create the initial invoice
- Pay the initial invoice
- Add acceptance tests for subscription purchases
**** TODO Endpoint: Update /webhooks to process subscriptions (iteration 2)
:PROPERTIES:
:Effort: 10d
:ALLOCATE: backend
:blocker: webhooks_1, order_subscriptions
:TASK_ID: webhooks_2
:END:
- Add logic to process a subscription
- Add the events corresponding to cancellation of a subscription
+ Process the remove tag/unsubscribe actions that are saved as metadata on the
subscription
+ Ensure that unconfirmed subscribers can be unsubscribed
- Need a way to differentiate fulfillment of a payment intent for a subscription vs a single payment
+ Pull the invoice for the payment intent with the expanded subscription details if they exist
**** TODO Database: Add database for event tracking
:PROPERTIES:
:Effort: 3d
:ALLOCATE: backend
:BLOCKER: milestone_management
:start: 2021-04-22
:task_id: database
:END:
- new postgres users ( one for the poller, one for the stripe-payments service )
- Needs the following information in an events table:
+ timestamp from the event
+ timestamp of action completion
+ webhook type
+ webhook id
+ stripe account id
+ aweber account id
+ and whether it was successfully processed (process state)
- dynamodb does not fit the use case because we need to retrieve all events in a time frame, and filter by account. We are also constantly deleting events.
+ we also will have two services updating the database and need transaction safety
- Scope assumes handing off the schema design to the DBA team
**** TODO Endpoint: Add event tracking to /webhooks
:PROPERTIES:
:Effort: 7d
:ALLOCATE: backend
:BLOCKER: database, webhooks_2
:TASK_ID: tracking
:END:
- Mark processed events in the database
+ Do we need transactions? Assume yes.
- Handling duplicate and out-of-order event cases
+ Fulfill only once (adding subscriber + tags)
+ Unsubscribe only once (removing subscriber / tags)
+ Don't fulfill if unsubscribed
*** Backend: Resiliency
We will need a job to periodically poll Stripe for unhandled webhooks for
processing.
- This job will need to share a data store with the stripe payments service to
track which webhooks have been processed.
+ Is this necessary, or does stripe expose the processed state of the
webhooks? Webhook attempts and responses are logged in the console.
- The job will need to maintain a lock to prevent concurrent runs.
- How will unhandled webhooks get processed?
+ Send them to the stripe payments service endpoint?
- Would have to store status or update it in Stripe.
+ Have Stripe send them?
- Is this possible?
**** TODO Poller: Create poller to check for unprocessed events
:PROPERTIES:
:EFFORT: 10d
:ALLOCATE: backend
:BLOCKER: milestone_subscriptions
:TASK_ID: poller
:END:
*** Backend: Cleanup
**** TODO Backfill products from existing landing pages into Stripe
:PROPERTIES:
:EFFORT: 10d
:ALLOCATE: backend, frontend
:blocker: milestone_subscriptions
:task_id: backfill2
:END:
**** TODO Endpoint: Remove /order and /fulfill
:PROPERTIES:
:EFFORT: 1d
:ALLOCATE: backend
:blocker: backfill2
:TASK_ID: cleanup
:END:
*** Frontend
Builder originally estimated at 25d combined.
**** TODO Frontend: Plugin
:PROPERTIES:
:Effort: 2d
:ALLOCATE: frontend
:TASK_ID: plugin
:BLOCKER:
:END:
**** TODO Frontend: Templates
:PROPERTIES:
:Effort: 1d
:ALLOCATE: frontend
:TASK_ID: templates
:BLOCKER: plugin
:END:
**** TODO Frontend: Builder (Product Management)
:PROPERTIES:
:EFFORT: 12d
:ALLOCATE: frontend
:TASK_ID: builder_products
:BLOCKER: plugin, templates
:END:
**** TODO Frontend: Builder (Recurring Payments)
:PROPERTIES:
:EFFORT: 28d
:ALLOCATE: frontend
:TASK_ID: builder_recurring
:BLOCKER: milestone_management
:start: 2021-04-22
:END:
** Milestones
*** Milestone 1: Product Management
:PROPERTIES:
:BLOCKER: api, create, update, fetch, order_both, migrate, backfill, builder_products
:TASK_ID: milestone_management
:Effort: 5d
:ALLOCATE: release
:END:
*** Milestone 2: Support Subscriptions
:PROPERTIES:
:BLOCKER: order_products, webhooks_1, webhooks_2, database, tracking, builder_recurring
:TASK_ID: milestone_subscriptions
:Effort: 10d
:ALLOCATE: release
:END:
*** Milestone 3: Add Resiliency
:PROPERTIES:
:BLOCKER: poller
:TASK_ID: milestone_resiliency
:END:
*** Milestone 4: Deprecate Product Name/Price
:PROPERTIES:
:BLOCKER: milestone_subscriptions, cleanup
:TASK_ID: milestone_cleanup
:END:
* Resources :taskjuggler_resource:
*** Backend
:PROPERTIES:
:resource_id: backend
:efficiency: 1.3
:END:
*** Frontend
:PROPERTIES:
:resource_id: frontend
:efficiency: 0.8
:END:
*** Release Validation
:PROPERTIES:
:resource_id: release
:efficiency: 1.0
:END:
* Variables :noexport:
Local Variables:
org-taskjuggler-reports-directory: "reports/recurring-and-split-stripe-payments"
End:

38
aweber/20210210113027-control_panel_http_requests.org
View file

@ -1,38 +0,0 @@
:PROPERTIES:
:ID: 33e47957-b3d0-41c9-8977-7243b42a76dd
:END:
#+title: Control Panel HTTP Requests
#+PROPERTY: header-args :exports both :eval no-export
#+PROPERTY: header-args:http :cookie .cookies :cookie-jar .cookies
* Cookies
| Name | Description |
|-------------+-------------|
| AUTORESPSID | Session ID |
Cookies for requests in this document are stored in cookie file by curl in
=~/.cookies= (https://curl.se/docs/http-cookies.html).
* AJAX Requests
Control Panel controller actions that expect to be called as AJAX endpoints
expect the =X-Requested-With= header to be present and set to =XMLHttpRequest=.
* Logging In
** Fetching a CSRF Token
#+name: login-csrf
#+begin_src http :pretty
GET localhost:8080/users/pub/csrf
X-Requested-With:XMLHttpRequest
#+end_src
#+RESULTS: login-csrf
: 63116e764c5d31cdd3e4f230ee3740527f6eb1c76aea1cb04e30da5d68e24d78
** Sending credentials
#+begin_src http :pretty :var csrf=login-csrf
POST localhost:8080/users/account/loginAjax
X-Requested-With: XMLHttpRequest
username=lookatme@example.com&password=testing&_csrf=${csrf}
#+end_src
#+RESULTS:
: {"submitStatus":{"code":200,"message":"\/users\/","category":"status_success"},"validationErrors":[]}

1166
aweber/20210302135423-stripe_api.org
View file

File diff suppressed because it is too large Load diff

383
aweber/20210316155320-stripe_payments_tracking_database.org
View file

@ -1,383 +0,0 @@
:PROPERTIES:
:ID: 6af95849-8f78-4697-ab48-3712ff2f5ee1
:END:
#+title: Stripe payments tracking database
#+OPTIONS: ^:nil
Database for tracking payments and subscriptions managed by the [[id:9b3ed74d-41d4-4784-89e3-6a9183903b9e][Stripe Payments Service]].
* State Tracking
** Legacy purchase
- Client initiates purchase
- Backend coordinates with Stripe and returns a payment intent
+ Track new purchase and related incomplete payment
- Client completes purchase with Stripe
- Stripe notifies backend of payment intent status (update purchase)
+ Track event
+ Track payment as completed
+ Fulfill purchase
+ Track purchase as fulfilled
#+begin_src plantuml :file stripe-legacy-purchase-tracking.svg
participant Stripe
actor Client
participant "stripe-payments" as Backend
database Tracking
Client -> Backend: Initiate purchase
Backend -> Tracking : <font color="blue">Store incomplete purchase</font>
Backend -> Stripe : Create payment intent
Backend -> Tracking : <font color="blue">Store incomplete payment</font>
Backend -> Client : Return payment intent
Client -> Stripe : Complete purchase
...
alt Success
Stripe ---> Backend : payment_intent.succeeded
Backend -> Tracking : <font color="green">Store event</font>
alt Event not previously handled
Backend -> Tracking : <font color="blue">Mark payment as succeeded</font>
Backend -> Backend : Fulfill purchase
Backend -> Tracking : <font color="blue">Mark purchase as fulfilled</font>
end
else Failure
Stripe --> Backend : payment_intent.payment_failed
Backend -> Tracking : <font color="green">Store event</font>
end
#+end_src
#+RESULTS:
[[file:stripe-legacy-purchase-tracking.svg]]
#+caption: Legacy purchase flow with events
#+RESULTS:
** Product purchase
- Client prepares payment method with Stripe
- Client initiates purchase
- Backend coordinates with Stripe to complete the purchase
+ Attaches the payment method to the customer
+ For single products
- Creates an invoice item
- Creates an invoice
- Pays the invoice
+ For subscriptions
- Creates the subscription
#+begin_src plantuml :file stripe-purchase-tracking.svg
participant Stripe
actor Client
participant "stripe-payments" as Backend
database Tracking
Client -> Stripe : Create payment method
Client -> Backend: Initiate purchase
Backend -> Tracking : <font color="blue">Store incomplete purchase</font>
alt Non-recurring
Backend -> Stripe : Create invoice item
Backend -> Stripe : Create invoice
Backend -> Stripe : Pay invoice
Backend -> Tracking : <font color="blue">Store completed payment</font>
else Recurring
Backend -> Stripe : Create subscription
Backend -> Tracking : <font color="blue">Store active subscription</font>
end
...
Stripe --> Backend : Subscription activated
...
alt
Stripe --> Backend : customer.subscription.updated (no longer active)
else
Stripe --> Backend : subscription_schedule.canceled
end
Backend -> Tracking : <font color="green">Store event</font>
Backend -> Backend : Trigger unsubscribe actions
Backend -> Tracking : <font color="blue">Mark subscription as terminated</font>
#+end_src
#+caption: Product purchase flow
#+RESULTS:
[[file:stripe-purchase-tracking.svg]]
#+begin_src plantuml :file stripe-purchase-tracking-payment-events.svg
participant Stripe
actor Client
participant "stripe-payments" as Backend
database Tracking
== Payment succeeded ==
Stripe --> Backend : payment_intent.succeeded
Backend -> Tracking : <font color="green">Store event</font>
alt Event not previously handled
Backend -> Tracking : <font color="blue">Mark payment as succeeded</font>
Backend -> Stripe : Look up invoice
note over Backend
Recurring if an invoice exists
and has an associated subscription
end note
alt Non-recurring
Backend -> Backend : Fulfill purchase
Backend -> Tracking : <font color="blue">Mark purchase as fulfilled</font>
else Recurring
end
end
== Payment failed ==
Stripe --> Backend : payment_intent.payment_failed
Backend -> Tracking : <font color="green">Store event</font>
#+end_src
#+caption: Payment events
#+RESULTS:
[[file:stripe-purchase-tracking-payment-events.svg]]
#+begin_src plantuml :file stripe-purchase-subscription-events.svg
participant Stripe
actor Client
participant "stripe-payments" as Backend
database Tracking
== Subscription activated ==
alt
Stripe --> Backend : customer.subscription.created (active)
else
Stripe --> Backend : customer.subscription.updated (active)
end
Backend -> Tracking : <font color="green">Store event</font>
alt New subscription
Backend -> Tracking : <font color="blue">Mark subscription as active</font>
Backend -> Backend : Fulfill purchase
Backend -> Tracking : <font color="blue">Mark purchase as fulfilled</font>
else Subscription already processed
note over Backend
One of the following is true:
<font color="green">- Already received an activation event for this subscription</font>
<font color="green">- Already receieved a termination event for this subscription</font>
<font color="blue">- Subscription already marked as active or terminated</font>
<font color="blue">- Purchase already marked as fulfilled</font>
end note
end
== Subscription terminated ==
alt
Stripe --> Backend : customer.subscription.updated (no longer active)
else
Stripe --> Backend : subscription_schedule.canceled
end
Backend -> Tracking : <font color="green">Store event</font>
alt Active subscription
Backend -> Backend : Trigger unsubscribe actions
Backend -> Tracking : <font color="blue">Mark subscription as terminated</font>
else Subscription not yet processed
note over Backend
One of the following is true:
<font color="green">- Did not receive an activation event for this subscription</font>
<font color="blue">- Subscription is not tracked</font>
end note
else Subscription already terminated
note over Backend
One of the following is true:
<font color="green">- Already received a termination event for this subscription</font>
<font color="blue">- Subscription already marked as terminated</font>
end note
end
#+end_src
#+caption: Subscription events
#+RESULTS:
[[file:stripe-purchase-subscription-events.svg]]
* Tables
** Purchases
Purchases made via the [[id:9b3ed74d-41d4-4784-89e3-6a9183903b9e][Stripe Payments Service]].
#+begin_src plantuml :file stripe-purchase-states.svg
[*] -> New
New -> Fulfilled
#+end_src
#+caption: Purchase state diagram
#+RESULTS:
[[file:stripe-purchase-states.svg]]
#+caption: Purchases
| Field | Type | Nullable | Description |
|----------------+-----------+----------+--------------------------------------------|
| id | UUID | N | Auto-generated purchase ID |
| created | timestamp | N | Time the purchase was initiated |
| last_updated | timestamp | N | Time the purchase was last updated |
| account | UUID | N | The AWeber customer account purchased from |
| stripe_account | text | N | The Stripe account of the AWeber customer |
| fulfilled | boolean | N | Was this purchase fulfilled |
- Store which automations were applied & when
** Subscriptions and Split Payments
Purchases made with recurring payments, managed using a Stripe Subscription.
#+begin_src plantuml :file stripe-subscription-states.svg
[*] -> New
New -> Active
Active --> Terminated : Payment Failed
Active --> Terminated : Unsubscribed
New --> Terminated : Payment Failed
#+end_src
#+caption: Subscription state diagram
#+RESULTS:
[[file:stripe-subscription-states.svg]]
#+caption: Subscriptions
| Field | Type | Nullable | Description |
|----------------+---------------------+----------+---------------------------------------------------------|
| id | text | N | The Stripe subscription ID |
| created | timestamp | N | Time the purchase was initiated |
| last_updated | timestamp | N | Time the purchase was last updated |
| account | UUID | N | The AWeber customer account the subscription belongs to |
| stripe_account | text | N | The Stripe account of the AWeber customer |
| status | [[subscription_status][subscription_status]] | N | Status of the subscription |
<<subscription_status>>
#+caption: ENUM: Subscription Status
| new |
| active |
| terminated |
** Payments
Payments collected from buyers.
#+begin_src plantuml :file stripe-payment-states.svg
[*] -> New
New --> Paid : Payment succeeded
New --> Failed : Payment failed
#+end_src
#+caption: Payment state diagram
#+RESULTS:
[[file:stripe-payment-states.svg]]
#+caption: Payments
| Field | Type | Nullable | Description |
|----------------+----------------+----------+-------------------------------------------|
| id | text | N | The Stripe payment intent ID |
| created | timestamp | N | Time the purchase was initiated |
| last_updated | timestamp | N | Time the purchase was last updated |
| account | UUID | N | The AWeber customer account that was paid |
| stripe_account | text | N | The Stripe account of the AWeber customer |
| purchase | UUID | N | |
| status | [[payment_status][payment_status]] | N | Status of the payment intent |
<<payment_status>>
#+caption: ENUM: Payment Status
| new |
| paid |
| failed |
** Events
#+caption: Events
| Field | Type | Nullable | Description |
|----------------+-----------+----------+---------------------------------------------------------|
| id | text | N | Stripe event id |
| type | text | N | Stripe event type |
| account | UUID | N | The AWeber customer account the subscription belongs to |
| stripe_account | text | N | The Stripe account of the AWeber customer |
| timestamp | timestamp | N | Time event was published |
| subscription | text | Y | Related Stripe subscription ID |
Events should expire out of the database over time. Stripe maintains events in
its database for up to 30 days.
*** Modeling in DynamoDB
**** Table
- Partition Key :: =id=
The table index provides for rapid lookup of individual events.
**** Attributes
| Field | Type | Required | Description |
|----------------+-------------------------+----------+-----------------------------------------------------------|
| id | String | Y | Stripe event id |
| type | String | Y | Stripe event type |
| stripe_account | String | Y | The Stripe account of the AWeber customer |
| timestamp | String (timestamp) | Y | Time event was published (e.g. "2020-01-01 01:02:34.567") |
| date | String (date) | Y | Date portion of timestamp (e.g. "2020-01-01") |
| time | String (time) | Y | Time portion of timestamp (e.g. "01:02:34.567") |
| expiration | Number (Unix timestamp) | Y | TTL expiration time of the event record |
| subscription | String | N | Related Stripe subscription ID |
**** TTL
The events table will automatically expire and remove rows for which
=expiration= has passed. We may set =expiration= to be the event creation time
plus thirty days to match Stripe's own expiration. This will prevent old items
from piling up in the table.
**** Time index (GSI)
- Partition Key :: =date=
- Sort Key :: =time=
- Projection :: Keys Only
The date and time portions of the event timestamp will be separated to allow for
scanning all events within a day, filterable by time. This should support our
polling job, which should run multiple times per day. We'll have to be careful
around date boundaries when looking up events (e.g. by including a query for
events from the previous day during the first run of the current day).
**** Stripe Account index (GSI)
- Partition Key :: =stripe_account=
- Sort Key :: =timestamp=
- Projection :: Include type, subscription
The account index provides for rapid lookup of events for a particular stripe
account, filterable by timestamp.
**** Subscription index (GSI)
- Partition Key :: =subscription=
- Sort Key :: =timestamp=
- Projection :: Include type, stripe_account
The account index provides for rapid lookup of events for a particular
subscription, filterable by timestamp.
**** IAM Policy
Put together by referencing [[id:ab2d34bf-97b1-4e50-8e9a-597d0f8fcf01][DynamoDB IAM Policies]].
#+caption: Stripe Payments DynamoDB IAM Policy
#+begin_src json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "dynamodb:ListTables",
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"dynamodb:BatchGetItem",
"dynamodb:BatchWriteItem",
"dynamodb:ConditionCheckItem",
"dynamodb:PutItem",
"dynamodb:DeleteItem",
"dynamodb:Scan",
"dynamodb:Query",
"dynamodb:UpdateItem",
"dynamodb:DescribeTimeToLive",
"dynamodb:CreateTable",
"dynamodb:DescribeTable",
"dynamodb:GetItem",
"dynamodb:UpdateTable",
"dynamodb:UpdateTimeToLive"
],
"Resource": [
"arn:aws:dynamodb:*:018154689201:table/*-stripe-payments-*/index/*",
"arn:aws:dynamodb:*:018154689201:table/*-stripe-payments-*"
]
},
]
}
#+end_src
* Notes
- How does the [[id:83d61eef-0781-46e0-b959-1a739cff5ea3][poller]] use these stored events?
+ /To identify events retrieved from Stripe which do not need to be replayed/
- How do we replay events?
+ Send it to the webhook endpoint? An internal webhook endpoint that shares
code with the public one?
- /The same webhook endpoint would be preferable, requires signing the
payload/
- Do we expose the processed events as an internal endpoint in the
stripe-payments service, or give the polling app a database connection?
+ /API seems preferable/
Following some discussion, the [[Events][Events]] table should be sufficient for now.

12
aweber/20210318110509-finding_number_of_subscribers_with_a_tag.org
View file

@ -1,12 +0,0 @@
:PROPERTIES:
:ID: fe6374ac-8fa6-4579-bd42-6d4de92de86a
:END:
#+title: Finding number of subscribers with a tag
* Retrieve number of subscribers per tag
- Tagging -> Kubernetes
- Add appdb connection to tagging
- Add endpoint to tagging
- Expose tagging endpoints via authenticated kong
* Hide tags
* Delete tags

23
aweber/20210318112158-tagging_roadmap.org
View file

@ -1,23 +0,0 @@
:PROPERTIES:
:ID: 8435e743-c092-43e9-bcc6-a8098aa4110c
:END:
#+title: Tagging Roadmap
* Retrieve number of subscribers per tag
** Migrate the tagging service from AWS to Kubernetes
** Expose tagging endpoints via authenticated kong
** Add appdb connection to tagging
** Create an AppDB user for the tagging service
- Full access to the subscriber_tags table
- Read access to the list.subscribers table
** Add endpoint to tagging to fetch subscribers on a tag
e.g.:
#+begin_example
GET tagging.service.production.consul/{tag}/account/{account}/subscribers
#+end_example
** Update the tagging service to update the subscriber_tags table directly.
** Retire the subscriber-tag-sync consumer
* Hide tags

46
aweber/20210323161636-moving_pages_out_of_sites.org
View file

@ -1,46 +0,0 @@
:PROPERTIES:
:ID: 193f7c04-0a03-4870-90c8-2b5e3c4c92ce
:END:
#+title: Moving pages out of Sites
#+filetags: :project:
A [[id:db322997-ff5e-416a-8dc8-f29e6a4928c8][Technical Initiative]] to rewrite pages in the [[id:57ee2f00-9bcd-4e0f-8a77-ae1f2d4cda89][Control Panel]] still built in PHP.
* Pages in Sites
- Tracking document :: https://confluence.aweber.io/display/BETL/PHP+to+React+Page+Needs
** TODO Dashboard
*** Broadcasts
*** Lists
*** Subscribers
In progress.
** Content Creation
*** TODO Follow-ups
*** TODO Blog Broadcasts
*** TODO Email Template Manager
** Subscribers
*** TODO Manage Subscribers
*** TODO Add Subscribers
*** TODO Import History
** Reports
*** TODO Classic Reports
*** TODO Report API
*** TODO Tracking
** Lists
*** TODO Manage Lists
*** TODO List Settings
*** TODO Custom Fields
*** TODO List Automations
** Accounts
*** TODO My Account
*** Billing
*** Notifications
** Advocates
- Create an affiliate management API
- Create an affiliate frontend application
- Embed the affiliate app in the CP and link them with CP accounts.
** TODO Help
Should this be separated?
** Integrations
😱

95
aweber/20210323162325-technical_initiative.org
View file

@ -1,95 +0,0 @@
:PROPERTIES:
:ID: db322997-ff5e-416a-8dc8-f29e6a4928c8
:END:
#+title: Technical Initiative
- [[https://confluence.aweber.io/display/~scottm/CP+Technical+Work+Brainstorming][2022 Brainstorming Document]]
- [[https://confluence.aweber.io/display/TCP/2022+Q1+CP+Priorities][2022 Q1 CP Priorities]]
- [[id:193f7c04-0a03-4870-90c8-2b5e3c4c92ce][Moving pages out of Sites]]
* Big
** Analytics View
- Coordinate on public URL structure with Dave S.
- Update dashboard and reports to use new endpoints as they're made available.
** [[id:11edd6c9-b976-403b-a419-b5542ddedaae][Subscriber Search Service]]
*** Store and paginate search results
:PROPERTIES:
:JIRA_ID: CCPANEL-7148
:END:
*** Rebuild Subscriber Management in React
:PROPERTIES:
:JIRA_ID: CCPANEL-11697
:END:
** Verifications
*** Updating the existing verification flow to use email-verifications
:PROPERTIES:
:JIRA_ID: CCPANEL-9416
:END:
*** Decommission Verifications
** Domain Validator
:PROPERTIES:
:JIRA_ID: CCPANEL-10554
:END:
** [[id:619b6c78-7be9-4ee4-a0b7-9d1a4d7536e2][Migrating services to use the new List service]]
*** Audit remaining services
*** Rebuild List Management in React
*** Rebuild List Settings in React
:PROPERTIES:
:JIRA_ID: CCPANEL-11694
:END:
*** Remove dependency on AWLists from Stripe
**** Stripe master branch does not allow null values in product recurrence
:PROPERTIES:
:JIRA_ID: CCPANEL-12072
:END:
*** Remove dependency on AWLists from Subscriber Import
:PROPERTIES:
:JIRA_ID: CCPANEL-12071
:END:
**** Update Subscriber Import client to fetch list data from the new lists service
:PROPERTIES:
:JIRA_ID: CCPANEL-12073
:END:
*** Remove dependency on AWLists from Sites
:PROPERTIES:
:JIRA_ID: CCPANEL-12074
:END:
*** Remove dependency on AWLists from Email Verification
:PROPERTIES:
:JIRA_ID: CCPANEL-12070
:END:
** Retire AWSubscribers in favor of Recipient
*** Back Recipient with AppDB
*** Retire sync consumers
*** Identify gaps between AWSubs and Recipient
Determine which endpoints need to have analogs in Recipient or could be replaced
with calls to other, more appropriate services.
*** Look into folding in edeliv's bulk subscriber service
** [[id:03e00c18-99c0-477c-b7fb-95ddc538755e][Addlead]] Python rewrite
https://jira.aweber.io/browse/TRAC-118
- Find / Build a test suite that can be run against old and new addlead?
- WHAT DOES IT DO?! https://jira.aweber.io/browse/CCPANEL-7614
- ACP? https://jira.aweber.io/browse/CCPANEL-7613
** Enlightener rewrite
- Investigate how to rebuild this
** Sites login / session management
- Should advocate users be migrated to user management?
*** Separate from the rest of the CP
** Advocate CP
*** Python service + react application
** Verify Opt-in Python rewrite
** Unsubscribe Python rewrite
** [[id:b4f579f7-f848-4a7b-b7bc-f34fec36346a][Cleaning up public endpoints in proxy services]]
* Small
** [[id:af4ae6ee-5201-49ee-aa01-6cf6a0801908][Migrating AWS services]]
** [[id:96d1d218-60cd-41d9-91ba-48359137d239][Decommission the mail-relay service]]
** KTLO
- User Management
- Stripe Payments
- Commissions Processor
* Ongoing
** Update project configuration and gitlab pathing to match our taxonomies in Imbi.

9
aweber/20210323162555-cleaning_up_public_endpoints_in_proxy_services.org
View file

@ -1,9 +0,0 @@
:PROPERTIES:
:ID: b4f579f7-f848-4a7b-b7bc-f34fec36346a
:END:
#+title: Cleaning up public endpoints in proxy services
A [[id:db322997-ff5e-416a-8dc8-f29e6a4928c8][Technical Initiative]] to move endpoints exposed in "proxy" services into more
relevant services and instead expose them via Kong.
Endpoints have been consolidated into Search Proxy.

4
aweber/20210324133247-imbi.org
View file

@ -1,4 +0,0 @@
:PROPERTIES:
:ID: 76933c22-fe7c-43e9-9ec9-62564377dd85
:END:
#+title: Imbi

13
aweber/20210406120229-troubleshooting_an_unresolvable_kubernetes_service_hostname.org
View file

@ -1,13 +0,0 @@
:PROPERTIES:
:ID: 9332ed8f-b669-4d3f-a25d-da751a8c2da1
:END:
#+title: Troubleshooting an unresolvable kubernetes service hostname
The service's selector wasn't matching any pods, therefore the service had no IP
to respond with. This can be troubleshooted by inspecting the endpoints for the
service. Due to the mismatch, none were available.
The ClusterIP service did not itself have an IP assigned even after fixing the
mismatch. This appears to be an optimization in that [[id:3ba1f581-c66c-492f-80fc-e7d2e488b362][Kubernetes]] won't bother
assigning an IP to the service if the service has only a single endpoint, as it
is more expedient to return the sole endpoint's IP address.

15
aweber/20210412153138-purchase_tracking.org
View file

@ -1,15 +0,0 @@
:PROPERTIES:
:ID: e2dab290-3e1b-4d5a-8628-61c2cb4896dc
:END:
#+title: Purchase tracking
* Related Confluence documents
- [[https://confluence.aweber.io/pages/viewpage.action?spaceKey=PD&title=2021-03-16+Record+purchase+details+for+AWeber+integrations][2021-03-16 Record purchase details for AWeber integrations]]
- [[https://confluence.aweber.io/pages/viewpage.action?spaceKey=API&title=Storing+PayPal+purchases+as+analytics+events][Storing PayPal purchases as analytics events]]
* Record purchases
- Track a separate event type for sales tracking.
- Changes to sales tracking URLs can break tracking currently.
- Separate URL profiles for Stripe and Paypal? (Stripe is currently using
ECommerce), and add an ECommerce event type for them.
* Segmenting on purchases

65
aweber/20210506110533-dynamodb_iam_policies.org
View file

@ -1,65 +0,0 @@
:PROPERTIES:
:ID: ab2d34bf-97b1-4e50-8e9a-597d0f8fcf01
:END:
#+title: DynamoDB IAM Policies
#+caption: DynamoDB access for the k8s-labs-application role
#+begin_src json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "dynamodb:ListTables",
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"dynamodb:BatchGetItem",
"dynamodb:BatchWriteItem",
"dynamodb:ConditionCheckItem",
"dynamodb:PutItem",
"dynamodb:DeleteItem",
"dynamodb:Scan",
"dynamodb:Query",
"dynamodb:UpdateItem",
"dynamodb:DescribeTimeToLive",
"dynamodb:CreateTable",
"dynamodb:DescribeTable",
"dynamodb:GetItem",
"dynamodb:UpdateTable"
],
"Resource": [
"arn:aws:dynamodb:*:018154689201:table/*-webhook-callbacks/index/*",
"arn:aws:dynamodb:*:018154689201:table/*-webhook-callbacks"
]
},
{
"Effect": "Allow",
"Action": [
"dynamodb:BatchGetItem",
"dynamodb:BatchWriteItem",
"dynamodb:ConditionCheckItem",
"dynamodb:PutItem",
"dynamodb:DeleteItem",
"dynamodb:Scan",
"dynamodb:Query",
"dynamodb:UpdateItem",
"dynamodb:DescribeTimeToLive",
"dynamodb:CreateTable",
"dynamodb:DescribeTable",
"dynamodb:GetItem",
"dynamodb:UpdateTable"
],
"Resource": [
"arn:aws:dynamodb:*:018154689201:table/*-webhooks",
"arn:aws:dynamodb:*:018154689201:table/*-webhooks/index/*"
]
}
]
}
#+end_src
- [[https://docs.amazonaws.cn/en_us/amazondynamodb/latest/developerguide/access-control-overview.html][Overview of Managing Access Permissions to Your Amazon DynamoDB Resources]]
- [[https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/api-permissions-reference.html][DynamoDB API Permissions: Actions, Resources, and Conditions Reference]]

41
aweber/20210512093626-rebuild_unsubscribe_page_as_a_react_application.org
View file

@ -1,41 +0,0 @@
:PROPERTIES:
:ID: bdc526d1-4f57-4210-93f2-12bb30d33ed9
:END:
#+title: Rebuild Unsubscribe Page as a React Application
#+filetags: :project:
* Flows
As described in [[https://confluence.aweber.io/display/~scottm/Unsubscribe+Flows][Unsubscribe Flows]]:
- View subscriptions
- Edit subscriptions
- Edit subscriber
+ Name
+ Email
+ Custom fields, if present
** Questions
- What do we do when the backend (database) is unavailable?
+ The backend could do some sort of queuing if the database is unavailable
(similar to the disk-persisted queue currently in use).
* Frontend
A new React application to support subscriber subscription management actions.
** Questions
- What will host the frontend application?
+ Deploy the application to S3 and have F5 route to it.
* Backend
A new python service to handle subscriber subscription management actions.
** Endpoints
*** Configuration
Returns account information and branding needed to display the page.
*** Get subscriptions
*** Edit subscriptions
*** Edit subscriber
Email verification needs to be processed if the subscriber updates their email
address in order for that change to take effect (addresses [[https://jira.aweber.io/browse/CCPANEL-11269][CCPANEL-11269]]).
Changes to other fields should be applied immediately.
**** Questions
- Do custom field changes need verification to change?
+ Probably not.
- Should we display a pending state until verification is complete?
+ We do not plan to store a pending state.

13
aweber/20210512133000-cobrowse_io.org
View file

@ -1,13 +0,0 @@
:PROPERTIES:
:ID: d0a802dd-3258-4a86-b53f-287f7f6df6e6
:END:
#+title: Cobrowse.io
#+filetags: :project:
- Allows CS team to view a session of one of our customers.
- This will initially be available to all customers as a link on the help page.
- User-initiated, share the code with the AW agent.
- We will be using their cloud solution.
- Look into an API for adding admin notes that the session took place
- Look into information to be redacted in session recordings
+ Information displayed will have to be tagged with a CSS class to be identified

16
aweber/20210520114208-migrating_aws_services.org
View file

@ -1,16 +0,0 @@
:PROPERTIES:
:ID: af4ae6ee-5201-49ee-aa01-6cf6a0801908
:END:
#+title: Migrating AWS services
#+filetags: :project:
A [[id:db322997-ff5e-416a-8dc8-f29e6a4928c8][Technical Initiative]] to migrate services from ECS deployments to Kubernetes
deployments. The purpose is to homogenize our deployments to use Kubernetes,
which could theoretically be deployed to our local Kubernetes cluster or to EKS.
We currently maintain two separate sets of clusters in AWS. Services should be
migrated to the new cluster, or to kubernetes, ideally.
- [[id:7e503917-646f-4275-aab9-3a125b99cbfd][Tagging Service]]
- Mapping
- Recipient

977
aweber/20210629133542-supporting_multiple_currencies_in_stripe.org
View file

@ -1,977 +0,0 @@
:PROPERTIES:
:ID: 022406d2-0480-4470-90d0-9533f6b9fa32
:END:
#+title: Supporting multiple currencies in Stripe
To be completed after [[id:a9835afc-e0be-4436-8274-c3898fdf119c][Recurring and split Stripe payments]].
* How does Stripe handle different currencies?
- https://stripe.com/docs/currencies
* Which currencies are supported by a particular customer?
- [[https://stripe.com/docs/api/country_specs/object#country_spec_object-supported_payment_currencies][Supported payment currencies]] can be retrieved via the Stripe API by looking up
their Stripe account's [[https://stripe.com/docs/api/accounts/object#account_object-country][configured country]].
- Stripe accounts have a [[https://stripe.com/docs/api/accounts/object#account_object-default_currency][default currency]] configured that we could refer to when
setting up an eCommerce widget.
** What are my supported currencies?
#+begin_src sh :results output :wrap src json :exports both :eval no-export
stripe get /v1/country_specs/US
#+end_src
#+RESULTS:
#+begin_src json
{
"id": "US",
"object": "country_spec",
"default_currency": "usd",
"supported_bank_account_currencies": {
"usd": [
"US"
]
},
"supported_payment_currencies": [
"usd",
"aed",
"afn",
"all",
"amd",
"ang",
"aoa",
"ars",
"aud",
"awg",
"azn",
"bam",
"bbd",
"bdt",
"bgn",
"bif",
"bmd",
"bnd",
"bob",
"brl",
"bsd",
"bwp",
"bzd",
"cad",
"cdf",
"chf",
"clp",
"cny",
"cop",
"crc",
"cve",
"czk",
"djf",
"dkk",
"dop",
"dzd",
"egp",
"etb",
"eur",
"fjd",
"fkp",
"gbp",
"gel",
"gip",
"gmd",
"gnf",
"gtq",
"gyd",
"hkd",
"hnl",
"hrk",
"htg",
"huf",
"idr",
"ils",
"inr",
"isk",
"jmd",
"jpy",
"kes",
"kgs",
"khr",
"kmf",
"krw",
"kyd",
"kzt",
"lak",
"lbp",
"lkr",
"lrd",
"lsl",
"mad",
"mdl",
"mga",
"mkd",
"mmk",
"mnt",
"mop",
"mro",
"mur",
"mvr",
"mwk",
"mxn",
"myr",
"mzn",
"nad",
"ngn",
"nio",
"nok",
"npr",
"nzd",
"pab",
"pen",
"pgk",
"php",
"pkr",
"pln",
"pyg",
"qar",
"ron",
"rsd",
"rub",
"rwf",
"sar",
"sbd",
"scr",
"sek",
"sgd",
"shp",
"sll",
"sos",
"srd",
"std",
"szl",
"thb",
"tjs",
"top",
"try",
"ttd",
"twd",
"tzs",
"uah",
"ugx",
"uyu",
"uzs",
"vnd",
"vuv",
"wst",
"xaf",
"xcd",
"xof",
"xpf",
"yer",
"zar",
"zmw"
],
"supported_payment_methods": [
"card",
"stripe"
],
"supported_transfer_countries": [
"US",
"AT",
"AR",
"AU",
"BE",
"BG",
"BO",
"CA",
"CH",
"CR",
"CY",
"CZ",
"DE",
"DK",
"DO",
"EE",
"EG",
"ES",
"FI",
"FR",
"GB",
"GR",
"HK",
"HR",
"HU",
"ID",
"IE",
"IL",
"IS",
"IT",
"LI",
"LT",
"LU",
"LV",
"MT",
"MX",
"NL",
"NO",
"NZ",
"PE",
"PL",
"PT",
"RO",
"SE",
"SG",
"SI",
"SK",
"TH",
"TT",
"UY"
],
"verification_fields": {
"company": {
"additional": [
"representative.verification.document"
],
"minimum": [
"business_profile.mcc",
"business_profile.url",
"business_type",
"company.address.city",
"company.address.line1",
"company.address.postal_code",
"company.address.state",
"company.name",
"company.owners_provided",
"company.phone",
"company.tax_id",
"external_account",
"owners.address.city",
"owners.address.line1",
"owners.address.postal_code",
"owners.address.state",
"owners.dob.day",
"owners.dob.month",
"owners.dob.year",
"owners.email",
"owners.first_name",
"owners.id_number",
"owners.last_name",
"owners.phone",
"owners.relationship.title",
"owners.verification.document",
"representative.address.city",
"representative.address.line1",
"representative.address.postal_code",
"representative.address.state",
"representative.dob.day",
"representative.dob.month",
"representative.dob.year",
"representative.email",
"representative.first_name",
"representative.id_number",
"representative.last_name",
"representative.phone",
"representative.relationship.executive",
"representative.relationship.title",
"tos_acceptance.date",
"tos_acceptance.ip"
]
},
"individual": {
"additional": [
"individual.verification.document"
],
"minimum": [
"business_profile.mcc",
"business_profile.url",
"business_type",
"external_account",
"individual.address.city",
"individual.address.line1",
"individual.address.postal_code",
"individual.address.state",
"individual.dob.day",
"individual.dob.month",
"individual.dob.year",
"individual.email",
"individual.first_name",
"individual.id_number",
"individual.last_name",
"individual.phone",
"tos_acceptance.date",
"tos_acceptance.ip"
]
}
}
}
#+end_src
#+begin_notes
Country data cannot, unfortunately, be expanded from an account:
#+begin_src sh :exports both :results output :wrap src json :eval no-export
stripe get acct_1IGnMkIoFf3wvXpR -d "expand[]=country"
#+end_src
#+RESULTS:
#+begin_src json
{
"error": {
"message": "This property cannot be expanded (country).",
"type": "invalid_request_error"
}
}
#+end_src
#+end_notes
* What happens when a payment is below the minimum allowed amount for a settlement currency?
- Verify the error when making a price with an unsupported currenccy
- Check that the flat fee is calculated correctly
** What happens when I create a price with an unsupported currency?
#+name: price-bhd
#+caption: Creating a price using Bahraini dinars (BHD)
#+header: :var product="prod_JgnmcPS2MRwDtp"
#+header: :var account="acct_1IGnMkIoFf3wvXpR"
#+header: :wrap src json
#+begin_src shell :var product=product :results output :exports both :eval no-export
stripe prices create \
--stripe-account=$account \
--product=$product \
--unit-amount=350 \
--currency=bhd \
-d "nickname"="Price in BHD"
#+end_src
#+RESULTS: price-bhd
#+begin_src json
{
"error": {
"message": "Invalid currency: bhd. Stripe accounts in US do not support bhd. Your account currently supports these currencies: usd, aed, afn, all, amd, ang, aoa, ars, aud, awg, azn, bam, bbd, bdt, bgn, bif, bmd, bnd, bob, brl, bsd, bwp, bzd, cad, cdf, chf, clp, cny, cop, crc, cve, czk, djf, dkk, dop, dzd, egp, etb, eur, fjd, fkp, gbp, gel, gip, gmd, gnf, gtq, gyd, hkd, hnl, hrk, htg, huf, idr, ils, inr, isk, jmd, jpy, kes, kgs, khr, kmf, krw, kyd, kzt, lak, lbp, lkr, lrd, lsl, mad, mdl, mga, mkd, mmk, mnt, mop, mro, mur, mvr, mwk, mxn, myr, mzn, nad, ngn, nio, nok, npr, nzd, pab, pen, pgk, php, pkr, pln, pyg, qar, ron, rsd, rub, rwf, sar, sbd, scr, sek, sgd, shp, sll, sos, srd, std, szl, thb, tjs, top, try, ttd, twd, tzs, uah, ugx, uyu, uzs, vnd, vuv, wst, xaf, xcd, xof, xpf, yer, zar, zmw.",
"param": "currency",
"type": "invalid_request_error"
}
}
#+end_src
** What do fees look like when using an alternate currency?
#+name: price-jpy
#+caption: Creating a price using Japanese yen (JPY)
#+header: :var product="prod_JgnmcPS2MRwDtp"
#+header: :var account="acct_1IGnMkIoFf3wvXpR"
#+header: :wrap src json
#+begin_src shell :var product=product :results output :exports both :eval no-export
stripe prices create \
--stripe-account=$account \
--product=$product \
--unit-amount=3500 \
--currency=jpy \
-d "nickname"="Price in JPY"
#+end_src
#+RESULTS: price-jpy
#+begin_src json
{
"id": "price_1JBOBRIoFf3wvXpR24iMvhoG",
"object": "price",
"active": true,
"billing_scheme": "per_unit",
"created": 1625854337,
"currency": "jpy",
"livemode": false,
"lookup_key": null,
"metadata": {
},
"nickname": "Price in JPY",
"product": "prod_JgnmcPS2MRwDtp",
"recurring": null,
"tiers_mode": null,
"transform_quantity": null,
"type": "one_time",
"unit_amount": 3500,
"unit_amount_decimal": "3500"
}
#+end_src
*** Flat fee in a one-time purchase invoice
#+caption: Invoicing a purchase in JPY for a US (USD) account
#+header: :var account="acct_1IGnMkIoFf3wvXpR"
#+header: :var price="price_1JBOBRIoFf3wvXpR24iMvhoG"
#+begin_src sh :exports both :results output :wrap src json :eval no-export
customer=$(stripe customers create \
--stripe-account=$account \
--description="JPY Invoice Customer" \
--name="Correl Roush" \
--email="correl+stripe.roam.docs@gmail.com" \
| jq -r .id)
stripe invoiceitems create \
--stripe-account=$account \
--customer=$customer \
--price=$price >/dev/null
invoice=$(stripe invoices create \
--stripe-account="$account" \
--customer="$customer" \
--application-fee-amount=500 \
| jq -r .id)
intent=$(stripe invoices finalize_invoice \
--stripe-account=$account \
$invoice \
| jq -r .payment_intent)
stripe payment_intents confirm $intent \
--stripe-account="$account" \
--payment-method=pm_card_visa
#+end_src
#+RESULTS:
#+begin_src json
{
"id": "pi_1JBOVQIoFf3wvXpRRJIbx8xf",
"object": "payment_intent",
"amount": 3500,
"amount_capturable": 0,
"amount_received": 3500,
"application": "ca_IIJmzFRoIG2rIdny5psryEvYurgcOmEP",
"application_fee_amount": 500,
"canceled_at": null,
"cancellation_reason": null,
"capture_method": "automatic",
"charges": {
"object": "list",
"data": [
{
"id": "ch_1JBOVRIoFf3wvXpRCYVnRz0y",
"object": "charge",
"amount": 3500,
"amount_captured": 3500,
"amount_refunded": 0,
"application": "ca_IIJmzFRoIG2rIdny5psryEvYurgcOmEP",
"application_fee": "fee_1JBOVSIoFf3wvXpR2DbO4Xxe",
"application_fee_amount": 500,
"balance_transaction": "txn_1JBOVSIoFf3wvXpRXaKfOsWB",
"billing_details": {
"address": {
"city": null,
"country": null,
"line1": null,
"line2": null,
"postal_code": null,
"state": null
},
"email": null,
"name": null,
"phone": null
},
"calculated_statement_descriptor": "CORRELS STUFF",
"captured": true,
"created": 1625855577,
"currency": "jpy",
"customer": "cus_Jp2a7jolV9KaII",
"description": "Payment for Invoice",
"destination": null,
"dispute": null,
"disputed": false,
"failure_code": null,
"failure_message": null,
"fraud_details": {
},
"invoice": "in_1JBOVPIoFf3wvXpRuM28s36R",
"livemode": false,
"metadata": {
},
"on_behalf_of": null,
"order": null,
"outcome": {
"network_status": "approved_by_network",
"reason": null,
"risk_level": "normal",
"risk_score": 15,
"seller_message": "Payment complete.",
"type": "authorized"
},
"paid": true,
"payment_intent": "pi_1JBOVQIoFf3wvXpRRJIbx8xf",
"payment_method": "pm_1JBOVRIoFf3wvXpR3JKo6m2a",
"payment_method_details": {
"card": {
"brand": "visa",
"checks": {
"address_line1_check": null,
"address_postal_code_check": null,
"cvc_check": null
},
"country": "US",
"exp_month": 7,
"exp_year": 2022,
"fingerprint": "C5qD8oSCGvVCbtcH",
"funding": "credit",
"installments": null,
"last4": "4242",
"network": "visa",
"three_d_secure": null,
"wallet": null
},
"type": "card"
},
"receipt_email": null,
"receipt_number": null,
"receipt_url": "https://pay.stripe.com/receipts/acct_1IGnMkIoFf3wvXpR/ch_1JBOVRIoFf3wvXpRCYVnRz0y/rcpt_Jp2aVQIwbWqn8CZ7T5uF9Wr7MX016cT",
"refunded": false,
"refunds": {
"object": "list",
"data": [
],
"has_more": false,
"total_count": 0,
"url": "/v1/charges/ch_1JBOVRIoFf3wvXpRCYVnRz0y/refunds"
},
"review": null,
"shipping": null,
"source": null,
"source_transfer": null,
"statement_descriptor": null,
"statement_descriptor_suffix": null,
"status": "succeeded",
"transfer_data": null,
"transfer_group": null
}
],
"has_more": false,
"total_count": 1,
"url": "/v1/charges?payment_intent=pi_1JBOVQIoFf3wvXpRRJIbx8xf"
},
"client_secret": "pi_1JBOVQIoFf3wvXpRRJIbx8xf_secret_sP4fa41NT86GIrhcyfEZEmFJE",
"confirmation_method": "automatic",
"created": 1625855576,
"currency": "jpy",
"customer": "cus_Jp2a7jolV9KaII",
"description": "Payment for Invoice",
"invoice": "in_1JBOVPIoFf3wvXpRuM28s36R",
"last_payment_error": null,
"livemode": false,
"metadata": {
},
"next_action": null,
"on_behalf_of": null,
"payment_method": "pm_1JBOVRIoFf3wvXpR3JKo6m2a",
"payment_method_options": {
"card": {
"installments": null,
"network": null,
"request_three_d_secure": "automatic"
}
},
"payment_method_types": [
"card"
],
"receipt_email": null,
"review": null,
"setup_future_usage": null,
"shipping": null,
"source": null,
"statement_descriptor": null,
"statement_descriptor_suffix": null,
"status": "succeeded",
"transfer_data": null,
"transfer_group": null
}
#+end_src
*** Percentage fee in a subscription
#+caption: Creating a subscription with a recurring JPY fee for a US (USD) account
#+header: :var account="acct_1IGnMkIoFf3wvXpR"
#+header: :var product="prod_JgnmcPS2MRwDtp"
#+header: :var price="price_1JBOBRIoFf3wvXpR24iMvhoG"
#+header: :exports both :results output :wrap src json :eval no-export
#+begin_src sh
price=$(stripe prices create \
--stripe-account=$account \
--product=$product \
--unit-amount=11500 \
--currency=vnd \
-d "recurring[interval]"="month" \
-d "nickname"="Recurring Price in VND" \
| jq -r .id)
payment_method=$(stripe payment_methods create \
--stripe-account=$account \
--type=card \
-d "card[number]"=4242424242424242 \
-d "card[exp_month]"=3 \
-d "card[exp_year]"=2022 \
-d "card[cvc]"=314 \
| jq -r '.id')
customer=$(stripe customers create \
--stripe-account=$account \
--description="JPY Subscription Customer" \
--name="Correl Roush" \
--email="correl+stripe.roam.docs@gmail.com" \
| jq -r .id)
stripe payment_methods attach "$payment_method" \
--stripe-account=$account \
--customer="$customer" >/dev/null
stripe customers update "$customer" \
--stripe-account=$account \
-d "invoice_settings[default_payment_method]=$payment_method" >/dev/null
stripe subscriptions create \
--stripe-account=$account \
--customer="$customer" \
--application-fee-percent=1 \
-d "items[0][price]"=$price
#+end_src
#+RESULTS:
#+begin_src json
{
"id": "sub_JqAatXRV9aIjfK",
"object": "subscription",
"application_fee_percent": 1.0,
"automatic_tax": {
"enabled": false
},
"billing_cycle_anchor": 1626115944,
"billing_thresholds": null,
"cancel_at": null,
"cancel_at_period_end": false,
"canceled_at": null,
"collection_method": "charge_automatically",
"created": 1626115944,
"current_period_end": 1628794344,
"current_period_start": 1626115944,
"customer": "cus_JqAaapAUdcr46Z",
"days_until_due": null,
"default_payment_method": null,
"default_source": null,
"default_tax_rates": [
],
"discount": null,
"ended_at": null,
"items": {
"object": "list",
"data": [
{
"id": "si_JqAaoXdAhBEisj",
"object": "subscription_item",
"billing_thresholds": null,
"created": 1626115944,
"metadata": {
},
"plan": {
"id": "price_1JCUEpIoFf3wvXpRNX67TZZB",
"object": "plan",
"active": true,
"aggregate_usage": null,
"amount": 11500,
"amount_decimal": "11500",
"billing_scheme": "per_unit",
"created": 1626115939,
"currency": "vnd",
"interval": "month",
"interval_count": 1,
"livemode": false,
"metadata": {
},
"nickname": "Recurring Price in VND",
"product": "prod_JgnmcPS2MRwDtp",
"tiers_mode": null,
"transform_usage": null,
"trial_period_days": null,
"usage_type": "licensed"
},
"price": {
"id": "price_1JCUEpIoFf3wvXpRNX67TZZB",
"object": "price",
"active": true,
"billing_scheme": "per_unit",
"created": 1626115939,
"currency": "vnd",
"livemode": false,
"lookup_key": null,
"metadata": {
},
"nickname": "Recurring Price in VND",
"product": "prod_JgnmcPS2MRwDtp",
"recurring": {
"aggregate_usage": null,
"interval": "month",
"interval_count": 1,
"trial_period_days": null,
"usage_type": "licensed"
},
"tiers_mode": null,
"transform_quantity": null,
"type": "recurring",
"unit_amount": 11500,
"unit_amount_decimal": "11500"
},
"quantity": 1,
"subscription": "sub_JqAatXRV9aIjfK",
"tax_rates": [
]
}
],
"has_more": false,
"total_count": 1,
"url": "/v1/subscription_items?subscription=sub_JqAatXRV9aIjfK"
},
"latest_invoice": "in_1JCUEuIoFf3wvXpRl1be4Ymf",
"livemode": false,
"metadata": {
},
"next_pending_invoice_item_invoice": null,
"pause_collection": null,
"pending_invoice_item_interval": null,
"pending_setup_intent": null,
"pending_update": null,
"plan": {
"id": "price_1JCUEpIoFf3wvXpRNX67TZZB",
"object": "plan",
"active": true,
"aggregate_usage": null,
"amount": 11500,
"amount_decimal": "11500",
"billing_scheme": "per_unit",
"created": 1626115939,
"currency": "vnd",
"interval": "month",
"interval_count": 1,
"livemode": false,
"metadata": {
},
"nickname": "Recurring Price in VND",
"product": "prod_JgnmcPS2MRwDtp",
"tiers_mode": null,
"transform_usage": null,
"trial_period_days": null,
"usage_type": "licensed"
},
"quantity": 1,
"schedule": null,
"start_date": 1626115944,
"status": "active",
"transfer_data": null,
"trial_end": null,
"trial_start": null
}
#+end_src
#+caption: Fetching the initial invoice
#+header: :var account="acct_1IGnMkIoFf3wvXpR"
#+header: :exports both :results output :wrap src json :eval no-export
#+begin_src sh
stripe get in_1JCUEuIoFf3wvXpRl1be4Ymf \
--stripe-account=$account
#+end_src
#+RESULTS:
#+begin_src json
{
"id": "in_1JCUEuIoFf3wvXpRl1be4Ymf",
"object": "invoice",
"account_country": "US",
"account_name": "Correl's Stuff",
"account_tax_ids": null,
"amount_due": 11500,
"amount_paid": 11500,
"amount_remaining": 0,
"application_fee_amount": 115,
"attempt_count": 1,
"attempted": true,
"auto_advance": false,
"automatic_tax": {
"enabled": false,
"status": null
},
"billing_reason": "subscription_create",
"charge": "ch_1JCUEuIoFf3wvXpRB1M6kEQz",
"collection_method": "charge_automatically",
"created": 1626115944,
"currency": "vnd",
"custom_fields": null,
"customer": "cus_JqAaapAUdcr46Z",
"customer_address": null,
"customer_email": "correl+stripe.roam.docs@gmail.com",
"customer_name": "Correl Roush",
"customer_phone": null,
"customer_shipping": null,
"customer_tax_exempt": "none",
"customer_tax_ids": [
],
"default_payment_method": null,
"default_source": null,
"default_tax_rates": [
],
"description": null,
"discount": null,
"discounts": [
],
"due_date": null,
"ending_balance": 0,
"footer": null,
"hosted_invoice_url": "https://invoice.stripe.com/i/acct_1IGnMkIoFf3wvXpR/invst_JqAahNswSL7xgr1R8bfwCjJA4V0UouX",
"invoice_pdf": "https://pay.stripe.com/invoice/acct_1IGnMkIoFf3wvXpR/invst_JqAahNswSL7xgr1R8bfwCjJA4V0UouX/pdf",
"last_finalization_error": null,
"lines": {
"object": "list",
"data": [
{
"id": "il_1JCUEuIoFf3wvXpRcrDHYRXJ",
"object": "line_item",
"amount": 11500,
"currency": "vnd",
"description": "1 × Example Product (Org-Roam Doc) (at ₫11,500 / month)",
"discount_amounts": [
],
"discountable": true,
"discounts": [
],
"livemode": false,
"metadata": {
},
"period": {
"end": 1628794344,
"start": 1626115944
},
"plan": {
"id": "price_1JCUEpIoFf3wvXpRNX67TZZB",
"object": "plan",
"active": true,
"aggregate_usage": null,
"amount": 11500,
"amount_decimal": "11500",
"billing_scheme": "per_unit",
"created": 1626115939,
"currency": "vnd",
"interval": "month",
"interval_count": 1,
"livemode": false,
"metadata": {
},
"nickname": "Recurring Price in VND",
"product": "prod_JgnmcPS2MRwDtp",
"tiers_mode": null,
"transform_usage": null,
"trial_period_days": null,
"usage_type": "licensed"
},
"price": {
"id": "price_1JCUEpIoFf3wvXpRNX67TZZB",
"object": "price",
"active": true,
"billing_scheme": "per_unit",
"created": 1626115939,
"currency": "vnd",
"livemode": false,
"lookup_key": null,
"metadata": {
},
"nickname": "Recurring Price in VND",
"product": "prod_JgnmcPS2MRwDtp",
"recurring": {
"aggregate_usage": null,
"interval": "month",
"interval_count": 1,
"trial_period_days": null,
"usage_type": "licensed"
},
"tiers_mode": null,
"transform_quantity": null,
"type": "recurring",
"unit_amount": 11500,
"unit_amount_decimal": "11500"
},
"proration": false,
"quantity": 1,
"subscription": "sub_JqAatXRV9aIjfK",
"subscription_item": "si_JqAaoXdAhBEisj",
"tax_amounts": [
],
"tax_rates": [
],
"type": "subscription"
}
],
"has_more": false,
"total_count": 1,
"url": "/v1/invoices/in_1JCUEuIoFf3wvXpRl1be4Ymf/lines"
},
"livemode": false,
"metadata": {
},
"next_payment_attempt": null,
"number": "CC4DCDC8-0001",
"on_behalf_of": null,
"paid": true,
"payment_intent": "pi_1JCUEuIoFf3wvXpRYSMrrCoL",
"payment_settings": {
"payment_method_options": null,
"payment_method_types": null
},
"period_end": 1626115944,
"period_start": 1626115944,
"post_payment_credit_notes_amount": 0,
"pre_payment_credit_notes_amount": 0,
"receipt_number": "2336-2115",
"starting_balance": 0,
"statement_descriptor": null,
"status": "paid",
"status_transitions": {
"finalized_at": 1626115944,
"marked_uncollectible_at": null,
"paid_at": 1626115944,
"voided_at": null
},
"subscription": "sub_JqAatXRV9aIjfK",
"subtotal": 11500,
"tax": null,
"total": 11500,
"total_discount_amounts": [
],
"total_tax_amounts": [
],
"transfer_data": null,
"webhooks_delivered_at": 1626115947
}
#+end_src
* What do we do about sales tracking?
- Start storing currency with sales
+ Add a currency column to the analytics database
+ Add a currency field to the avro event
- +Hide non-usd transactions from reports, etc. until they are ready?+
- Beware of mixed-currency totals
* Tasks
** Add an endpoint exposing an account's supported currencies
** Remove =usd= restriction to the currency field in Stripe requests
** Add acceptance tests for multiple currencies
** Reporting
*** Add a currency column to the analytics database
*** Add a currency field to the pageview avro event
*** Update page hits consumer to store the currency field
*** Update reports
Both classic and redesigned reports need to be updated to handle multiple
currencies.
- Totals must be broken down by currency
- Values must be displayed in a manner appropriate for its currency
It'll be more straightforward to duplicate the graph/table for each currency or
otherwise filter them rather than attempt to restructure them to accomodate
another dimension (currency).
- Do we want to eliminate the old reports? Or at least the old sales over time
report? The new sales over time report will need the activity detail section.
+ /Yes, provided it's doable in a similar or shorter amount of time./
**** Revenue Over Time
https://gitlab.aweber.io/CP/applications/sites/-/blob/master/aweber_app/controllers/analytics_charts_controller.php
**** Sales over time
*** Update display of sale value in subscriber details
We'll want to either find a way to mimic the behavior of the frontend currency
display library, or pass the value through in such a way that we could use that
library.

199
aweber/20210630150008-stripe_poller.org
View file

@ -1,199 +0,0 @@
:PROPERTIES:
:ID: 83d61eef-0781-46e0-b959-1a739cff5ea3
:END:
#+title: Stripe Poller
Identify webhook events tracked by Stripe that have not yet been processed by
our service, and replay them against it.
* Investigation
** Fetching events on behalf of each connected account
On the afternoon of [[id:8c85f055-9d0c-4b7b-991e-1e32905d38ba][2021-07-02]] I fetched the 860 connected accounts and fetched
webhook events from the previous two hours. Each request to Stripe took, on
average, 0.38 seconds, requiring one request per account, totalling 323.77
seconds, or nearly five and a half minutes. No degradation nor failure of Stripe
functionality was noted. From all of these requests, a total of 19 events were
retrieved.
** Alternatives
*** The Stripe dashboard
Viewing a particular webhook within the Stripe dashboard allows us to view the
events sent specifically to that webhook, independent of the account to which it
relates. Regrettably, this is fed from a dashboard-specific API which does not
appear to be exposed elsewhere for consumption.
*** Email notification
We are emailed when a webhook event fails to process. It may make sense to alert
when this occurs and correct the issue manually, or find a way to automate its
replay.
** Conclusion
Barring the release of an API to fetch all events sent to a single webhook,
ideally filtered by delivery status, fetching these events from Stripe is
terribly inefficient. Though we can process recent events in a timely manner, we
will likely be further slowed as more customers connect with Stripe, and there
is no good solution at this time for dealing with that problem. Given the
infrequency with which Stripe fails to send us an event via their retry
mechanisms, I expect we will be better served by reacting to notifications of
those failures than to routinely slam their API with inefficient requests.
* Tasks
** Build Poller
*** Prepare a rundeck job for the Stripe poller
- Create the project
- Build the rundeck job
*** Add a stripe-payments endpoint to fetch logged events
- Needs only return event ids
- Events are partitioned by date and sorted by time in the time-index GSI, which projects keys only
#+begin_src yaml
paths:
/stripe/events:
get:
summary: Fetch webhook events
tags:
- Webhooks
parameters:
- name: last_id
in: query
schema:
$ref: '#/components/schemas/EventId'
- name: limit
in: query
description: Number of events to return per page of results
schema:
type: integer
minimum: 1
maximum: 1000
default: 100
responses:
'200':
description: Event list
headers:
Link:
description: Pagination links
schema:
type: string
example: >-
<{{ base_url }}/stripe/events?last_id=evt_1234abcdef>; rel="next"
content:
application/json:
schema:
type: array
items:
$ref: '#/components/schemas/EventId'
components:
schemas:
EventId:
type: string
description: Unique identifier for the webhook event
example: "evt_1IHAsBIoFf3wvXpR7VLvGfae"
#+end_src
*** Add integrations endpoint enumerating connected Stripe accounts
*** Reprocess unlogged Stripe events
- Iterate processed event ids into memory
- Iterate over published events, sending them to the stripe-payments webhook
endpoint if they are not in the processed set
#+begin_src plantuml :file stripe-poller.svg
loop until last page of results
Poller -> StripePayments : GET /stripe/events?since=<timestamp>[&last_id=<last_id>]
StripePayments --> Poller : Return list of event IDs
end
loop until last page of results
Poller -> Integrations : Get list of connected accounts
Integrations --> Poller : Return list of Stripe account IDs
loop for each account
loop until last page of results
Poller -> Stripe : GET /v1/events?created[gte]=<timestamp>&types[]=<...>[&starting_after=<last_id>]
StripePayments --> Poller : Return list of event IDs
alt is an AWeber Ecommerce event and event not in processed
Poller -> StripePayments : POST /stripe/webhooks
end
end
end
end
#+end_src
#+RESULTS:
[[file:stripe-poller.svg]]
**** Determining how far back to look for events
Stripe events are retained for a maximum of thirty days, on their end and also
in our event database.
The job could use consul to store the time of the latest fetched event to be
referenced in subsequent runs. If it is set, it should collect all events newer
than one hour prior to that time (this window may also be configurable). If that
time is not set, it should collect all available events. This value should be
updated in consul only when the job completes successfully.
Hit the Rundeck API to get the time of the last successful execution.
**** Retrieving webhook events
https://stripe.com/docs/api/events/list
+ Events can be filtered by type, creation time, and whether they were
successfully delivered (the webhook endpoint returned a =200 OK= response)
- Wait, how does it know it was succesfully delivered to /our/ webhook
endpoint?
+ Should we set up another "client" with its own rate limiting with Stripe for
the poller's requests?
#+CAPTION: Support chat on [2021-07-01 Thu]
#+begin_quote
- My team is hoping to build an automated process to check via the Stripe API
for any events that weren't successfully delivered to our webhook endpoint and
see that they're handled appropriately. I've been trying out the events list
endpoint, and it does not seem to return all of the events that were sent to
our webhook, nor does there appear to be any way to identify a webhook to that
endpoint to find and filter events for it. We assume this is because the
events are triggered from connected accounts. We noticed that the Stripe
dashboard is able to show events sent to a webhook and their status, is there
some way to achieve this through the API?
+ /Hector Otero has joined./
+ Hi, there. I am glad to help.
- Hello. Are you able to see my previous message?
+ Yes, I am just reading through it.
+ Ok, so your main question is if there is a way to show events sent to a webhook and their status using the API, similar to how it is done on the Stripe dashboard.
+ Is that correct?
- Correct
+ Ok, my brief look into the documentation has not turned up anything regarding how to implement this. I am going to consult with my team members regarding whether any of them knows how to implement this functionality. Once I have more info I can reach out to you via email, is that alright?
- Yes, thank you.
+ Sure, no worries. Please keep an eye on your email inbox awapi@aweber.com for further correspondence regarding this issue.
+ Have a nice day! we will be in contact soon.
- Thanks!
+ /Hector Otero has left./
#+end_quote
**** Signing the webhook request
Webhook events [[https://stripe.com/docs/webhooks/signatures][must be signed using a secret key]], which we store [[http://consul.service.production.consul/ui/production/kv/services/cp/services/stripe-payments/stripe_webhook_secret/edit][in consul]]. The
following /should/ result in a valid signature header:
#+CAPTION: Stripe signature generation example
#+begin_src python :results output :exports code
import hmac
import hashlib
import time
unix_timestamp = int(time.time())
json_payload = '{ ... }'
secret_key = 'secret'
message = f'{unix_timestamp}.{json_payload}'
signature = hmac.new(bytes(secret_key, 'utf-8'),
msg=bytes(message, 'utf-8'),
digestmod=hashlib.sha256).hexdigest()
print(f'Stripe-Signature: t={unix_timestamp},v1={signature}')
#+end_src
#+RESULTS:
: Stripe-Signature: t=1625084385,v1=dce1ef0332969bce98fd76b5fd08d1b07af0d0fd5f9788d9f8435537e5c3cd12
*** Create playbook and dashboard for the Stripe poller
- Track how many events are resent for processing vs how many are already processed
** KILL Create an event processing endpoint
Create an endpoint in the Stripe Payments service for internal use that will,
given an event id, fetch that event from Stripe and process it as though it had
been sent to the webhook endpoint.
** TODO Document how to find and replay a Stripe event
- Use the Stripe UI to find failed events within the past 15 days
- Include steps for fetching an event from more than 15 days ago from the API
and sending that to the webhook endpoint.

5
aweber/20210706161607-supporting_multiple_time_zones.org
View file

@ -1,5 +0,0 @@
:PROPERTIES:
:ID: 89c2dda6-46d7-41c9-8af7-18ce604a2daf
:END:
#+title: Supporting multiple time zones
* Initial investigation

31
aweber/20210714102625-deploying_projects.org
View file

@ -1,31 +0,0 @@
:PROPERTIES:
:ID: e06b26c8-9227-4fcc-8f0a-9b83c64693b4
:END:
#+title: Deploying projects
* Environments
- Testing :: Newly merged code is automatically deployed to this environment to
be tested.
- Staging :: Tagged releases are automatically deployed to this environment for
spot-checking prior to production release.
- Production :: Tagged releases are manually deployed to the live production
environment.
* Deployment methods
** Gitlab CI
Projects define pipelines in a =.gitlab-ci.yml= file to automate running tests,
building the project, and deploying it to our three platform environments.
** Jenkins :deprecated:
Pipelines are defined in Jenkins to react to pushed and tagged code in source
control to run tests and deploy projects to our platform environments.
** Chef / Puppet :deprecated:
* Deployment targets
** Buzzops (Local Kubernetes)
https://confluence.aweber.io/display/STD/Kubernetes+Application+Deployment
** Amazon Web Services
When appropriate, dockerized applications may be deployed to Amazon ECS
** Chef and Puppet managed virtual machines :deprecated:
* Further information
** Front-End Applications
https://confluence.aweber.io/display/FEBOF/Web+App+Deployment
** Control Panel (Sites)
https://confluence.aweber.io/display/AR/Control+Panel+%28Sites%29+Playbook

6
aweber/20210714105000-team_member_onboarding.org
View file

@ -1,6 +0,0 @@
:PROPERTIES:
:ID: bf5d1146-9481-4710-8143-61086f263a7a
:END:
#+title: Team Member Onboarding
- [[id:e06b26c8-9227-4fcc-8f0a-9b83c64693b4][Deploying projects]]

23
aweber/20210810095435-deploying_s4_to_kubernetes.org
View file

@ -1,23 +0,0 @@
:PROPERTIES:
:ID: 9cfd85fd-998e-4f21-b82e-c7963576c202
:END:
#+title: Deploying S4 to Kubernetes
Tasks for deploying [[id:c7322400-c6e6-4595-87e2-7db6e57b6a2b][S4]] to Kubernetes.
* DONE Deploying the service
* Migrating the Redis database
* Deploying the workers
** DONE Use environment variables for configuration rather than consul
https://gitlab.aweber.io/CP/Rundeck/s4-utils/-/commit/8d4dd3b8196bcd716a70e1c751b4743fdaa62646
** DONE Set up all the rundeck jobs in testing
- Configure the rundeck jobs
- Ensure all ACL tokens are accounted for
** TODO Update the appdb credentials
- [ ] Testing
- [ ] Staging
- [ ] Production
** TODO Set up all the rundeck jobs in staging
** TODO Set up all the rundeck jobs in production
** TODO Add missing config values
- IP ranges =lock_key= and =lock_minutes=

108
aweber/20210813142844-projects.org
View file

@ -1,108 +0,0 @@
:PROPERTIES:
:ID: 207560cc-7700-4d06-918d-cc01ae530146
:END:
#+title: Projects
#+STARTUP: indent logdrawer
#+COLUMNS: %50ITEM %JIRA_ID
#+PROPERTY: Effort_ALL 0:30 1:00 0.5d 1d 2d 3d 4d 1w
#+PROPERTY: ClassificationOfWork_ALL backend frontend ops product design
#+TODO: BACKLOG(b!) TODO(t!) | DONE(d@!) CANCELLED(c@!)
#+TAGS: { SPRINT(S) EPIC(e) STORY(s) BUG(b) TASK(t) }
#+OPTIONS: num:nil toc:t arch:nil p:t prop:t
#+LINK: jira https://jira.aweber.io/browse/
* Priorities
#+BEGIN: columnview :id global :match "TODO=\"TODO\""
| ITEM | JIRA_ID |
|------------------------------------------------+---------|
| Create the [[id:11edd6c9-b976-403b-a419-b5542ddedaae][Subscriber Search Service]] | |
| Create the [[id:c45881de-46f2-4f76-9579-063626c5956c][Analytics View Service]] | |
| [[id:619b6c78-7be9-4ee4-a0b7-9d1a4d7536e2][Migrating services to use the new List service]] | |
#+END:
* Service Upgrades
** DONE Deploy GeoIP to Kubernetes
:PROPERTIES:
:JIRA_ID: CCPANEL-11592
:END:
:LOGBOOK:
- State "TODO" from [2021-09-01 Wed 13:44]
:END:
** DONE [[id:9cfd85fd-998e-4f21-b82e-c7963576c202][Deploying S4 to Kubernetes]]
:PROPERTIES:
:JIRA_ID: CCPANEL-10549
:END:
:LOGBOOK:
- State "TODO" from [2021-09-01 Wed 13:42]
:END:
** DONE [[id:6413d680-ee2e-43e6-b7c7-10f14e0873c2][Deploying Bulk Tagging to Kubernetes]]
:PROPERTIES:
:JIRA_ID: CCPANEL-11615
:END:
:LOGBOOK:
- State "TODO" from [2021-09-01 Wed 13:42]
:END:
** BACKLOG Deploying Domain Validator to Kubernetes
:PROPERTIES:
:JIRA_ID: CCPANEL-10554
:END:
:LOGBOOK:
- State "BACKLOG" from "TODO" [2021-10-20 Wed 15:53]
- State "TODO" from [2021-09-01 Wed 13:42]
:END:
** BACKLOG Deploy Notification Bar to Kubernetes
:LOGBOOK:
- State "BACKLOG" from [2021-10-26 Tue 14:02]
:END:
** DONE Deploying Recipient Service to Kubernetes
:LOGBOOK:
- State "BACKLOG" from "TODO" [2021-10-20 Wed 15:53]
- State "TODO" from [2021-10-13 Wed 16:26]
:END:
** BACKLOG Deploying Tagging Service to Kubernetes
:LOGBOOK:
- State "BACKLOG" from "TODO" [2021-10-20 Wed 15:53]
- State "TODO" from [2021-10-13 Wed 16:26]
:END:
* [[id:f633f967-11d2-432c-b5ff-ad842c88a51c][Decommissioning Sites]]
** [[id:3cc8bd09-dd02-4950-8c89-a737f92809fd][Tracking progress of moving pages out of Sites]]
** Creating a new Control Panel shell application
* TODO Create the [[id:11edd6c9-b976-403b-a419-b5542ddedaae][Subscriber Search Service]]
:LOGBOOK:
- State "TODO" from [2021-10-20 Wed 15:57]
:END:
* TODO Create the [[id:c45881de-46f2-4f76-9579-063626c5956c][Analytics View Service]]
:LOGBOOK:
- State "TODO" from [2021-10-20 Wed 15:57]
:END:
* [[id:ee5b8d5f-e3d4-45c2-9ce6-bcd8c7a63376][Retire Redcache]]
* [[id:4df15f2f-d2e1-40f4-8acd-dbfb78fe304f][Deploy CoreAPI to Kubernetes]]
* Replacing CAPI Services
** TODO [[id:619b6c78-7be9-4ee4-a0b7-9d1a4d7536e2][Migrating services to use the new List service]]
:LOGBOOK:
- State "TODO" from [2021-10-20 Wed 15:58]
:END:
*** DONE Set EOL date for awlists
- [2021-08-13 Fri 15:21] :: Discussed this. Also talked about separation of
concerns about account status vs list status. Also discussed how an
entitlements service might fit into our architecture and how we handle state
transitions and reverals (e.g. cancellations).
- [2021-08-17 Tue 16:44] :: Set a one-year time limit? Should the public list
endpoints be in the new service as well, deprecating public api lists?
- [2021-10-18 Mon] :: The expectation is set to be migrated to the new list service exclusively by the end of Q2 2022
** Subscribers API
*** [[id:2c1a7b1d-8726-4b88-9534-2f5abfec35f0][Use AppDB as the source of truth for subscriber data in Recipient]]
* Frontend Client Upgrades
** Upgrade Dashboard to React
*** BACKLOG Create an API for broadcasts and sent messages across lists
:PROPERTIES:
:JIRA_ID: CCPANEL-11609
:END:
:LOGBOOK:
- State "BACKLOG" from "TODO" [2021-10-20 Wed 15:57]
- State "TODO" from [2021-09-01 Wed 13:33]
:END:
** Upgrade other non-React projects to React
*** Add subscriber
** [[id:fab0cf8f-7c54-4848-882b-dba5e087760d][Redesigned Reports]]
* New List Management Interface

19
aweber/20210826152454-deploying_bulk_tagging_to_kubernetes.org
View file

@ -1,19 +0,0 @@
:PROPERTIES:
:ID: 6413d680-ee2e-43e6-b7c7-10f14e0873c2
:END:
#+title: Deploying Bulk Tagging to Kubernetes
#+filetags: project
* DONE Deploy to kubernetes in production
* Update services to use the new consul hostname
** DONE Subscriber Proxy
Configured in consul
** DONE Bulk Tagging Consumers
Configured in consul
** TODO AWSubscribers
https://gitlab.aweber.io/CP/Services/awsubscribers
** TODO Bulk Tagging Acceptance Tests
[[https://gitlab.aweber.io/CP/automation/bulk-tagging-acceptance/]]
** TODO QA Regression Tests
- https://gitlab.aweber.io/BoFs/QA/QAWeber

202
aweber/20210914142000-subscriber_search_service.org
View file

@ -1,202 +0,0 @@
:PROPERTIES:
:ID: 11edd6c9-b976-403b-a419-b5542ddedaae
:END:
#+title: Subscriber Search Service
#+LINK: jira https://jira.aweber.io/browse/
A replacement for the current [[id:d9cb2b55-3b0e-4ab3-8369-f71ebc3cd882][Sites Subscriber Search]] and [[id:f74e335d-577f-4749-bf32-1c025795b039][Broadcast Segment Search]] implementations.
- [[https://jira.aweber.io/issues/?jql=project%20%3D%20CCPANEL%20AND%20component%20%3D%20%22Search%20Service%22][JIRA tickets]]
* Architecture Notes
#+begin_quote
Added: [2019-08-06 Tue 10:34]
- Broadcast sending manipulated queries before performing them to suit
its needs
- *Handle list exclusion in the search service?* ([[jira:CCPANEL-9557][CCPANEL-9557]])
- Blocked emails
- De-duplication
- *Lead view ids -> segment ids -> search service segments?* ([[jira:CCPANEL-9556][CCPANEL-9556]])
- Side-by-side comparison
- Use broadcast-segment to compare its results to the search api
results
- *Recipient-style representation or leads?*
- *Define API* ([[jira:CCPANEL-9554][CCPANEL-9554]])
- Include saved searches (segments)
- *Iterate release* ([[jira:CCPANEL-9555][CCPANEL-9555]])
- Current UI
- New UI with same capabilities of old UI
- Still writing old-style segments
- Research getting off the leads table (aurora?)
- https://confluence.aweber.io/display/AR/Search+Service+Using+Existing+Databases
- Distinct from a materialized search-optimized db
- Deal with =subscriber_tags= table bloat in AppDB
#+end_quote
** Component Diagram
#+BEGIN_SRC plantuml :file search-components.svg
database Analytics {
database Ana as Ana01
database Ana as Ana02
database Ana as Ana03
}
database App
database "Results Cache" as ResultsCache
component "Search Service" as Service {
component Search
component Results
Search -- Analytics
Search -- App
Search --> ResultsCache
Results <-- ResultsCache
}
#+END_SRC
#+RESULTS:
[[file:search-components.svg]]
* Concerns
** Performance
Email delivery has cases
*** Analytics query speedups for broadcasts
https://jira.aweber.io/browse/EDELIV-8318
** Fitness to Purpose
This service will need to fulfill the needs of both end-user subscriber searches
and segment emailing.
* Plan
:PROPERTIES:
:COLUMNS: %40ITEM %Effort{:}
:END:
#+BEGIN: columnview :id local
| ITEM | Effort |
|--------------------------------------------------------+----------|
| Plan | 18d 0:00 |
| Search Centralization | 18d 0:00 |
| Expose search inputs backed with the existing database | 2d |
| Enable dblink on the search master database | 2d |
| Create new unlogged search results table | 2d |
| Define the search result format | 1d |
| Perform search using new search DSL | 5d |
| Perform search using legacy segment ID | 3d |
| Manage segments using the existing database | 3d |
| Migrate to an updated schema | |
| Migrate to new search service | |
| Create new subscriber management React application | |
| Update broadcast-segment to use new search service | |
| Milestone 3: Add new search features | |
#+END:
** A Dedicated Service for Subscriber Search
:PROPERTIES:
:JIRA_ID: CCPANEL-10169
:END:
Create a dedicated, publicly exposed service for performing searches on
subscribers using subscriber and analytics criteria. The goal of this project is
to replace the current implementation from sites and in the broadcast segment
service.
*** Expose search inputs backed with the existing database
:PROPERTIES:
:Effort: 2d
:END:
- Include IDs required to build existing POST format
*** Enable dblink on the search master database
:PROPERTIES:
:Effort: 2d
:END:
[[jira:CCPANEL-7147][CCPANEL-7147]]
*** NO Create new unlogged search results table
:PROPERTIES:
:Effort: 2d
:END:
https://jira.aweber.io/browse/CCPANEL-7077
*** Define the search result format
:PROPERTIES:
:Effort: 1d
:JIRA_ID: CCPANEL-10440
:END:
https://xd.adobe.com/view/ae8fb2b2-c039-4e88-8ade-ff2562a8c8cf-fbdc/screen/c03f09c7-187e-4a6f-8b8c-571d131daee1/ (ignore engagement column)
- name
- email
- source
- status
- date added
- last updated
*** Perform search using new search DSL
:PROPERTIES:
:Effort: 5d
:END:
**** DONE Perform search using text comparisons
- Is / Is Not
- Contains / Does not contain
- Starts with / Does not start with
- Ends with / Does not end with
**** DONE Perform search using numeric comparisons
**** DONE Perform search using tag sets
**** DONE Perform search using enumerated values
**** Add support for remaining static AppDB filters
- Date range fields
- Remaining string / numeric fields
**** Add support for custom field filters
- Add all custom field columns as supported filters
- Fetch the custom fields for the list and use them when building the list of
available filters.
**** Add support for Analytics filters
- Clicks
- Opens
- Messages
- Web pages (links)
*** Perform search using legacy segment ID
:PROPERTIES:
:Effort: 3d
:END:
Provided with a legacy segment ID, execute a search using its stored parameters.
*** Manage segments using the existing database
:PROPERTIES:
:Effort: 3d
:END:
Create, retrieve, update, and delete legacy segments using the new search DSL.
*** Manage segments stored using the new DSL
*** Migrate legacy segments to the new DSL
*** Migrate to an updated schema
*** Support filtering options used by broadcast-segment
** Centralizing Subscriber Search
Applications making use of subscriber search will be updated to use the new
dedicated service, eliminating multiple search implementations.
*** Create new subscriber management React application
*** Update broadcast-segment to use new search service
** Milestone 3: Add new search features
* Implementation
** [[id:7b0f97f3-9037-4d05-9170-a478e97c8d1f][Modeling the new search DSL]]
** Constructing SQL queries programmatically
** Translating legacy segments
** Gathering results
** Reaching into Analytics
* Resources
- [[https://confluence.aweber.io/display/AR/PostgreSQL+Backed+Search][PostgreSQL Backed Search]] (Rejected ACP)
- [[https://confluence.aweber.io/display/AR/Search+Proxy+Service][Search Proxy Service]]
- +[[https://confluence.aweber.io/display/~robink/SoT+-+ElasticSearch+Next+Steps][SoT - ElasticSearch Next Steps]]+
- [[https://confluence.aweber.io/display/~robink/Alternative+Search+Proposal][Alternative Search Proposal]]
- [[https://confluence.aweber.io/display/AR/Search+Service+Using+Existing+Databases][Search Service Using Existing Databases]] (Approved ACP)
+ [[https://confluence.aweber.io/display/~victorc/Search+Service+-+Proof+of+Concept+Findings][Search Service - Proof of Concept Findings]] (Benchmarks different approaches)
+ [[https://gitlab.aweber.io/CP/archive/victorc-search-prototype][Search Prototype]]
- [[https://confluence.aweber.io/display/AR/Search+DSL+JSON+Schema][Search DSL JSON Schema]]

42
aweber/20210914143542-sites_subscriber_search.org
View file

@ -1,42 +0,0 @@
:PROPERTIES:
:ID: d9cb2b55-3b0e-4ab3-8369-f71ebc3cd882
:END:
#+title: Sites Subscriber Search
* Sorting
Added: [2020-04-14 Tue 13:34]
The current sites search code includes the following functioning code for
setting a sort order on a search based on form input:
https://gitlab.aweber.io/CP/applications/sites/blob/52d1d944854554c5818ef9a46c8a12493599eb55/aweber_app/controllers/queries_controller.php#L386-402
#+begin_src php :exports code :eval never
// Look up the column for the order by clause. There are no SQL column name values passed publicly.
if (!empty($this->data['SearchOrder']['SearchInput'])){
$this->SearchInput->recursive = -1;
if ($time = $this->SearchMutex->lock($aId, '6')) {
$orderCol = $this->SearchInput->find(array('SearchInput.id' => $this->data['SearchOrder']['SearchInput']));
$this->SearchMutex->unlock($aId, '6', $time);
}
if (!empty($orderCol['SearchInput']['column'])){
//Case-insensitive text sorting.
// Lower text fields so that ordering is case insensitive. SearchInputs 5, 23, and 24 are actually integers, despite
// having a text search input. refs #3275
if($orderCol['SearchInput']['input_type'] == 'text' && !in_array($orderCol['SearchInput']['id'], array(5,23,24))) {
$orderCol['SearchInput']['column'] = 'lower('.$orderCol['SearchInput']['column'].')';
}
$this->data['SearchOrder']['column'] = $orderCol['SearchInput']['column'];
}
}
#+end_src
- It is saved with the segment
- It is passed back to the front-end when loading a saved segment
- The =SearchCriteria= class incorporates the selected ordering and column when
building its query for a targeted search database.
- The broadcast segment service ignores the selected ordering, opting for its
own for deliverability reasons.
- All search inputs NOT in the analytics database are available for sorting (https://gitlab.aweber.io/CP/applications/sites/blob/f7ea2e9431e3ed2e694730f6446b4b3828d7c8fe/aweber_app/views/helpers/search_form.php#L54-62).
- Performance degrades with list size, likely due to memory constraints and
unindexed sort fields
(https://www.cybertec-postgresql.com/en/postgresql-improving-sort-performance/).

7
aweber/20210914144833-broadcast_segment_search.org
View file

@ -1,7 +0,0 @@
:PROPERTIES:
:ID: f74e335d-577f-4749-bf32-1c025795b039
:END:
#+title: Broadcast Segment Search
Performs a search using a stored segment and builds an iterable list of
recipients for a broadcast email. Implemented in the service's [[https://gitlab.aweber.io/edeliv/Applications/broadcast-segment/-/blob/master/broadcastsegment/handlers.py#L357][BroadcastHandler]].

7
aweber/20210915132045-redesigned_reports.org
View file

@ -1,7 +0,0 @@
:PROPERTIES:
:ID: fab0cf8f-7c54-4848-882b-dba5e087760d
:END:
#+title: Redesigned Reports
Currently dependent on the [[id:3ddc4e32-932f-4748-bfe9-7025d4d6b352][Report API Controller]] in [[id:57ee2f00-9bcd-4e0f-8a77-ae1f2d4cda89][sites]] for data, which we
hope to move into an [[id:c45881de-46f2-4f76-9579-063626c5956c][Analytics View Service]].

737
aweber/20210915133101-analytics_view_service.org
View file

@ -1,737 +0,0 @@
:PROPERTIES:
:ID: c45881de-46f2-4f76-9579-063626c5956c
:END:
#+title: Analytics View Service
#+TODO: WAITING(w) READY(r) | DONE(d)
The Analytics View Service provides a collection of report endpoints. These
endpoints handle querying the analytics databases or, in the longer term,
exposing efficient materialized data views.
* Plan
- Parent ticket :: [[https://jira.aweber.io/browse/CCPANEL-11781][CCPANEL-11781]]
** Create the analytics view service
- New project using cookie cutter
- Deployed to kubernetes
- Grafana dashboard created
** Create the analytics view service playbook
** Plan API structure
- Pathing (=/reports/*=)?
- Report versioning? (=/reports/$NAME.v$VERSION=)?
** Create endpoints for existing reports
- Based on the endpoints provided in the [[id:3ddc4e32-932f-4748-bfe9-7025d4d6b352][Report API Controller]]
- Are all of these report endpoints in use?
*** Opens over time
#+attr_confluence: :as-table t
- Name :: daily-opens
- Parameters ::
- List (default: all lists)
- Date range (default: last 30 days)
- Report API controller endpoints ::
- opens_all_range
- opens_list_range
#+caption: Sample response
#+begin_src json
{
"2021-11-02T00:00:00Z": {
"broadcasts": 2499,
"followups": 2547,
"unique": 2923,
"total": 5046
},
"2021-11-03T00:00:00Z": {
"broadcasts": 25808,
"followups": 2430,
"unique": 24876,
"total": 28238
},
"2021-11-04T00:00:00Z": {
"broadcasts": 16733,
"followups": 1437,
"unique": 14780,
"total": 18170
}
}
#+end_src
*** Clicks over time
#+attr_confluence: :as-table t
- Name :: daily-clicks
- Parameters ::
- List (default: all lists)
- Date range (default: last 30 days)
- Report API controller endpoints ::
- clicks_all_range
- clicks_list_range
#+caption: Sample response
#+begin_src json
{
"2021-11-02T00:00:00Z": {
"broadcasts": 105,
"followups": 137,
"unique": 130,
"total": 242
},
"2021-11-03T00:00:00Z": {
"broadcasts": 636,
"followups": 185,
"unique": 622,
"total": 821
},
"2021-11-04T00:00:00Z": {
"broadcasts": 480,
"followups": 109,
"unique": 426,
"total": 589
}
}
#+end_src
*** Sales over time (events)
#+attr_confluence: :as-table t
- Name :: sale-events
- Parameters ::
- Date range (default: last 30 days)
- Currency (default: USD)
- Report API controller endpoints ::
- sales_tracked_events
#+caption: Sample response
#+begin_src json
[
{
"time": "2021-11-02 09:37:36-04",
"type": "followup",
"currency": "USD",
"revenue": "19.00",
"note": "",
"description": "Upgraded to Pro",
"source_url": "https://www.aweber.com/users/#upgraded",
"email": "team@harmoniamedia.com"
},
{
"time": "2021-11-02 09:37:37-04",
"type": "followup",
"currency": "USD",
"revenue": "19.00",
"note": "",
"description": "Upgraded to Pro",
"source_url": "https://www.aweber.com/users/#upgraded",
"email": "team@harmoniamedia.com"
},
{
"time": "2021-11-02 12:01:17-04",
"type": "followup",
"currency": "USD",
"revenue": "19.00",
"note": "",
"description": "Upgraded to Pro",
"source_url": "https://www.aweber.com/users/#upgraded",
"email": "giuliagiardino12@gmail.com"
},
{
"time": "2021-11-02 12:01:19-04",
"type": "followup",
"currency": "USD",
"revenue": "19.00",
"note": "",
"description": "Upgraded to Pro",
"source_url": "https://www.aweber.com/users/#upgraded",
"email": "giuliagiardino12@gmail.com"
},
{
"time": "2021-11-04 05:21:35-04",
"type": "broadcast",
"currency": "USD",
"revenue": "19.00",
"note": "",
"description": "Upgraded to Pro",
"source_url": "https://www.aweber.com/users/#upgraded",
"email": "jeremy@jeremy-quick.com"
},
{
"time": "2021-11-04 05:21:36-04",
"type": "broadcast",
"currency": "USD",
"revenue": "19.00",
"note": "",
"description": "Upgraded to Pro",
"source_url": "https://www.aweber.com/users/#upgraded",
"email": "jeremy@jeremy-quick.com"
}
]
#+end_src
*** Sales over time (summary)
#+attr_confluence: :as-table t
- Name :: daily-sales
- Parameters ::
- Date range (default: last 60 days)
- Currency (default: USD)
#+caption: Sample response
#+begin_src json
{
"2021-11-02T00:00:00Z": {
"broadcast": 0,
"followup": 76,
"pageview": 76,
"ecommerce": 0,
"total": 76
},
"2021-11-03T00:00:00Z": {
"broadcast": 0,
"followup": 0,
"pageview": 0,
"ecommerce": 0,
"total": 0
},
"2021-11-04T00:00:00Z": {
"broadcast": 38,
"followup": 0,
"pageview": 38,
"ecommerce": 0,
"total": 38
}
}
#+end_src
*** Lifetime Sale Totals
#+attr_confluence: :as-table t
- Name :: sale-totals
- Parameters ::
- Currency (default: USD)
- Report API controller endpoints ::
- sales_tracked_total
#+caption: Sample response
#+begin_src json
{
"count": 94924,
"pagehits": 1820122.79,
"revenue": 2181498.96
}
#+end_src
*** Sale Currencies
#+attr_confluence: :as-table t
- Name :: sale-currencies
- Parameters ::
- Date range (default: last 60 days)
- Report API controller endpoints ::
- sales_tracked_currencies
#+caption: Sample response
#+begin_src json
["USD", "CAD"]
#+end_src
*** Pending Broadcasts
#+attr_confluence: :as-table t
- Name :: pending-broadcasts
- Parameters ::
- List (default: all lists)
- Report API controller endpoints ::
- broadcasts_pending
#+caption: Sample response
#+begin_src json
[
{
"unit_id": "3854",
"send_date": "01/01/25 12:00am",
"broadcast_id": "1800243",
"subject": "Scheduled Broadcast Test",
"status": "Queue",
"percent_done": "0",
"mesg_encoding": "utf-8",
"campaign_id": "1802217",
"unit": "awlist3854",
"orig_send_date": "2025-01-01 00:00:00-05",
"friendly_list_name": "Fluff Cafe",
"id": "1802217",
"list_id": "3854",
"account_id": "778",
"campaign_type_id": "b",
"uses_block_editor": "t"
}
]
#+end_src
*** Completed Broadcasts
#+attr_confluence: :as-table t
- Name :: completed-broadcasts
- Parameters ::
- List (default: all lists)
- Report API controller endpoints ::
- broadcasts_completed
- broadcasts_completed_all
#+caption: Sample response
#+begin_src json
[
{
"a_id": "778",
"broadcast_id": "1258855",
"for_sent_date": "08/14/20 03:41 PM",
"track_click_rate": "1",
"unit_id": "3854",
"assassin_pts": "0",
"mesg_type": "HTML",
"mesg_encoding": "utf-8",
"sent_date": "2020-08-14 15:41:05.24086-04",
"num_emailed": "1",
"num_undeliv": 0,
"num_opened": 0,
"num_attachments": 0,
"num_complaints": 0,
"subject": "Testing a bad segment",
"created_date": "2020-08-14 15:38:57.59072-04",
"status": "Sent Composer",
"lead_view_id": "42564",
"unit": "awlist3854",
"pct_opened": "0",
"show_opens_warning": true,
"pct_undeliv": "0",
"total_clicks": 0,
"clicks_analytics_type": "premium",
"pct_click": "0",
"pct_complaints": "0",
"segment_id": "42564",
"extra_lists": "",
"excluded_lists": "",
"extra_lists_count": 0,
"excluded_lists_count": 0,
"friendly_list_name": "Fluff Cafe"
},
{
"a_id": "778",
"broadcast_id": "1243650",
"for_sent_date": "08/05/20 06:33 PM",
"track_click_rate": "1",
"unit_id": "3854",
"assassin_pts": "0",
"mesg_type": "Text/HTML",
"mesg_encoding": "utf-8",
"sent_date": "2020-08-05 18:33:48.638875-04",
"num_emailed": "27",
"num_undeliv": 0,
"num_opened": 0,
"num_attachments": 0,
"num_complaints": 0,
"subject": "Buggssss 🐛🐛🐛🐛🐛🐛🐛",
"created_date": "2020-08-05 18:31:10.567418-04",
"status": "Sent Composer",
"lead_view_id": "8",
"unit": "awlist3854",
"pct_opened": "0",
"show_opens_warning": true,
"pct_undeliv": "0",
"total_clicks": 0,
"clicks_analytics_type": "premium",
"pct_click": "0",
"pct_complaints": "0",
"segment_id": "8",
"extra_lists": "",
"excluded_lists": "",
"extra_lists_count": 0,
"excluded_lists_count": 0,
"friendly_list_name": "Fluff Cafe"
}
]
#+end_src
*** Cities, States, and Countries
#+attr_confluence: :as-table t
- Name :: subscribers-by-location
- Parameters ::
- List (default: all lists)
- Report API controller endpoints ::
- city_state_countries
#+caption: Sample response
#+begin_src json
[
{
"country": null,
"state": null,
"city": null,
"pending": 0,
"unsubscribed": 457262,
"subscribed": 276245,
"total": 733507
},
{
"country": "US",
"state": "PA",
"city": "Newtown",
"pending": 0,
"unsubscribed": 279304,
"subscribed": 3236,
"total": 282540
},
{
"country": "US",
"state": null,
"city": null,
"pending": 0,
"unsubscribed": 109074,
"subscribed": 2456,
"total": 111530
},
{
"country": "US",
"state": "PA",
"city": "Philadelphia",
"pending": 0,
"unsubscribed": 48485,
"subscribed": 657,
"total": 49142
},
{
"country": "US",
"state": "PA",
"city": "Huntingdon Valley",
"pending": 0,
"unsubscribed": 4839,
"subscribed": 90,
"total": 4929
},
{
"country": "VN",
"state": "64",
"city": "Hanoi",
"pending": 0,
"unsubscribed": 204,
"subscribed": 2,
"total": 206
},
{
"country": "VN",
"state": "65",
"city": "Ho Chi Minh City",
"pending": 0,
"unsubscribed": 176,
"subscribed": 1,
"total": 177
}
]
#+end_src
*** Campaign Statistics
Add the campaign start endpoint into analytics-view including
- DynamoDB fixtures
- Dynamo dbhelpers (currently there is no dynamo connectivity in analytics-view)
- CampaignStarted Handler
https://gitlab.aweber.io/CP/Services/campaignstats/-/blob/master/campaignstats/handlers.py
- Grafana dashboard updated
- Alerts configured
- Confluence docs updated
*** Followups
#+attr_confluence: :as-table t
- Name :: followup-totals
- Parameters ::
- List (default: all lists)
- Report API controller endpoints ::
- followups
Uses the =freq_mesg= and =freq_mesg_stats= tables in AppDB combined with data
from the =messages= table in Analytics.
=editor_path= is hard-coded in the current report endpoint.
#+caption: Sample Response (Production AID 91)
#+begin_src json
[
{
"followup_message": "1",
"num_emailed": 83024,
"num_opened": 89950,
"open_percentage": 23.659423781075,
"clicks": "13198",
"message_id": "28613652",
"click_percentage": 0,
"unique_opens": "19643",
"unique_clicks": "1584",
"clicks_percentage": 23.659423781075,
"editor_path": "messages#/active"
},
{
"followup_message": "2",
"num_emailed": 73714,
"num_opened": 49979,
"open_percentage": 13.675828200884,
"clicks": "6169",
"message_id": "28613655",
"click_percentage": 0,
"unique_opens": "10081",
"unique_clicks": "649",
"clicks_percentage": 13.675828200884,
"editor_path": "messages#/active"
},
{
"followup_message": "3",
"num_emailed": 70880,
"num_opened": 38472,
"open_percentage": 10.272291196388,
"clicks": "4322",
"message_id": "28613659",
"click_percentage": 0,
"unique_opens": "7281",
"unique_clicks": "409",
"clicks_percentage": 10.272291196388,
"editor_path": "messages#/active"
},
{
"followup_message": "4",
"num_emailed": 69375,
"num_opened": 37496,
"open_percentage": 10.105945945946,
"clicks": "4419",
"message_id": "28613661",
"click_percentage": 0,
"unique_opens": "7011",
"unique_clicks": "447",
"clicks_percentage": 10.105945945946,
"editor_path": "messages#/active"
},
{
"followup_message": "5",
"num_emailed": 67478,
"num_opened": 30996,
"open_percentage": 8.2752897240582,
"clicks": "3236",
"message_id": "28613662",
"click_percentage": 0,
"unique_opens": "5584",
"unique_clicks": "257",
"clicks_percentage": 8.2752897240582,
"editor_path": "messages#/active"
},
{
"followup_message": "6",
"num_emailed": 65414,
"num_opened": 32483,
"open_percentage": 9.2334974164552,
"clicks": "3127",
"message_id": "28613664",
"click_percentage": 0,
"unique_opens": "6040",
"unique_clicks": "323",
"clicks_percentage": 9.2334974164552,
"editor_path": "messages#/active"
},
{
"followup_message": "7",
"num_emailed": 63871,
"num_opened": 28756,
"open_percentage": 8.1883797028385,
"clicks": "1964",
"message_id": "28613665",
"click_percentage": 0,
"unique_opens": "5230",
"unique_clicks": "166",
"clicks_percentage": 8.1883797028385,
"editor_path": "messages#/active"
}
]
#+end_src
*** New Subscribers Daily
#+attr_confluence: :as-table t
- Name :: daily-new-subscribers
- Parameters ::
- List (default: all lists)
- Report API controller endpoints ::
- new_subscribers_daily
Uses the =public.leads_stats_day= table in AppDB.
#+caption: Sample Response (Production AID 91)
#+begin_src json
{
"2022-01-09T00:00:00Z": {
"subscribed": 324,
"unsubscribed": 83
},
"2022-01-10T00:00:00Z": {
"subscribed": 417,
"unsubscribed": 80
},
"2022-01-11T00:00:00Z": {
"subscribed": 433,
"unsubscribed": 92
},
...
}
#+end_src
*** New Subscribers Weekly
#+attr_confluence: :as-table t
- Name :: weekly-new-subscribers
- Parameters ::
- List (default: all lists)
- Report API controller endpoints ::
- new_subscribers_weekly
Uses the =public.leads_stats_day= table in AppDB.
#+caption: Sample Response (Production AID 91)
#+begin_src json
{
"2021-03-26T00:00:00Z": {
"subscribed": 3182,
"unsubscribed": 1249
},
"2021-04-02T00:00:00Z": {
"subscribed": 3423,
"unsubscribed": 1497
},
"2021-04-09T00:00:00Z": {
"subscribed": 3052,
"unsubscribed": 1217
},
...
}
#+end_src
*** New Subscribers Monthly
#+attr_confluence: :as-table t
- Name :: monthly-new-subscribers
- Parameters ::
- List (default: all lists)
- Report API controller endpoints ::
- new_subscribers_monthly
Uses the =public.leads_stats_day= table in AppDB.
#+caption: Sample Response (Production AID 91)
#+begin_src json
{
"2021-03-01T00:00:00Z": {
"subscribed": 14972,
"unsubscribed": 4770
},
"2021-04-01T00:00:00Z": {
"subscribed": 14973,
"unsubscribed": 7181
},
"2021-05-01T00:00:00Z": {
"subscribed": 12652,
"unsubscribed": 5243
},
...
}
#+end_src
*** Subscriber Totals Daily
#+attr_confluence: :as-table t
- Name :: subscribers-by-location
- Parameters ::
- List (default: all lists)
- Report API controller endpoints ::
- subscriber_totals_daily
Uses the =public.leads_stats_day= table in AppDB.
#+caption: Sample Response (Production AID 91)
#+begin_src json
{
"2022-01-09T00:00:00Z": {
"subscribed": 289192,
"unsubscribed": 920812
},
"2022-01-10T00:00:00Z": {
"subscribed": 289609,
"unsubscribed": 920892
},
"2022-01-11T00:00:00Z": {
"subscribed": 290042,
"unsubscribed": 920984
},
...
}
#+end_src
*** Subscriber Totals Weekly
#+attr_confluence: :as-table t
- Name :: subscribers-by-location
- Parameters ::
- List (default: all lists)
- Report API controller endpoints ::
- subscriber_totals_weekly
Uses the =public.leads_stats_day= table in AppDB.
#+caption: Sample Response (Production AID 91)
#+begin_src json
{
"2021-02-08T00:00:00Z": {
"subscribed": 191745,
"unsubscribed": 887713
},
"2021-02-15T00:00:00Z": {
"subscribed": 193972,
"unsubscribed": 888733
},
"2021-02-22T00:00:00Z": {
"subscribed": 196200,
"unsubscribed": 889717
},
...
}
#+end_src
*** Subscriber Totals Monthly
#+attr_confluence: :as-table t
- Name :: subscribers-by-location
- Parameters ::
- List (default: all lists)
- Report API controller endpoints ::
- subscriber_totals_monthly
Uses the =public.leads_stats_day= table in AppDB.
#+caption: Sample Response (Production AID 91)
#+begin_src json
{
"2021-02-01T00:00:00Z": {
"subscribed": 196200,
"unsubscribed": 889717
},
"2021-03-01T00:00:00Z": {
"subscribed": 206569,
"unsubscribed": 894298
},
"2021-04-01T00:00:00Z": {
"subscribed": 215813,
"unsubscribed": 900020
},
...
}
#+end_src
** Migrate reports to the Analytics View Service
*** READY Use the new analytics view endpoints for the opens over time reports
Update the opens over time report for [[https://www.aweber.com/users/report/opens_all][all lists]] and the [[https://www.aweber.com/users/report/opens][current list]] to use the
new [[http://analytics-view.service.production.consul/static/index.html#tag/Reports/paths/~1reports~1opens/get][daily opens endpoint]] in the Analytics View service.
*** READY Use the new analytics view endpoints for the clicks over time reports
Update the clicks over time report for [[https://www.aweber.com/users/report/clicks_all][all lists]] and the [[https://www.aweber.com/users/report/clicks][current list]] to use the
new [[http://analytics-view.service.production.consul/static/index.html#tag/Reports/paths/~1reports~1clicks/get][daily clicks endpoint]] in the Analytics View service.
*** WAITING Use the new analytics view endpoints for the sales over time report
Update the [[https://www.aweber.com/users/report/sales_tracked_all][sales over time report]] to use the new [[http://analytics-view.service.production.consul/static/index.html#tag/Reports/paths/~1reports~1sales-by-day/get][daily sales endpoint]] for graph
summary data and the sale events endpoint for the sales table.
#+begin_notes
The sale events endpoint present, but is not yet documented!
#+end_notes
*** DONE New subscribers
Update the new subscribers report to use the new [[http://analytics-view.service.production.consul/static/index.html#tag/Reports/paths/~1reports~1new-subscribers-daily/get][daily]], [[http://analytics-view.service.production.consul/static/index.html#tag/Reports/paths/~1reports~1new-subscribers-weekly/get][weekly]], and [[http://analytics-view.service.production.consul/static/index.html#tag/Reports/paths/~1reports~1new-subscribers-monthly/get][monthly]] new
subscriber endpoints in the Analytics View service.
*** DONE Subscriber totals
Update the subscriber totals report to use the new [[http://analytics-view.service.production.consul/static/index.html#tag/Reports/paths/~1reports~1total-subscribers-daily/get][daily]], [[http://analytics-view.service.production.consul/static/index.html#tag/Reports/paths/~1reports~1total-subscribers-weekly/get][weekly]], and [[http://analytics-view.service.production.consul/static/index.html#tag/Reports/paths/~1reports~1total-subscribers-monthly/get][monthly]]
total subscriber endpoints in the Analytics View service.
*** WAITING Use the new analytics view endpoints for the broadcast totals report
Update the [[https://www.aweber.com/users/report/broadcast_totals][broadcast totals report]] to use the new completed broadcasts endpoint
in the Analytics View service.
#+begin_notes
The completed broadcasts endpoint is [[https://jira.aweber.io/browse/CCPANEL-11788][not yet complete]].
#+end_notes
*** Use the new analytics view endpoints for the follow-up totals report
Update the [[https://www.aweber.com/users/report/followup_totals][followup totals report]] to use the new [[http://analytics-view.service.production.consul/static/index.html#tag/Reports/paths/~1reports~1followup-totals/get][followup totals]] endpoint in the
Analytics View service.
*** Use the new analytics view endpoints for the location totals report
Update the [[https://www.aweber.com/users/report/subscribers_by_location][location totals report]] to use the new [[http://analytics-view.service.production.consul/static/index.html#tag/Reports/paths/~1reports~1subscribers-by-location/get][subscribers by location]]
endpoint in the Analytics View service.

39
aweber/20210915133448-report_api_controller.org
View file

@ -1,39 +0,0 @@
:PROPERTIES:
:ID: 3ddc4e32-932f-4748-bfe9-7025d4d6b352
:ROAM_REFS: https://gitlab.aweber.io/CP/applications/sites/-/blob/master/aweber_app/controllers/report_api_controller.php
:END:
#+title: Report API Controller
Contains JSON report endpoints used by the [[id:fab0cf8f-7c54-4848-882b-dba5e087760d][Redesigned Reports]] and the [[id:0d24c57a-fd56-43b0-8209-497320bf79f7][Dashboard]].
These make use of the App and Analytics databases.
| Endpoint | Description | Report | Dashboard |
|--------------------------------------+-----------------+-----------------------+----------------------|
| opens_all | | | |
| opens_all_range | All lists | Opens over time | |
| opens | | | |
| opens_list_range | Current list | Opens over time | |
| clicks_all | | | |
| clicks_all_range | All lists | Clicks over time | |
| clicks | | | |
| clicks_list_range | Current list | Clicks over time | |
| sales_tracked_all | | | |
| sales_tracked_all_range | | | |
| sales_tracked_summary | | Sales over time | |
| sales_tracked_currencies | | Sales over time | Sales |
| sales_tracked_events | | Sales over time | |
| sales_tracked_total | Lifetime totals | | Sales |
| broadcasts_pending | | | Scheduled Broadcasts |
| broadcasts_completed | Current list | Broadcast totals | |
| broadcasts_completed_all | All lists | | Sent Broadcasts |
| broadcasts_completed_range | | | |
| city_state_countries | | Location totals | |
| followups | | Followup totals | |
| new_subscribers_daily | | New subscribers added | |
| new_subscribers_weekly | | New subscribers added | |
| new_subscribers_monthly | | New subscribers added | |
| subscriber_totals_daily | | Subscriber totals | |
| subscriber_totals_weekly | | Subscriber totals | |
| subscriber_totals_monthly | | Subscriber totals | |
| account_subscriber_totals | | | People |
| account_subscriber_totals_past_month | | | People |

9
aweber/20210916162334-ceeecs.org
View file

@ -1,9 +0,0 @@
:PROPERTIES:
:ID: 6ee0e3f3-1df2-4f8a-bc1b-659d8c01e2b5
:ROAM_ALIASES: CEEECS
:END:
#+title: Customer Empathy and Excellence via Customer Solutions
A program in which AWeber team members shadow a CS specialist as they field
customer support work, with the goal of finding areas in our platform where we
can improve the customer experience.

5
aweber/20210920134622-suspicious_submission_spam_service.org
View file

@ -1,5 +0,0 @@
:PROPERTIES:
:ID: c7322400-c6e6-4595-87e2-7db6e57b6a2b
:ROAM_ALIASES: S4
:END:
#+title: Suspicious Submission Spam Service

88
aweber/20210921090818-bulk_tagging_service.org
View file

@ -1,88 +0,0 @@
:PROPERTIES:
:ID: 321075e7-db53-4676-b785-7c77ed9d1150
:END:
#+title: Bulk Tagging Service
The Bulk Tagging service queues the application of tags to a set of subscribers
and tracks the overall progress of the operation. This allows customers to
select a large number of subscribers and apply tags to all of them in a single
operation, while also giving them visibility into how their tagging request is
progressing.
* Problems Solved
** Background processing of tagging operations
Tagging operations are assigned a bulk-tagging job with one task per subscriber
to add or remove the specified tags. These operations are performed
asynchronously by consumers such that the end-user need not wait for all
operations to be completed before moving on.
** Rate-limited application of tags
Consumers are configured to consume tasks no faster than a configured maximum
rate to control the load placed upon downstream services (e.g. rules engine,
etc.). The ideal rate is divided amongst the number of consumers available.
** Serial application of operations within an account
There is an expectation that operations a customer applies to subscribers will
be performed sequentially. To allow this while still allowing jobs for /other/
accounts to be worked upon simultaneously, jobs are divided into queues
deterministically using a consistent hash of their account IDs.
** Tracking overall job progress
The job itself may be in one of the following states, which is updated as tasks
are acted upon:
- Pending :: No tasks have yet been acted upon
- Processing :: Some, but not all, tasks have been acted upon
- Succeeded :: All tasks have completed successfully
- Failed :: All tasks have completed, but at least one task was not successful
** Tracking of individual task status
Any particular task, once acted upon, will be updated as having succeeded or
failed with a message explaining the issue.
** Jobs must survive queue failures
A guarantee of the service is such that if a job is successfully submitted to
the Bulk Tagging service, we will not lose it, and can take steps manually if
necessary to ensure its completion. To account for unexpected failures when
submitting tasks to a queue or consuming a task from the queue, the job request
is archived to S3. This archive contains sufficient information to requeue the
job for processing as it was requested without further input.
** Customer visibility into job progress
End users are able to, via the service API, fetch any and all jobs stored for
their account, as well as their associated tasks.
** Administrative visibility into job progress
Administrative (internal) users are able to, via the service API, fetch any and
all jobs stored for any or all accounts. This is used to populate dashboards for
insight into the progress of Bulk Tagging as a whole, and whether jobs from
different accounts are holding each other up.
Administrative users also have access to delete or requeue jobs.
** Prioritizing smaller jobs over larger jobs
Tasks for jobs affecting a number of subscribers lower than a defined threshold
are assigned higher priority, causing them to be processed /ahead/ of any other
ongoing jobs in their respective queues. This avoids leaving a customer waiting
long periods of time for quick operations, which they may not expect to be held
up by other jobs outside their control.
* Known Problems
** Jobs appearing "stuck"
Jobs may appear stuck (either failing to start processing, or failing to
complete processing).
*** Job has not yet begun processing
This is caused by one of two things. Either a job is stuck behind other jobs
(this or another account's job could be in front of it in the same queue), or
the tasks failed to be written successfully to the queue. In the former case,
the job will complete normally once the tasks ahead of it in the queue are
processed. In the latter case, the job will need to be requeued.
*** Job is stuck in an in-progress state
This is typically due to the overall job progress being out of sync with the
actual state of its consituent tasks ([[https://jira.aweber.io/browse/CCPANEL-8660][CCPANEL-8660]]).
The overall status of a job and the status of its individual tasks are stored in
separate Dynamo tables. Because Dynamo tables are independent, there is no way
to update an individual task and the job's overall status in a single, atomic
operation. This means they may (and do) at times get out of sync, if
infrequently. The indexes on the tables are not designed to make reconciliation
easy, nor does an automated reconciliation process exist.
This could be solved by adding indexes to allow rapid computation of the actual
state of all tasks in a job and creating a scheduled task to synchronize these
counts, or by changing the underlying storage to a relational database which
would make it easier to compute these states.
* Resources
- ACP :: https://confluence.aweber.io/display/AR/ACP+Bulk+Tagging
- Playbook :: https://confluence.aweber.io/display/AR/Bulk+Tagging+Service+Playbook

4
aweber/20210928132316-dashboard.org
View file

@ -1,4 +0,0 @@
:PROPERTIES:
:ID: 0d24c57a-fd56-43b0-8209-497320bf79f7
:END:
#+title: Dashboard

253
aweber/20210928155832-translating_the_search_dsl.org
View file

@ -1,253 +0,0 @@
:PROPERTIES:
:ID: 7b0f97f3-9037-4d05-9170-a478e97c8d1f
:END:
#+title: Modeling the new search DSL
Defining and translating the Search DSL for the [[id:11edd6c9-b976-403b-a419-b5542ddedaae][Subscriber Search Service]].
* Searches
** A search is a collection of groupings
#+begin_src python :noweb-ref search
@dataclasses.dataclass
class Search:
group: Group
# TODO: sorting : Sorting
#+end_src
#+begin_src yaml :noweb-ref search-yaml
Search:
type: object
properties:
group:
$ref: "#/components/schemas/Group"
#+end_src
** A grouping is a collection of conditions
#+begin_src python :noweb-ref group
class GroupType(enum.Enum):
AND = 1
# TODO: OR = 2
@dataclasses.dataclass
class Group:
group_type: GroupType
conditions: typing.List[Condition]
#+end_src
#+begin_src yaml :noweb-ref group-yaml
Group:
type: object
properties:
group_type:
enum:
- "AND"
conditions:
type: array
items:
$ref: "#/components/schemas/Condition"
#+end_src
** A condition is a filter applied to a field
#+begin_src python :noweb-ref condition
@dataclasses.dataclass
class Condition:
filter: Filter
match : str
#+end_src
#+begin_src yaml :noweb-ref condition-yaml
Condition:
type: object
properties:
filter:
$ref: "#/components/schemas/Filter"
match:
type: string
#+end_src
** A filter is a boolean expression applied to a field with an optional argument
#+begin_src python :noweb-ref filter
class InputType(enum.Enum):
Nothing = 1
String = 2
Date = 3
Tag = 4
TagSet = 5
Message = 6
@dataclasses.dataclass
class Filter:
operator: str
field: Field
input_type: InputType
#+end_src
** A field refers to a specific database field somewhere in our system
#+begin_src python :noweb-ref field
class Database(enum.Enum):
AppDB = 1
Analytics = 2
@dataclasses.dataclass
class FieldType:
name: str
@dataclasses.dataclass
class Field:
name: str
column: str
table: str
database: Database
#+end_src
** Available filters
*** Subscriber email is x
#+begin_src python :noweb-ref fields
email = Field(
name="email",
column="email",
table="subscribers",
database=Database.AppDB,
)
#+end_src
#+begin_src python :noweb-ref filters
email = Filter(field=fields.email, operator="is", input_type=InputType.String)
#+end_src
#+begin_src yaml :noweb-ref filters-spec
#+end_src
** Sample searches
*** Match subscriber email
#+begin_src python :noweb-ref searches
Search(
group=Group(
group_type=GroupType.AND,
conditions=[Condition(filter=filters.email, match="test@example.org")],
)
)
#+end_src
* SQL Generation
#+begin_src python :noweb-ref builder
def to_sql(search: Search) -> str:
tables: typing.Set[str] = {"subscribers"}
tables = tables | {
condition.filter.field.table for condition in search.group.conditions
}
def condition_to_sql(condition: Condition):
field = ".".join([condition.filter.field.table, condition.filter.field.column])
return f"{field} {condition.filter.operator} {condition.match}"
def group_to_sql(group: Group) -> str:
operator = "AND" if search.group.group_type == GroupType.AND else "OR"
clauses = f" {operator} ".join(
[condition_to_sql(condition) for condition in group.conditions]
)
return f"({clauses})"
where = group_to_sql(search.group)
return f"""SELECT * FROM {', '.join(tables)} WHERE {where}"""
#+end_src
* Decisions
** DONE Should the input type presented to the end-user be tied to the database field or the conditional operator?
Seems it should be the operator, as an "equals" operator would match a single
value, whereas an "in" operator would match against multiple. That said, it
could be /parameterized/ by the field's type (e.g. a tag has type =str=, its
"equals" operator has type =str=, its "in" operator has type =List[str]=).
--------------------------------------------------------------------------------
The input type will be defined as a property of the filter being applied.
** DONE Should the search service maintain a set of filters, or field types and operators?
- A filter is a combination of a field, an operator, and a type
- A field has a type, and operators could be defined that work with a type or set of types
For the former, the service would have total control over the search filters
available to the UI, and the UI would be coupled to the filter collection. With
the latter, the UI would have total control over which fields it's able to
search on and how, provided the fields are available.
--------------------------------------------------------------------------------
The search service will maintain a set of filters.
** TODO How should the values of each filter be represented in the request schema?
Should they be normalized to strings, or should we allow any type and validate
it when we attempt to build the search data model? If the latter, could the
available filters be baked into the OpenAPI schema?
** TODO How should the SQL be generated for each filter?
Should a SQL template or generation function be attached to each filter?
** TODO How do we want to define the joins for the various tables that may come into play?
We'll have to know, one way or another, how to narrow the records from the
joined table. Will they all be joined by the subscriber id, or will we need to
maintain a map?
* Code
** Python
#+begin_src python :noweb yes :noweb-ref final :exports code :results silent
import dataclasses
import enum
import typing
<<field>>
<<filter>>
<<condition>>
<<group>>
<<search>>
<<builder>>
class fields:
<<fields>>
class filters:
<<filters>>
searches = [
<<searches>>,
]
#+end_src
#+RESULTS:
#+caption: Mypy analysis
#+begin_src bash :noweb yes :results output :exports results
mypy <(cat <<'EOF'
<<final>>
EOF) 2>&1 || true
#+end_src
#+RESULTS:
: Success: no issues found in 1 source file
** OpenAPI
* Output
#+caption: Generated queries
#+begin_src python :noweb yes :exports results
<<final>>
return [[to_sql(search)] for search in searches]
#+end_src
#+RESULTS:
| SELECT * FROM subscribers WHERE (subscribers.email is test@example.org) |

166
aweber/20211001095858-tracking_progress_of_moving_pages_out_of_sites.org
View file

@ -1,166 +0,0 @@
:PROPERTIES:
:ID: 3cc8bd09-dd02-4950-8c89-a737f92809fd
:header-args:bash: :dir ~/sites-clean :exports both :eval no-export
:header-args:python: :exports results :eval no-export
:END:
#+title: Tracking progress of moving pages out of Sites
- [[https://jira.aweber.io/browse/CCPANEL-11608][Initiative parent ticket in JIRA]]
* Metrics
#+caption: Migrated controllers in the CP
#+begin_src python :var total=controller-count done=js-controller-count :results file
import matplotlib.pyplot as plt
total = float(total)
fig1, ax1 = plt.subplots()
ax1.pie(
[100 * (total - done) / total, 100 * done / total],
explode=[0.0, 0.1],
labels=["Legacy", "JavaScript"],
autopct="%1.1f%%",
shadow=True,
startangle=90,
)
ax1.axis("equal")
plt.title("Controller Types")
filename = "controllers-migrated-in-sites.png"
plt.savefig(filename)
return filename
#+end_src
#+RESULTS:
[[file:None]]
** Controllers in Sites
#+caption: Identifying the total number of public controllers in the CP
#+name: controller-count
#+begin_src bash
grep -l AppController aweber_app/controllers/*_controller.php | wc -l
#+end_src
#+RESULTS: controller-count
: 85
** Controllers loading JavaScript applications
#+caption: Identifying the number of controllers loading JS applications
#+name: js-controller-count
#+begin_src bash
egrep -l '\bappName\b' aweber_app/controllers/*_controller.php | wc -l
#+end_src
#+RESULTS: js-controller-count
: 25
* Progress over time
#+caption: Percentage of controllers migrated over time
#+begin_src python :var progress=progress :results file
from datetime import date
import matplotlib.pyplot as plt
progress = [[date.fromisoformat(row[0]), 100.0 * row[2] / row[1]] for row in progress]
x = [p[0] for p in progress]
y = [p[1] for p in progress]
plt.plot(x, y)
plt.fill_between(x, y, alpha=0.3)
plt.gcf().autofmt_xdate()
plt.title("% Controllers Migrated Over Time")
filename = "controllers-migrated-in-sites-over-time.png"
plt.savefig(filename)
return filename
#+end_src
#+RESULTS:
[[file:None]]
#+caption: Identifying the last tagged release each month
#+name: tags
#+begin_src bash :results silent :exports code
git log --tags \
--simplify-by-decoration \
--pretty="format:%as#%S" \
--after="2018-01-01" \
| sort -r -u -t- -k1,2 # Last tag of each month
#+end_src
#+caption: Gathering progress over time
#+name: progress
#+begin_src bash :noweb yes :cache yes :exports code
controller_total () {
<<controller-count>>
}
controller_done () {
<<js-controller-count>>
}
tags () {
<<tags>>
}
git checkout -q master
for taginfo in $(tags); do
date=$(echo $taginfo | cut -d '#' -f 1)
tag=$(echo $taginfo | cut -d '#' -f 2)
git checkout $tag
echo $date $(controller_total) $(controller_done)
done
git checkout -q master
#+end_src
#+RESULTS[b2f17a7946c030068f7ef85189b10bd0c5cb6a0a]: progress
| 2021-09-30 | 85 | 24 |
| 2021-08-31 | 85 | 23 |
| 2021-07-28 | 85 | 23 |
| 2021-06-28 | 85 | 23 |
| 2021-05-27 | 85 | 23 |
| 2021-04-28 | 85 | 23 |
| 2021-03-04 | 84 | 22 |
| 2021-02-25 | 84 | 22 |
| 2021-01-28 | 83 | 16 |
| 2020-12-29 | 83 | 16 |
| 2020-11-20 | 83 | 16 |
| 2020-10-29 | 83 | 16 |
| 2020-09-30 | 83 | 16 |
| 2020-08-27 | 83 | 16 |
| 2020-07-31 | 83 | 16 |
| 2020-06-30 | 82 | 15 |
| 2020-05-29 | 81 | 15 |
| 2020-04-30 | 81 | 15 |
| 2020-03-31 | 81 | 15 |
| 2020-02-28 | 81 | 15 |
| 2020-01-30 | 82 | 15 |
| 2019-12-18 | 82 | 15 |
| 2019-11-25 | 82 | 15 |
| 2019-10-31 | 81 | 14 |
| 2019-09-30 | 81 | 14 |
| 2019-08-27 | 81 | 14 |
| 2019-07-31 | 81 | 14 |
| 2019-06-27 | 81 | 14 |
| 2019-05-31 | 81 | 14 |
| 2019-04-26 | 80 | 13 |
| 2019-03-29 | 79 | 13 |
| 2019-02-28 | 78 | 12 |
| 2019-01-30 | 78 | 12 |
| 2018-12-27 | 77 | 10 |
| 2018-11-29 | 76 | 10 |
| 2018-10-31 | 75 | 9 |
| 2018-09-28 | 75 | 9 |
| 2018-08-31 | 74 | 8 |
| 2018-07-26 | 74 | 8 |
| 2018-06-29 | 73 | 6 |
| 2018-05-31 | 73 | 6 |
| 2018-04-30 | 73 | 6 |
| 2018-03-29 | 73 | 6 |
| 2018-02-28 | 73 | 6 |
| 2018-01-24 | 73 | 6 |
* Comparing with the CP URL Inventory
https://docs.google.com/spreadsheets/d/1bRKL1zRe_SjePD1QKSHrbgiW9paMjLyCIX_2wu2FXwE/edit#gid=1209260269

6
aweber/20211005091910-tech_initiative_workshop.org
View file

@ -1,6 +0,0 @@
:PROPERTIES:
:ID: 071f551f-56d9-425c-bfde-af80cd7c26f7
:END:
#+title: Tech Initiative Workshop
A bi-weekly meeting to swarm on various [[id:db322997-ff5e-416a-8dc8-f29e6a4928c8][Technical Initiative]] issues.

4
aweber/20211005153541-tagging_service.org
View file

@ -1,4 +0,0 @@
:PROPERTIES:
:ID: 7e503917-646f-4275-aab9-3a125b99cbfd
:END:
#+title: Tagging Service

4
aweber/20211005153555-tag_publisher_consumer.org
View file

@ -1,4 +0,0 @@
:PROPERTIES:
:ID: 131dde93-60d3-4813-a16e-7568c79ba6c4
:END:
#+title: Tag Publisher Consumer

5
aweber/20211011102943-creating_remarkable_experiences_via_application_support_engineering.org
View file

@ -1,5 +0,0 @@
:PROPERTIES:
:ID: bdea0611-e377-4378-a118-aef6d4a70bdf
:ROAM_ALIASES: CREASE
:END:
#+title: Creating Remarkable Experiences via Application Support Engineering

9
aweber/20211013143400-staging_account_information.org
View file

@ -1,9 +0,0 @@
:PROPERTIES:
:ID: 87f76e97-fbdc-49e4-9af8-588ceee85a5c
:END:
#+title: Staging account information
[[id:57ee2f00-9bcd-4e0f-8a77-ae1f2d4cda89][Control Panel]] staging account.
* Lists
- Fluff Cafe :: 9f1db623-fbc3-4112-a2f3-ab563e37e131 (3854)

21
aweber/20211013161201-validating_and_sanitizing_tags.org
View file

@ -1,21 +0,0 @@
:PROPERTIES:
:ID: 16298b74-f9a2-48ac-a84c-118af70d834c
:END:
#+title: Validating and sanitizing tags
* Sanitizing tag display
** DONE In the autocomplete of the tag input box
Fixes [[https://jira.aweber.io/browse/CCPANEL-11654][CCPANEL-11654]].
https://gitlab.aweber.io/BoFs/FE/libraries/tagbox/-/merge_requests/29
* Validating tags on creation
** TODO Lead controller in [[id:57ee2f00-9bcd-4e0f-8a77-ae1f2d4cda89][Control Panel]]
** TODO [[id:03e00c18-99c0-477c-b7fb-95ddc538755e][Addlead]]
** TODO [[id:cd4a8a83-be53-4ec9-8cca-b6f34b59ba35][Subscriber Proxy]]
** TODO [[id:321075e7-db53-4676-b785-7c77ed9d1150][Bulk Tagging]]
** TODO [[id:7e503917-646f-4275-aab9-3a125b99cbfd][Tagging]]
*** TODO Add inbound validation
*** TODO Remove outbound sanitization

10
aweber/20211013162152-addlead.org
View file

@ -1,10 +0,0 @@
:PROPERTIES:
:ID: 03e00c18-99c0-477c-b7fb-95ddc538755e
:END:
#+title: Addlead
A business-critical nightmare in Perl.
https://confluence.aweber.io/display/~erict/Addlead+Notes
* Create ACP to rewrite Addlead as a Python service
* Break down tickets

4
aweber/20211013162215-subscriber_proxy_service.org
View file

@ -1,4 +0,0 @@
:PROPERTIES:
:ID: cd4a8a83-be53-4ec9-8cca-b6f34b59ba35
:END:
#+title: Subscriber Proxy Service

6
aweber/20211019140007-cp_leads_and_product_sync_up.org
View file

@ -1,6 +0,0 @@
:PROPERTIES:
:ID: 0e5f578f-96a2-47d8-8dd9-d0d7f1e4fc35
:END:
#+title: CP Leads and Product Sync-Up
A weekly discussion on team priorities.

4
aweber/20211019142241-manager_one_on_one.org
View file

@ -1,4 +0,0 @@
:PROPERTIES:
:ID: 0a1e48ec-e132-4ec4-81a1-124711330b5a
:END:
#+title: Manager one-on-one

14
aweber/20211019163438-migrating_services_to_use_the_new_list_service.org
View file

@ -1,14 +0,0 @@
:PROPERTIES:
:ID: 619b6c78-7be9-4ee4-a0b7-9d1a4d7536e2
:END:
#+title: Migrating services to use the new List service
- Parent ticket :: [[https://jira.aweber.io/browse/CCPANEL-11745][CCPANEL-11745]]
As part of our effort to deprecate the old Core API services and iterate towards
modern, domain-oriented APIs, the Control Panel team is deprecating usage of
AWLists in favor of a new List API.
AWLists is planned to be sunsetted at the end of Q2 2022. Applications and
services dependent upon AWLists must be migrated to use the new List API by that
time.

7
aweber/20211019163502-deploy_coreapi_to_kubernetes.org
View file

@ -1,7 +0,0 @@
:PROPERTIES:
:ID: 4df15f2f-d2e1-40f4-8acd-dbfb78fe304f
:END:
#+title: Deploy CoreAPI to Kubernetes
- Merge the sub-projects into CAPI?
- API Suspenders replacement?

6
aweber/20211020091712-redash.org
View file

@ -1,6 +0,0 @@
:PROPERTIES:
:ID: 77ea54db-0c35-47ad-84b3-5c08ae5ac347
:END:
#+title: Redash
Tool for interrogating and graphing data points from the event stream.

11
aweber/20211022094829-validation_and_sanitization_guidelines.org
View file

@ -1,11 +0,0 @@
:PROPERTIES:
:ID: 38457ac3-ba81-4727-9a65-5de22059c175
:END:
#+title: Validation and Sanitization Guidelines
- [[id:2ba04972-f498-41c2-970e-a64c7f3f1c3b][Data sanitization]]
- [[id:9914d09e-99fe-46a6-95be-676c5b78ed90][Input validation]]
- All content being displayed to a web browser MUST be appropriately sanitized
(unsafe characters should be escaped using their respective html entities)
-

24
aweber/20211026141640-decommissioning_sites.org
View file

@ -1,24 +0,0 @@
:PROPERTIES:
:ID: f633f967-11d2-432c-b5ff-ad842c88a51c
:END:
#+title: Decommissioning Sites
The goal of this project is the elimination of the [[https://gitlab.aweber.io/CP/applications/sites][sites repository]], which is
built upon Perl and PHP code that is well past it's end-of-life date, and a
modernization of our public-facing application stack.
The project will engage multiple teams to coordinate the following three
efforts:
* [[id:193f7c04-0a03-4870-90c8-2b5e3c4c92ce][Moving applications out of Sites]]
- Individual pages will be replaced with React applications using public APIs
- Static content will be moved to CDN hosting
- Independent applications will be broken out into separate services
* Replacing the top-level application
The CakePHP Control Panel application will be replaced with a modern alternative
handling routing and React application loading.
* [[id:0328a202-376d-4e97-b0e3-031eaad2a557][Overhauling logins and session management]]
The session-based login mechanism of the legacy Control Panel application is to
be replaced with OAuth, which is used for public API requests.

82
aweber/20211103100927-updating_projects_using_tagbox.org
View file

@ -1,82 +0,0 @@
:PROPERTIES:
:ID: d399955b-894c-4d44-82ed-892009b4aa4f
:END:
#+title: Updating projects using Tagbox
#+OPTIONS: prop:("JIRA_ID")
#+todo: TODO(t) INVESTIGATE(i) TESTING(s) AWAITING-RELEASE(a) | DONE(d) NO-ACTION(n)
The Tagbox component has been updated with a fix addressing an XSS security
vulnerability in its tag label auto-completion which needs to be propogated out
to projects using the widget ([[https://jira.aweber.io/browse/CCPANEL-11654][CCPANEL-11654]]).
- New version of Tagbox is 5.0.4 (AWeberUI 8.7)
- Aweber UI is updated, includes other breaking changes
- Adobe spectrum resource picker breaks stuff (present in some versions)
- Sites is already updated
* Notes on upgrading AWeberUI
Target versions:
- =@aw-int/components= :: =^9.0.5=
- =@aw-int/icons= :: =^8.3.0=
- =@aw-int/aweber-webapp-scripts= :: =^10.8.0=
* Projects
Parent ticket: [[https://jira.aweber.io/browse/CCPANEL-11762][CCPANEL-11762]]
** TESTING Campaign Builder (Direct)
:PROPERTIES:
:JIRA_ID: CC-7550
:END:
- Tagbox :: 5.0.3
** NO-ACTION GoToWebinar Client (Direct)
:PROPERTIES:
:JIRA_ID: INT-5508
:END:
- Tagbox :: 3.0.0
This project does not use the Tagbox component.
** TESTING Subscriber Import (Direct)
:PROPERTIES:
:JIRA_ID: CCPANEL-11768
:END:
- Tagbox :: 5.0.3
- AWeberUI :: 2
Uses ramda, etc.
** TESTING List Automation Client (AWeberUI)
:PROPERTIES:
:JIRA_ID: CCPANEL-11769
:END:
- AWeberUI :: 8.3
** NO-ACTION User Management Client (AWeberUI)
This project does not use the Tagbox component.
** TESTING Landing Page Editor (AWeberUI)
:PROPERTIES:
:JIRA_ID: CC-7551
:END:
** TESTING Draft Bin (Direct)
:PROPERTIES:
:JIRA_ID: CC-7552
:END:
- Tagbox :: 5.0.0
** TESTING Subscribers Client (AWeber UI)
:PROPERTIES:
:JIRA_ID: CCPANEL-11770
:END:
- AWeberUI :: 8.6.1
** TESTING Webform Generator (Standalone file)
:PROPERTIES:
:JIRA_ID: CC-7553
:END:
- Tagbox :: 5.0.3
** NO-ACTION Integrations Platform Client (AWeberUI)
:PROPERTIES:
:JIRA_ID: INT-5509
:END:
- AWeberUI :: 5.14.0
This is a prototype project that has not been deployed anywhere.
** NO-ACTION Tag Management Client (Direct)
:PROPERTIES:
:JIRA_ID: CCPANEL-11771
:END:
- Tagbox :: 5.0.3
This project does not use the Tagbox component.

4
aweber/20211108150155-bulk_job_service.org
View file

@ -1,4 +0,0 @@
:PROPERTIES:
:ID: 1463cf0a-e2b2-490c-b1b3-40249b483ca8
:END:
#+title: Bulk Job Service

4
aweber/20211202143853-appdb.org
View file

@ -1,4 +0,0 @@
:PROPERTIES:
:ID: dd113e53-6144-4cb2-a4aa-da3dc2e3e6ea
:END:
#+title: AppDB

517
aweber/20211208140356-legacy_search_filters.org
View file

@ -1,517 +0,0 @@
:PROPERTIES:
:ID: 2f42c362-ae96-41d5-988b-329f8c162e45
:END:
#+title: Legacy Search Filters
Search filters provided by [[id:d9cb2b55-3b0e-4ab3-8369-f71ebc3cd882][Sites Subscriber Search]] and supported in the new
[[id:11edd6c9-b976-403b-a419-b5542ddedaae][Subscriber Search Service]].
#+name: search-boxes
#+header: :engine postgresql
#+header: :dbhost 127.0.0.1 :dbport 56893
#+header: :dbuser postgres
#+begin_src sql :cache yes :eval no-export
select box.id, input.description, input.column, term.description
from search_boxes as box
join search_inputs as input on (box.search_input_id = input.id)
join search_terms as term on (box.search_term_id = term.id)
#+end_src
#+RESULTS[50d2c2ce018c9fd37f4093af3646725bcd98131f]: search-boxes
| id | description | column | description |
|-----+-----------------------+----------------------+-----------------------------|
| 1 | Email | email | is |
| 2 | Email | email | is not |
| 3 | Email | email | contains |
| 4 | Email | email | does not contain |
| 5 | Email | email | starts with |
| 6 | Email | email | ends with |
| 7 | Email | email | does not start with |
| 8 | Email | email | does not end with |
| 9 | Name | name | is |
| 10 | Name | name | is not |
| 11 | Name | name | contains |
| 12 | Name | name | does not contain |
| 13 | Name | name | starts with |
| 14 | Name | name | ends with |
| 15 | Name | name | does not start with |
| 16 | Name | name | does not end with |
| 17 | Ad Tracking | note | is |
| 18 | Ad Tracking | note | is not |
| 19 | Ad Tracking | note | contains |
| 20 | Ad Tracking | note | does not contain |
| 21 | Ad Tracking | note | starts with |
| 22 | Ad Tracking | note | ends with |
| 23 | Ad Tracking | note | does not start with |
| 24 | Ad Tracking | note | does not end with |
| 25 | Additional Notes | name2 | is |
| 26 | Additional Notes | name2 | is not |
| 27 | Additional Notes | name2 | contains |
| 28 | Additional Notes | name2 | does not contain |
| 29 | Additional Notes | name2 | starts with |
| 30 | Additional Notes | name2 | ends with |
| 31 | Additional Notes | name2 | does not start with |
| 32 | Additional Notes | name2 | does not end with |
| 33 | Last Message # | message | is |
| 34 | Last Message # | message | is not |
| 35 | Last Message # | message | contains |
| 36 | Last Message # | message | does not contain |
| 37 | Last Message # | message | starts with |
| 38 | Last Message # | message | ends with |
| 39 | Last Message # | message | does not start with |
| 40 | Last Message # | message | does not end with |
| 41 | Last Message # | message | is less than |
| 42 | Last Message # | message | is less than or equal to |
| 43 | Last Message # | message | is greater than |
| 44 | Last Message # | message | is greater than or equal to |
| 45 | Stop Status | stop_status | is |
| 46 | Stop Status | stop_status | is not |
| 47 | Stop Method | stop_method | is |
| 48 | Stop Method | stop_method | is not |
| 49 | Confirmed? | verified | is |
| 50 | Confirmed? | verified | is not |
| 51 | Add Method | add_method | is |
| 52 | Add Method | add_method | is not |
| 53 | Add URL | add_url | is |
| 54 | Add URL | add_url | is not |
| 55 | Add URL | add_url | contains |
| 56 | Add URL | add_url | does not contain |
| 57 | Add URL | add_url | starts with |
| 58 | Add URL | add_url | ends with |
| 59 | Add URL | add_url | does not start with |
| 60 | Add URL | add_url | does not end with |
| 86 | Date Stopped | stop_time | date is on or before |
| 61 | Add IP | add_notes | is |
| 62 | Add IP | add_notes | is not |
| 63 | Add IP | add_notes | contains |
| 64 | Add IP | add_notes | does not contain |
| 65 | Add IP | add_notes | starts with |
| 66 | Add IP | add_notes | ends with |
| 67 | Add IP | add_notes | does not start with |
| 68 | Add IP | add_notes | does not end with |
| 69 | Confirmation IP | verification_notes | is |
| 70 | Confirmation IP | verification_notes | is not |
| 71 | Confirmation IP | verification_notes | contains |
| 72 | Confirmation IP | verification_notes | does not contain |
| 73 | Confirmation IP | verification_notes | starts with |
| 74 | Confirmation IP | verification_notes | ends with |
| 75 | Confirmation IP | verification_notes | does not start with |
| 76 | Confirmation IP | verification_notes | does not end with |
| 87 | Date Stopped | stop_time | date is after |
| 88 | Date Stopped | stop_time | date is on or after |
| 89 | Date Confirmed | verification_time | date is before |
| 90 | Date Confirmed | verification_time | date is on or before |
| 91 | Date Confirmed | verification_time | date is after |
| 92 | Date Confirmed | verification_time | date is on or after |
| 127 | Latitude (from IP) | geog_lat | is |
| 128 | Latitude (from IP) | geog_lat | is not |
| 129 | Latitude (from IP) | geog_lat | contains |
| 130 | Latitude (from IP) | geog_lat | does not contain |
| 131 | Latitude (from IP) | geog_lat | starts with |
| 132 | Latitude (from IP) | geog_lat | ends with |
| 133 | Latitude (from IP) | geog_lat | does not start with |
| 134 | Latitude (from IP) | geog_lat | does not end with |
| 135 | Latitude (from IP) | geog_lat | is less than |
| 136 | Latitude (from IP) | geog_lat | is less than or equal to |
| 93 | Country (from IP) | geog_country | is |
| 94 | Country (from IP) | geog_country | is not |
| 95 | Country (from IP) | geog_country | contains |
| 96 | Country (from IP) | geog_country | does not contain |
| 97 | Country (from IP) | geog_country | starts with |
| 98 | Country (from IP) | geog_country | ends with |
| 99 | Country (from IP) | geog_country | does not start with |
| 100 | Country (from IP) | geog_country | does not end with |
| 101 | Region (from IP) | geog_region | is |
| 102 | Region (from IP) | geog_region | is not |
| 103 | Region (from IP) | geog_region | contains |
| 104 | Region (from IP) | geog_region | does not contain |
| 105 | Region (from IP) | geog_region | starts with |
| 106 | Region (from IP) | geog_region | ends with |
| 107 | Region (from IP) | geog_region | does not start with |
| 108 | Region (from IP) | geog_region | does not end with |
| 137 | Latitude (from IP) | geog_lat | is greater than |
| 138 | Latitude (from IP) | geog_lat | is greater than or equal to |
| 111 | City (from IP) | geog_city | is |
| 112 | City (from IP) | geog_city | is not |
| 113 | City (from IP) | geog_city | contains |
| 114 | City (from IP) | geog_city | does not contain |
| 115 | City (from IP) | geog_city | starts with |
| 116 | City (from IP) | geog_city | ends with |
| 117 | City (from IP) | geog_city | does not start with |
| 118 | City (from IP) | geog_city | does not end with |
| 119 | Postal Code (from IP) | geog_postal | is |
| 120 | Postal Code (from IP) | geog_postal | is not |
| 121 | Postal Code (from IP) | geog_postal | contains |
| 122 | Postal Code (from IP) | geog_postal | does not contain |
| 123 | Postal Code (from IP) | geog_postal | starts with |
| 124 | Postal Code (from IP) | geog_postal | ends with |
| 125 | Postal Code (from IP) | geog_postal | does not start with |
| 126 | Postal Code (from IP) | geog_postal | does not end with |
| 77 | Date Added | timehit | date is before |
| 78 | Date Added | timehit | date is on or before |
| 79 | Date Added | timehit | date is after |
| 80 | Date Added | timehit | date is on or after |
| 81 | Date Last Follow Up | followuptime | date is before |
| 82 | Date Last Follow Up | followuptime | date is on or before |
| 83 | Date Last Follow Up | followuptime | date is after |
| 84 | Date Last Follow Up | followuptime | date is on or after |
| 85 | Date Stopped | stop_time | date is before |
| 139 | Longitude (from IP) | geog_lon | is |
| 144 | Longitude (from IP) | geog_lon | starts with |
| 140 | Longitude (from IP) | geog_lon | is not |
| 141 | Longitude (from IP) | geog_lon | contains |
| 142 | Longitude (from IP) | geog_lon | does not contain |
| 145 | Longitude (from IP) | geog_lon | ends with |
| 146 | Longitude (from IP) | geog_lon | does not start with |
| 147 | Longitude (from IP) | geog_lon | does not end with |
| 148 | Longitude (from IP) | geog_lon | is less than |
| 149 | Longitude (from IP) | geog_lon | is less than or equal to |
| 150 | Longitude (from IP) | geog_lon | is greater than |
| 151 | Longitude (from IP) | geog_lon | is greater than or equal to |
| 152 | Area Code (from IP) | geog_area_code | is |
| 153 | Area Code (from IP) | geog_area_code | is not |
| 154 | Area Code (from IP) | geog_area_code | contains |
| 155 | Area Code (from IP) | geog_area_code | does not contain |
| 156 | Area Code (from IP) | geog_area_code | starts with |
| 157 | Area Code (from IP) | geog_area_code | ends with |
| 158 | Area Code (from IP) | geog_area_code | does not start with |
| 159 | Area Code (from IP) | geog_area_code | does not end with |
| 160 | Area Code (from IP) | geog_area_code | is less than |
| 161 | Area Code (from IP) | geog_area_code | is less than or equal to |
| 162 | Area Code (from IP) | geog_area_code | is greater than |
| 163 | Area Code (from IP) | geog_area_code | is greater than or equal to |
| 164 | DMA Code (from IP) | geog_dma_code | is |
| 165 | DMA Code (from IP) | geog_dma_code | is not |
| 166 | DMA Code (from IP) | geog_dma_code | contains |
| 167 | DMA Code (from IP) | geog_dma_code | does not contain |
| 168 | DMA Code (from IP) | geog_dma_code | starts with |
| 169 | DMA Code (from IP) | geog_dma_code | ends with |
| 170 | DMA Code (from IP) | geog_dma_code | does not start with |
| 171 | DMA Code (from IP) | geog_dma_code | does not end with |
| 172 | DMA Code (from IP) | geog_dma_code | is less than |
| 173 | DMA Code (from IP) | geog_dma_code | is less than or equal to |
| 174 | DMA Code (from IP) | geog_dma_code | is greater than |
| 175 | DMA Code (from IP) | geog_dma_code | is greater than or equal to |
| 176 | Message not opened | app_message_id | is |
| 177 | Message opened | app_message_id | is |
| 226 | datum1 | datum1 | is |
| 227 | datum1 | datum1 | is not |
| 228 | datum1 | datum1 | contains |
| 229 | datum1 | datum1 | does not contain |
| 230 | datum1 | datum1 | starts with |
| 231 | datum1 | datum1 | ends with |
| 232 | datum1 | datum1 | does not start with |
| 233 | datum1 | datum1 | does not end with |
| 234 | datum2 | datum2 | is |
| 235 | datum2 | datum2 | is not |
| 236 | datum2 | datum2 | contains |
| 237 | datum2 | datum2 | does not contain |
| 238 | datum2 | datum2 | starts with |
| 239 | datum2 | datum2 | ends with |
| 240 | datum2 | datum2 | does not start with |
| 241 | datum2 | datum2 | does not end with |
| 242 | datum3 | datum3 | is |
| 243 | datum3 | datum3 | is not |
| 244 | datum3 | datum3 | contains |
| 245 | datum3 | datum3 | does not contain |
| 246 | datum3 | datum3 | starts with |
| 247 | datum3 | datum3 | ends with |
| 248 | datum3 | datum3 | does not start with |
| 249 | datum3 | datum3 | does not end with |
| 250 | datum4 | datum4 | is |
| 251 | datum4 | datum4 | is not |
| 252 | datum4 | datum4 | contains |
| 253 | datum4 | datum4 | does not contain |
| 254 | datum4 | datum4 | starts with |
| 255 | datum4 | datum4 | ends with |
| 256 | datum4 | datum4 | does not start with |
| 257 | datum4 | datum4 | does not end with |
| 258 | datum5 | datum5 | is |
| 259 | datum5 | datum5 | is not |
| 260 | datum5 | datum5 | contains |
| 261 | datum5 | datum5 | does not contain |
| 262 | datum5 | datum5 | starts with |
| 263 | datum5 | datum5 | ends with |
| 264 | datum5 | datum5 | does not start with |
| 265 | datum5 | datum5 | does not end with |
| 266 | datum6 | datum6 | is |
| 267 | datum6 | datum6 | is not |
| 268 | datum6 | datum6 | contains |
| 269 | datum6 | datum6 | does not contain |
| 270 | datum6 | datum6 | starts with |
| 271 | datum6 | datum6 | ends with |
| 272 | datum6 | datum6 | does not start with |
| 273 | datum6 | datum6 | does not end with |
| 274 | datum7 | datum7 | is |
| 275 | datum7 | datum7 | is not |
| 276 | datum7 | datum7 | contains |
| 277 | datum7 | datum7 | does not contain |
| 278 | datum7 | datum7 | starts with |
| 279 | datum7 | datum7 | ends with |
| 280 | datum7 | datum7 | does not start with |
| 281 | datum7 | datum7 | does not end with |
| 282 | datum8 | datum8 | is |
| 283 | datum8 | datum8 | is not |
| 284 | datum8 | datum8 | contains |
| 285 | datum8 | datum8 | does not contain |
| 286 | datum8 | datum8 | starts with |
| 287 | datum8 | datum8 | ends with |
| 288 | datum8 | datum8 | does not start with |
| 289 | datum8 | datum8 | does not end with |
| 290 | datum9 | datum9 | is |
| 291 | datum9 | datum9 | is not |
| 292 | datum9 | datum9 | contains |
| 293 | datum9 | datum9 | does not contain |
| 294 | datum9 | datum9 | starts with |
| 295 | datum9 | datum9 | ends with |
| 296 | datum9 | datum9 | does not start with |
| 297 | datum9 | datum9 | does not end with |
| 298 | datum10 | datum10 | is |
| 299 | datum10 | datum10 | is not |
| 300 | datum10 | datum10 | contains |
| 301 | datum10 | datum10 | does not contain |
| 302 | datum10 | datum10 | starts with |
| 303 | datum10 | datum10 | ends with |
| 304 | datum10 | datum10 | does not start with |
| 305 | datum10 | datum10 | does not end with |
| 306 | datum11 | datum11 | is |
| 307 | datum11 | datum11 | is not |
| 308 | datum11 | datum11 | contains |
| 309 | datum11 | datum11 | does not contain |
| 310 | datum11 | datum11 | starts with |
| 311 | datum11 | datum11 | ends with |
| 312 | datum11 | datum11 | does not start with |
| 313 | datum11 | datum11 | does not end with |
| 314 | datum12 | datum12 | is |
| 315 | datum12 | datum12 | is not |
| 316 | datum12 | datum12 | contains |
| 317 | datum12 | datum12 | does not contain |
| 318 | datum12 | datum12 | starts with |
| 319 | datum12 | datum12 | ends with |
| 320 | datum12 | datum12 | does not start with |
| 321 | datum12 | datum12 | does not end with |
| 322 | datum13 | datum13 | is |
| 323 | datum13 | datum13 | is not |
| 324 | datum13 | datum13 | contains |
| 325 | datum13 | datum13 | does not contain |
| 326 | datum13 | datum13 | starts with |
| 327 | datum13 | datum13 | ends with |
| 328 | datum13 | datum13 | does not start with |
| 329 | datum13 | datum13 | does not end with |
| 330 | datum14 | datum14 | is |
| 331 | datum14 | datum14 | is not |
| 332 | datum14 | datum14 | contains |
| 333 | datum14 | datum14 | does not contain |
| 334 | datum14 | datum14 | starts with |
| 335 | datum14 | datum14 | ends with |
| 336 | datum14 | datum14 | does not start with |
| 337 | datum14 | datum14 | does not end with |
| 338 | datum15 | datum15 | is |
| 339 | datum15 | datum15 | is not |
| 340 | datum15 | datum15 | contains |
| 341 | datum15 | datum15 | does not contain |
| 342 | datum15 | datum15 | starts with |
| 343 | datum15 | datum15 | ends with |
| 344 | datum15 | datum15 | does not start with |
| 345 | datum15 | datum15 | does not end with |
| 346 | datum16 | datum16 | is |
| 347 | datum16 | datum16 | is not |
| 348 | datum16 | datum16 | contains |
| 349 | datum16 | datum16 | does not contain |
| 350 | datum16 | datum16 | starts with |
| 351 | datum16 | datum16 | ends with |
| 352 | datum16 | datum16 | does not start with |
| 353 | datum16 | datum16 | does not end with |
| 354 | datum17 | datum17 | is |
| 355 | datum17 | datum17 | is not |
| 356 | datum17 | datum17 | contains |
| 357 | datum17 | datum17 | does not contain |
| 358 | datum17 | datum17 | starts with |
| 359 | datum17 | datum17 | ends with |
| 360 | datum17 | datum17 | does not start with |
| 361 | datum17 | datum17 | does not end with |
| 362 | datum18 | datum18 | is |
| 363 | datum18 | datum18 | is not |
| 364 | datum18 | datum18 | contains |
| 365 | datum18 | datum18 | does not contain |
| 366 | datum18 | datum18 | starts with |
| 367 | datum18 | datum18 | ends with |
| 368 | datum18 | datum18 | does not start with |
| 369 | datum18 | datum18 | does not end with |
| 370 | datum19 | datum19 | is |
| 371 | datum19 | datum19 | is not |
| 372 | datum19 | datum19 | contains |
| 373 | datum19 | datum19 | does not contain |
| 374 | datum19 | datum19 | starts with |
| 375 | datum19 | datum19 | ends with |
| 376 | datum19 | datum19 | does not start with |
| 377 | datum19 | datum19 | does not end with |
| 534 | Link clicked | link_id | is |
| 535 | Link not clicked | link_id | is |
| 429 | datum1 | datum1 | is greater than |
| 433 | datum2 | datum2 | is greater than |
| 437 | datum3 | datum3 | is greater than |
| 441 | datum4 | datum4 | is greater than |
| 445 | datum5 | datum5 | is greater than |
| 449 | datum6 | datum6 | is greater than |
| 453 | datum7 | datum7 | is greater than |
| 457 | datum8 | datum8 | is greater than |
| 461 | datum9 | datum9 | is greater than |
| 465 | datum10 | datum10 | is greater than |
| 469 | datum11 | datum11 | is greater than |
| 473 | datum12 | datum12 | is greater than |
| 477 | datum13 | datum13 | is greater than |
| 430 | datum1 | datum1 | is greater than or equal to |
| 434 | datum2 | datum2 | is greater than or equal to |
| 438 | datum3 | datum3 | is greater than or equal to |
| 442 | datum4 | datum4 | is greater than or equal to |
| 446 | datum5 | datum5 | is greater than or equal to |
| 450 | datum6 | datum6 | is greater than or equal to |
| 454 | datum7 | datum7 | is greater than or equal to |
| 458 | datum8 | datum8 | is greater than or equal to |
| 462 | datum9 | datum9 | is greater than or equal to |
| 466 | datum10 | datum10 | is greater than or equal to |
| 470 | datum11 | datum11 | is greater than or equal to |
| 474 | datum12 | datum12 | is greater than or equal to |
| 478 | datum13 | datum13 | is greater than or equal to |
| 482 | datum14 | datum14 | is greater than or equal to |
| 486 | datum15 | datum15 | is greater than or equal to |
| 490 | datum16 | datum16 | is greater than or equal to |
| 494 | datum17 | datum17 | is greater than or equal to |
| 498 | datum18 | datum18 | is greater than or equal to |
| 502 | datum19 | datum19 | is greater than or equal to |
| 432 | datum1 | datum1 | is less than or equal to |
| 436 | datum2 | datum2 | is less than or equal to |
| 440 | datum3 | datum3 | is less than or equal to |
| 444 | datum4 | datum4 | is less than or equal to |
| 448 | datum5 | datum5 | is less than or equal to |
| 452 | datum6 | datum6 | is less than or equal to |
| 456 | datum7 | datum7 | is less than or equal to |
| 460 | datum8 | datum8 | is less than or equal to |
| 464 | datum9 | datum9 | is less than or equal to |
| 468 | datum10 | datum10 | is less than or equal to |
| 472 | datum11 | datum11 | is less than or equal to |
| 476 | datum12 | datum12 | is less than or equal to |
| 480 | datum13 | datum13 | is less than or equal to |
| 484 | datum14 | datum14 | is less than or equal to |
| 488 | datum15 | datum15 | is less than or equal to |
| 492 | datum16 | datum16 | is less than or equal to |
| 496 | datum17 | datum17 | is less than or equal to |
| 500 | datum18 | datum18 | is less than or equal to |
| 504 | datum19 | datum19 | is less than or equal to |
| 431 | datum1 | datum1 | is less than |
| 435 | datum2 | datum2 | is less than |
| 439 | datum3 | datum3 | is less than |
| 443 | datum4 | datum4 | is less than |
| 447 | datum5 | datum5 | is less than |
| 451 | datum6 | datum6 | is less than |
| 455 | datum7 | datum7 | is less than |
| 459 | datum8 | datum8 | is less than |
| 463 | datum9 | datum9 | is less than |
| 467 | datum10 | datum10 | is less than |
| 471 | datum11 | datum11 | is less than |
| 475 | datum12 | datum12 | is less than |
| 479 | datum13 | datum13 | is less than |
| 483 | datum14 | datum14 | is less than |
| 487 | datum15 | datum15 | is less than |
| 491 | datum16 | datum16 | is less than |
| 495 | datum17 | datum17 | is less than |
| 499 | datum18 | datum18 | is less than |
| 503 | datum19 | datum19 | is less than |
| 529 | Sale Amount | monetary_value | is |
| 530 | Sale Amount | monetary_value | is less than |
| 531 | Sale Amount | monetary_value | is less than or equal to |
| 532 | Sale Amount | monetary_value | is greater than |
| 533 | Sale Amount | monetary_value | is greater than or equal to |
| 481 | datum14 | datum14 | is greater than |
| 485 | datum15 | datum15 | is greater than |
| 489 | datum16 | datum16 | is greater than |
| 493 | datum17 | datum17 | is greater than |
| 497 | datum18 | datum18 | is greater than |
| 501 | datum19 | datum19 | is greater than |
| 536 | Undeliverable | unit_id | is |
| 537 | Web Page Visited | web_page_id | is |
| 538 | No Opens | created | since |
| 587 | datum20 | datum20 | is |
| 588 | datum20 | datum20 | is not |
| 589 | datum20 | datum20 | contains |
| 590 | datum20 | datum20 | does not contain |
| 591 | datum20 | datum20 | starts with |
| 592 | datum20 | datum20 | ends with |
| 593 | datum20 | datum20 | does not start with |
| 594 | datum20 | datum20 | does not end with |
| 595 | datum20 | datum20 | is less than |
| 596 | datum20 | datum20 | is less than or equal to |
| 597 | datum20 | datum20 | is greater than |
| 598 | datum20 | datum20 | is greater than or equal to |
| 599 | datum21 | datum21 | is |
| 600 | datum21 | datum21 | is not |
| 601 | datum21 | datum21 | contains |
| 602 | datum21 | datum21 | does not contain |
| 603 | datum21 | datum21 | starts with |
| 604 | datum21 | datum21 | ends with |
| 605 | datum21 | datum21 | does not start with |
| 606 | datum21 | datum21 | does not end with |
| 607 | datum21 | datum21 | is less than |
| 608 | datum21 | datum21 | is less than or equal to |
| 609 | datum21 | datum21 | is greater than |
| 610 | datum21 | datum21 | is greater than or equal to |
| 611 | datum22 | datum22 | is |
| 612 | datum22 | datum22 | is not |
| 613 | datum22 | datum22 | contains |
| 614 | datum22 | datum22 | does not contain |
| 615 | datum22 | datum22 | starts with |
| 616 | datum22 | datum22 | ends with |
| 617 | datum22 | datum22 | does not start with |
| 618 | datum22 | datum22 | does not end with |
| 619 | datum22 | datum22 | is less than |
| 620 | datum22 | datum22 | is less than or equal to |
| 621 | datum22 | datum22 | is greater than |
| 622 | datum22 | datum22 | is greater than or equal to |
| 623 | datum23 | datum23 | is |
| 624 | datum23 | datum23 | is not |
| 625 | datum23 | datum23 | contains |
| 626 | datum23 | datum23 | does not contain |
| 627 | datum23 | datum23 | starts with |
| 628 | datum23 | datum23 | ends with |
| 629 | datum23 | datum23 | does not start with |
| 630 | datum23 | datum23 | does not end with |
| 631 | datum23 | datum23 | is less than |
| 632 | datum23 | datum23 | is less than or equal to |
| 633 | datum23 | datum23 | is greater than |
| 634 | datum23 | datum23 | is greater than or equal to |
| 635 | datum24 | datum24 | is |
| 636 | datum24 | datum24 | is not |
| 637 | datum24 | datum24 | contains |
| 638 | datum24 | datum24 | does not contain |
| 639 | datum24 | datum24 | starts with |
| 640 | datum24 | datum24 | ends with |
| 641 | datum24 | datum24 | does not start with |
| 642 | datum24 | datum24 | does not end with |
| 643 | datum24 | datum24 | is less than |
| 644 | datum24 | datum24 | is less than or equal to |
| 645 | datum24 | datum24 | is greater than |
| 646 | datum24 | datum24 | is greater than or equal to |
| 647 | datum25 | datum25 | is |
| 648 | datum25 | datum25 | is not |
| 649 | datum25 | datum25 | contains |
| 650 | datum25 | datum25 | does not contain |
| 651 | datum25 | datum25 | starts with |
| 652 | datum25 | datum25 | ends with |
| 653 | datum25 | datum25 | does not start with |
| 654 | datum25 | datum25 | does not end with |
| 655 | datum25 | datum25 | is less than |
| 656 | datum25 | datum25 | is less than or equal to |
| 657 | datum25 | datum25 | is greater than |
| 658 | datum25 | datum25 | is greater than or equal to |
| 659 | Tag | subscriber_tags.tags | is |
| 660 | Tag | subscriber_tags.tags | is not |
| 661 | Any Opens | opens | since |
| 662 | Any Clicks | clicks | since |
| 663 | Any Clicks | clicks | before |
| 664 | Any Opens | opens | before |
| 665 | No Opens | created | before |
| 666 | Tag | subscriber_tags.tags | is any of these |
| 667 | Tag | subscriber_tags.tags | includes all of these |

8
aweber/20211209115433-admin_restore_deleted_controller.org
View file

@ -1,8 +0,0 @@
:PROPERTIES:
:ID: 4bf81e33-8020-40f2-b6ed-ce4e1eae2234
:ROAM_ALIASES: "Restoring deleted subscribers"
:END:
#+title: Admin restore deleted controller
Lists have a deleted subscriber history view, from which deleted subscribers can
be restored.

74
aweber/20211210114117-legacy_search_segment_construction.org
View file

@ -1,74 +0,0 @@
:PROPERTIES:
:ID: e6a2c650-ff59-4b72-b073-970731796888
:END:
#+title: Legacy search segment construction
Thoughts on constructing a segment representation from legacy data in the
[[id:11edd6c9-b976-403b-a419-b5542ddedaae][Subscriber Search Service]], referencing a [[https://gitlab.aweber.io/CP/Services/subscriber-search/-/merge_requests/15/diffs?commit_id=4d203dfec800cf49825d32a7ea663c4b862693bb][WIP commit]] loading legacy segments from
the database.
- The legacy segment stores field and operator information encoded in a search
box ID, and a match value in a list view criteria field.
- Legacy segments only support a top-level AND grouping of criteria
Therefore, given a list of criteria on a segment, a Search can be formed as
follows:
#+begin_src python
import dataclasses
from subscribersearch import legacy, search
@dataclasses.dataclass
class Segment:
"""A saved search on a list representing a segment of subscribers.
Likely belongs in the search module.
"""
id: typing.Optional[int]
name: str
search: search.Search
@staticmethod
def from_dict(value: dict) -> Search:
return Segment(
id=value.get("id"),
name=value["name"],
search=search.from_dict(value["search"]),
)
def to_dict(self) -> typing.Dict[str, typing.Any]:
return {
"id": self.id,
"name": self.name,
"search": self.search.to_dict(),
}
# segment = fetch legacy segment info
# rows = fetch legacy segment criteria
segment = Segment(
id=segment["id"],
name=segment["name"],
search=search.Search(
list_id=segment["list_id"],
group=search.Group(
group_type=search.GroupType.AND,
conditions=[
search.Condition(
field=filter.field,
operator=filter.operator,
match=match,
)
for filter, match in [
(legacy.filter_from_id(row["sb_id"]), row["lvc_criteria"]) in rows
]
],
),
),
)
#+end_src
By having functions that can retrieve a Segment object from the database and
save one back to it, we have an interface that we can then copy later for
storing and retrieving segments in an updated schema, and migrate between the
two.

32
aweber/20211227135649-move_cache1_cache4_to_mdf.org
View file

@ -1,32 +0,0 @@
:PROPERTIES:
:ID: b6a4fa1c-c07b-4834-8ab8-5ebe79334729
:END:
#+title: Move Cache1-Cache4 to MDF
#+filetags: :project:
- https://jira.aweber.io/browse/DCM-259
- https://confluence.aweber.io/pages/viewpage.action?spaceKey=PSE&title=DCM2+Analysis%3A+Cache+Servers
| Team | Project | Redis Usage |
|------+------------------------------+----------------------------------------|
| cp | bulk-tagging | aiomappinglib |
| cp | bulk-tagging-consumer | aiomappinglib |
| cp | coiconsumer | aiomappinglib |
| cp | newsubnotifier | aiomappinglib |
| cp | sosconsumer | aiomappinglib |
| cp | stripe-payments | aiomappinglib |
| cp | subscriber-import-evaluation | imports |
| cp | subscriber-import-processor | imports |
| cp | subscriber-rebuild | aiomappinglib |
| cp | subscriber-sync-create | aiomappinglib, subscriber-lead-mapping |
| cp | subscriber-sync-delete | aiomappinglib, subscriber-lead-mapping |
| cp | subscriber-sync-update | aiomappinglib, subscriber-lead-mapping |
| cp | subscriber-tag-sync | aiomappinglib |
| cp | subscriberimportapi | imports |
| cp | subscriberproxy | aiomappinglib |
| cp | uosconsumer | aiomappinglib |
| cp | uouconsumer | aiomappinglib |
TTLs:
- imports :: 86400 seconds
- subscriber-lead-mapping :: None

5
aweber/20220118100827-retire_capi.org
View file

@ -1,5 +0,0 @@
:PROPERTIES:
:ID: 46515cfd-3e6c-46ac-a8f7-7fc722141338
:END:
#+title: Retire CAPI

8
aweber/20220118102133-decommission_the_mail_relay_service.org
View file

@ -1,8 +0,0 @@
:PROPERTIES:
:ID: 96d1d218-60cd-41d9-91ba-48359137d239
:END:
#+title: Decommission the mail-relay service
- https://jira.aweber.io/browse/CCPANEL-10580
Tackle remaining legacy emails and retire the mail relay container.

12
aweber/20220304110425-cs_lead_ticket_questionairre.org
View file

@ -1,12 +0,0 @@
:PROPERTIES:
:ID: 311f56fc-4404-4e25-a764-d7e455cd406e
:END:
#+title: CS Lead Ticket Questionairre
- *What was broken?* :: _
- *When was it broken?* :: _
- *What did you do to fix the problem?* :: _
- *How many customers did it likely impact?* :: _
- *Is the issue automatically fixed for all customers now?* :: _
- *Does the customer or CS need to manually do something to fix their account?* :: _
- *Should a new monitoring check, metric, or test be created to prevent this from happening again?* :: _

148
aweber/20220308141431-tag_normalization.org
View file

@ -1,148 +0,0 @@
:PROPERTIES:
:ID: d06d3ab4-c2d0-47c3-aae1-4395567fc3d2
:END:
#+title: Tag Normalization
#+OPTIONS: prop:t
https://jira.aweber.io/browse/CCPANEL-11888
Tags allow extra spaces between words and characters like commas and quotes and
then rendering is misinterpreting the actual tag save in the database.
A lot of campaign issues come from inconsistencies in how tags are saved and
visualized.
* Normalization Rules
- Spaces
- Multiple spaces turned into a single space.
- No leading or trailing spaces.
- No non-printable characters (nbsp, line breaks, etc).
- Other Characters
- No commas.
- No quotes (single or double).
- Lowercase all characters.
* Roll-out
:PROPERTIES:
:COLUMNS: %50ITEM %JIRA_ID %Effort{:}
:END:
#+BEGIN: columnview :indent t
| ITEM | JIRA_ID | Effort |
|-----------------------------------------------------------------------------------------+---------------+----------|
| Roll-out | | 21d 0:00 |
| \_ Normalize when comparing tags | CCPANEL-12033 | 4d 4:00 |
| \_ Update Campaign Engine to apply normalization rules when comparing tags | CCPANEL-12031 | 1.5d |
| \_ Update Analytics Search DB Terms to apply normalization rules when comparing tags | CCPANEL-12034 | 3d |
| \_ Normalize incoming data | CCPANEL-12009 | 5d 0:00 |
| \_ Update [[id:7e503917-646f-4275-aab9-3a125b99cbfd][Tagging Service]] to normalize tags | CCPANEL-12010 | 2d |
| \_ Update [[id:aa9b1fdc-d766-41bb-ab7b-11c35bca54fa][AWSubscribers]] to normalize tags | CCPANEL-12011 | 2d |
| \_ Update TagBox to normalize tags | CCPANEL-12035 | |
| \_ Normalize tags when storing in rule sets | CCPANEL-12036 | 1d |
| \_ Normalize existing tag data | CCPANEL-12013 | 10d 0:00 |
| \_ Update tags in rulesets to conform to normalization rules | CCPANEL-12014 | 3d |
| \_ Update tags in segments to conform to normalization rules | CCPANEL-12015 | 3d |
| \_ Update tags in the tagging database to conform to normalization rules | CCPANEL-12016 | 2d |
| \_ Update tags in the subscriber tags table to conform to normalization rules | CCPANEL-12037 | 2d |
| \_ Remove comparison normalization logic | CCPANEL-12038 | 1d 4:00 |
| \_ Remove normalized comparisons from Campaign Engine | CCPANEL-12039 | 4h |
| \_ Remove normalized comparisons from Analytics DB Search Terms | CCPANEL-12040 | 1d |
| \_ Normalize tags stored outside of tagging and campaigns | CCPANEL-12041 | |
| \_ Normalize tags when storing in Stripe | | |
| \_ Integrations | | |
#+END:
** Normalize when comparing tags
:PROPERTIES:
:JIRA_ID: CCPANEL-12033
:END:
*** Update Campaign Engine to apply normalization rules when comparing tags
:PROPERTIES:
:JIRA_ID: CCPANEL-12031
:EFFORT: 1.5d
:END:
*** Update Analytics Search DB Terms to apply normalization rules when comparing tags
:PROPERTIES:
:EFFORT: 3d
:JIRA_ID: CCPANEL-12034
:END:
** Normalize incoming data
:PROPERTIES:
:JIRA_ID: CCPANEL-12009
:END:
*** Update [[id:7e503917-646f-4275-aab9-3a125b99cbfd][Tagging Service]] to normalize tags
:PROPERTIES:
:Effort: 2d
:JIRA_ID: CCPANEL-12010
:END:
*** Update [[id:aa9b1fdc-d766-41bb-ab7b-11c35bca54fa][AWSubscribers]] to normalize tags
:PROPERTIES:
:Effort: 2d
:JIRA_ID: CCPANEL-12011
:END:
*** Update TagBox to normalize tags
:PROPERTIES:
:JIRA_ID: CCPANEL-12035
:END:
- Needs UX
- Should this also change the display of non-normalized tags?
*** Normalize tags when storing in rule sets
:PROPERTIES:
:EFFORT: 1d
:JIRA_ID: CCPANEL-12036
:END:
Campaign proxy / rule service
** Normalize existing tag data
:PROPERTIES:
:JIRA_ID: CCPANEL-12013
:EFFORT: 10d 0:00
:END:
- Takes an account as input and updates tags in all tables.
- Store existing records before modification.
*** Update tags in rulesets to conform to normalization rules
:PROPERTIES:
:JIRA_ID: CCPANEL-12014
:EFFORT: 3d
:END:
*** Update tags in segments to conform to normalization rules
:PROPERTIES:
:JIRA_ID: CCPANEL-12015
:EFFORT: 3d
:END:
*** Update tags in the tagging database to conform to normalization rules
:PROPERTIES:
:JIRA_ID: CCPANEL-12016
:EFFORT: 2d
:END:
*** Update tags in the subscriber tags table to conform to normalization rules
:PROPERTIES:
:Effort: 2d
:JIRA_ID: CCPANEL-12037
:END:
** Remove comparison normalization logic
:PROPERTIES:
:JIRA_ID: CCPANEL-12038
:END:
*** Remove normalized comparisons from Campaign Engine
:PROPERTIES:
:EFFORT: 4h
:JIRA_ID: CCPANEL-12039
:END:
*** Remove normalized comparisons from Analytics DB Search Terms
:PROPERTIES:
:EFFORT: 1d
:JIRA_ID: CCPANEL-12040
:END:
** [#B] Normalize tags stored outside of tagging and campaigns
:PROPERTIES:
:JIRA_ID: CCPANEL-12041
:END:
Optional based on product decisions. Tags will be normalized when ingested.
This should be done, but is not a blocker for the tag normalization project.
(Could this be managed just by updating the display of non-normalized tags in
TagBox?)
*** Normalize tags when storing in Stripe
*** Integrations

6
aweber/20220308142926-awsubscribers.org
View file

@ -1,6 +0,0 @@
:PROPERTIES:
:ID: aa9b1fdc-d766-41bb-ab7b-11c35bca54fa
:END:
#+title: AWSubscribers
A [[id:e4d00c11-da8a-4c91-8f38-ce939846e5cb][CoreAPI]] service dedicated to mailing list subscriber data.

4
aweber/20220310152925-overhauling_logins_and_session_management.org
View file

@ -1,4 +0,0 @@
:PROPERTIES:
:ID: 0328a202-376d-4e97-b0e3-031eaad2a557
:END:
#+title: Overhauling logins and session management

29
aweber/20220329103252-identifying_active_accounts.org
View file

@ -1,29 +0,0 @@
:PROPERTIES:
:ID: e97adcf4-86ad-4d97-9c63-41476b52b111
:END:
#+title: Identifying active accounts
A query to identify active accounts in [[id:dd113e53-6144-4cb2-a4aa-da3dc2e3e6ea][AppDB]]
Per [[file:~/git/appdb/functions/coreapi_account/get_account_status.yaml][coreapi_account.get_account_status]]:
#+begin_src sql :exports code :eval never
CREATE FUNCTION coreapi_account.get_account_status(in_account_id integer) RETURNS coreapi_account.account_status
LANGUAGE sql STRICT
AS $_$
SELECT accounts.status_id,
accounts.status_id = 7,
CASE accounts.status_id
WHEN 1 THEN 'New Order'
WHEN 4 THEN 'Unpaid - Un-notified'
WHEN 5 THEN 'Paid'
WHEN 6 THEN 'Unpaid - Notified'
WHEN 7 THEN 'Cancelled'
WHEN 8 THEN 'Unpaid - Overdue'
WHEN 9 THEN 'Place Holder'
ELSE 'Unknown'
END
FROM public.accounts
WHERE accounts.a_id = $1;
#+end_src

136
aweber/20220407112615-null_and_postgresql_array_operators.org
View file

@ -1,136 +0,0 @@
:PROPERTIES:
:ID: 00cf1628-bc60-451d-bd30-d11d6b92992f
:header-args:sql: :engine postgresql :cmdline "-U postgres postgres" :dir /docker:postgres: :exports both :cache yes :eval no-export
:END:
#+title: Null and PostgreSQL array operators
Researching an issue in subscriber search and broadcast sending resulted in
learning some interesting things about [[id:af84ed59-96a4-4f9c-b34c-b79178ad20cb][PostgreSQL]]'s generation and handling of
=NULL= values, despite its otherwise strict type checking.
* The symptom
[[https://jira.aweber.io/browse/ASE-8617][ASE-8617]] describes a scenario where a customer ([[https://admin.aweber.io/account/index/1018872][AID 1018872]]) sends broadcasts to
a segment ("no tags subscribers") on their list (List ID 5830776), defined with
the following criteria:
#+caption: Segment definition: "no tags subscribers"
| Tag | is not | wr |
| Tag | is not | cb |
| Tag | is not | cb2 |
| Tag | is not | fsc |
| Tag | is not | flp |
| Tag | is not | lsa |
Normally, they expect this segment to reach >1000 subscribers. Following a
release of our [[id:d06d3ab4-c2d0-47c3-aae1-4395567fc3d2][Tag Normalization]] changes, they sent a broadcast to this segment
that only reached 7 subscribers. Rolling back the [[https://gitlab.aweber.io/DBA/ddl/schema-deploy/-/merge_requests/257][change to segment search terms]]
corrected the issue for the customer.
* The implementation
** The "Tag Is Not" search filter
The change updated the tag "is not" filter from
#+begin_src sql :exports code :eval never
(NOT(tags @> ARRAY[=:$1:=]) or tags is null)
#+end_src
to
#+begin_src sql :exports code :eval never
(NOT(public.normalize_tags(tags) @> public.normalize_tags(ARRAY[=:$1:=])) or tags is null)
#+end_src
** The =normalize_tags= function
The =normalize_tags= function is defined as:
#+begin_src sql :exports code :eval never
CREATE OR REPLACE FUNCTION public.normalize_tags(in_tags text[])
RETURNS text[]
LANGUAGE SQL STRICT IMMUTABLE AS $$
SELECT ARRAY_AGG(public.normalize_tag(tag))
FROM UNNEST(in_tags) AS tag;
$$;
#+end_src
* The cause
When testing to see what the criteria SQL would evaluate to, I discovered
something ... interesting.
#+begin_src sql
SELECT (NOT(public.normalize_tags(ARRAY[]::text[]) @> ARRAY['foo']::text[]))
#+end_src
#+RESULTS[b0e468c2d8fc47126ee77eb463b2525ff9dca266]:
| ?column? |
|----------|
| |
That hardly seems right. It's not even returning a boolean. Could it be?
#+begin_src sql
SELECT (NOT(public.normalize_tags(ARRAY[]::text[]) @> ARRAY['foo']::text[])) IS NULL
#+end_src
#+RESULTS[c30ad761b00b6d3c038ec008010f17ae5228befd]:
| ?column? |
|----------|
| t |
Yep. It's returning =NULL=, Tony Hoare's "[[https://www.infoq.com/presentations/Null-References-The-Billion-Dollar-Mistake-Tony-Hoare/][billion-dollar mistake]]". The only
thing different in this implementation is the call to =normalize_tags=, so let's
take a look at that.
#+begin_src sql
SELECT public.normalize_tags(ARRAY[]::text[]) IS NULL
#+end_src
#+RESULTS[8a85412be393d1e19d576db8ef321f0ab1380261]:
| ?column? |
|----------|
| t |
Sure enough, it is returning =NULL= when called with an empty array. That =NULL= then escapes to our array comparison...
#+begin_src sql
SELECT (NULL @> ARRAY['foo'::text]) IS NULL
#+end_src
#+RESULTS[80d59a923b98d6ec7adc7af29a53b7557c7c0efc]:
| ?column? |
|----------|
| t |
... which also returns =NULL=. That then gets passed to =NOT=...
#+begin_src sql
SELECT (NOT(NULL)) IS NULL
#+end_src
#+RESULTS[77051c05ab0f6df48e3c13be6ff0651904f8b6da]:
| ?column? |
|----------|
| t |
... which also returns =NULL=. =NULL= is treated as falsy by [[id:af84ed59-96a4-4f9c-b34c-b79178ad20cb][PostgreSQL]],
and so the search fails to match subscribers without tags.
* The resolution
Using =COALESCE= on the result of =ARRAY_AGG= to ensure we get an empty array
when the result is =NULL=, which lets us avoid all of the above problems caused
by =NULL= escaping into our comparisons.
#+begin_src sql :exports code :eval no-export
CREATE OR REPLACE FUNCTION public.normalize_tags(in_tags text[])
RETURNS text[]
LANGUAGE SQL STRICT IMMUTABLE AS $$
SELECT COALESCE(ARRAY_AGG(public.normalize_tag(tag)), ARRAY[]::text[])
FROM UNNEST(in_tags) AS tag
WHERE public.normalize_tag(tag) <> '';
$$;
#+end_src
#+RESULTS[f49761b0fa0e3074cf739102361da7272672f0a1]:
| CREATE FUNCTION |
|-----------------|
#+begin_src sql
SELECT public.normalize_tags(ARRAY[]::text[])
#+end_src
#+RESULTS[a37fd5ae0eaedaf2b33b4886d60d2db0cf632fb5]:
| normalize_tags |
|----------------|
| {} |

5
aweber/20220520154427-recipient_service.org
View file

@ -1,5 +0,0 @@
:PROPERTIES:
:ID: 1ff6586e-2dba-41a2-a887-753cc5ac27c9
:ROAM_REFS: https://gitlab.aweber.io/CP/Services/recipient
:END:
#+title: Recipient Service

150
aweber/20220609145347-retire_redcache.org
View file

@ -1,150 +0,0 @@
:PROPERTIES:
:ID: ee5b8d5f-e3d4-45c2-9ce6-bcd8c7a63376
:ROAM_REFS: https://jira.aweber.io/browse/AWEB-378
:END:
#+title: Retire RedCache
The goal of this initiative is to identify uses of the shared RedCache servers
and eliminate them. If a key/value store is required, use a sidecar Redis or
memcached instance as appropriate.
Any keys that will require cross-team coordination to remove should be
documented in the [[https://confluence.aweber.io/display/AWD/RedCache+Inventory][RedCache Inventory]] page in Confluence.
* Usage in the [[id:57ee2f00-9bcd-4e0f-8a77-ae1f2d4cda89][Control Panel]]
** Caching
These use cases treat Redis as a temporary cache. They could be safely and
seamlessly switched over to a new Redis instance.
*** Cake Cache
The default Cake framework cache engine for the application is [[https://gitlab.aweber.io/CP/applications/sites/-/blob/master/aweber_app/config/core.php#L236-243][configured to use
RedCache]].
*** Mapping
The [[https://gitlab.aweber.io/CP/applications/sites/-/blob/master/components/mapping.php][Mapping component]] caches mapping lookups.
*** Avro Schemas
The [[https://gitlab.aweber.io/CP/applications/sites/-/blob/master/components/avro.php][Avro component]] caches schema documents.
*** Private Labeling
A [[https://gitlab.aweber.io/CP/applications/sites/-/blob/master/php5-vendors/vendors/purge_pl_memcache.php][script to purge the private label cache]] exists, though it does not appear to
be used.
#+begin_notes
Private labeling is no longer used in the CP.
#+end_notes
*** Click Tracking
The [[https://gitlab.aweber.io/CP/applications/sites/-/blob/master/php5-vendors/vendors/click_tracker.php][Click tracking component]] caches tracking url lookups in AppDB.
*** Showcased Applications
The [[https://gitlab.aweber.io/CP/applications/sites/-/blob/master/models/application.php][Application model]] caches the number of customers using each application from
queries to AppDB for six hours.
*** Web Form Templates
The [[https://gitlab.aweber.io/CP/applications/sites/-/blob/master/models/web_form_template_category.php][Web Form Template Category model]] caches the top ten web form template
families using the =template_directory_web_form_popular= key.
*** Active Lists
The [[https://gitlab.aweber.io/CP/applications/sites/-/blob/master/models/auto_responder.php][Auto Responder model]] caches the active lists for an account using the key
format =aweber_app_lists_{$a_id}_{$aServId}=, where =$a_id= is the integer
account ID and =$aServId= is an integer list ID or =false=.
*** Web Form Chicklet Images
The [[https://gitlab.aweber.io/CP/applications/sites/-/blob/master/aweber_app/webroot/form/ci/index.php][Web form chicklet image handler]] caches image data stored in AppDB using the
key format =aweber_app_chicklet_$id= where =$id= is the integer ID of the
chicklet image.
*** Template Gallery
The [[https://gitlab.aweber.io/CP/applications/sites/-/blob/master/aweber_app/views/helpers/template_gallery.php][Template Gallery view helper]] caches email templates from the [[http://template-directory.service.production.consul][Template
Directory Service]] using the =template_directory_{$type}_family_data= key format,
where =$type= is =web_form= or =block=.
*** List Settings
The [[https://gitlab.aweber.io/CP/applications/sites/-/blob/master/aweber_app/controllers/settings_controller.php][Settings controller]] clears cached list settings using the key pattern
=aweber_app_{$namespace}_remove_options_{$list_id}=, where =$namespace= is the
value retrieved from the key =aweber_app_remove_options_namespace=.
*** Web Form Serving
The [[https://gitlab.aweber.io/CP/applications/sites/-/blob/master/aweber_app/vendors/web_form_server.php][Web Form Server component]] caches web form split tests loaded from AppDB
using the key format =aweber_app_{$namespace}_web-form_split_{$split_id}= where
=$namespace= is the value retrieved from the key
=aweber_app_web_form_namespace=.
** Other
These use cases treat Redis as a key/value store with specific expectations
around if/when the key is cleared. Data migration may be necessary for a move to
a separate Redis instance.
*** Throttling
The [[https://gitlab.aweber.io/CP/applications/sites/-/blob/master/php5-vendors/vendors/throttler.php][Throttler component]] uses cache keys with a TTL to rate-limit various actions
in the CP including logins. An older [[https://gitlab.aweber.io/CP/applications/sites/-/blob/master/components/throttle.php][Throttle component]] also exists with
references to redcache, but appears unused.
*** Verify-Optin Processing
The [[https://gitlab.aweber.io/CP/applications/sites/-/blob/master/php5-vendors/vendors/vo_processor.php][VO Processor component]] uses the =aweber_app_db_down= cache key to determine
whether the control panel is under scheduled maintenance.
This component also resides separately in the [[https://gitlab.aweber.io/CP/applications/verify-optin/-/blob/master/verify-optin/include/vo_processor.php][Verify-Optin]] project, doing the
same thing, and should be removed from the sites repository.
*** Unsubscribe / Manage Subscriptions
The [[https://gitlab.aweber.io/CP/applications/sites/-/blob/master/aweber_app/webroot/z/r/index.htm][Manage Subscriptions page]] uses the =aweber_app_db_down= cache key to
determine whether the control panel is under scheduled maintenance.
This component also resides separately in the [[https://gitlab.aweber.io/CP/applications/unsubscribe][Unsubscribe]] project, doing the
same thing, and should be removed from the sites repository.
*** Refer a Friend
The [[https://gitlab.aweber.io/CP/applications/sites/-/blob/master/components/refer_a_friend.php][Refer a Friend component]] uses the =refer_a_friend:access_token= key to store
and verify an access token.
*** One-Click Unsubscribe
The [[https://gitlab.aweber.io/CP/applications/sites/-/blob/master/aweber_app/webroot/z/r/one_click_remove.php][One-click remove handler]] uses the =aweber_app_db_down= cache key to
determine whether the control panel is under scheduled maintenance.
*** Blocked Orders
The [[https://gitlab.aweber.io/CP/applications/sites/-/blob/master/aweber_app/controllers/order_controller.php][Order controller]] checks and stores IP addresses for blocking orders with the
key pattern =orders_blocked_for_{$_SERVER['REMOTE_ADDR']}=
*** Feed Broadcasts
The [[https://gitlab.aweber.io/CP/applications/sites/-/blob/master/aweber_app/controllers/feedbroadcaster_controller.php][Feed Broadcasts controller]] tracks feed gearman job status by setting and
retrieving the Redis key patterns =aweber_app_process_feed_{$sid}_complete= and
=aweber_app_process_feed_{$sid}_status=, where =$sid= is the current session ID.
*** Preferences
The [[https://gitlab.aweber.io/CP/applications/sites/-/blob/master/aweber_app/controllers/components/preferences.php][Preferences controller component]] stores and retrieves account-level
preferences using the key pattern
=aweber_app_{$namespace}_cp_preference_{$aId}=.
*** Subscriber Search Locking
The [[https://gitlab.aweber.io/CP/applications/sites/-/blob/master/aweber_app/controllers/components/search_mutex.php][Search Mutex controller component]] sets and clears a lock preventing
concurrent searches within an account using the key pattern
=subscriber_search_lock_{$accountId}=.
*** Login Email Verification
The [[https://gitlab.aweber.io/CP/applications/sites/-/blob/master/aweber_app/controllers/account_controller.php][Account controller]] stores an email verification token using the key pattern
=login_email:verification_token:{$user['id']}=
*** Sift Login Verification
The [[https://gitlab.aweber.io/CP/applications/sites/-/blob/master/aweber_app/controllers/account_controller.php][Account controller]] stores a flag allowing previously verified logins to
bypass the Sift score check using the key pattern
=login_verification:verified_bypass:{$login}:{$ipAddress}=.
*** Application maintenance
The [[https://gitlab.aweber.io/CP/applications/sites/-/blob/master/aweber_app/app_controller.php][App controller]] uses the =aweber_app_db_down= cache key to determine whether
the control panel is under scheduled maintenance.
*** Web Form Generation
The [[https://gitlab.aweber.io/CP/applications/sites/-/blob/master/aweber_app/vendors/web_form_generator.php][Web Form Generator component]] stores and retrieves web form settings using the key pattern ={$namespace}_web_form_{$webFormId}{$type}= where =$namespace= is the value retrieved from the key =aweber_app_web_form_namespace=, and type is one of the following:
- =_js= (JavaScript)
- =s_js= (split JavaScript)
- =_htm=
- =_html=
** Unclear
These use cases read or clear keys in the key/value store, but the keys may be
managed elsewhere. It is unsafe to migrate these to another Redis instance until
they are fully understood.
*** List Twitter Account
The [[https://gitlab.aweber.io/CP/applications/sites/-/blob/master/models/vendor_account_list.php][Vendor Account List model]] clears the twitter account on a list using the key
format =orm.list.by_id.$listId.twitter_account=. It is unclear what sets that
key.
*** Flagged Credit Card Numbers
The [[https://gitlab.aweber.io/CP/applications/sites/-/blob/master/aweber_app/controllers/order_controller.php][Order controller]] retrieves flagged credit card numbers using the key
=aweber_app_flagged_cc_bins=. Used to check the first six digits of a card for
flagging.
*** Message Templates
The [[https://gitlab.aweber.io/CP/applications/sites/-/blob/master/aweber_app/controllers/message_templates_controller.php][Message Templates controller]] clears the key
=template_directory_block_family_data= when a message template is saved.
*** Lead Editing
The [[https://gitlab.aweber.io/CP/applications/sites/-/blob/master/aweber_app/controllers/leads_controller.php][Leads controller]] checks for blocked emails and email domains using the key
patterns =0-$email= and =0-$domain=.
* Usage in Verify-Optin
** Control Panel Maintenance
The [[https://gitlab.aweber.io/CP/applications/verify-optin/-/blob/master/verify-optin/include/vo_processor.php][VO Processor component]] uses the =aweber_app_db_down= cache key to determine
whether the control panel is under scheduled maintenance.
* Usage in Unsubscribe
** Control Panel Maintenance
The [[https://gitlab.aweber.io/CP/applications/unsubscribe/-/blob/master/unsubscribe/webroot/z/r/index.php][Manage Subscriptions page]] uses the =aweber_app_db_down= cache key to
determine whether the control panel is under scheduled maintenance.
* Usage in [[id:03e00c18-99c0-477c-b7fb-95ddc538755e][Addlead]]
** Control Panel Maintenance
Uses the =aweber_app_db_down= cache key to determine whether the control panel
is under scheduled maintenance.

32
aweber/20220609150836-use_appdb_as_the_source_of_truth_for_subscriber_data_in_recipient.org
View file

@ -1,32 +0,0 @@
:PROPERTIES:
:ID: 2c1a7b1d-8726-4b88-9534-2f5abfec35f0
:END:
#+title: Use AppDB as the source of truth for subscriber data in Recipient
The [[id:1ff6586e-2dba-41a2-a887-753cc5ac27c9][Recipient Service]] shall replace [[id:aa9b1fdc-d766-41bb-ab7b-11c35bca54fa][AWSubscribers]] as the API for mailing list
subscribers. Currently, [[id:aa9b1fdc-d766-41bb-ab7b-11c35bca54fa][AWSubscribers]] is responsible for all subscriber creation
and updating and uses [[id:dd113e53-6144-4cb2-a4aa-da3dc2e3e6ea][AppDB]] as its data store, which is the current source of
truth for subscriber data. The [[id:1ff6586e-2dba-41a2-a887-753cc5ac27c9][Recipient Service]] maintains its own copy of
subscriber data in DynamoDB along with its additional recipient data, which is
kept up to date via the [[id:b285adee-2dab-48ed-b2d8-2df5594f9d30][Subscriber Sync Consumer]]. Having both services sharing
the same data store will allow us to migrate calls towards the [[id:1ff6586e-2dba-41a2-a887-753cc5ac27c9][Recipient Service]]
without risking data becoming desynchronized between them.
* Read subscriber data from AppDB by default
Campaigns may fail as they race to retrieve information from the [[id:1ff6586e-2dba-41a2-a887-753cc5ac27c9][Recipient
Service]] for a newly created subscriber, which may not have synchronized over
from [[id:dd113e53-6144-4cb2-a4aa-da3dc2e3e6ea][AppDB]]. Reads are also the least impactful change to make, as it will only
affect the [[id:b285adee-2dab-48ed-b2d8-2df5594f9d30][Subscriber Sync Consumer]] as it checks whether to create or update a
record. This makes this change the ideal starting point for this project.
The current DynamoDB behavior will be kept intact so that the [[id:b285adee-2dab-48ed-b2d8-2df5594f9d30][Subscriber Sync
Consumer]] can continue to keep the data stored in DynamoDB up to date.
** DONE Add support for reading subscriber data from AppDB to Recipient
** DONE Update subscriber-sync to use flag to read from DynamoDB
** DONE Switch recipient to read subscriber data from AppDB by default
* Write subscriber data to AppDB
* Retire subscriber data in DynamoDB

5
aweber/20220609151255-subscriber_sync_consumer.org
View file

@ -1,5 +0,0 @@
:PROPERTIES:
:ID: b285adee-2dab-48ed-b2d8-2df5594f9d30
:ROAM_REFS: https://gitlab.aweber.io/CP/Consumers/subscriber-sync
:END:
#+title: Subscriber Sync Consumer

10
aweber/20220630170620-addlead_rewrite.org
View file

@ -1,10 +0,0 @@
:PROPERTIES:
:ID: 9adaef5d-9cfa-424f-966b-64fd558a5122
:END:
#+title: Addlead Rewrite
A [[id:207560cc-7700-4d06-918d-cc01ae530146][Project]] to rewrite [[id:03e00c18-99c0-477c-b7fb-95ddc538755e][Addlead]] in Python as a modern, highly available service.
- [[id:d30d8d19-f9bd-4c98-827a-5895e3902688][Addlead Rewrite ACP]]
* New Product Requirements

38
aweber/20220706110652-addlead_rewrite_acp.org
View file

@ -1,38 +0,0 @@
:PROPERTIES:
:ID: d30d8d19-f9bd-4c98-827a-5895e3902688
:END:
#+title: Addlead Rewrite ACP
* Purpose
The purpose of this ACP is to replace the existing subscriber web form endpoint
with a modern implementation using our current language, library, and deployment
standards.
* Problem
Addlead is currently written in Perl, relying on old libraries, and is running
on deprecated VM hardware deployed with Chef.
* Current State
Addlead is far behind our current technology stack and standard practices:
- It is written in Perl, which is not one of our supported languages.
Increasingly fewer engineers are familiar enough with it to make modifications
to it.
- Its libraries are no longer being actively maintained.
- It is running on old hardware which would be difficult to reprovision if lost.
- It is deployed using Chef, which we have deprecated and are actively removing
from our stack.
* Requirements
* Terminology
* Proposed Solution
** Addlead HTTP Endpoint
- Accept web form POST
- Validate request
- If validation services are unavailable, proceed as though validation
succeeded. Validation will be applied when the event is processed.
- Submit subscriber event to RabbitMQ
- If RabbitMQ is unavailable, store the event in a failover queue (S3, shared
volume?)
** Failover Queue Consumer
This will be a job that runs periodically to take any events in the failover
queue and deliver them to RabbitMQ, removing them once they've been accepted by
RabbitMQ.
** Addlead Consumer
Listens for subscription requests and processes them using the subscriber API.

BIN
aweber/controllers-migrated-in-sites-over-time.png
View file

Binary file not shown.
Side by side

Before

Width:  |  Height:  |  Size: 20 KiB

BIN
aweber/controllers-migrated-in-sites.png
View file

Binary file not shown.
Side by side

Before

Width:  |  Height:  |  Size: 25 KiB

BIN
aweber/data/02/2406d2-0480-4470-90d0-9533f6b9fa32/_20210708_1637107cd58bb6-dba2-4ab6-a03e-150aa7f5c216.jpeg
View file

Binary file not shown.
Side by side

Before

Width:  |  Height:  |  Size: 92 KiB

22
aweber/ecommerce-cancellation-webhooks.svg
View file

@ -1,22 +0,0 @@
<?xml version="1.0" encoding="UTF-8" standalone="no"?><svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" contentScriptType="application/ecmascript" contentStyleType="text/css" height="242px" preserveAspectRatio="none" style="width:554px;height:242px;background:#FFFFFF;" version="1.1" viewBox="0 0 554 242" width="554px" zoomAndPan="magnify"><defs><filter height="300%" id="f1g5bqu9vn3r6s" width="300%" x="-1" y="-1"><feGaussianBlur result="blurOut" stdDeviation="2.0"/><feColorMatrix in="blurOut" result="blurOut2" type="matrix" values="0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 .4 0"/><feOffset dx="4.0" dy="4.0" in="blurOut2" result="blurOut3"/><feBlend in="SourceGraphic" in2="blurOut3" mode="normal"/></filter></defs><g><rect fill="#DDDDDD" height="230.5293" style="stroke:#A80036;stroke-width:1.0;" width="85" x="395.5" y="6"/><text fill="#000000" font-family="sans-serif" font-size="13" font-weight="bold" lengthAdjust="spacing" textLength="53" x="411.5" y="18.5684">Internal</text><line style="stroke:#A80036;stroke-width:1.0;stroke-dasharray:5.0,5.0;" x1="132" x2="132" y1="60.7988" y2="198.041"/><line style="stroke:#A80036;stroke-width:1.0;stroke-dasharray:5.0,5.0;" x1="437.5" x2="437.5" y1="60.7988" y2="198.041"/><line style="stroke:#A80036;stroke-width:1.0;stroke-dasharray:5.0,5.0;" x1="515.5" x2="515.5" y1="60.7988" y2="198.041"/><rect fill="#FEFECE" filter="url(#f1g5bqu9vn3r6s)" height="30.4883" style="stroke:#A80036;stroke-width:1.5;" width="250" x="5" y="25.3105"/><text fill="#000000" font-family="sans-serif" font-size="14" lengthAdjust="spacing" textLength="236" x="12" y="45.8457">Stripe Payments (Unauthenticated)</text><rect fill="#FEFECE" filter="url(#f1g5bqu9vn3r6s)" height="30.4883" style="stroke:#A80036;stroke-width:1.5;" width="250" x="5" y="197.041"/><text fill="#000000" font-family="sans-serif" font-size="14" lengthAdjust="spacing" textLength="236" x="12" y="217.5762">Stripe Payments (Unauthenticated)</text><rect fill="#FEFECE" filter="url(#f1g5bqu9vn3r6s)" height="30.4883" style="stroke:#A80036;stroke-width:1.5;" width="73" x="399.5" y="25.3105"/><text fill="#000000" font-family="sans-serif" font-size="14" lengthAdjust="spacing" textLength="59" x="406.5" y="45.8457">Core API</text><rect fill="#FEFECE" filter="url(#f1g5bqu9vn3r6s)" height="30.4883" style="stroke:#A80036;stroke-width:1.5;" width="73" x="399.5" y="197.041"/><text fill="#000000" font-family="sans-serif" font-size="14" lengthAdjust="spacing" textLength="59" x="406.5" y="217.5762">Core API</text><rect fill="#FEFECE" filter="url(#f1g5bqu9vn3r6s)" height="30.4883" style="stroke:#A80036;stroke-width:1.5;" width="54" x="486.5" y="25.3105"/><text fill="#000000" font-family="sans-serif" font-size="14" lengthAdjust="spacing" textLength="40" x="493.5" y="45.8457">Stripe</text><rect fill="#FEFECE" filter="url(#f1g5bqu9vn3r6s)" height="30.4883" style="stroke:#A80036;stroke-width:1.5;" width="54" x="486.5" y="197.041"/><text fill="#000000" font-family="sans-serif" font-size="14" lengthAdjust="spacing" textLength="40" x="493.5" y="217.5762">Stripe</text><polygon fill="#A80036" points="143,88.1094,133,92.1094,143,96.1094,139,92.1094" style="stroke:#A80036;stroke-width:1.0;"/><line style="stroke:#A80036;stroke-width:1.0;" x1="137" x2="514.5" y1="92.1094" y2="92.1094"/><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacing" textLength="359" x="149" y="87.3672">POST /stripe/webhooks (customer.subscription.updated)</text><polygon fill="#A80036" points="503.5,117.4199,513.5,121.4199,503.5,125.4199,507.5,121.4199" style="stroke:#A80036;stroke-width:1.0;"/><line style="stroke:#A80036;stroke-width:1.0;" x1="132" x2="509.5" y1="121.4199" y2="121.4199"/><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacing" textLength="149" x="139" y="116.6777">Fetch product metadata</text><polygon fill="#A80036" points="426,146.7305,436,150.7305,426,154.7305,430,150.7305" style="stroke:#A80036;stroke-width:1.0;"/><line style="stroke:#A80036;stroke-width:1.0;" x1="132" x2="432" y1="150.7305" y2="150.7305"/><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacing" textLength="282" x="139" y="145.9883">Remove tags from subscriber or unsubscribe</text><polygon fill="#A80036" points="503.5,176.041,513.5,180.041,503.5,184.041,507.5,180.041" style="stroke:#A80036;stroke-width:1.0;"/><line style="stroke:#A80036;stroke-width:1.0;" x1="132" x2="509.5" y1="180.041" y2="180.041"/><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacing" textLength="46" x="139" y="175.2988">200 OK</text><!--MD5=[550573097974f7680661b673e70c4d3c]
@startuml
participant "Stripe Payments (Unauthenticated)" as sp
box "Internal"
participant "Core API" as capi
end box
participant "Stripe" as stripe
stripe -> sp : POST /stripe/webhooks (customer.subscription.updated)
sp -> stripe : Fetch product metadata
sp -> capi : Remove tags from subscriber or unsubscribe
sp -> stripe : 200 OK
@enduml
PlantUML version 1.2021.10(Mon Aug 30 09:43:48 EDT 2021)
(GPL source distribution)
Java Runtime: Java(TM) SE Runtime Environment
JVM: Java HotSpot(TM) 64-Bit Server VM
Default Encoding: UTF-8
Language: en
Country: US
--></g></svg>
Side by side

Before

Width:  |  Height:  |  Size: 5.1 KiB

22
aweber/ecommerce-payment-webhooks.svg
View file

@ -1,22 +0,0 @@
<?xml version="1.0" encoding="UTF-8" standalone="no"?><svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" contentScriptType="application/ecmascript" contentStyleType="text/css" height="242px" preserveAspectRatio="none" style="width:638px;height:242px;background:#FFFFFF;" version="1.1" viewBox="0 0 638 242" width="638px" zoomAndPan="magnify"><defs><filter height="300%" id="fd0b0cv41yrbd" width="300%" x="-1" y="-1"><feGaussianBlur result="blurOut" stdDeviation="2.0"/><feColorMatrix in="blurOut" result="blurOut2" type="matrix" values="0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 .4 0"/><feOffset dx="4.0" dy="4.0" in="blurOut2" result="blurOut3"/><feBlend in="SourceGraphic" in2="blurOut3" mode="normal"/></filter></defs><g><rect fill="#DDDDDD" height="230.5293" style="stroke:#A80036;stroke-width:1.0;" width="93" x="471.5" y="6"/><text fill="#000000" font-family="sans-serif" font-size="13" font-weight="bold" lengthAdjust="spacing" textLength="53" x="491.5" y="18.5684">Internal</text><line style="stroke:#A80036;stroke-width:1.0;stroke-dasharray:5.0,5.0;" x1="132" x2="132" y1="60.7988" y2="198.041"/><line style="stroke:#A80036;stroke-width:1.0;stroke-dasharray:5.0,5.0;" x1="517.5" x2="517.5" y1="60.7988" y2="198.041"/><line style="stroke:#A80036;stroke-width:1.0;stroke-dasharray:5.0,5.0;" x1="599.5" x2="599.5" y1="60.7988" y2="198.041"/><rect fill="#FEFECE" filter="url(#fd0b0cv41yrbd)" height="30.4883" style="stroke:#A80036;stroke-width:1.5;" width="250" x="5" y="25.3105"/><text fill="#000000" font-family="sans-serif" font-size="14" lengthAdjust="spacing" textLength="236" x="12" y="45.8457">Stripe Payments (Unauthenticated)</text><rect fill="#FEFECE" filter="url(#fd0b0cv41yrbd)" height="30.4883" style="stroke:#A80036;stroke-width:1.5;" width="250" x="5" y="197.041"/><text fill="#000000" font-family="sans-serif" font-size="14" lengthAdjust="spacing" textLength="236" x="12" y="217.5762">Stripe Payments (Unauthenticated)</text><rect fill="#FEFECE" filter="url(#fd0b0cv41yrbd)" height="30.4883" style="stroke:#A80036;stroke-width:1.5;" width="81" x="475.5" y="25.3105"/><text fill="#000000" font-family="sans-serif" font-size="14" lengthAdjust="spacing" textLength="67" x="482.5" y="45.8457">RabbitMQ</text><rect fill="#FEFECE" filter="url(#fd0b0cv41yrbd)" height="30.4883" style="stroke:#A80036;stroke-width:1.5;" width="81" x="475.5" y="197.041"/><text fill="#000000" font-family="sans-serif" font-size="14" lengthAdjust="spacing" textLength="67" x="482.5" y="217.5762">RabbitMQ</text><rect fill="#FEFECE" filter="url(#fd0b0cv41yrbd)" height="30.4883" style="stroke:#A80036;stroke-width:1.5;" width="54" x="570.5" y="25.3105"/><text fill="#000000" font-family="sans-serif" font-size="14" lengthAdjust="spacing" textLength="40" x="577.5" y="45.8457">Stripe</text><rect fill="#FEFECE" filter="url(#fd0b0cv41yrbd)" height="30.4883" style="stroke:#A80036;stroke-width:1.5;" width="54" x="570.5" y="197.041"/><text fill="#000000" font-family="sans-serif" font-size="14" lengthAdjust="spacing" textLength="40" x="577.5" y="217.5762">Stripe</text><polygon fill="#A80036" points="143,88.1094,133,92.1094,143,96.1094,139,92.1094" style="stroke:#A80036;stroke-width:1.0;"/><line style="stroke:#A80036;stroke-width:1.0;" x1="137" x2="598.5" y1="92.1094" y2="92.1094"/><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacing" textLength="330" x="149" y="87.3672">POST /stripe/webhooks (payment_intent.succeeded)</text><polygon fill="#A80036" points="506,117.4199,516,121.4199,506,125.4199,510,121.4199" style="stroke:#A80036;stroke-width:1.0;"/><line style="stroke:#A80036;stroke-width:1.0;" x1="132" x2="512" y1="121.4199" y2="121.4199"/><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacing" textLength="215" x="139" y="116.6777">Sales tracking event (pageview.v4)</text><polygon fill="#A80036" points="506,146.7305,516,150.7305,506,154.7305,510,150.7305" style="stroke:#A80036;stroke-width:1.0;"/><line style="stroke:#A80036;stroke-width:1.0;" x1="132" x2="512" y1="150.7305" y2="150.7305"/><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacing" textLength="362" x="139" y="145.9883">Payment succeeded event (stripe_payment_succeeded.v1)</text><polygon fill="#A80036" points="587.5,176.041,597.5,180.041,587.5,184.041,591.5,180.041" style="stroke:#A80036;stroke-width:1.0;"/><line style="stroke:#A80036;stroke-width:1.0;" x1="132" x2="593.5" y1="180.041" y2="180.041"/><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacing" textLength="46" x="139" y="175.2988">200 OK</text><!--MD5=[75ac5feda4e5cecbf340b2717033f540]
@startuml
participant "Stripe Payments (Unauthenticated)" as sp
box "Internal"
participant "RabbitMQ" as amqp
end box
participant "Stripe" as stripe
stripe -> sp : POST /stripe/webhooks (payment_intent.succeeded)
sp -> amqp : Sales tracking event (pageview.v4)
sp -> amqp : Payment succeeded event (stripe_payment_succeeded.v1)
sp -> stripe : 200 OK
@enduml
PlantUML version 1.2021.10(Mon Aug 30 09:43:48 EDT 2021)
(GPL source distribution)
Java Runtime: Java(TM) SE Runtime Environment
JVM: Java HotSpot(TM) 64-Bit Server VM
Default Encoding: UTF-8
Language: en
Country: US
--></g></svg>
Side by side

Before

Width:  |  Height:  |  Size: 5.2 KiB

32
aweber/ecommerce-products.svg
View file

File diff suppressed because one or more lines are too long
Side by side

Before

Width:  |  Height:  |  Size: 11 KiB

34
aweber/ecommerce-subscribing.svg
View file

File diff suppressed because one or more lines are too long
Side by side

Before

Width:  |  Height:  |  Size: 9.9 KiB

BIN
aweber/icons/details.png
View file

Binary file not shown.
Side by side

Before

Width:  |  Height:  |  Size: 744 B

BIN
aweber/icons/flag-green.png
View file

Binary file not shown.
Side by side

Before

Width:  |  Height:  |  Size: 583 B

BIN
aweber/icons/flag-red.png
View file

Binary file not shown.
Side by side

Before

Width:  |  Height:  |  Size: 578 B

BIN
aweber/icons/flag-yellow.png
View file

Binary file not shown.
Side by side

Before

Width:  |  Height:  |  Size: 577 B

BIN
aweber/icons/resource.png
View file

Binary file not shown.
Side by side

Before

Width:  |  Height:  |  Size: 722 B

BIN
aweber/icons/resourcegroup.png
View file

Binary file not shown.
Side by side

Before

Width:  |  Height:  |  Size: 866 B

BIN
aweber/icons/task.png
View file

Binary file not shown.
Side by side

Before

Width:  |  Height:  |  Size: 494 B

Some files were not shown because too many files have changed in this diff Show more