correlr/roam
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

updates

Browse source
This commit is contained in:
Correl Roush 2021-10-26 00:56:58 -04:00
parent 38b362dbc8
commit 9a5564f3cb
6 changed files with 264 additions and 17 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

56
aweber/20210813142844-projects.org
View file

@ -3,16 +3,32 @@
:END:
#+title: Projects
#+STARTUP: indent logdrawer
#+COLUMNS: %TAGS %JIRA_ID %50ITEM %TODO %4StoryPoints{+} %COMPONENT %BLOCKER
#+PROPERTY: StoryPoints_ALL 0 1 2 3 5 8 13 20 40 100
#+COLUMNS: %50ITEM %JIRA_ID
#+PROPERTY: Effort_ALL 0:30 1:00 0.5d 1d 2d 3d 4d 1w
#+PROPERTY: ClassificationOfWork_ALL backend frontend ops product design
#+TODO: TODO(t!) BACKLOG(b!) RE-EVALUATE(r!) | DONE(d@!) CANCELLED(c@!)
#+TODO: BACKLOG(b!) TODO(t!) | DONE(d@!) CANCELLED(c@!)
#+TAGS: { SPRINT(S) EPIC(e) STORY(s) BUG(b) TASK(t) }
#+OPTIONS: num:nil toc:t arch:nil p:t prop:t
#+LINK: jira https://jira.aweber.io/browse/
* Priorities
#+BEGIN: columnview :id global :match "TODO=\"TODO\""
| ITEM | JIRA_ID |
|------------------------------------------------+---------------|
| Deploy GeoIP to Kubernetes | CCPANEL-11592 |
| Create the [[id:11edd6c9-b976-403b-a419-b5542ddedaae][Subscriber Search Service]] | |
| Create the [[id:c45881de-46f2-4f76-9579-063626c5956c][Analytics View Service]] | |
| [[id:619b6c78-7be9-4ee4-a0b7-9d1a4d7536e2][Migrating services to use the new List service]] | |
#+END:
* Service Upgrades
** TODO Deploy GeoIP to Kubernetes
:PROPERTIES:
:JIRA_ID: CCPANEL-11592
:END:
:LOGBOOK:
- State "TODO" from [2021-09-01 Wed 13:44]
:END:
** DONE [[id:9cfd85fd-998e-4f21-b82e-c7963576c202][Deploying S4 to Kubernetes]]
:PROPERTIES:
:JIRA_ID: CCPANEL-10549
@ -27,34 +43,39 @@
:LOGBOOK:
- State "TODO" from [2021-09-01 Wed 13:42]
:END:
** TODO GeoIP
:PROPERTIES:
:JIRA_ID: CCPANEL-11592
:END:
:LOGBOOK:
- State "TODO" from [2021-09-01 Wed 13:44]
:END:
** TODO Deploying Domain Validator to Kubernetes
** BACKLOG Deploying Domain Validator to Kubernetes
:PROPERTIES:
:JIRA_ID: CCPANEL-10554
:END:
:LOGBOOK:
- State "BACKLOG" from "TODO" [2021-10-20 Wed 15:53]
- State "TODO" from [2021-09-01 Wed 13:42]
:END:
** TODO Deploying Recipient Service to Kubernetes
** BACKLOG Deploying Recipient Service to Kubernetes
:LOGBOOK:
- State "BACKLOG" from "TODO" [2021-10-20 Wed 15:53]
- State "TODO" from [2021-10-13 Wed 16:26]
:END:
** TODO Deploying Tagging Service to Kubernetes
** BACKLOG Deploying Tagging Service to Kubernetes
:LOGBOOK:
- State "BACKLOG" from "TODO" [2021-10-20 Wed 15:53]
- State "TODO" from [2021-10-13 Wed 16:26]
:END:
* [[id:3cc8bd09-dd02-4950-8c89-a737f92809fd][Tracking progress of moving pages out of Sites]]
* [[id:11edd6c9-b976-403b-a419-b5542ddedaae][Subscriber Search Service]]
* [[id:c45881de-46f2-4f76-9579-063626c5956c][Analytics View Service]]
* TODO Create the [[id:11edd6c9-b976-403b-a419-b5542ddedaae][Subscriber Search Service]]
:LOGBOOK:
- State "TODO" from [2021-10-20 Wed 15:57]
:END:
* TODO Create the [[id:c45881de-46f2-4f76-9579-063626c5956c][Analytics View Service]]
:LOGBOOK:
- State "TODO" from [2021-10-20 Wed 15:57]
:END:
* [[id:4df15f2f-d2e1-40f4-8acd-dbfb78fe304f][Deploy CoreAPI to Kubernetes]]
* Replacing CAPI Services
** [[id:619b6c78-7be9-4ee4-a0b7-9d1a4d7536e2][Migrating services to use the new List service]]
** TODO [[id:619b6c78-7be9-4ee4-a0b7-9d1a4d7536e2][Migrating services to use the new List service]]
:LOGBOOK:
- State "TODO" from [2021-10-20 Wed 15:58]
:END:
*** DONE Set EOL date for awlists
- [2021-08-13 Fri 15:21] :: Discussed this. Also talked about separation of
concerns about account status vs list status. Also discussed how an
@ -66,11 +87,12 @@
** Subscribers API
* Frontend Client Upgrades
** Upgrade Dashboard to React
*** TODO Need an API for broadcasts and sent messages across lists
*** BACKLOG Create an API for broadcasts and sent messages across lists
:PROPERTIES:
:JIRA_ID: CCPANEL-11609
:END:
:LOGBOOK:
- State "BACKLOG" from "TODO" [2021-10-20 Wed 15:57]
- State "TODO" from [2021-09-01 Wed 13:33]
:END:
** Upgrade other non-React projects to React

10
aweber/20211019163438-migrating_services_to_use_the_new_list_service.org
View file

@ -2,3 +2,13 @@
:ID: 619b6c78-7be9-4ee4-a0b7-9d1a4d7536e2
:END:
#+title: Migrating services to use the new List service
- Parent ticket :: [[https://jira.aweber.io/browse/CCPANEL-11745][CCPANEL-11745]]
As part of our effort to deprecate the old Core API services and iterate towards
modern, domain-oriented APIs, the Control Panel team is deprecating usage of
AWLists in favor of a new List API.
AWLists is planned to be sunsetted at the end of Q2 2022. Applications and
services dependent upon AWLists must be migrated to use the new List API by that
time.

11
aweber/20211022094829-validation_and_sanitization_guidelines.org Normal file
View file

@ -0,0 +1,11 @@
:PROPERTIES:
:ID: 38457ac3-ba81-4727-9a65-5de22059c175
:END:
#+title: Validation and Sanitization Guidelines
- [[id:2ba04972-f498-41c2-970e-a64c7f3f1c3b][Data sanitization]]
- [[id:9914d09e-99fe-46a6-95be-676c5b78ed90][Input validation]]
- All content being displayed to a web browser MUST be appropriately sanitized
(unsafe characters should be escaped using their respective html entities)
-

27
daily/2021-10-20.org
View file

@ -2,6 +2,33 @@
:ID: 29e51b04-ce89-4934-b17f-1f64bffc2069
:END:
#+title: 2021-10-20
* [[id:bdea0611-e377-4378-a118-aef6d4a70bdf][CREASE]]
Started a morning session with Steve Eisner, whose notes are available at
https://confluence.aweber.io/pages/viewpage.action?pageId=155890548.
Working off a JIRA structure board of prioritized customer requests.
** Etsy imports
- Problematic since July
- We've got an MVP service for performing import requests and loading them up
for a rundeck job to process in chunks via the Public API
- Lots more customers than anticipated have Etsy accounts, sometimes with up to
10k purchases (which overloads the rate limiting)
- Steve has been busy manually requeuing jobs (using a Jupyter notebook), only
just got caught up recently (had been slowed down by a rate limit error
handling bug)
- Alex may work on an admin interface to streamline the import process
** CSLead tickets
- Most of the CP tickets have already been triaged
*** Customer not recieving test emails reliably
- Checking for sendtest events sent to [[id:e1b95d0e-366e-4ecf-b867-409b6b6c6ee8][Momentum]] in [[id:77ea54db-0c35-47ad-84b3-5c08ae5ac347][Redash]]
- Viewing the interaction in Fullstory
** Code Red - Custom Domains is down
[[id:bdea0611-e377-4378-a118-aef6d4a70bdf][CREASE]] Session interrupted by outage.
* [[id:0a1e48ec-e132-4ec4-81a1-124711330b5a][Manager one-on-one]]
- Discuss better ways of:
- Capturing new projects

31
daily/2021-10-22.org Normal file
View file

@ -0,0 +1,31 @@
:PROPERTIES:
:ID: 6df725cd-289d-4c22-b359-c9e4d4167468
:END:
#+title: 2021-10-22
* Huddle to discuss XSS validation in the new-list client
- Concerns [[id:05698e38-65b2-496c-b02b-1db376ae734c][Validation vs Sanitization]]
- Sanitize on input in the frontend? (Ignore / remove invalid characters as
data is being typed)
- Warn when invalid characters are present?
- This is preferable
- It should be expected that the backend will return an error if/when invalid
content is submitted
** How does this apply to tag entry?
- Tagging doesn't currently have an error state for invalid input that we can
leverage
** Action items
- Chris V will bring tag input validation to the product meeting on Tuesday ([[file:~/Nextcloud/org/aweber.org::*Follow up with Chris V on tagging input validation][TODO]])
- I will start a thread in the backend channel re: backend validation of
unwanted characters (vs sanitizing with [[https://github.com/mozilla/bleach][Bleach]]).
** Thread posting
Posting here for feedback / discussion:
While dealing with XSS vulnerabilities, besides updating HTML rendering to
properly escape data, we are taking the additional measure of disallowing
unwanted characters (specifically, =<= and =>=) in our back-end input
validation. I looked at the Bleach library, but that only makes sense to me in
cases where we want to accept, sanitize, and store HTML content.

146
daily/2021-10-25.org Normal file
View file

@ -0,0 +1,146 @@
:PROPERTIES:
:ID: 83d8f082-c376-407a-916c-19bbdf3b84e8
:END:
#+title: 2021-10-25
* Discussing Sites removal
Targetting the complete removal and archival of the sites repo / PHP codebase.
- Front-end
- Back-end
- Cron jobs
** [[id:193f7c04-0a03-4870-90c8-2b5e3c4c92ce][Moving pages out of Sites]]
*** Required APIs
- Prefer to update the existing public api vs creating additional services
*** Move static content out
*** Breaking apart separate applications/areas
**** addlead
**** webforms
**** control panel
**** static content
**** login
**** order
**** affiliate
**** public site (not static)
**** admin (not an issue)
**** analytics opens
**** analytics clicks
*** Integrations
** React Shell Application
- David R is researching "Federated components" in webpack to tie multiple
smaller react applications together
*** Move routing from the F5 to the shell application
- Use www as the base for loading the container HTML / routing?
- Jim will investigate where we'll go with this
** Session & Authentication
- Replace sessions with OAuth login
- Identify how to handle authentication and exposure of "hidden" scopes and
permissions.
- How do we manage different usage flows?
- Logged into same account in multiple tabs
- Different accounts in different tabs?
- Admin logging in as an account?
- Investigate abuse vectors
* Planning Recipient Service Migration
Migrating the recipient service into the production AWS cluster.
#+begin_quote
Eric Toner Today at 2:04 PM
@pse-team Does anyone know of any docs regarding the new AWS cluster? Planning to migrate recipient to the new cluster but I’m not sure what the differences are re: production cluster v. prod-us-east-1-ecs-1
20 replies
Edward Long:spiral_calendar_pad: 1 hour ago
(how long will this migration take, will there be any downtime?)
Gavin M Roy 1 hour ago
@Eric Toner I don't know that I documented differences. They are:
The VPC they reside in
The version of the AWS Linux distribution
The name
The EC2 instance types used
(edited)
:thankyou:
1
Gavin M Roy 1 hour ago
Tangentially, you should also be using the "newer" ECS base Images, which get rid of the com-aweber-config behavior.
https://gitlab.aweber.io/PSE/Docker/ecs
:+1:
2
Ryan Steele 1 hour ago
I tried looking for docs on standing up a new service in AWS, but the closest thing I could find in Confluence was https://confluence.aweber.io/display/STD/AWS+Due+Diligence. It seems more high-level in nature though, and the Setting up a New Service page it links to was last modified in 2015 and contains a fair number of broken links.
Ryan Steele 1 hour ago
Wouldn’t hurt to have a canonical source of truth for that info (in a similar vein as our CDE AWS docs).
Correl Roush (she/her) 1 hour ago
I had detailed notes on AWS setup in the bulk tagging playbook that I tend to reference back on, might split that into its own doc as that’s moving into k8s
Correl Roush (she/her) 1 hour ago
https://confluence.aweber.io/display/AR/Bulk+Tagging+Service+Playbook#BulkTaggingServicePlaybook-DeploymentChecklist
Gavin M Roy 1 hour ago
@Ryan Steele we are not adding new services to AWS
Gavin M Roy 1 hour ago
we are moving old services to the new cluster or to k8s
Gavin M Roy 1 hour ago
ideally ECS would be gone.
Gavin M Roy 1 hour ago
sans perhaps the Infrastructure cluster due to what it's done.
Ryan Steele 1 hour ago
Okay, that’s fine. But that documenting our current policy/recommendations would be preferable to having it buried in a Slack thread. (If that information exists already, my Jira search-fu is too weak to find it.) (edited)
Gavin M Roy 42 minutes ago
https://confluence.aweber.io/display/STD/Where+to+Provision+New+Projects
:docs-props:
1
Correl Roush (she/her) 30 minutes ago
@gavinr Is there any documentation on migrating from the old cluster to the new one? (e.g. what needs to get updated, how, and in what order to avoid downtime)
Correl Roush (she/her) 28 minutes ago
My gut says we’ll need to set up an elb, get the service running with it in the new cluster, then update route 53 and remove the old stuff.
Gavin M Roy 28 minutes ago
Not sure, not that I created, but I've walked @dangrosso and others through it multiple times. Am happy to do so with you and Eric when you're ready to do one.
High Level:
Using the "New" single stack load balancer cloud-formation template
New task definitions
Making sure the project uses the ecs deploy docker image in the CI pipleine
Deploying the new services to the new clusters
Moving DNS manually
Removing the old service definitions from the old clusters once migrated
Deleting the old Cloudformation stacks
:thankyou:
1
Gavin M Roy 27 minutes ago
The old clusters used one task definition across environments and promoted it on deploy. The new ones use a task-definition per environment and copy that on deployments for that environment only.
Correl Roush (she/her) 27 minutes ago
Ah, that’s good to know.
Gavin M Roy 26 minutes ago
Oh and the newer ECS images work exclusively against SSM Parameter store instead of the old com-aweber-config repos.
Gavin M Roy 26 minutes ago
(though that's independent of cluster)
#+end_quote