correlr/roam
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

updates

Browse source
This commit is contained in:
Correl Roush 2021-11-04 16:52:09 -04:00
parent 51dc784f84
commit 457de828f7
8 changed files with 402 additions and 12 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

18
20211104164651-input_data_properties.org Normal file
View file

@ -0,0 +1,18 @@
:PROPERTIES:
:ID: d6d1f003-5aaa-4a36-867c-02d43cdf2475
:END:
#+title: Input data properties
How could we describe data that we wish to [[id:9914d09e-99fe-46a6-95be-676c5b78ed90][validate]] and possibly [[id:2ba04972-f498-41c2-970e-a64c7f3f1c3b][sanitize]]
differently, and present different experiences to end users for providing that
data?
- What causes us to want to trim or prevent leading or trailing whitespace from
some inputs?
- What causes us to limit the characters a field accepts?
- What sort of data would we want to sanitize or otherwise alter /as/ we accept
it (e.g. accepting but also sanitizing HTML to be stored as trusted HTML)?
Is there a good language for describing these categories of data, can we better
understand what differentiates them, and use that knowledge to build more
consistent guidelines for UX and input handling?

3
aweber/20210323161636-moving_pages_out_of_sites.org
View file

@ -7,6 +7,9 @@
A [[id:db322997-ff5e-416a-8dc8-f29e6a4928c8][Technical Initiative]] to rewrite pages in the [[id:57ee2f00-9bcd-4e0f-8a77-ae1f2d4cda89][Control Panel]] still built in PHP.
* Pages in Sites
- Tracking document :: https://confluence.aweber.io/display/BETL/PHP+to+React+Page+Needs
** TODO Dashboard
*** Broadcasts
*** Lists

2
aweber/20210813142844-projects.org
View file

@ -64,7 +64,7 @@
- State "BACKLOG" from "TODO" [2021-10-20 Wed 15:53]
- State "TODO" from [2021-10-13 Wed 16:26]
:END:
* Eliminating Sites
* [[id:f633f967-11d2-432c-b5ff-ad842c88a51c][Decommissioning Sites]]
** [[id:3cc8bd09-dd02-4950-8c89-a737f92809fd][Tracking progress of moving pages out of Sites]]
** Creating a new Control Panel shell application
* TODO Create the [[id:11edd6c9-b976-403b-a419-b5542ddedaae][Subscriber Search Service]]

143
aweber/20210915133101-analytics_view_service.org
View file

@ -61,6 +61,29 @@
- Last 30 days
- 30 day range
#+caption: opens_all
#+begin_example
{
"2021-11-02T00:00:00Z": {
"broadcasts": 2499,
"followups": 2547,
"unique": 2923,
"total": 5046
},
"2021-11-03T00:00:00Z": {
"broadcasts": 25808,
"followups": 2430,
"unique": 24876,
"total": 28238
},
"2021-11-04T00:00:00Z": {
"broadcasts": 16733,
"followups": 1437,
"unique": 14780,
"total": 18170
}
}
#+end_example
*** Clicks
:PROPERTIES:
:Effort: 1d
@ -72,6 +95,29 @@
- Last 30 days
- 30 day range
#+caption: clicks_all
#+begin_example
{
"2021-11-02T00:00:00Z": {
"broadcasts": 105,
"followups": 137,
"unique": 130,
"total": 242
},
"2021-11-03T00:00:00Z": {
"broadcasts": 636,
"followups": 185,
"unique": 622,
"total": 821
},
"2021-11-04T00:00:00Z": {
"broadcasts": 480,
"followups": 109,
"unique": 426,
"total": 589
}
}
#+end_example
*** Sales
:PROPERTIES:
:Effort: 1d
@ -80,6 +126,72 @@
- Last 30 days
- 30 day range
- Currency
#+caption: sales_tracked_events
#+begin_example
[
{
"time": "2021-11-02 09:37:36-04",
"type": "followup",
"currency": "USD",
"revenue": "19.00",
"note": "",
"description": "Upgraded to Pro",
"source_url": "https://www.aweber.com/users/#upgraded",
"email": "team@harmoniamedia.com"
},
{
"time": "2021-11-02 09:37:37-04",
"type": "followup",
"currency": "USD",
"revenue": "19.00",
"note": "",
"description": "Upgraded to Pro",
"source_url": "https://www.aweber.com/users/#upgraded",
"email": "team@harmoniamedia.com"
},
{
"time": "2021-11-02 12:01:17-04",
"type": "followup",
"currency": "USD",
"revenue": "19.00",
"note": "",
"description": "Upgraded to Pro",
"source_url": "https://www.aweber.com/users/#upgraded",
"email": "giuliagiardino12@gmail.com"
},
{
"time": "2021-11-02 12:01:19-04",
"type": "followup",
"currency": "USD",
"revenue": "19.00",
"note": "",
"description": "Upgraded to Pro",
"source_url": "https://www.aweber.com/users/#upgraded",
"email": "giuliagiardino12@gmail.com"
},
{
"time": "2021-11-04 05:21:35-04",
"type": "broadcast",
"currency": "USD",
"revenue": "19.00",
"note": "",
"description": "Upgraded to Pro",
"source_url": "https://www.aweber.com/users/#upgraded",
"email": "jeremy@jeremy-quick.com"
},
{
"time": "2021-11-04 05:21:36-04",
"type": "broadcast",
"currency": "USD",
"revenue": "19.00",
"note": "",
"description": "Upgraded to Pro",
"source_url": "https://www.aweber.com/users/#upgraded",
"email": "jeremy@jeremy-quick.com"
}
]
#+end_example
*** Sales Totals
:PROPERTIES:
:Effort: 1d
@ -88,6 +200,32 @@
- Last 60 days
- 60 day range
#+caption: sales_tracked_summary
#+begin_example
{
"2021-11-02T00:00:00Z": {
"broadcast": 0,
"followup": 76,
"pageview": 76,
"ecommerce": 0,
"total": 76
},
"2021-11-03T00:00:00Z": {
"broadcast": 0,
"followup": 0,
"pageview": 0,
"ecommerce": 0,
"total": 0
},
"2021-11-04T00:00:00Z": {
"broadcast": 38,
"followup": 0,
"pageview": 38,
"ecommerce": 0,
"total": 38
}
}
#+end_example
*** Sale Currencies
:PROPERTIES:
:Effort: 1d
@ -95,7 +233,10 @@
- Time
- 60 day range
- All time
#+caption: sales_tracked_currencies
#+begin_example
["USD"]
#+end_example
*** Sale Events
:PROPERTIES:
:Effort: 1d

84
aweber/20210921090818-bulk_tagging_service.org
View file

@ -2,3 +2,87 @@
:ID: 321075e7-db53-4676-b785-7c77ed9d1150
:END:
#+title: Bulk Tagging Service
The Bulk Tagging service queues the application of tags to a set of subscribers
and tracks the overall progress of the operation. This allows customers to
select a large number of subscribers and apply tags to all of them in a single
operation, while also giving them visibility into how their tagging request is
progressing.
* Problems Solved
** Background processing of tagging operations
Tagging operations are assigned a bulk-tagging job with one task per subscriber
to add or remove the specified tags. These operations are performed
asynchronously by consumers such that the end-user need not wait for all
operations to be completed before moving on.
** Rate-limited application of tags
Consumers are configured to consume tasks no faster than a configured maximum
rate to control the load placed upon downstream services (e.g. rules engine,
etc.). The ideal rate is divided amongst the number of consumers available.
** Serial application of operations within an account
There is an expectation that operations a customer applies to subscribers will
be performed sequentially. To allow this while still allowing jobs for /other/
accounts to be worked upon simultaneously, jobs are divided into queues
deterministically using a consistent hash of their account IDs.
** Tracking overall job progress
The job itself may be in one of the following states, which is updated as tasks
are acted upon:
- Pending :: No tasks have yet been acted upon
- Processing :: Some, but not all, tasks have been acted upon
- Succeeded :: All tasks have completed successfully
- Failed :: All tasks have completed, but at least one task was not successful
** Tracking of individual task status
Any particular task, once acted upon, will be updated as having succeeded or
failed with a message explaining the issue.
** Jobs must survive queue failures
A guarantee of the service is such that if a job is successfully submitted to
the Bulk Tagging service, we will not lose it, and can take steps manually if
necessary to ensure its completion. To account for unexpected failures when
submitting tasks to a queue or consuming a task from the queue, the job request
is archived to S3. This archive contains sufficient information to requeue the
job for processing as it was requested without further input.
** Customer visibility into job progress
End users are able to, via the service API, fetch any and all jobs stored for
their account, as well as their associated tasks.
** Administrative visibility into job progress
Administrative (internal) users are able to, via the service API, fetch any and
all jobs stored for any or all accounts. This is used to populate dashboards for
insight into the progress of Bulk Tagging as a whole, and whether jobs from
different accounts are holding each other up.
Administrative users also have access to delete or requeue jobs.
** Prioritizing smaller jobs over larger jobs
Tasks for jobs affecting a number of subscribers lower than a defined threshold
are assigned higher priority, causing them to be processed /ahead/ of any other
ongoing jobs in their respective queues. This avoids leaving a customer waiting
long periods of time for quick operations, which they may not expect to be held
up by other jobs outside their control.
* Known Problems
** Jobs appearing "stuck"
Jobs may appear stuck (either failing to start processing, or failing to
complete processing).
*** Job has not yet begun processing
This is caused by one of two things. Either a job is stuck behind other jobs
(this or another account's job could be in front of it in the same queue), or
the tasks failed to be written successfully to the queue. In the former case,
the job will complete normally once the tasks ahead of it in the queue are
processed. In the latter case, the job will need to be requeued.
*** Job is stuck in an in-progress state
This is typically due to the overall job progress being out of sync with the
actual state of its consituent tasks ([[https://jira.aweber.io/browse/CCPANEL-8660][CCPANEL-8660]]).
The overall status of a job and the status of its individual tasks are stored in
separate Dynamo tables. Because Dynamo tables are independent, there is no way
to update an individual task and the job's overall status in a single, atomic
operation. This means they may (and do) at times get out of sync, if
infrequently. The indexes on the tables are not designed to make reconciliation
easy, nor does an automated reconciliation process exist.
This could be solved by adding indexes to allow rapid computation of the actual
state of all tasks in a job and creating a scheduled task to synchronize these
counts, or by changing the underlying storage to a relational database which
would make it easier to compute these states.
* Resources
- ACP :: https://confluence.aweber.io/display/AR/ACP+Bulk+Tagging
- Playbook :: https://confluence.aweber.io/display/AR/Bulk+Tagging+Service+Playbook

2
aweber/20211026141640-decommissioning_sites.org
View file

@ -10,7 +10,7 @@ modernization of our public-facing application stack.
The project will engage multiple teams to coordinate the following three
efforts:
* Moving applications out of Sites
* [[id:193f7c04-0a03-4870-90c8-2b5e3c4c92ce][Moving applications out of Sites]]
- Individual pages will be replaced with React applications using public APIs
- Static content will be moved to CDN hosting
- Independent applications will be broken out into separate services

20
aweber/20211103100927-updating_projects_using_tagbox.org
View file

@ -3,7 +3,7 @@
:END:
#+title: Updating projects using Tagbox
#+OPTIONS: prop:("JIRA_ID")
#+todo: TODO(t) INVESTIGATE(i) AWAITING-RELEASE(a) | DONE(d) NO-ACTION(n)
#+todo: TODO(t) INVESTIGATE(i) TESTING(s) AWAITING-RELEASE(a) | DONE(d) NO-ACTION(n)
The Tagbox component has been updated with a fix addressing an XSS security
vulnerability in its tag label auto-completion which needs to be propogated out
@ -21,17 +21,19 @@ Target versions:
* Projects
Parent ticket: [[https://jira.aweber.io/browse/CCPANEL-11762][CCPANEL-11762]]
** TODO Campaign Builder (Direct)
** TESTING Campaign Builder (Direct)
:PROPERTIES:
:JIRA_ID: CC-7550
:END:
- Tagbox :: 5.0.3
** TODO GoToWebinar Client (Direct)
** NO-ACTION GoToWebinar Client (Direct)
:PROPERTIES:
:JIRA_ID: INT-5508
:END:
- Tagbox :: 3.0.0
** TODO Subscriber Import (Direct)
This project does not use the Tagbox component.
** TESTING Subscriber Import (Direct)
:PROPERTIES:
:JIRA_ID: CCPANEL-11768
:END:
@ -39,28 +41,28 @@ Parent ticket: [[https://jira.aweber.io/browse/CCPANEL-11762][CCPANEL-11762]]
- AWeberUI :: 2
Uses ramda, etc.
** AWAITING-RELEASE List Automation Client (AWeberUI)
** TESTING List Automation Client (AWeberUI)
:PROPERTIES:
:JIRA_ID: CCPANEL-11769
:END:
- AWeberUI :: 8.3
** NO-ACTION User Management Client (AWeberUI)
This project does not use the Tagbox component.
** INVESTIGATE Landing Page Editor (AWeberUI)
** TESTING Landing Page Editor (AWeberUI)
:PROPERTIES:
:JIRA_ID: CC-7551
:END:
** TODO Draft Bin (Direct)
** TESTING Draft Bin (Direct)
:PROPERTIES:
:JIRA_ID: CC-7552
:END:
- Tagbox :: 5.0.0
** INVESTIGATE Subscribers Client (AWeber UI)
** TESTING Subscribers Client (AWeber UI)
:PROPERTIES:
:JIRA_ID: CCPANEL-11770
:END:
- AWeberUI :: 8.6.1
** TODO Webform Generator (Standalone file)
** TESTING Webform Generator (Standalone file)
:PROPERTIES:
:JIRA_ID: CC-7553
:END:

142
daily/2021-11-04.org Normal file
View file

@ -0,0 +1,142 @@
:PROPERTIES:
:ID: a20323e3-fc41-496c-8acb-cf62cdb3ba27
:END:
#+title: 2021-11-04
* Backend BOF
- Scott M
- Kevin V
- Josh E
- Pycares was failing to install due to a dependency (safety virtual
environment did not have wheel)
- Josh B
- Redis issue in sessions resuilting in rescheduling kubenetes node workers
- May need to change how we're running certain Redises in k8s; the old
workload took time to shut down, the new instance read old data, failures
ensued.
- We need some measure of fault tolerance / HA
- Reasonable way to run a 3-pod redis that would be durable to that kind of
failure?
- Be really clear about the use cases and failure cases of Redis or any
persistent store in k8s.
- Redis usually used as a cache or database. Applications should be able to
work without a cache. Databases should have durability.
- (Dave S) We're treating redis like a cache, but failing to consider
connection timeouts and failures (blocking connect). Lack of data didn't
affect the service, but lack of connectivity did.
- (Amber H) We have existing VMs, is that an option?
- (Josh B) It is. There are challenges with client behaviors using HA Redis
and handling failover. VMs aren't my first choice.
- (Gavin R) We should focus on figuring out a workable k8s solution first.
The underlying storage solution is in essence the same thing.
- (Dave S) Aioredis doesn't have direct support for sharding or replication
(needs an exception handling wrapper). Simliar with TRedis.
- (Gavin R) A fork of redis implements transparent clustering, which directs
the client to the correct instance. (https://keydb.dev/)
- (Josh B) Also has multi-master
- (Dave S) The trouble is writes result in a denial response, leaving it to
the client to find the correct instance it can write to
- (Josh B) HAProxy could be told to talk the Redis protocol to find the
primary and send traffic there for clients that can't handle the read-only
response well.
- (Dave S) In k8s even when running as a cache, Redis will occasionally (4-5
times past year) fail to write to disk when there's some confusion between
CEPH and the underlying mount
- (Josh B) This was a lower-level CEPH issue; a race condition based on our
specific configuration.
- I will take time to put a Redis pattern together
- Ihar H
- Recently got feedback updating one of the service pipelines to meet
standards, want to clarify expectations. We'd decided to split the stages
- (Gavin R) I believe you can accomplish what you want to accomplish without
creating so many explicit stages
- (Ihar H) You have to click on a pipeline stage to see whether one of the nested pipeline tasks failed, and further stages could be run without the prior one passing
- (Amber H / Gavin R) We do need to sometimes get deploys out even when
acceptance tests fail
- (Ihar H) Even if code is deployed to production, the pipeline will be marked
as failed in some cases
- (Gavin R) Your proposal is fine, we need to clarify best practices
documentation to cover all cases.
- (Dave S) Gitlab visualization has changed over time, we can see if
anything can be improved to make things visually clearer
- (Gavin R) Since we are using immutable image building, there's little
value in running unit and integration tests in staging
- (Dave S) ... they /shouldn't/
- (Gavin R) The prior method was tying stages to environments, which we
don't want to do, they are different things.
- (Ihar H) I will update the confluence page and follow up with Dave S
- (Amber H) I want to see more of the /why/ in that document
- Gavin R
- Update on experiences with psycopg3, fixed the bug I found, I've been using
it pretty heavily. Seems to work well. API is backwards compatible and has
also evolved. Idiomatic row factory usage is different. One of the cool
things is if you use dataclasses, you can use a dataclass row factory.
- (Amber H) We just need them and Pydantic to have a baby
- (Gavin R) That probably wouldn't be too hard to do
- (Andrew R) I really don't like what has to be done to pass a value to the
=IN= keyword. In Psycopg, you have to format the comma-separated value yourself.
- (Gavin R) You can specify an array
- (Andrew R) That works with ~= ANY (...)~, but is much less performant than
using =IN=.
- (Alex C) A newer version of Postgres may fix the performance issue
- Eric T
- David R
- Been doing a lot of front-end experimentation with the concept of module
federation to power out micro-frontend architecture for our react clients to
build an app shell to compose them into a single MFA. We should discuss soon
what it'd look like to deploy something like that. Grossly simplifying,
every MFE exposes as part of its build a =remoteEntry.js= file that the
shell application needs to be told as part of its configuration where that
lives so it can load the module when necessary. The route willi then use the
remote to on-demand load the module and its dependencies to load that
application. I believe we just need that remote to be built and available as
part of our deployment strategy. Will want to get the POC I've been working
on into staging to start getting it working outside of local development.
- (Gavin R) This is the effort to replace Sites as our application shell? (Yes)
- https://confluence.aweber.io/display/BETL/Decommissioning+Sites
- (Gavin R) The larger implication to backend is sessions and sites going
away and the front-end application using the Public API going forward.
- (Scott M) Something will need to fetch AWVars data to provide to the
applications, from the session service for now, as a dynamic service
hosting the shell application?
- (David R) It would just be static content / JavaScript, provided it is
able to discover the contained applications.
- (Gavin R) JS should be on aweber-static, the shell application should be
hosted on something TBD, and we'll need to bridge the transition between
using sessions for stored data to something else.
- (David R) I see the session service as a middle state (stepping stone)
to work towards no longer needing it.
- (Scott M) We should look into how it can get direct access to the session service
- (Gavin R) I don't think we want that to be a dependency. We need a
global state managed in a better way
- (Dave S) We should start putting thought into how we're using local
storage, etc. in a consistent way so data can be cached and not
re-fetched, avoiding having each service have its own data model.
- (Josh B) The shell app could provide a common interface to access
shared cache data.
- (David R) Let's make sure there isn't an assumption that the scout file and the
remote entry file are the same thing.
- Dave S
- Had something, will post it; Updating docs on aweber API and what's behind it.
- Updated endpoint map on
https://confluence.aweber.io/display/STD/api.aweber.com+Endpoint+Management
- If anybody really understands CORS, we need someone in the company that
does. We're having CORS failures with companies like FaceBook, etc. If
anyone has experience, please speak up in BoF or elsewhere.
- Cedric W
- Alex and I drafted an ACP for bulk actions, will post a link for review.
Will probably be more after tomorrow's meeting.
- https://confluence.aweber.io/display/AR/Bulk-Action+Consumers+ACP
- Arnela M
- Andrew R
- Pydantic's awesome!
- Amber H
- Working with the analytics ingestion service, it's pretty awesome for opens
and clicks reporting. Currently in the development staging
- Talking with Scott the other day about how to do some cross-team code
reviews and wanted to float the idea here on having other people on other
teams to commit to some time during a sprint to review code from other
teams.
- Alex C
- Correl R
- Perl stinks. Derefencing data structures from scalars with weird symbols
stinks.