This commit is contained in:
Correl Roush 2021-09-14 23:50:54 -04:00
parent 9775046d26
commit f56869feed
9 changed files with 231 additions and 1 deletions

View file

@ -42,7 +42,7 @@
- State "TODO" from [2021-09-01 Wed 13:44]
:END:
* Tracking live vs dead / removed code branches in Sites
* Search Service
* [[id:11edd6c9-b976-403b-a419-b5542ddedaae][Subscriber Search Service]]
* Analytics View Service
* Replace CAPI Services
** List API

View file

@ -0,0 +1 @@
#+title: Test

View file

@ -0,0 +1,51 @@
:PROPERTIES:
:ID: 11edd6c9-b976-403b-a419-b5542ddedaae
:END:
#+title: Subscriber Search Service
A replacement for the current [[id:d9cb2b55-3b0e-4ab3-8369-f71ebc3cd882][Sites Subscriber Search]] and [[id:f74e335d-577f-4749-bf32-1c025795b039][Broadcast Segment Search]] implementations.
- [[file:~/org-aweber/worklog-archive.org::#2019-08-06-tuesday-discuss-subscriber-management-architecture][file:~/org-aweber/worklog-archive.org::#2019-08-06-tuesday-discuss-subscriber-management-architecture]]
- https://jira.aweber.io/issues/?jql=project%20%3D%20CCPANEL%20AND%20component%20%3D%20%22Search%20Service%22
* Concerns
** Performance
** Fitness to Purpose
This service will need to fulfill the needs of both end-user subscriber searches
and segment emailing.
* Interim Solution
** Component Diagram
#+BEGIN_SRC plantuml :file search-components.svg
database Analytics {
database Ana as Ana01
database Ana as Ana02
database Ana as Ana03
}
database App
database "Results Cache" as ResultsCache
component "Search Service" as Service {
component Search
component Results
Search -- Analytics
Search -- App
Search --> ResultsCache
Results <-- ResultsCache
}
#+END_SRC
#+RESULTS:
[[file:search-components.svg]]
* Resources
- [[https://confluence.aweber.io/display/AR/PostgreSQL+Backed+Search][PostgreSQL Backed Search]] (Rejected ACP)
- [[https://confluence.aweber.io/display/AR/Search+Proxy+Service][Search Proxy Service]]
- +[[https://confluence.aweber.io/display/~robink/SoT+-+ElasticSearch+Next+Steps][SoT - ElasticSearch Next Steps]]+
- [[https://confluence.aweber.io/display/~robink/Alternative+Search+Proposal][Alternative Search Proposal]]
- [[https://confluence.aweber.io/display/AR/Search+Service+Using+Existing+Databases][Search Service Using Existing Databases]] (Approved ACP)
- [[https://confluence.aweber.io/display/AR/Search+DSL+JSON+Schema][Search DSL JSON Schema]]

View file

@ -0,0 +1,42 @@
:PROPERTIES:
:ID: d9cb2b55-3b0e-4ab3-8369-f71ebc3cd882
:END:
#+title: Sites Subscriber Search
* Sorting
Added: [2020-04-14 Tue 13:34]
The current sites search code includes the following functioning code for
setting a sort order on a search based on form input:
https://gitlab.aweber.io/CP/applications/sites/blob/52d1d944854554c5818ef9a46c8a12493599eb55/aweber_app/controllers/queries_controller.php#L386-402
#+begin_src php :exports code :eval never
// Look up the column for the order by clause. There are no SQL column name values passed publicly.
if (!empty($this->data['SearchOrder']['SearchInput'])){
$this->SearchInput->recursive = -1;
if ($time = $this->SearchMutex->lock($aId, '6')) {
$orderCol = $this->SearchInput->find(array('SearchInput.id' => $this->data['SearchOrder']['SearchInput']));
$this->SearchMutex->unlock($aId, '6', $time);
}
if (!empty($orderCol['SearchInput']['column'])){
//Case-insensitive text sorting.
// Lower text fields so that ordering is case insensitive. SearchInputs 5, 23, and 24 are actually integers, despite
// having a text search input. refs #3275
if($orderCol['SearchInput']['input_type'] == 'text' && !in_array($orderCol['SearchInput']['id'], array(5,23,24))) {
$orderCol['SearchInput']['column'] = 'lower('.$orderCol['SearchInput']['column'].')';
}
$this->data['SearchOrder']['column'] = $orderCol['SearchInput']['column'];
}
}
#+end_src
- It is saved with the segment
- It is passed back to the front-end when loading a saved segment
- The =SearchCriteria= class incorporates the selected ordering and column when
building its query for a targeted search database.
- The broadcast segment service ignores the selected ordering, opting for its
own for deliverability reasons.
- All search inputs NOT in the analytics database are available for sorting (https://gitlab.aweber.io/CP/applications/sites/blob/f7ea2e9431e3ed2e694730f6446b4b3828d7c8fe/aweber_app/views/helpers/search_form.php#L54-62).
- Performance degrades with list size, likely due to memory constraints and
unindexed sort fields
(https://www.cybertec-postgresql.com/en/postgresql-improving-sort-performance/).

View file

@ -0,0 +1,7 @@
:PROPERTIES:
:ID: f74e335d-577f-4749-bf32-1c025795b039
:END:
#+title: Broadcast Segment Search
Performs a search using a stored segment and builds an iterable list of
recipients for a broadcast email. Implemented in the service's [[https://gitlab.aweber.io/edeliv/Applications/broadcast-segment/-/blob/master/broadcastsegment/handlers.py#L357][BroadcastHandler]].

View file

@ -0,0 +1,47 @@
<?xml version="1.0" encoding="UTF-8" standalone="no"?><svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" contentScriptType="application/ecmascript" contentStyleType="text/css" height="375px" preserveAspectRatio="none" style="width:444px;height:375px;background:#FFFFFF;" version="1.1" viewBox="0 0 444 375" width="444px" zoomAndPan="magnify"><defs><filter height="300%" id="fcycqyl6g7owb" width="300%" x="-1" y="-1"><feGaussianBlur result="blurOut" stdDeviation="2.0"/><feColorMatrix in="blurOut" result="blurOut2" type="matrix" values="0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 .4 0"/><feOffset dx="4.0" dy="4.0" in="blurOut2" result="blurOut3"/><feBlend in="SourceGraphic" in2="blurOut3" mode="normal"/></filter></defs><g><!--MD5=[6629dc9890c0dd0cccadf803c7f40a36]
cluster Analytics--><path d="M240,146 C240,136 338.5,136 338.5,136 C338.5,136 437,136 437,146 L437,358 C437,368 338.5,368 338.5,368 C338.5,368 240,368 240,358 L240,146 " fill="#FFFFFF" filter="url(#fcycqyl6g7owb)" style="stroke:#000000;stroke-width:1.5;"/><path d="M240,146 C240,156 338.5,156 338.5,156 C338.5,156 437,156 437,146 " fill="none" style="stroke:#000000;stroke-width:1.5;"/><text fill="#000000" font-family="sans-serif" font-size="14" font-weight="bold" lengthAdjust="spacing" textLength="66" x="305.5" y="171.5352">Analytics</text><!--MD5=[9dd75fd92da113d30380b27734fd2f01]
cluster Service--><rect fill="#FFFFFF" filter="url(#fcycqyl6g7owb)" height="97" style="stroke:#000000;stroke-width:1.5;" width="243" x="7" y="7"/><rect fill="#FFFFFF" height="10" style="stroke:#000000;stroke-width:1.5;" width="15" x="230" y="12"/><rect fill="#FFFFFF" height="2" style="stroke:#000000;stroke-width:1.5;" width="4" x="228" y="14"/><rect fill="#FFFFFF" height="2" style="stroke:#000000;stroke-width:1.5;" width="4" x="228" y="18"/><text fill="#000000" font-family="sans-serif" font-size="14" font-weight="bold" lengthAdjust="spacing" textLength="101" x="78" y="33.5352">Search Service</text><!--MD5=[3e6ed1d6220ce02acacd6309d9871975]
entity Ana01--><path d="M283.5,204 C283.5,194 307,194 307,194 C307,194 330.5,194 330.5,204 L330.5,229.4883 C330.5,239.4883 307,239.4883 307,239.4883 C307,239.4883 283.5,239.4883 283.5,229.4883 L283.5,204 " fill="#FEFECE" filter="url(#fcycqyl6g7owb)" style="stroke:#000000;stroke-width:1.5;"/><path d="M283.5,204 C283.5,214 307,214 307,214 C307,214 330.5,214 330.5,204 " fill="none" style="stroke:#000000;stroke-width:1.5;"/><text fill="#000000" font-family="sans-serif" font-size="14" lengthAdjust="spacing" textLength="27" x="293.5" y="231.5352">Ana</text><!--MD5=[aa71d2519cf930747497ce848711e8ea]
entity Ana02--><path d="M365.5,204 C365.5,194 389,194 389,194 C389,194 412.5,194 412.5,204 L412.5,229.4883 C412.5,239.4883 389,239.4883 389,239.4883 C389,239.4883 365.5,239.4883 365.5,229.4883 L365.5,204 " fill="#FEFECE" filter="url(#fcycqyl6g7owb)" style="stroke:#000000;stroke-width:1.5;"/><path d="M365.5,204 C365.5,214 389,214 389,214 C389,214 412.5,214 412.5,204 " fill="none" style="stroke:#000000;stroke-width:1.5;"/><text fill="#000000" font-family="sans-serif" font-size="14" lengthAdjust="spacing" textLength="27" x="375.5" y="231.5352">Ana</text><!--MD5=[3e19a41a4d5464172e5bdab1838a7d86]
entity Ana03--><path d="M283.5,309 C283.5,299 307,299 307,299 C307,299 330.5,299 330.5,309 L330.5,334.4883 C330.5,344.4883 307,344.4883 307,344.4883 C307,344.4883 283.5,344.4883 283.5,334.4883 L283.5,309 " fill="#FEFECE" filter="url(#fcycqyl6g7owb)" style="stroke:#000000;stroke-width:1.5;"/><path d="M283.5,309 C283.5,319 307,319 307,319 C307,319 330.5,319 330.5,309 " fill="none" style="stroke:#000000;stroke-width:1.5;"/><text fill="#000000" font-family="sans-serif" font-size="14" lengthAdjust="spacing" textLength="27" x="293.5" y="336.5352">Ana</text><!--MD5=[f08d99a5d954cab98c847de775e5956b]
entity Search--><rect fill="#FEFECE" filter="url(#fcycqyl6g7owb)" height="46.4883" style="stroke:#A80036;stroke-width:1.5;" width="86" x="148" y="42"/><rect fill="#FEFECE" height="10" style="stroke:#A80036;stroke-width:1.5;" width="15" x="214" y="47"/><rect fill="#FEFECE" height="2" style="stroke:#A80036;stroke-width:1.5;" width="4" x="212" y="49"/><rect fill="#FEFECE" height="2" style="stroke:#A80036;stroke-width:1.5;" width="4" x="212" y="53"/><text fill="#000000" font-family="sans-serif" font-size="14" lengthAdjust="spacing" textLength="46" x="163" y="75.5352">Search</text><!--MD5=[9b8af6697c917a00185d52e9ead47c2e]
entity Results--><rect fill="#FEFECE" filter="url(#fcycqyl6g7owb)" height="46.4883" style="stroke:#A80036;stroke-width:1.5;" width="89" x="23.5" y="42"/><rect fill="#FEFECE" height="10" style="stroke:#A80036;stroke-width:1.5;" width="15" x="92.5" y="47"/><rect fill="#FEFECE" height="2" style="stroke:#A80036;stroke-width:1.5;" width="4" x="90.5" y="49"/><rect fill="#FEFECE" height="2" style="stroke:#A80036;stroke-width:1.5;" width="4" x="90.5" y="53"/><text fill="#000000" font-family="sans-serif" font-size="14" lengthAdjust="spacing" textLength="49" x="38.5" y="75.5352">Results</text><!--MD5=[6e8f311b0aa23032184ffba0ef926f91]
entity App--><path d="M165,204 C165,194 189,194 189,194 C189,194 213,194 213,204 L213,229.4883 C213,239.4883 189,239.4883 189,239.4883 C189,239.4883 165,239.4883 165,229.4883 L165,204 " fill="#FEFECE" filter="url(#fcycqyl6g7owb)" style="stroke:#000000;stroke-width:1.5;"/><path d="M165,204 C165,214 189,214 189,214 C189,214 213,214 213,204 " fill="none" style="stroke:#000000;stroke-width:1.5;"/><text fill="#000000" font-family="sans-serif" font-size="14" lengthAdjust="spacing" textLength="28" x="175" y="231.5352">App</text><!--MD5=[2b02850483498e4c1932461fd42eefc1]
entity ResultsCache--><path d="M11.5,204 C11.5,194 69,194 69,194 C69,194 126.5,194 126.5,204 L126.5,229.4883 C126.5,239.4883 69,239.4883 69,239.4883 C69,239.4883 11.5,239.4883 11.5,229.4883 L11.5,204 " fill="#FEFECE" filter="url(#fcycqyl6g7owb)" style="stroke:#000000;stroke-width:1.5;"/><path d="M11.5,204 C11.5,214 69,214 69,214 C69,214 126.5,214 126.5,204 " fill="none" style="stroke:#000000;stroke-width:1.5;"/><text fill="#000000" font-family="sans-serif" font-size="14" lengthAdjust="spacing" textLength="95" x="21.5" y="231.5352">Results Cache</text><!--MD5=[ee7cec6c4bf92f69866868e8bd4a1fee]
link Search to Analytics--><path d="M211.06,88.19 C218.41,97.43 226.11,108.64 231,120 C233.0663,124.795 234.8814,129.8291 236.476,134.974 C237.2733,137.5464 238.0154,140.1466 238.7062,142.7585 C239.0516,144.0644 239.3842,145.3733 239.7044,146.683 C239.7844,147.0105 239.8637,147.338 239.9422,147.6655 " fill="none" id="Search-Analytics" style="stroke:#A80036;stroke-width:1.0;"/><!--MD5=[f6c2da6e0062f1e8cda340dd7d0901ce]
link Search to App--><path d="M190.7,88.06 C190.32,116.49 189.67,165.6 189.29,193.84 " fill="none" id="Search-App" style="stroke:#A80036;stroke-width:1.0;"/><!--MD5=[bb3be57bc03efa3271a0c82c99357d97]
link Search to ResultsCache--><path d="M173,88.06 C150.89,115.15 113.44,161.04 90.03,189.73 " fill="none" id="Search-to-ResultsCache" style="stroke:#A80036;stroke-width:1.0;"/><polygon fill="#A80036" points="86.68,193.84,95.4598,189.3773,89.833,189.9594,89.2509,184.3326,86.68,193.84" style="stroke:#A80036;stroke-width:1.0;"/><!--MD5=[2957b18b021e11d45836d926c3fa7b01]
reverse link Results to ResultsCache--><path d="M68.18,93.26 C68.38,122.17 68.68,167.26 68.86,193.84 " fill="none" id="Results-backto-ResultsCache" style="stroke:#A80036;stroke-width:1.0;"/><polygon fill="#A80036" points="68.15,88.06,64.2015,97.0827,68.1786,93.0599,72.2014,97.037,68.15,88.06" style="stroke:#A80036;stroke-width:1.0;"/><!--MD5=[d0b8168f66f151212937210d487ad831]
link Ana01 to Ana02--><!--MD5=[5701837461d1fac35dc83f629dc4aea3]
link Ana01 to Ana03--><!--MD5=[79c493abef169456e4e0dc8848d939c0]
@startuml
database Analytics {
database Ana as Ana01
database Ana as Ana02
database Ana as Ana03
}
database App
database "Results Cache" as ResultsCache
component "Search Service" as Service {
component Search
component Results
Search - - Analytics
Search - - App
Search - -> ResultsCache
Results <- - ResultsCache
}
@enduml
PlantUML version 1.2021.10(Mon Aug 30 09:43:48 EDT 2021)
(GPL source distribution)
Java Runtime: Java(TM) SE Runtime Environment
JVM: Java HotSpot(TM) 64-Bit Server VM
Default Encoding: UTF-8
Language: en
Country: US
--></g></svg>

After

Width:  |  Height:  |  Size: 8.2 KiB

63
daily/2021-09-03.org Normal file
View file

@ -0,0 +1,63 @@
:PROPERTIES:
:ID: 30330ca0-3a44-4b56-8bce-6f6cce9ab115
:END:
#+title: 2021-09-03
* CP crashes
Looking into [[id:57ee2f00-9bcd-4e0f-8a77-ae1f2d4cda89][Control Panel]] OOM issues in Kubernetes.
Average of ~138mb memory usage per request.
#+begin_example
ps aufx |grep apache | awk '{print "cat /proc/" $1 "/statm"}' | sh | grep -v open | awk '{print $0}'
#+end_example
Memory limit should account for the 256MB APC cache.
#+CAPTION: Josh Benner doing math on likely requirements
#+begin_quote
rps = 32
ram per worker = 150 MB
minimum workers = rps => 32
num workers = minimum workers * 2 => 64
desired pod count = 8
workers per pod = num workers / desired pod count => 8
apc ram = 256 MB
ram per pod = ram per worker * workers per pod + apc ram => 1,456 MB
rps = 32
ram per worker = 150 MB
minimum workers = rps => 32
num workers = minimum workers * 2 => 64
desired pod count = 8
workers per pod = num workers / desired pod count * 2 => 16
apc ram = 256 MB
ram per pod = ram per worker * workers per pod + apc ram => 2,656 MB
Making assumptions on baseline requests per second and duration of request.
active workers = 20 * 8 => 160
apc ram = 256 MB
ram per worker = 150 MB
pod count = 8
workers per pod = active workers / pod count => 20
ram per pod = workers per pod * ram per worker => 3,000 MB
active workers = 20 * 8 * 2 => 320
apc ram = 256 MB
ram per worker = 150 MB
pod count = 20
workers per pod = active workers / pod count => 16
ram per pod = workers per pod * ram per worker => 2,400 MB
#+end_quote
Actions taken:
- Made per-pod worker count configurable via consul (easier to change if needed)
- Tuned pod worker counts and memory allocations based on memory usage metrics (helps avoid OOMKills)
- Removed the k8s liveness probe (avoids k8s-kills when loaded)
- Revised the k8s readiness probe to just check TCP socket availability (avoids removing pod from rotation during load)
- Load-tested these changes in staging: confirmed that pods stay up under load, and scaling more readily mitigates pod
- Monitoring added to draw attention to load-induced symptoms of control-panel pods.
- Auto-scaling will be investigated next week as an additional load mitigation tool.
(details: 32 workers per pod, 1Gi RAM per pod (for now), 8 pods)

12
daily/2021-09-14.org Normal file
View file

@ -0,0 +1,12 @@
:PROPERTIES:
:ID: d6eaaf94-d72b-4e8e-b76c-318c1fab5ada
:END:
#+title: 2021-09-14
* Grooming with Time Estimates
Is this fundamentally any better than tracking velocity of arbitrary point
completion? It seems it'd be worse, in that time-based estimates can give a
false sense of actual time to completion.
- How does time estimation affect sprint capacity expectations?
- Will this blur the lines between estimate and commitment?

View file

@ -0,0 +1,7 @@
:PROPERTIES:
:ID: 67c1db33-2a17-4bcb-8b15-1b5e108ff9f6
:ROAM_REFS: https://thoughtbot.com/blog/lessons-learned-avoiding-primitives-in-elm
:END:
#+title: Lessons Learned: Avoiding Primitives in Elm
Discusses using specialized wrapper types in Elm.