updates

aweber/20210813142844-projects.org

@@ -42,7 +42,7 @@
 - State "TODO"       from              [2021-09-01 Wed 13:44]
 :END:
 * Tracking live vs dead / removed code branches in Sites
-* Search Service
+* [[id:11edd6c9-b976-403b-a419-b5542ddedaae][Subscriber Search Service]]
 * Analytics View Service
 * Replace CAPI Services
 ** List API

aweber/20210901164718-test.org

@@ -0,0 +1 @@
+#+title: Test

aweber/20210914142000-subscriber_search_service.org

@@ -0,0 +1,51 @@
+:PROPERTIES:
+:ID:       11edd6c9-b976-403b-a419-b5542ddedaae
+:END:
+#+title: Subscriber Search Service
+
+A replacement for the current [[id:d9cb2b55-3b0e-4ab3-8369-f71ebc3cd882][Sites Subscriber Search]] and [[id:f74e335d-577f-4749-bf32-1c025795b039][Broadcast Segment Search]] implementations.
+
+- [[file:~/org-aweber/worklog-archive.org::#2019-08-06-tuesday-discuss-subscriber-management-architecture][file:~/org-aweber/worklog-archive.org::#2019-08-06-tuesday-discuss-subscriber-management-architecture]]
+- https://jira.aweber.io/issues/?jql=project%20%3D%20CCPANEL%20AND%20component%20%3D%20%22Search%20Service%22
+
+* Concerns
+** Performance
+** Fitness to Purpose
+This service will need to fulfill the needs of both end-user subscriber searches
+and segment emailing.
+
+* Interim Solution
+** Component Diagram
+#+BEGIN_SRC plantuml :file search-components.svg
+  database Analytics {
+    database Ana as Ana01
+    database Ana as Ana02
+    database Ana as Ana03
+  }
+  database App
+  database "Results Cache" as ResultsCache
+
+  component "Search Service" as Service {
+
+    component Search
+    component Results
+
+    Search -- Analytics
+    Search -- App
+    Search --> ResultsCache
+
+    Results <-- ResultsCache
+
+  }
+#+END_SRC
+
+#+RESULTS:
+[[file:search-components.svg]]
+
+* Resources
+- [[https://confluence.aweber.io/display/AR/PostgreSQL+Backed+Search][PostgreSQL Backed Search]] (Rejected ACP)
+- [[https://confluence.aweber.io/display/AR/Search+Proxy+Service][Search Proxy Service]]
+- +[[https://confluence.aweber.io/display/~robink/SoT+-+ElasticSearch+Next+Steps][SoT - ElasticSearch Next Steps]]+
+- [[https://confluence.aweber.io/display/~robink/Alternative+Search+Proposal][Alternative Search Proposal]]
+- [[https://confluence.aweber.io/display/AR/Search+Service+Using+Existing+Databases][Search Service Using Existing Databases]] (Approved ACP)
+- [[https://confluence.aweber.io/display/AR/Search+DSL+JSON+Schema][Search DSL JSON Schema]]

aweber/20210914143542-sites_subscriber_search.org

@@ -0,0 +1,42 @@
+:PROPERTIES:
+:ID:       d9cb2b55-3b0e-4ab3-8369-f71ebc3cd882
+:END:
+#+title: Sites Subscriber Search
+
+* Sorting
+Added: [2020-04-14 Tue 13:34]
+
+The current sites search code includes the following functioning code for
+setting a sort order on a search based on form input:
+
+https://gitlab.aweber.io/CP/applications/sites/blob/52d1d944854554c5818ef9a46c8a12493599eb55/aweber_app/controllers/queries_controller.php#L386-402
+#+begin_src php :exports code :eval never
+  // Look up the column for the order by clause.  There are no SQL column name values passed publicly.
+  if (!empty($this->data['SearchOrder']['SearchInput'])){
+     $this->SearchInput->recursive = -1;
+     if ($time = $this->SearchMutex->lock($aId, '6')) {
+         $orderCol = $this->SearchInput->find(array('SearchInput.id' => $this->data['SearchOrder']['SearchInput']));
+         $this->SearchMutex->unlock($aId, '6', $time);
+     }
+     if (!empty($orderCol['SearchInput']['column'])){
+       //Case-insensitive text sorting.
+       // Lower text fields so that ordering is case insensitive. SearchInputs 5, 23, and 24 are actually integers, despite
+       // having a text search input. refs #3275
+       if($orderCol['SearchInput']['input_type'] == 'text' && !in_array($orderCol['SearchInput']['id'], array(5,23,24))) {
+         $orderCol['SearchInput']['column'] = 'lower('.$orderCol['SearchInput']['column'].')';
+       }
+        $this->data['SearchOrder']['column'] = $orderCol['SearchInput']['column'];
+     }
+  }
+#+end_src
+
+- It is saved with the segment
+- It is passed back to the front-end when loading a saved segment
+- The =SearchCriteria= class incorporates the selected ordering and column when
+  building its query for a targeted search database.
+- The broadcast segment service ignores the selected ordering, opting for its
+  own for deliverability reasons.
+- All search inputs NOT in the analytics database are available for sorting (https://gitlab.aweber.io/CP/applications/sites/blob/f7ea2e9431e3ed2e694730f6446b4b3828d7c8fe/aweber_app/views/helpers/search_form.php#L54-62).
+- Performance degrades with list size, likely due to memory constraints and
+  unindexed sort fields
+  (https://www.cybertec-postgresql.com/en/postgresql-improving-sort-performance/).

aweber/20210914144833-broadcast_segment_search.org

@@ -0,0 +1,7 @@
+:PROPERTIES:
+:ID:       f74e335d-577f-4749-bf32-1c025795b039
+:END:
+#+title: Broadcast Segment Search
+
+Performs a search using a stored segment and builds an iterable list of
+recipients for a broadcast email. Implemented in the service's [[https://gitlab.aweber.io/edeliv/Applications/broadcast-segment/-/blob/master/broadcastsegment/handlers.py#L357][BroadcastHandler]].

aweber/search-components.svg

@@ -0,0 +1,47 @@
+<?xml version="1.0" encoding="UTF-8" standalone="no"?><svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" contentScriptType="application/ecmascript" contentStyleType="text/css" height="375px" preserveAspectRatio="none" style="width:444px;height:375px;background:#FFFFFF;" version="1.1" viewBox="0 0 444 375" width="444px" zoomAndPan="magnify"><defs><filter height="300%" id="fcycqyl6g7owb" width="300%" x="-1" y="-1"><feGaussianBlur result="blurOut" stdDeviation="2.0"/><feColorMatrix in="blurOut" result="blurOut2" type="matrix" values="0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 .4 0"/><feOffset dx="4.0" dy="4.0" in="blurOut2" result="blurOut3"/><feBlend in="SourceGraphic" in2="blurOut3" mode="normal"/></filter></defs><g><!--MD5=[6629dc9890c0dd0cccadf803c7f40a36]
+cluster Analytics--><path d="M240,146 C240,136 338.5,136 338.5,136 C338.5,136 437,136 437,146 L437,358 C437,368 338.5,368 338.5,368 C338.5,368 240,368 240,358 L240,146 " fill="#FFFFFF" filter="url(#fcycqyl6g7owb)" style="stroke:#000000;stroke-width:1.5;"/><path d="M240,146 C240,156 338.5,156 338.5,156 C338.5,156 437,156 437,146 " fill="none" style="stroke:#000000;stroke-width:1.5;"/><text fill="#000000" font-family="sans-serif" font-size="14" font-weight="bold" lengthAdjust="spacing" textLength="66" x="305.5" y="171.5352">Analytics</text><!--MD5=[9dd75fd92da113d30380b27734fd2f01]
+cluster Service--><rect fill="#FFFFFF" filter="url(#fcycqyl6g7owb)" height="97" style="stroke:#000000;stroke-width:1.5;" width="243" x="7" y="7"/><rect fill="#FFFFFF" height="10" style="stroke:#000000;stroke-width:1.5;" width="15" x="230" y="12"/><rect fill="#FFFFFF" height="2" style="stroke:#000000;stroke-width:1.5;" width="4" x="228" y="14"/><rect fill="#FFFFFF" height="2" style="stroke:#000000;stroke-width:1.5;" width="4" x="228" y="18"/><text fill="#000000" font-family="sans-serif" font-size="14" font-weight="bold" lengthAdjust="spacing" textLength="101" x="78" y="33.5352">Search Service</text><!--MD5=[3e6ed1d6220ce02acacd6309d9871975]
+entity Ana01--><path d="M283.5,204 C283.5,194 307,194 307,194 C307,194 330.5,194 330.5,204 L330.5,229.4883 C330.5,239.4883 307,239.4883 307,239.4883 C307,239.4883 283.5,239.4883 283.5,229.4883 L283.5,204 " fill="#FEFECE" filter="url(#fcycqyl6g7owb)" style="stroke:#000000;stroke-width:1.5;"/><path d="M283.5,204 C283.5,214 307,214 307,214 C307,214 330.5,214 330.5,204 " fill="none" style="stroke:#000000;stroke-width:1.5;"/><text fill="#000000" font-family="sans-serif" font-size="14" lengthAdjust="spacing" textLength="27" x="293.5" y="231.5352">Ana</text><!--MD5=[aa71d2519cf930747497ce848711e8ea]
+entity Ana02--><path d="M365.5,204 C365.5,194 389,194 389,194 C389,194 412.5,194 412.5,204 L412.5,229.4883 C412.5,239.4883 389,239.4883 389,239.4883 C389,239.4883 365.5,239.4883 365.5,229.4883 L365.5,204 " fill="#FEFECE" filter="url(#fcycqyl6g7owb)" style="stroke:#000000;stroke-width:1.5;"/><path d="M365.5,204 C365.5,214 389,214 389,214 C389,214 412.5,214 412.5,204 " fill="none" style="stroke:#000000;stroke-width:1.5;"/><text fill="#000000" font-family="sans-serif" font-size="14" lengthAdjust="spacing" textLength="27" x="375.5" y="231.5352">Ana</text><!--MD5=[3e19a41a4d5464172e5bdab1838a7d86]
+entity Ana03--><path d="M283.5,309 C283.5,299 307,299 307,299 C307,299 330.5,299 330.5,309 L330.5,334.4883 C330.5,344.4883 307,344.4883 307,344.4883 C307,344.4883 283.5,344.4883 283.5,334.4883 L283.5,309 " fill="#FEFECE" filter="url(#fcycqyl6g7owb)" style="stroke:#000000;stroke-width:1.5;"/><path d="M283.5,309 C283.5,319 307,319 307,319 C307,319 330.5,319 330.5,309 " fill="none" style="stroke:#000000;stroke-width:1.5;"/><text fill="#000000" font-family="sans-serif" font-size="14" lengthAdjust="spacing" textLength="27" x="293.5" y="336.5352">Ana</text><!--MD5=[f08d99a5d954cab98c847de775e5956b]
+entity Search--><rect fill="#FEFECE" filter="url(#fcycqyl6g7owb)" height="46.4883" style="stroke:#A80036;stroke-width:1.5;" width="86" x="148" y="42"/><rect fill="#FEFECE" height="10" style="stroke:#A80036;stroke-width:1.5;" width="15" x="214" y="47"/><rect fill="#FEFECE" height="2" style="stroke:#A80036;stroke-width:1.5;" width="4" x="212" y="49"/><rect fill="#FEFECE" height="2" style="stroke:#A80036;stroke-width:1.5;" width="4" x="212" y="53"/><text fill="#000000" font-family="sans-serif" font-size="14" lengthAdjust="spacing" textLength="46" x="163" y="75.5352">Search</text><!--MD5=[9b8af6697c917a00185d52e9ead47c2e]
+entity Results--><rect fill="#FEFECE" filter="url(#fcycqyl6g7owb)" height="46.4883" style="stroke:#A80036;stroke-width:1.5;" width="89" x="23.5" y="42"/><rect fill="#FEFECE" height="10" style="stroke:#A80036;stroke-width:1.5;" width="15" x="92.5" y="47"/><rect fill="#FEFECE" height="2" style="stroke:#A80036;stroke-width:1.5;" width="4" x="90.5" y="49"/><rect fill="#FEFECE" height="2" style="stroke:#A80036;stroke-width:1.5;" width="4" x="90.5" y="53"/><text fill="#000000" font-family="sans-serif" font-size="14" lengthAdjust="spacing" textLength="49" x="38.5" y="75.5352">Results</text><!--MD5=[6e8f311b0aa23032184ffba0ef926f91]
+entity App--><path d="M165,204 C165,194 189,194 189,194 C189,194 213,194 213,204 L213,229.4883 C213,239.4883 189,239.4883 189,239.4883 C189,239.4883 165,239.4883 165,229.4883 L165,204 " fill="#FEFECE" filter="url(#fcycqyl6g7owb)" style="stroke:#000000;stroke-width:1.5;"/><path d="M165,204 C165,214 189,214 189,214 C189,214 213,214 213,204 " fill="none" style="stroke:#000000;stroke-width:1.5;"/><text fill="#000000" font-family="sans-serif" font-size="14" lengthAdjust="spacing" textLength="28" x="175" y="231.5352">App</text><!--MD5=[2b02850483498e4c1932461fd42eefc1]
+entity ResultsCache--><path d="M11.5,204 C11.5,194 69,194 69,194 C69,194 126.5,194 126.5,204 L126.5,229.4883 C126.5,239.4883 69,239.4883 69,239.4883 C69,239.4883 11.5,239.4883 11.5,229.4883 L11.5,204 " fill="#FEFECE" filter="url(#fcycqyl6g7owb)" style="stroke:#000000;stroke-width:1.5;"/><path d="M11.5,204 C11.5,214 69,214 69,214 C69,214 126.5,214 126.5,204 " fill="none" style="stroke:#000000;stroke-width:1.5;"/><text fill="#000000" font-family="sans-serif" font-size="14" lengthAdjust="spacing" textLength="95" x="21.5" y="231.5352">Results Cache</text><!--MD5=[ee7cec6c4bf92f69866868e8bd4a1fee]
+link Search to Analytics--><path d="M211.06,88.19 C218.41,97.43 226.11,108.64 231,120 C233.0663,124.795 234.8814,129.8291 236.476,134.974 C237.2733,137.5464 238.0154,140.1466 238.7062,142.7585 C239.0516,144.0644 239.3842,145.3733 239.7044,146.683 C239.7844,147.0105 239.8637,147.338 239.9422,147.6655 " fill="none" id="Search-Analytics" style="stroke:#A80036;stroke-width:1.0;"/><!--MD5=[f6c2da6e0062f1e8cda340dd7d0901ce]
+link Search to App--><path d="M190.7,88.06 C190.32,116.49 189.67,165.6 189.29,193.84 " fill="none" id="Search-App" style="stroke:#A80036;stroke-width:1.0;"/><!--MD5=[bb3be57bc03efa3271a0c82c99357d97]
+link Search to ResultsCache--><path d="M173,88.06 C150.89,115.15 113.44,161.04 90.03,189.73 " fill="none" id="Search-to-ResultsCache" style="stroke:#A80036;stroke-width:1.0;"/><polygon fill="#A80036" points="86.68,193.84,95.4598,189.3773,89.833,189.9594,89.2509,184.3326,86.68,193.84" style="stroke:#A80036;stroke-width:1.0;"/><!--MD5=[2957b18b021e11d45836d926c3fa7b01]
+reverse link Results to ResultsCache--><path d="M68.18,93.26 C68.38,122.17 68.68,167.26 68.86,193.84 " fill="none" id="Results-backto-ResultsCache" style="stroke:#A80036;stroke-width:1.0;"/><polygon fill="#A80036" points="68.15,88.06,64.2015,97.0827,68.1786,93.0599,72.2014,97.037,68.15,88.06" style="stroke:#A80036;stroke-width:1.0;"/><!--MD5=[d0b8168f66f151212937210d487ad831]
+link Ana01 to Ana02--><!--MD5=[5701837461d1fac35dc83f629dc4aea3]
+link Ana01 to Ana03--><!--MD5=[79c493abef169456e4e0dc8848d939c0]
+@startuml
+database Analytics {
+  database Ana as Ana01
+  database Ana as Ana02
+  database Ana as Ana03
+}
+database App
+database "Results Cache" as ResultsCache
+
+component "Search Service" as Service {
+
+  component Search
+  component Results
+
+  Search - - Analytics
+  Search - - App
+  Search - -> ResultsCache
+
+  Results <- - ResultsCache
+
+}
+@enduml
+
+PlantUML version 1.2021.10(Mon Aug 30 09:43:48 EDT 2021)
+(GPL source distribution)
+Java Runtime: Java(TM) SE Runtime Environment
+JVM: Java HotSpot(TM) 64-Bit Server VM
+Default Encoding: UTF-8
+Language: en
+Country: US
+--></g></svg>

daily/2021-09-03.org

@@ -0,0 +1,63 @@
+:PROPERTIES:
+:ID:       30330ca0-3a44-4b56-8bce-6f6cce9ab115
+:END:
+#+title: 2021-09-03
+
+* CP crashes
+Looking into [[id:57ee2f00-9bcd-4e0f-8a77-ae1f2d4cda89][Control Panel]] OOM issues in Kubernetes.
+
+Average of ~138mb memory usage per request.
+
+#+begin_example
+  ps aufx |grep apache | awk '{print "cat /proc/" $1 "/statm"}' | sh | grep -v open | awk '{print $0}'
+#+end_example
+
+Memory limit should account for the 256MB APC cache.
+
+#+CAPTION: Josh Benner doing math on likely requirements
+#+begin_quote
+rps = 32
+ram per worker = 150 MB
+minimum workers = rps => 32
+num workers = minimum workers * 2 => 64
+desired pod count = 8
+workers per pod = num workers / desired pod count => 8
+apc ram = 256 MB
+ram per pod = ram per worker * workers per pod + apc ram => 1,456 MB
+
+rps = 32
+ram per worker = 150 MB
+minimum workers = rps => 32
+num workers = minimum workers * 2 => 64
+desired pod count = 8
+workers per pod = num workers / desired pod count * 2 => 16
+apc ram = 256 MB
+ram per pod = ram per worker * workers per pod + apc ram => 2,656 MB
+
+Making assumptions on baseline requests per second and duration of request.
+
+active workers = 20 * 8 => 160
+apc ram = 256 MB
+ram per worker = 150 MB
+pod count = 8
+workers per pod = active workers / pod count => 20
+ram per pod = workers per pod * ram per worker => 3,000 MB
+
+active workers = 20 * 8 * 2 => 320
+apc ram = 256 MB
+ram per worker = 150 MB
+pod count = 20
+workers per pod = active workers / pod count => 16
+ram per pod = workers per pod * ram per worker => 2,400 MB
+#+end_quote
+
+Actions taken:
+- Made per-pod worker count configurable via consul (easier to change if needed)
+- Tuned pod worker counts and memory allocations based on memory usage metrics (helps avoid OOMKills)
+- Removed the k8s liveness probe (avoids k8s-kills when loaded)
+- Revised the k8s readiness probe to just check TCP socket availability (avoids removing pod from rotation during load)
+- Load-tested these changes in staging: confirmed that pods stay up under load, and scaling more readily mitigates pod
+- Monitoring added to draw attention to load-induced symptoms of control-panel pods.
+- Auto-scaling will be investigated next week as an additional load mitigation tool.
+
+(details: 32 workers per pod, 1Gi RAM per pod (for now), 8 pods)

daily/2021-09-14.org

@@ -0,0 +1,12 @@
+:PROPERTIES:
+:ID:       d6eaaf94-d72b-4e8e-b76c-318c1fab5ada
+:END:
+#+title: 2021-09-14
+* Grooming with Time Estimates
+Is this fundamentally any better than tracking velocity of arbitrary point
+completion? It seems it'd be worse, in that time-based estimates can give a
+false sense of actual time to completion.
+
+
+- How does time estimation affect sprint capacity expectations?
+- Will this blur the lines between estimate and commitment?

lessons_learned_avoiding_primitives_in_elm.org

@@ -0,0 +1,7 @@
+:PROPERTIES:
+:ID:       67c1db33-2a17-4bcb-8b15-1b5e108ff9f6
+:ROAM_REFS: https://thoughtbot.com/blog/lessons-learned-avoiding-primitives-in-elm
+:END:
+#+title: Lessons Learned: Avoiding Primitives in Elm
+
+Discusses using specialized wrapper types in Elm.