:PROPERTIES: :ID: 11edd6c9-b976-403b-a419-b5542ddedaae :END: #+title: Subscriber Search Service #+LINK: jira https://jira.aweber.io/browse/ A replacement for the current [[id:d9cb2b55-3b0e-4ab3-8369-f71ebc3cd882][Sites Subscriber Search]] and [[id:f74e335d-577f-4749-bf32-1c025795b039][Broadcast Segment Search]] implementations. - [[https://jira.aweber.io/issues/?jql=project%20%3D%20CCPANEL%20AND%20component%20%3D%20%22Search%20Service%22][JIRA tickets]] * Architecture Notes #+begin_quote Added: [2019-08-06 Tue 10:34] - Broadcast sending manipulated queries before performing them to suit its needs - *Handle list exclusion in the search service?* ([[jira:CCPANEL-9557][CCPANEL-9557]]) - Blocked emails - De-duplication - *Lead view ids -> segment ids -> search service segments?* ([[jira:CCPANEL-9556][CCPANEL-9556]]) - Side-by-side comparison - Use broadcast-segment to compare its results to the search api results - *Recipient-style representation or leads?* - *Define API* ([[jira:CCPANEL-9554][CCPANEL-9554]]) - Include saved searches (segments) - *Iterate release* ([[jira:CCPANEL-9555][CCPANEL-9555]]) - Current UI - New UI with same capabilities of old UI - Still writing old-style segments - Research getting off the leads table (aurora?) - https://confluence.aweber.io/display/AR/Search+Service+Using+Existing+Databases - Distinct from a materialized search-optimized db - Deal with =subscriber_tags= table bloat in AppDB #+end_quote ** Component Diagram #+BEGIN_SRC plantuml :file search-components.svg database Analytics { database Ana as Ana01 database Ana as Ana02 database Ana as Ana03 } database App database "Results Cache" as ResultsCache component "Search Service" as Service { component Search component Results Search -- Analytics Search -- App Search --> ResultsCache Results <-- ResultsCache } #+END_SRC #+RESULTS: [[file:search-components.svg]] * Concerns ** Performance ** Fitness to Purpose This service will need to fulfill the needs of both end-user subscriber searches and segment emailing. * Plan :PROPERTIES: :COLUMNS: %40ITEM %Effort{:} :END: #+BEGIN: columnview :id local | ITEM | Effort | |--------------------------------------------------------+----------| | Plan | 18d 0:00 | | Search Centralization | 18d 0:00 | | Expose search inputs backed with the existing database | 2d | | Enable dblink on the search master database | 2d | | Create new unlogged search results table | 2d | | Define the search result format | 1d | | Perform search using new search DSL | 5d | | Perform search using legacy segment ID | 3d | | Manage segments using the existing database | 3d | | Migrate to an updated schema | | | Migrate to new search service | | | Create new subscriber management React application | | | Update broadcast-segment to use new search service | | | Milestone 3: Add new search features | | #+END: ** A Dedicated Service for Subscriber Search :PROPERTIES: :JIRA_ID: CCPANEL-10169 :END: Create a dedicated, publicly exposed service for performing searches on subscribers using subscriber and analytics criteria. The goal of this project is to replace the current implementation from sites and in the broadcast segment service. *** Expose search inputs backed with the existing database :PROPERTIES: :Effort: 2d :END: - Include IDs required to build existing POST format *** Enable dblink on the search master database :PROPERTIES: :Effort: 2d :END: [[jira:CCPANEL-7147][CCPANEL-7147]] *** Create new unlogged search results table :PROPERTIES: :Effort: 2d :END: https://jira.aweber.io/browse/CCPANEL-7077 *** Define the search result format :PROPERTIES: :Effort: 1d :JIRA_ID: CCPANEL-10440 :END: https://xd.adobe.com/view/ae8fb2b2-c039-4e88-8ade-ff2562a8c8cf-fbdc/screen/c03f09c7-187e-4a6f-8b8c-571d131daee1/ (ignore engagement column) - name - email - source - status - date added - last updated *** Perform search using new search DSL :PROPERTIES: :Effort: 5d :END: **** Perform search using text comparisons - Is / Is Not - Contains / Does not contain - Starts with / Does not start with - Ends with / Does not end with **** Perform search using numeric comparisons **** Perform search using date ranges **** Perform search using message and link IDs **** Perform search using tag sets *** Perform search using legacy segment ID :PROPERTIES: :Effort: 3d :END: - Include flags for broadcast segment-required filtering? *** Manage segments using the existing database :PROPERTIES: :Effort: 3d :END: - Translate a DSL search into a legacy stored segment *** Migrate to an updated schema ** Centralizing Subscriber Search Applications making use of subscriber search will be updated to use the new dedicated service, eliminating multiple search implementations. *** Create new subscriber management React application *** Update broadcast-segment to use new search service ** Milestone 3: Add new search features * Resources - [[https://confluence.aweber.io/display/AR/PostgreSQL+Backed+Search][PostgreSQL Backed Search]] (Rejected ACP) - [[https://confluence.aweber.io/display/AR/Search+Proxy+Service][Search Proxy Service]] - +[[https://confluence.aweber.io/display/~robink/SoT+-+ElasticSearch+Next+Steps][SoT - ElasticSearch Next Steps]]+ - [[https://confluence.aweber.io/display/~robink/Alternative+Search+Proposal][Alternative Search Proposal]] - [[https://confluence.aweber.io/display/AR/Search+Service+Using+Existing+Databases][Search Service Using Existing Databases]] (Approved ACP) + [[https://confluence.aweber.io/display/~victorc/Search+Service+-+Proof+of+Concept+Findings][Search Service - Proof of Concept Findings]] (Benchmarks different approaches) + [[https://gitlab.aweber.io/CP/archive/victorc-search-prototype][Search Prototype]] - [[https://confluence.aweber.io/display/AR/Search+DSL+JSON+Schema][Search DSL JSON Schema]]