:PROPERTIES: :ID: 11edd6c9-b976-403b-a419-b5542ddedaae :END: #+title: Subscriber Search Service #+LINK: jira https://jira.aweber.io/browse/ A replacement for the current [[id:d9cb2b55-3b0e-4ab3-8369-f71ebc3cd882][Sites Subscriber Search]] and [[id:f74e335d-577f-4749-bf32-1c025795b039][Broadcast Segment Search]] implementations. - [[https://jira.aweber.io/issues/?jql=project%20%3D%20CCPANEL%20AND%20component%20%3D%20%22Search%20Service%22][JIRA tickets]] * Architecture Notes #+begin_quote Added: [2019-08-06 Tue 10:34] - Broadcast sending manipulated queries before performing them to suit its needs - *Handle list exclusion in the search service?* ([[jira:CCPANEL-9557][CCPANEL-9557]]) - Blocked emails - De-duplication - *Lead view ids -> segment ids -> search service segments?* ([[jira:CCPANEL-9556][CCPANEL-9556]]) - Side-by-side comparison - Use broadcast-segment to compare its results to the search api results - *Recipient-style representation or leads?* - *Define API* ([[jira:CCPANEL-9554][CCPANEL-9554]]) - Include saved searches (segments) - *Iterate release* ([[jira:CCPANEL-9555][CCPANEL-9555]]) - Current UI - New UI with same capabilities of old UI - Still writing old-style segments - Research getting off the leads table (aurora?) - https://confluence.aweber.io/display/AR/Search+Service+Using+Existing+Databases - Distinct from a materialized search-optimized db - Deal with =subscriber_tags= table bloat in AppDB #+end_quote ** Component Diagram #+BEGIN_SRC plantuml :file search-components.svg database Analytics { database Ana as Ana01 database Ana as Ana02 database Ana as Ana03 } database App database "Results Cache" as ResultsCache component "Search Service" as Service { component Search component Results Search -- Analytics Search -- App Search --> ResultsCache Results <-- ResultsCache } #+END_SRC #+RESULTS: [[file:search-components.svg]] * Concerns ** Performance ** Fitness to Purpose This service will need to fulfill the needs of both end-user subscriber searches and segment emailing. * Plan :PROPERTIES: :COLUMNS: %40ITEM %Effort{:} :END: #+BEGIN: columnview :id local | ITEM | Effort | |--------------------------------------------------------+----------| | Plan | 18d 0:00 | | Search Centralization | 18d 0:00 | | Expose search inputs backed with the existing database | 2d | | Enable dblink on the search master database | 2d | | Create new unlogged search results table | 2d | | Define the search result format | 1d | | Perform search using new search DSL | 5d | | Perform search using legacy segment ID | 3d | | Manage segments using the existing database | 3d | | Migrate to an updated schema | | | Migrate to new search service | | | Create new subscriber management React application | | | Update broadcast-segment to use new search service | | | Milestone 3: Add new search features | | #+END: ** A Dedicated Service for Subscriber Search :PROPERTIES: :JIRA_ID: CCPANEL-10169 :END: Create a dedicated, publicly exposed service for performing searches on subscribers using subscriber and analytics criteria. The goal of this project is to replace the current implementation from sites and in the broadcast segment service. *** Expose search inputs backed with the existing database :PROPERTIES: :Effort: 2d :END: - Include IDs required to build existing POST format *** Enable dblink on the search master database :PROPERTIES: :Effort: 2d :END: [[jira:CCPANEL-7147][CCPANEL-7147]] *** NO Create new unlogged search results table :PROPERTIES: :Effort: 2d :END: https://jira.aweber.io/browse/CCPANEL-7077 *** Define the search result format :PROPERTIES: :Effort: 1d :JIRA_ID: CCPANEL-10440 :END: https://xd.adobe.com/view/ae8fb2b2-c039-4e88-8ade-ff2562a8c8cf-fbdc/screen/c03f09c7-187e-4a6f-8b8c-571d131daee1/ (ignore engagement column) - name - email - source - status - date added - last updated *** Perform search using new search DSL :PROPERTIES: :Effort: 5d :END: **** DONE Perform search using text comparisons - Is / Is Not - Contains / Does not contain - Starts with / Does not start with - Ends with / Does not end with **** DONE Perform search using numeric comparisons **** DONE Perform search using tag sets **** DONE Perform search using enumerated values **** Add support for remaining static AppDB filters - Date range fields - Remaining string / numeric fields **** Add support for custom field filters - Add all custom field columns as supported filters - Fetch the custom fields for the list and use them when building the list of available filters. **** Add support for Analytics filters - Clicks - Opens - Messages - Web pages (links) *** Perform search using legacy segment ID :PROPERTIES: :Effort: 3d :END: Provided with a legacy segment ID, execute a search using its stored parameters. *** Manage segments using the existing database :PROPERTIES: :Effort: 3d :END: Create, retrieve, update, and delete legacy segments using the new search DSL. *** Manage segments stored using the new DSL *** Migrate legacy segments to the new DSL *** Migrate to an updated schema *** Support filtering options used by broadcast-segment ** Centralizing Subscriber Search Applications making use of subscriber search will be updated to use the new dedicated service, eliminating multiple search implementations. *** Create new subscriber management React application *** Update broadcast-segment to use new search service ** Milestone 3: Add new search features * Implementation ** [[id:7b0f97f3-9037-4d05-9170-a478e97c8d1f][Modeling the new search DSL]] ** Constructing SQL queries programmatically ** Translating legacy segments ** Gathering results ** Reaching into Analytics * Resources - [[https://confluence.aweber.io/display/AR/PostgreSQL+Backed+Search][PostgreSQL Backed Search]] (Rejected ACP) - [[https://confluence.aweber.io/display/AR/Search+Proxy+Service][Search Proxy Service]] - +[[https://confluence.aweber.io/display/~robink/SoT+-+ElasticSearch+Next+Steps][SoT - ElasticSearch Next Steps]]+ - [[https://confluence.aweber.io/display/~robink/Alternative+Search+Proposal][Alternative Search Proposal]] - [[https://confluence.aweber.io/display/AR/Search+Service+Using+Existing+Databases][Search Service Using Existing Databases]] (Approved ACP) + [[https://confluence.aweber.io/display/~victorc/Search+Service+-+Proof+of+Concept+Findings][Search Service - Proof of Concept Findings]] (Benchmarks different approaches) + [[https://gitlab.aweber.io/CP/archive/victorc-search-prototype][Search Prototype]] - [[https://confluence.aweber.io/display/AR/Search+DSL+JSON+Schema][Search DSL JSON Schema]]