roam/aweber/20210914142000-subscriber_search_service.org

:PROPERTIES:
:ID:       11edd6c9-b976-403b-a419-b5542ddedaae
:END:
#+title: Subscriber Search Service
#+LINK: jira https://jira.aweber.io/browse/

A replacement for the current [[id:d9cb2b55-3b0e-4ab3-8369-f71ebc3cd882][Sites Subscriber Search]] and [[id:f74e335d-577f-4749-bf32-1c025795b039][Broadcast Segment Search]] implementations.

- [[https://jira.aweber.io/issues/?jql=project%20%3D%20CCPANEL%20AND%20component%20%3D%20%22Search%20Service%22][JIRA tickets]]

* Architecture Notes
#+begin_quote
Added: [2019-08-06 Tue 10:34]

- Broadcast sending manipulated queries before performing them to suit
  its needs
  - *Handle list exclusion in the search service?* ([[jira:CCPANEL-9557][CCPANEL-9557]])
  - Blocked emails
  - De-duplication
- *Lead view ids -> segment ids -> search service segments?* ([[jira:CCPANEL-9556][CCPANEL-9556]])
- Side-by-side comparison
  - Use broadcast-segment to compare its results to the search api
    results
- *Recipient-style representation or leads?*
- *Define API* ([[jira:CCPANEL-9554][CCPANEL-9554]])
  - Include saved searches (segments)
- *Iterate release* ([[jira:CCPANEL-9555][CCPANEL-9555]])
  - Current UI
  - New UI with same capabilities of old UI
    - Still writing old-style segments
- Research getting off the leads table (aurora?)
  - https://confluence.aweber.io/display/AR/Search+Service+Using+Existing+Databases
  - Distinct from a materialized search-optimized db
- Deal with =subscriber_tags= table bloat in AppDB
#+end_quote
** Component Diagram
#+BEGIN_SRC plantuml :file search-components.svg
  database Analytics {
    database Ana as Ana01
    database Ana as Ana02
    database Ana as Ana03
  }
  database App
  database "Results Cache" as ResultsCache

  component "Search Service" as Service {

    component Search
    component Results

    Search -- Analytics
    Search -- App
    Search --> ResultsCache

    Results <-- ResultsCache

  }
#+END_SRC

#+RESULTS:
[[file:search-components.svg]]

* Concerns
** Performance
Email delivery has cases

*** Analytics query speedups for broadcasts
https://jira.aweber.io/browse/EDELIV-8318
** Fitness to Purpose
This service will need to fulfill the needs of both end-user subscriber searches
and segment emailing.

* Plan
:PROPERTIES:
:COLUMNS:  %40ITEM %Effort{:}
:END:
#+BEGIN: columnview :id local
| ITEM                                                   |   Effort |
|--------------------------------------------------------+----------|
| Plan                                                   | 18d 0:00 |
| Search Centralization                                  | 18d 0:00 |
| Expose search inputs backed with the existing database |       2d |
| Enable dblink on the search master database            |       2d |
| Create new unlogged search results table               |       2d |
| Define the search result format                        |       1d |
| Perform search using new search DSL                    |       5d |
| Perform search using legacy segment ID                 |       3d |
| Manage segments using the existing database            |       3d |
| Migrate to an updated schema                           |          |
| Migrate to new search service                          |          |
| Create new subscriber management React application     |          |
| Update broadcast-segment to use new search service     |          |
| Milestone 3: Add new search features                   |          |
#+END:

** A Dedicated Service for Subscriber Search
:PROPERTIES:
:JIRA_ID:  CCPANEL-10169
:END:
Create a dedicated, publicly exposed service for performing searches on
subscribers using subscriber and analytics criteria. The goal of this project is
to replace the current implementation from sites and in the broadcast segment
service.

*** Expose search inputs backed with the existing database
:PROPERTIES:
:Effort:   2d
:END:
- Include IDs required to build existing POST format

*** Enable dblink on the search master database
:PROPERTIES:
:Effort:   2d
:END:
[[jira:CCPANEL-7147][CCPANEL-7147]]

*** NO Create new unlogged search results table
:PROPERTIES:
:Effort:   2d
:END:
https://jira.aweber.io/browse/CCPANEL-7077

*** Define the search result format
:PROPERTIES:
:Effort:   1d
:JIRA_ID:  CCPANEL-10440
:END:
https://xd.adobe.com/view/ae8fb2b2-c039-4e88-8ade-ff2562a8c8cf-fbdc/screen/c03f09c7-187e-4a6f-8b8c-571d131daee1/ (ignore engagement column)

- name
- email
- source
- status
- date added
- last updated
*** Perform search using new search DSL
:PROPERTIES:
:Effort:   5d
:END:
**** DONE Perform search using text comparisons
- Is / Is Not
- Contains / Does not contain
- Starts with / Does not start with
- Ends with / Does not end with
**** DONE Perform search using numeric comparisons
**** DONE Perform search using tag sets
**** DONE Perform search using enumerated values
**** Add support for remaining static AppDB filters
- Date range fields
- Remaining string / numeric fields
**** Add support for custom field filters
- Add all custom field columns as supported filters
- Fetch the custom fields for the list and use them when building the list of
  available filters.
**** Add support for Analytics filters
- Clicks
- Opens
- Messages
- Web pages (links)
*** Perform search using legacy segment ID
:PROPERTIES:
:Effort:   3d
:END:

Provided with a legacy segment ID, execute a search using its stored parameters.
*** Manage segments using the existing database
:PROPERTIES:
:Effort:   3d
:END:
Create, retrieve, update, and delete legacy segments using the new search DSL.
*** Manage segments stored using the new DSL
*** Migrate legacy segments to the new DSL
*** Migrate to an updated schema
*** Support filtering options used by broadcast-segment
** Centralizing Subscriber Search
Applications making use of subscriber search will be updated to use the new
dedicated service, eliminating multiple search implementations.
*** Create new subscriber management React application
*** Update broadcast-segment to use new search service
** Milestone 3: Add new search features

* Implementation

** [[id:7b0f97f3-9037-4d05-9170-a478e97c8d1f][Modeling the new search DSL]]

** Constructing SQL queries programmatically

** Translating legacy segments

** Gathering results

** Reaching into Analytics

* Resources
- [[https://confluence.aweber.io/display/AR/PostgreSQL+Backed+Search][PostgreSQL Backed Search]] (Rejected ACP)
- [[https://confluence.aweber.io/display/AR/Search+Proxy+Service][Search Proxy Service]]
- +[[https://confluence.aweber.io/display/~robink/SoT+-+ElasticSearch+Next+Steps][SoT - ElasticSearch Next Steps]]+
- [[https://confluence.aweber.io/display/~robink/Alternative+Search+Proposal][Alternative Search Proposal]]
- [[https://confluence.aweber.io/display/AR/Search+Service+Using+Existing+Databases][Search Service Using Existing Databases]] (Approved ACP)
  + [[https://confluence.aweber.io/display/~victorc/Search+Service+-+Proof+of+Concept+Findings][Search Service - Proof of Concept Findings]] (Benchmarks different approaches)
  + [[https://gitlab.aweber.io/CP/archive/victorc-search-prototype][Search Prototype]]
- [[https://confluence.aweber.io/display/AR/Search+DSL+JSON+Schema][Search DSL JSON Schema]]