:PROPERTIES:
:ID:       11edd6c9-b976-403b-a419-b5542ddedaae
:END:
#+title: Subscriber Search Service
#+LINK: jira https://jira.aweber.io/browse/

A replacement for the current [[id:d9cb2b55-3b0e-4ab3-8369-f71ebc3cd882][Sites Subscriber Search]] and [[id:f74e335d-577f-4749-bf32-1c025795b039][Broadcast Segment Search]] implementations.

- [[https://jira.aweber.io/issues/?jql=project%20%3D%20CCPANEL%20AND%20component%20%3D%20%22Search%20Service%22][JIRA tickets]]

* Architecture Notes
#+begin_quote
Added: [2019-08-06 Tue 10:34]

- Broadcast sending manipulated queries before performing them to suit
  its needs
  - *Handle list exclusion in the search service?* ([[jira:CCPANEL-9557][CCPANEL-9557]])
  - Blocked emails
  - De-duplication
- *Lead view ids -> segment ids -> search service segments?* ([[jira:CCPANEL-9556][CCPANEL-9556]])
- Side-by-side comparison
  - Use broadcast-segment to compare its results to the search api
    results
- *Recipient-style representation or leads?*
- *Define API* ([[jira:CCPANEL-9554][CCPANEL-9554]])
  - Include saved searches (segments)
- *Iterate release* ([[jira:CCPANEL-9555][CCPANEL-9555]])
  - Current UI
  - New UI with same capabilities of old UI
    - Still writing old-style segments
- Research getting off the leads table (aurora?)
  - https://confluence.aweber.io/display/AR/Search+Service+Using+Existing+Databases
  - Distinct from a materialized search-optimized db
- Deal with =subscriber_tags= table bloat in AppDB
#+end_quote
** Component Diagram
#+BEGIN_SRC plantuml :file search-components.svg
  database Analytics {
    database Ana as Ana01
    database Ana as Ana02
    database Ana as Ana03
  }
  database App
  database "Results Cache" as ResultsCache

  component "Search Service" as Service {

    component Search
    component Results

    Search -- Analytics
    Search -- App
    Search --> ResultsCache

    Results <-- ResultsCache

  }
#+END_SRC

#+RESULTS:
[[file:search-components.svg]]

* Concerns
** Performance
** Fitness to Purpose
This service will need to fulfill the needs of both end-user subscriber searches
and segment emailing.

* Plan
:PROPERTIES:
:COLUMNS:  %40ITEM %Effort{:}
:END:
#+BEGIN: columnview :id local
| ITEM                                                   |   Effort |
|--------------------------------------------------------+----------|
| Plan                                                   | 18d 0:00 |
| Search Centralization                                  | 18d 0:00 |
| Expose search inputs backed with the existing database |       2d |
| Enable dblink on the search master database            |       2d |
| Create new unlogged search results table               |       2d |
| Define the search result format                        |       1d |
| Perform search using new search DSL                    |       5d |
| Perform search using legacy segment ID                 |       3d |
| Manage segments using the existing database            |       3d |
| Migrate to an updated schema                           |          |
| Migrate to new search service                          |          |
| Create new subscriber management React application     |          |
| Update broadcast-segment to use new search service     |          |
| Milestone 3: Add new search features                   |          |
#+END:

** Search Centralization
*** Expose search inputs backed with the existing database
:PROPERTIES:
:Effort:   2d
:END:
- Include IDs required to build existing POST format

*** Enable dblink on the search master database
:PROPERTIES:
:Effort:   2d
:END:
[[jira:CCPANEL-7147][CCPANEL-7147]]

*** Create new unlogged search results table
:PROPERTIES:
:Effort:   2d
:END:
https://jira.aweber.io/browse/CCPANEL-7077

*** Define the search result format
:PROPERTIES:
:Effort:   1d
:END:
https://xd.adobe.com/view/ae8fb2b2-c039-4e88-8ade-ff2562a8c8cf-fbdc/screen/c03f09c7-187e-4a6f-8b8c-571d131daee1/ (ignore engagement column)

- name
- email
- source
- status
- date added
- last updated
*** Perform search using new search DSL
:PROPERTIES:
:Effort:   5d
:END:
*** Perform search using legacy segment ID
:PROPERTIES:
:Effort:   3d
:END:
- Include flags for broadcast segment-required filtering?
*** Manage segments using the existing database
:PROPERTIES:
:Effort:   3d
:END:
- Translate a DSL search into a legacy stored segment
** Migrate to an updated schema
** Migrate to new search service
*** Create new subscriber management React application
*** Update broadcast-segment to use new search service
** Milestone 3: Add new search features

* Resources
- [[https://confluence.aweber.io/display/AR/PostgreSQL+Backed+Search][PostgreSQL Backed Search]] (Rejected ACP)
- [[https://confluence.aweber.io/display/AR/Search+Proxy+Service][Search Proxy Service]]
- +[[https://confluence.aweber.io/display/~robink/SoT+-+ElasticSearch+Next+Steps][SoT - ElasticSearch Next Steps]]+
- [[https://confluence.aweber.io/display/~robink/Alternative+Search+Proposal][Alternative Search Proposal]]
- [[https://confluence.aweber.io/display/AR/Search+Service+Using+Existing+Databases][Search Service Using Existing Databases]] (Approved ACP)
  + [[https://confluence.aweber.io/display/~victorc/Search+Service+-+Proof+of+Concept+Findings][Search Service - Proof of Concept Findings]] (Benchmarks different approaches)
  + [[https://gitlab.aweber.io/CP/archive/victorc-search-prototype][Search Prototype]]
- [[https://confluence.aweber.io/display/AR/Search+DSL+JSON+Schema][Search DSL JSON Schema]]