:PROPERTIES:
:ID:       8f824b4a-65df-44a6-a9f9-d500e90cd70e
:END:
#+title: 2021-06-30
* CP Outage retro
- CP experienced a login DDOS resulting in an outage on [2021-06-25 Fri]
  + [[id:d17e934b-b340-4246-88f0-9b36527100c0][Login Throttling]] flagged most via Sift ID
- Ops BOF discussed Apache possibly permitting PHP processes more memory than
  the pod allows, resulting in them getting OOM-killed
- How much memory is the login endpoint using?
- ini set request body limit per path
- [ ] look into pod memory limits
- why was there so much cpu usage for a login attack?
- is there an opportunity to short circuit login attacks by IP?
  + could it trigger something in the F5?
  + could it be enhanced to look at CIDR blocks?
    - assume everything is a =/24=?
- Add an intermediary tool or service to handle throttling?
  + Put login behind Kong?
- Separate the login page and give it its own scaling rules?