2021-06-30

CP Outage retro

CP experienced a login DDOS resulting in an outage on [2021-06-25 Fri]
- Login Throttling flagged most via Sift ID
Ops BOF discussed Apache possibly permitting PHP processes more memory than the pod allows, resulting in them getting OOM-killed
How much memory is the login endpoint using?
ini set request body limit per path
look into pod memory limits
why was there so much cpu usage for a login attack?
is there an opportunity to short circuit login attacks by IP?
- could it trigger something in the F5?
- could it be enhanced to look at CIDR blocks?
  - assume everything is a /24?
Add an intermediary tool or service to handle throttling?
- Put login behind Kong?
Separate the login page and give it its own scaling rules?