Anomalia Machina 9 — Massively Scalable Anomaly Detection with Apache Kafka and Cassandra

Anomalia Machina - Poseidon stirs up a storm

Kubernetes Resource Management (Replicas, Pods, Nodes)

I ended my initial Kubernetes experiments (Anomalia Machina 7) by accidentally exploding my Kubernetes cluster — I overloaded the Kubernetes worker nodes with too many Pods. This time we’ll try and scale correctly. As we discovered, Kubernetes by default assumes pods require no resources, so will (try) to create an infinite number of Pods on a finite number of Nodes. Here’s the Kubernetes resource management documentation. In the kubernetes configuration file, you can specify the minimum (requests) and maximum (limits) resources a Pod requires to run in terms of CPU and/or memory resources. This is how Pods are scheduled to Nodes:

apiVersion: apps/v1
kind: Deployment
metadata:
name: consumer-deployment
labels:
app: consumer
spec:
replicas: 1
selector:
matchLabels:
app: consumer
template:
metadata:
labels:
app: consumer
spec:
containers:
- name: consumer
image: myimages/consumer:latest
ports:
- containerPort: 1235
resources:
requests:
cpu: "2000m"

Anomaly Detection at Scale

Checking a few events for anomalies is easy, but what happens when there are many events to check?

Anomalia Machina - Sheeps
Anomalia Machina - Odysseus nasty encounter with Polyphemus

Results: 18,000 Anomaly Checks per second

Finally, we can reveal some results. I spun up (small) production sized Cassandra (version 3.11.3) and Kafka (version 2.1.0) clusters in the same region as my AWS EKS cluster. Here are the cluster and the worker nodes and producer instance details:

kubectl scale deployment consumer-deployment --replicas=10
Anomalia Machina - Cassandra Cluster results
Anomalia Machina 9 - Kubernetes Cluster Throughput
Anomalia Machina - System events

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store