FloCon 2022

Modern micro-services leverage native cloud capabilities to automatically manage workload deployment and scalability. In this model, end-to-end visibility becomes more complex, endpoint-centric, and difficult to execute, especially when deployment and upgrades are continuous. Endpoint monitors, therefore, need to adapt to unpredictable workloads without jeopardizing the performance and stability of the production environment.

Typically, these monitors collect system telemetry data, such as application logs or system calls. Collecting system call data is advantageous because it can provide a detailed view of each process running on a host. Unfortunately, this data is often too large to store and analyze in any meaningful way, forcing practitioners to write complex filtering rules to make the telemetry footprint more manageable. Such filters require constant tuning as new applications and updates are deployed, and often result in important contextual attack data being filtered out needlessly. Telemetry formats such as SysFlow make data collection more palatable by lifting raw system call information into a more semantic summarization of system behaviors. However, SysFlow does not maintain enough system state to further reduce away noisy, redundant process behaviors that can occlude security analyses and create long-term storage headaches.

In this talk, we describe a new system graph data model that encodes process behaviors into hierarchical summarized views of system activity and demonstrate how such data structure can be used to implement a self-modulating telemetry stream that adapts to the monitored environment, and drastically reduces event fatigue, optimizes storage, and provides important contextual information for security investigations.

Attendees Will Learn:
- Challenges in system telemetry for distributed cloud endpoints
- A data modeling approach to address event fatigue in system call monitoring
- Design principles for cloud-native observability