Tail Based Sampling Framework for Distributed Tracing Using Stream Processing

Abstract: In recent years, microservice architecture has surpassed monolithic architecture in popularity among developers by providing a flexible way of developing complex distributed applications. Whereas a monolithic application functions as a single indivisible unit, a microservices-based application comprises a collection of loosely coupled services that communicate with each other to fulfill the requirements of the application. Consequently, different services in a microservices-based application can be developed and deployed independently. However, this flexibility is achieved at the expense of reduced observability of microservices-based applications complicating the debugging of such applications. The reduction of observability can be compensated by performing distributed tracing in microservices-based applications. Distributed tracing refers to observing requests propagating through a distributed system to collect observability data that can aid in understanding the interactions among the services and pinpoint failures and performance issues in the system. Open- Telemetry, an open-source observability framework supported by Cloud Native Computing Foundation (CNCF), defines a standardized specification for generating observability data. Nevertheless, instrumenting an application with an observability framework incurs performance overhead. To tackle this deterioration of performance and to reduce the cost of persisting observability data, only a subset of the requests are typically traced by performing head-based or tail-based sampling. In this work, we present a tail-based sampling framework using stream processing techniques. The developed framework demonstrated promising performance in our experiments by saving approximately a third of memory-based storage compared to an OpenTelemetry tail-based sampling module. Moreover, being compliant with the OpenTelemetry specifications, our framework aligns well with the OpenTelemetry ecosystem.