Introducing the SignalFx Smart Agent: Enabling Auto-Discovery and Dynamic Configuration of Metrics for Any Environment

For several years, collectd has been the preferred metric collection agent for SignalFx. It is a great open-source tool with a relatively small footprint, that is very stable, flexible, and extensible. We love it, and we are continuing to extend its functionality to make it even easier for you to collect metrics from all your favorite services. Ease of use and speed to deployment are the inspiration behind our new Smart Agent. The SignalFx Smart Agent helps accelerate your time-to-value with 40+ of our existing integrations, a good portion of which include our Python based plug-ins such as Mongo, Redis, and Docker. And as always, we believe in ensuring no agent lock-in and treating custom metrics as a first-class citizen.

The Smart Agent

Initially developed as a Kubernetes integration, the SignalFx Smart Agent now supports metrics collection from any Linux environment, whether containerized or not. You’re probably wondering how this is different from our collectd agent, below we dive into the key differentiators. 

  • Seamless install: Installing the right collectd plug-in for each service takes significant time, and so does periodically updating them. To simplify installation and updates, the Smart Agent offers what we call seamless installation. This means all the common collectd plug-ins are bundled into a single package that’s easy to install, track, and upgrade. We’ll continue to expand which plug-ins are pre-bundled in the Smart Agent based on customer demand.
  • Dynamic configuration: We’re pushing the boundaries of what agents have traditionally been able to do by including an all-encompassing service discovery feature. This allows the agent to identify each of the services within your environment and automatically configure the appropriate integration plug-ins from within its bundle. This is particularly valuable in large ephemeral container environments that experience high-churn and dynamic service creation – as new services are automatically discovered, installed, and configured within minutes. However, we have extended this capability to non-containerized environments as well. 

Smart Agent Framework

The Smart Agent has two main sets of components: monitors and observers.


Monitors go out and gather metrics from the environment.  For host-level metrics, this can involve looking at virtual files exposed in the /proc filesystem on Linux or making syscalls such as statvfs.  For services and applications, this usually means hitting some network endpoint exposed by the application and converting that response to metrics.  

With the initial release of Smart Agent, the core set of monitors we offer are focused on collectd.  We’ve built a monitor framework that can easily accommodate both collectd and non-collectd monitors so that they share a universal interface with the rest of the agent. There are currently two non-collectd monitors; the kubernetes-cluster monitor that gets metrics pertaining to K8 clusters, and the cadvisor monitor for container-level metrics.  We will continue to add non-collectd monitors based on customer demand. The agents flexibility also makes it easy to build your own custom monitors. For example, adapting Telegraf plug-ins to work with the agent is easy since they are both written in Go.


Observers enable the Smart Agent’s automatic service discovery feature. They look at what is running in the surrounding environment and report any discovered endpoints to the agent, which are tested against the set of discovery rules associated with a particular monitor.  If a monitor is configured with a discovery rule that matches a particular endpoint discovered by an observer, it will be configured with that endpoint and begin monitoring that service. 

For example, the discovery rule "containerImage=~"mysql"&&port==3306" would match the main endpoint of a MySQL database server where metrics relevant to that database come from.

Our initial release offers observers for Kubernetes, Docker, and the host directly (which means we use kernel interfaces to determine what is running on the same machine as the agent).  New observers will be added based on customer demand.

What’s in Store for Smart Agent

Moving forward we plan to enable the capability to pre-aggregate metrics before being sent to SignalFx. Generally, we encourage users to let our backend do the heavy lifting of processing aggregations, but some customers prefer to limit DPM usage by pre-aggregating the data. If you’re interested in adding this functionality immediately, collectd has an aggregation plug-in that can be used for your existing collectd-based monitors. Once again, we will be adding plug-ins based on customer demand so be on the look out for those.

The Smart Agent is available today, so get started with a free trial. For more information please reference our GitHub repository for information on installation, and our FAQ for information on how to convert from our current collectd agent to the new Smart Agent.

Join our live weekly demo on cloud monitoring »

Ryan Powers

Posted by



Introducing the SignalFx Smart Agent: Enabling Auto-Discovery and Dynamic Configuration of Metrics for Any Environment

Show All Tags
Show Less Tags

Join the Discussion