[opamp] This client needs extension points (SPIs)

[breedx-splk]

OpenTelemetry Java users or vendors who create their own distros are able to wire up this opamp client. This is very useful.

The challenge comes in because opamp is a pretty generic protocol. It doesn't have opinionated minutia about what config/file formats are supported, how servers and clients should behave, or what commands even do, etc. etc. As a result, we can build clients and connect them to the server, but the semantic actions resulting from the client operations are mostly up in the air.

I'd like to propose that we introduce some SPIs that allow common/generic extension points around specific OpAMP behaviors. I suspect there ought to be several more, but the first ones that come to mind for me are:

• Identity provider - maybe something that helps to provide or discriminate/filter what items should or should not be in the identifying/non-identifying attributes.
• Effective config provider - If the client is configured to report effective config, when the server asks for it, this SPI can allow component-specific configuration providers to yield their current settings.
• Command handlers - Yup, if the opamp server sends us a command, where should it go? This SPI would allow for extensible command handling based on some criteria...probably starting with the command name or whatever.

This should allow specific, custom, and yet still undetermined use cases to be satisfied.

[LikeTheSalad]

Hi @breedx-splk

The examples you provided sound good, but in terms of how we provide them, they seem to be configurable via the existing builder at first glance (the effective config is already one of those options). Do you have a use case that requires these config options to be provided via SPIs?

[breedx-splk]

The examples you provided sound good, but in terms of how we provide them, they seem to be configurable via the existing builder at first glance (the effective config is already one of those options). Do you have a use case that requires these config options to be provided via SPIs?

Right, well the builder is really responsible for creating the client and passing in the callback/handlers...but the rest of the implementation is left to the user. In many cases, an instrumentation agent (vendor distro) will be creating the opamp client. Much like the sdk, the agent has extension points that allow it to remain modular and to have well encapsulated components hooked up. You know how sdk autoconfigure has the ResourceProvider SPI right? That allows implementations to customize the resource as it's being built. I want to do something analogous to that, but around certain parts of the opamp client behavior.

Use case 1: AgentDescription customization

When the agent is sending its description to the server, it includes a set of identifying and non-identifying attributes. These attributes may not be known by the agent in advance, especially when agent extensions or a highly modular design is in play. If the agent provided an SPI, any extensions that wished to help customize the AgentDescription could do so, without the agent having to know those specific details.

Use case 2: Custom capabilities

The agent can report CustomCapabilities to the server to indicate additional capabilities that it supports. These capabilities could be provided through a library or agent extension that the agent creating the opamp client is unaware of. An SPI would allow these components to report their extra capabilities to the server.

Use case 3: Custom message handling

A server can send a custom message to the agent, and the agent can handle these. The handlers could live in a library or agent extension and might not be known in advance by the agent at the time it creates the opamp client. If the agent provided an SPI, then these extra components could hook in an provide custom message handling functionality.

Use case 4: Effective config customizations

When the agent reports its "effective configuration" to the server, it does so with a "file" and a content type. Some configuration may be encapsulated in libraries or read from an extension, an these would be unknown to the agent when it creates the opamp client. If we provided SPIs, these components could hook in and report their configuration in addition to whatever other config the agent might be providing. Alternatively, they could customize the config provided by the agent internals/core.

[LikeTheSalad]

Thanks for the detailed response, @breedx-splk. However, I think it mostly explains what SPIs are in general, rather than what use cases for OpAMP are needed right now that require using that mechanism. Also, it seems like we're talking about configuring features that aren't even available in this implementation yet, so I'm confused about what the request is about, as it seems like we're touching on many points here. I'm not saying you don't have a valid point, though, but at least right now, I don't understand it. So please bear with me while I further elaborate on what I mean below.

On the comparison to the SDK autoconfigure

You mentioned the OTel Java SDK autoconfigure feature as a reference for what you're looking for in OpAMP. From the autoconfigure readme:

This artifact implements environment-based autoconfiguration of the OpenTelemetry SDK. This can be an alternative to programmatic configuration using the normal SDK builders.

What I understand from it is that the autoconfigure feature is about creating an OTel instance without touching the SDK's code (at least not from the host application), a "zero-code" object creation option. It's probably worth mentioning that the configuration still has to exist in a programmatic way; the SPI is just a wrapper for the config. I think this makes sense for the OTel SDK, as it can work independently; however, in the case of an OpAMP client, so far, the use cases I'm aware of are always complementary, so the OpAMP client is always a tool used by an agent, and it's the agent's responsibility to create and use it. This makes sense because what you'll get from an OpAMP client are instructions/configs for an OTel agent.

Another way to look at it is that an SPI mechanism would be needed to control a component that's supposed to work on its own, detached from any other component by design. Which is not my understanding of how an OpAMP client is supposed to work.

What I've seen so far is:

What this issue seems to suggest:

So, based on the above, if we consider that we already added an SPI mechanism to the OpAMP client, I would have the following questions:

• What OTel agent would the OpAMP client object be attached to?
• If there's no OTel agent available at runtime, would the OpAMP client still run? What for?
• How would the OTel agent connect to the OpAMP client?

A potential answer to the last question could be "the agent will use SPIs to connect to the OpAMP client". But the thing about it is, to use SPIs, you still have to include OpAMP dependencies in the agent's code, so a follow-up question to that would be: Why can't the agent directly initialize its OpAMP client instance by using the builder, since it has to include the OpAMP dependency anyway? - The OTel Java Instrumentation agent can get away with it and not having to enforce dependencies on the host app thanks to the javaagent mechanism, which adds the agent's JAR at runtime, and my understanding is that the agent comes with all it needs to work, which might include its OpAMP client dependency.

On existing and missing configurations

A quick and up-to-date way to know which of the OpAMP features are currently supported by this implementation is to take a look at this enum. Right now, out of the configurations you mentioned, only Agent Description and Effective Config are supported. These are configurable via the OpAMP builder, and for the Effective Config one, you can provide an observable config provider here, which will allow consumers of the OpAMP client to let it know when the effective config has changed, allowing them to provide it on demand.

We can and should add support for the other OpAMP fields and provide configurations for them as well, though I think those should be addressed separately in their own issues, as they are feature additions.

How I see the OpAMP client and the OTel Agent

In the examples you provided, the word "agent" comes up a lot as the reason why an SPI is needed, for example:

When the agent is sending its description to the server, it includes a set of identifying and non-identifying attributes. These attributes may not be known by the agent in advance, especially when agent extensions or a highly modular design is in play. If the agent provided an SPI, any extensions that wished to help customize the AgentDescription could do so, without the agent having to know those specific details.

I agree with that statement; however, I believe my confusion stems from the use of "Agent" and "OpAMP client" across this issue as if they were the same thing, or as if the OpAMP client should be aware of the agent that uses it.

From the OpAMP spec introduction:

OpAMP allows Agents to report their status to and receive configuration from a Server

The way I see it is that the OpAMP client is a very simple tool that agents use to communicate with a server. A proper comparison for the OpAMP client would be an HTTP client tool, such as OkHttp, for example. Based on that, I find it very confusing to picture a configuration mechanism in which a client communication tool should allow libraries to provide parameters for its consumer, in this case, the OTel Agent. It's like asking OkHttp to provide SPIs so that any library can add custom headers to all of your app's HTTP requests.

To summarise. I do believe there is room for an SPI mechanism to address the concerns you mentioned above, is just that I find it difficult to understand why that would be the responsibility of a network protocol client. If an agent that uses an OpAMP client internally wants to enable third-party code to modify its OpAMP payloads, I believe that's an SPI that should be provided by the agent itself.

[breedx-splk]

Thanks @LikeTheSalad. So I think I hear what you're saying -- that there is room for SPI extension points, but they belong at a higher level of abstraction around the opamp client, rather than inside the opamp client. That's fair, and I think from a software design standpoint I'd agree with that.

I think a prototype might help to think about this stuff. Will have to try and make one sometime.

[jackshirazi]

Does this help