Stream lineage

Visualize how data flows between Kafka topics and the service accounts that produce or consume them.

Overview

Stream lineage shows the flow of data through your Kafka estate, mapping the producers, consumers, and topics involved in each stream. Use it to:

• Trace how a topic is being read and written across applications
• Identify which service accounts depend on a given topic
• Investigate the downstream impact of a topic or application change
• Watch live produce and consume throughput on Gateway clusters

Lineage is derived from the Kafka ACLs granted to each service account. If you use Self-service, those ACLs are generated from your ApplicationInstance and ApplicationInstancePermission resources (including approved Topic Access Requests), so the resulting lineage reflects the applications and permissions you’ve modeled in Console.

On Confluent Cloud or Confluent Platform clusters configured to use role bindings instead of ACLs, the edges of the lineage graph are populated from those role bindings. See the guides for migrating to role bindings on Confluent Cloud and Confluent Platform.

Lineage view entry point

Open Lineage from the Console sidebar to land on the entry point for the selected cluster. From here, choose which dimension of your platform’s data flow you want to explore:

• Application instance — view the flow built from Self-service data, with ApplicationInstance resources as the nodes
• Service account — view the flow built from the underlying Kafka authorization data (ACLs, or role bindings on Confluent Cloud and Confluent Platform), with service accounts as the nodes

The search bar at the top lets you jump straight to a known Application instance or service account without browsing.

Application instance view

The Application instance view shows how your Self-service applications connect to one another through the topics they produce to and consume from. Each card on the left lists an instance with its read and write counts; selecting one focuses the graph on that instance and its neighbours.

The Topics toggle in the top right controls how connections are rendered:

• On — every topic that links two applications is shown as a node on the graph, so you can see exactly which topics carry data between services
• Off — topics are hidden and the graph collapses to a compressed view showing only how applications are transitively connected through any shared topics

Switch to Data flow in the top right to see a left-to-right view of the same relationships: the topics an instance reads from on the left, the instance in the middle, and the topics it writes to on the right, alongside the service accounts producing the data.

Service account view

The Service account view shows the flow derived directly from Kafka authorization data, with each service account as a node and edges drawn from the ACLs (or role bindings) granting read or write access to a topic. This is the view to use when you want to reason about access at the authorization layer rather than the Self-service model — for example, to spot service accounts that have permissions outside of any registered application.

The Topics toggle and the Graph / Data flow switch behave the same as in the Application instance view.

The data flow view lays the same relationships out left to right, with the topics a service account reads from on the left and the topics it writes to on the right:

Live lineage

On Gateway clusters, the Live toggle at the top of the Lineage view overlays live throughput on the graph. Each connection shows its current rate in bytes per second, updated every few seconds: connections carrying more traffic are drawn thicker and animate in the direction the data flows, while connections with no traffic stay thin and grey. A badge on the canvas shows the state of the live feed: Live, Connecting or Reconnecting.

Live throughput works in the Graph and Data flow views and in the drill-down views. With the Topics toggle off, the collapsed application-to-application connections keep their topic counts but don’t show a rate, because traffic can’t be attributed to a single collapsed link.

To use live lineage:

• The cluster has to be a Gateway cluster. On other cluster types the Live toggle doesn’t appear.
• The observability Interceptor has to be deployed on Gateway. This is the same Interceptor that Chargeback uses for Gateway throughput, so if you’ve already set that up, live lineage needs no extra configuration.
• Rates are averaged over the Interceptor’s flush interval (five minutes by default), so set flushIntervalInSecond to a low value, such as 10 seconds, for numbers that respond quickly.

The live feed updates the numbers on the graph you’re viewing. New topics or connections appear the next time the graph loads.

Large graphs and drill-down views

For very large graphs, the view is trimmed to stay responsive: Console keeps the 5,000 most connected nodes and draws the most significant connections between them, with a banner telling you when the graph has been trimmed. To see everything around a specific resource, use the drill-down views: select an individual Application instance or service account from the sidebar list and click Expand to open a sub-graph that centers on that resource and shows only its transitive relationships to other services and topics.

The drill-down view focuses on the selected node, with read and write counts, the ACLs that produce each edge, and a HOPS control in the top right to widen or narrow how many relationship steps out from the node are included in the graph.

Performance impact

Console caches each cluster’s lineage graph in memory to keep the views responsive, which carries a small JVM heap cost. From our internal benchmarks, a cluster with a graph of around 15,000 nodes adds roughly 150 MB to the Console JVM heap while that cluster sits in the cache.

To keep the overhead bounded:

• Graphs are cached with a 15 minute TTL, after which Console rebuilds them on the next request
• At most three clusters’ graphs live in the cache at any one time; cycling to a fourth cluster evicts the least recently used graph

To disable Stream lineage entirely or change the refresh frequency, set the relevant CDK_STREAMLINEAGE_* options listed in the Console environment variables reference.

Related resources

• Service accounts and ACLs
• Self-service
• Deploy the observability Interceptor