> ## Documentation Index > Fetch the complete documentation index at: https://docs.conduktor.io/llms.txt > Use this file to discover all available pages before exploring further. # Gateway metrics reference > Prometheus metrics exposed by Conduktor Gateway: connection, latency, throughput, thread, per-user and storage metrics. Use with Grafana, Cortex, Mimir. Gateway exposes metrics in the Prometheus format that you can scrape and send to your external log management system. The metrics endpoint is `:/metrics` of your deployed Gateway instance. For example: `localhost:8888/metrics`. If `GATEWAY_SECURED_METRICS` is enabled (which is the default), you'll have to use the credentials specified in `GATEWAY_ADMIN_API_USERS` to access the metrics. ## Prometheus metric suffixes When metrics are exported to Prometheus, suffixes are automatically appended based on the metric type: * **Counter**: `_total` suffix (e.g., `gateway_bytes_exchanged_total`) * **Timer (summary)**: `_seconds_count`, `_seconds_sum` and `_seconds_max` suffixes (e.g., `gateway_apiKeys_latency_request_response_seconds_count`) * **Timer (histogram)**: same as summary, plus `_seconds_bucket` with a `le` label for histogram buckets (e.g., `gateway_latency_request_response_seconds_bucket`). Only `gateway_latency_request_response` is exported as a histogram. * **Gauge**: no suffix (e.g., `gateway_active_connections_vcluster`) Metrics that declare a base unit also carry it as a suffix, which is why JVM memory appears as `jvm_memory_used_bytes` and garbage collection pauses as `jvm_gc_pause_seconds`. The table below lists the Prometheus metric names as they appear when scraped. Dots (`.`) in the underlying metric name are converted to underscores (`_`). | Prometheus metric name | Metric type | Labels | Description | | --------------------------------------------------------- | ----------- | ------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Upstream connection metrics** | | | | | `gateway_upstream_connection_close_rate_seconds_count` | summary | | The number of connections closed | | `gateway_upstream_connection_close_rate_seconds_sum` | summary | | Total time spent closing connections | | `gateway_upstream_connection_close_rate_seconds_max` | gauge | | Maximum time spent closing a connection | | `gateway_upstream_connection_close_total` | counter | | The total number of connections closed | | `gateway_upstream_connection_creation_rate_seconds_count` | summary | | The number of new connections established | | `gateway_upstream_connection_creation_rate_seconds_sum` | summary | | Total time spent creating connections | | `gateway_upstream_connection_creation_rate_seconds_max` | gauge | | Maximum time spent creating a connection | | `gateway_upstream_connection_creation_total` | counter | | The total number of new connections established | | `gateway_upstream_select_rate_seconds_count` | summary | | The number of times the I/O layer checked for new I/O to perform | | `gateway_upstream_select_rate_seconds_sum` | summary | | Total time spent on I/O select operations | | `gateway_upstream_select_rate_seconds_max` | gauge | | Maximum time spent on a single I/O select operation | | `gateway_upstream_select_total` | counter | | The total number of times the I/O layer checked for new I/O to perform | | `gateway_upstream_io_wait_rate_seconds_count` | summary | | The number of times the I/O thread spent waiting | | `gateway_upstream_io_wait_rate_seconds_sum` | summary | | Total time the I/O thread spent waiting | | `gateway_upstream_io_wait_rate_seconds_max` | gauge | | Maximum time the I/O thread spent waiting in a single wait | | `gateway_upstream_io_wait_total` | counter | | The total time the I/O thread spent waiting | | **Connection metrics** | | | | | `gateway_active_connections_vcluster` | gauge | `vcluster` | The number of active connections per Virtual Cluster | | `gateway_upstream_connections_upstream_connected` | gauge | `name`, `threadId` | The number of connections from Gateway to the backing Kafka cluster | | `gateway_upstream_io_nodes` | gauge | `clusterId` | The number of Kafka nodes | | `gateway_backend_brokered_active_connections` | gauge | | The number of active backend brokers on the MAIN cluster | | **Latency metrics** | | | | | `gateway_latency_request_response_seconds_count` | histogram | | The number of requests processed | | `gateway_latency_request_response_seconds_sum` | histogram | | The total round-trip latency from Gateway to Kafka and back | | `gateway_latency_request_response_seconds_max` | gauge | | The maximum round-trip latency from Gateway to Kafka and back | | `gateway_latency_request_response_seconds_bucket` | histogram | `le` | Histogram buckets for round-trip latency distribution | | `gateway_latency_request_response_seconds` | gauge | `quantile` | Round-trip latency percentiles per Gateway instance (p30, p50, p95, p99) | | `gateway_apiKeys_latency_request_response_seconds_count` | summary | `apiKeys` | The number of requests processed per API key | | `gateway_apiKeys_latency_request_response_seconds_sum` | summary | `apiKeys` | The total latency per API key | | `gateway_apiKeys_latency_request_response_seconds_max` | gauge | `apiKeys` | The maximum latency per API key | | `gateway_apiKeys_throttle_ms_seconds_count` | summary | `apiKeys` | The number of throttled responses per API key | | `gateway_apiKeys_throttle_ms_seconds_sum` | summary | `apiKeys` | The total throttle time in Kafka broker responses per API key. Reflects throttling imposed by Kafka (for example, due to quota violations). | | `gateway_apiKeys_throttle_ms_seconds_max` | gauge | `apiKeys` | The maximum throttle time in Kafka broker responses per API key | | **Throughput metrics** | | | | | `gateway_bytes_exchanged_total` | counter | `direction` | The total bytes exchanged between clients and Gateway. Tagged by direction: `upstream` (from clients toward Kafka) or `downstream` (from Kafka toward clients). | | `gateway_bytes_exchanged_vcluster_total` | counter | `vcluster`, `direction` | The total bytes exchanged between Gateway and Kafka brokers for a Virtual Cluster | | `gateway_bytes_exchanged_topic_total` | counter | `vcluster`, `direction`, `topic`, `apiKeys` | The total bytes exchanged within the context of a specific topic | | **Thread metrics** | | | | | `gateway_thread_request_received_total` | counter | `threadId`, `name` | Client requests received and forwarded to Kafka, per Gateway thread | | `gateway_thread_request_rebuild_total` | counter | `api_key`, `name`, `threadId`, `vcluster` | Requests rebuilt per Gateway thread | | `gateway_request_expired_total` | counter | `thread`, `apiKey`, `vcluster`\*, `username`\*, `clusterId`\*, `nodeId`\* | The number of client requests that timed out waiting for a response from Kafka. Labels marked with \* are conditional and only present when available. | | `gateway_thread_tasks` | gauge | `threadId`, `name` | The number of pending tasks on the Gateway thread (where all request/response rebuilding happens) | | **Per-user metrics** | | | | | `gateway_error_per_apiKeys_total` | counter | `vcluster`, `username`, `apiKeys` | The total Gateway processing exceptions per API key for the specified Virtual Cluster and username | | `gateway_current_inflight_apiKeys_total` | counter | `vcluster`, `username`, `apiKeys` | The current inflight API keys of the specified Virtual Cluster and username | | `gateway_failed_authentications_total` | counter | `vcluster`, `user`, `apiKeys` | The number of authentication attempts that failed for each user | | **Storage metrics** | | | | | `gateway_kcache_size` | gauge | `type` | The size of the kcache (equal to the number of key-value pairs in the cache) | | `gateway_topic_log_end_offset` | gauge | `vcluster`, `topic`, `partition` | The log end offset of logical topics | | `gateway_topic_current_offset` | gauge | `vcluster`, `group`, `topic`, `partition` | The current offset of a consumer group on a logical topic | | **License metrics** | | | | | `gateway_license_remaining_days` | gauge | | The number of days remaining on your Gateway license | The `gateway_topic_log_end_offset` and `gateway_topic_current_offset` metrics are currently only populated for topics using [topic concentration](/guide/conduktor-concepts/logical-topics). For non-concentrated topics, use standard Kafka consumer lag monitoring. ## JVM and system metrics The same endpoint also exposes the standard JVM, system and Netty metrics of the Gateway process. Use them to size your deployment, to watch memory and CPU headroom and to tell a Gateway problem apart from a Kafka problem. | Prometheus metric name | Metric type | Labels | Description | | ------------------------------------------- | ----------- | ------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Memory metrics** | | | | | `jvm_memory_used_bytes` | gauge | `area`, `id` | Memory currently used, per pool. `area` is `heap` or `nonheap`, `id` is the pool name (for example `G1 Old Gen` or `Metaspace`) | | `jvm_memory_committed_bytes` | gauge | `area`, `id` | Memory committed to the JVM by the operating system, per pool | | `jvm_memory_max_bytes` | gauge | `area`, `id` | Maximum memory the pool can reach. Returns `-1` when the pool has no defined maximum | | `jvm_buffer_memory_used_bytes` | gauge | `id` | Memory used by direct and mapped buffer pools | | `jvm_buffer_count_buffers` | gauge | `id` | Number of buffers in the pool | | `jvm_buffer_total_capacity_bytes` | gauge | `id` | Total capacity of the buffers in the pool | | **Garbage collection metrics** | | | | | `jvm_gc_pause_seconds_count` | summary | `action`, `cause`, `gc` | Number of garbage collection pauses | | `jvm_gc_pause_seconds_sum` | summary | `action`, `cause`, `gc` | Total time spent paused for garbage collection | | `jvm_gc_pause_seconds_max` | gauge | `action`, `cause`, `gc` | Longest single garbage collection pause in the current rolling window | | `jvm_gc_memory_allocated_bytes_total` | counter | | Bytes allocated in the young generation since Gateway started | | `jvm_gc_memory_promoted_bytes_total` | counter | | Bytes promoted from the young to the old generation | | `jvm_gc_live_data_size_bytes` | gauge | | Size of memory used in the long lived heap after its most recent reclamation | | `jvm_gc_max_data_size_bytes` | gauge | | Maximum size of the long-lived heap | | **CPU metrics** | | | | | `process_cpu_usage` | gauge | | Recent CPU usage of the Gateway process, as a fraction of the CPUs available to it | | `system_cpu_usage` | gauge | | Recent CPU usage of the container or host environment where Gateway runs | | `system_cpu_count` | gauge | | Number of processors available to the JVM | | `system_load_average_1m` | gauge | | System load average over the last minute | | **Thread and class metrics** | | | | | `jvm_threads_live_threads` | gauge | | Current number of live threads, daemon threads included | | `jvm_threads_daemon_threads` | gauge | | Current number of live daemon threads | | `jvm_threads_peak_threads` | gauge | | Peak number of live threads since the JVM started | | `jvm_threads_started_threads_total` | counter | | Total number of threads started since the JVM started | | `jvm_threads_states_threads` | gauge | `state` | Number of threads per state, such as `runnable`, `blocked` or `waiting` | | `jvm_classes_loaded_classes` | gauge | | Number of classes currently loaded | | `jvm_classes_unloaded_classes_total` | counter | | Number of classes unloaded since the JVM started | | **Process metrics** | | | | | `process_files_open_files` | gauge | | Number of open file descriptors. Compare it against `process_files_max_files`, because Gateway holds one descriptor per client and per upstream connection | | `process_files_max_files` | gauge | | File descriptor limit of the process | | **Netty metrics** | | | | | `netty_eventexecutor_tasks_pending` | gauge | `name` | Tasks queued on a Netty event loop. A growing queue means Gateway accepts work faster than it processes it | | `netty_allocator_memory_used` | gauge | `allocator_type`, `id`, `memory_type` | Memory the Netty allocator holds, split by `memory_type` (`heap` or `direct`) | | `netty_allocator_memory_pinned` | gauge | `allocator_type`, `id`, `memory_type` | Memory in use out of the memory the allocator holds | | `netty_allocator_pooled_arenas` | gauge | `allocator_type`, `id`, `memory_type` | Number of arenas in a pooled allocator | | `netty_allocator_pooled_cache_size` | gauge | `allocator_type`, `cache_type`, `id` | Buffer-cache capacity per thread and allocation size | | `netty_allocator_pooled_threadlocal_caches` | gauge | `allocator_type`, `id` | Number of thread-local caches of a pooled allocator | | `netty_allocator_pooled_chunk_size` | gauge | `allocator_type`, `id` | Chunk size of a pooled allocator | The `netty_allocator_pooled_*` metrics only appear when Gateway runs with the pooled allocator, which is the default. Setting [`GATEWAY_NETWORK_USE_POOLED_ALLOCATOR`](/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/environment-variables) to `false` removes them. The arena-count, cache-size and chunk-size metrics describe allocator configuration and remain constant for the allocator lifetime. Netty allocates most of its buffers off-heap, so the memory the Gateway container uses is larger than `jvm_memory_used_bytes`. This endpoint reports what the JVM sees, not what the container uses, so it doesn't show the resident memory of the container or CPU throttling. On Kubernetes, read `container_memory_working_set_bytes` and `container_cpu_cfs_throttled_seconds_total` from cAdvisor alongside these metrics. ## Related resources * [Set up alerts and notifications](/guide/monitor-brokers-apps/alerts) * [Deploy Cortex](/guide/conduktor-in-production/deploy-artifacts/deploy-cortex) * [Give us feedback/request a feature](https://conduktor.io/roadmap)