Brokers

The Brokers section shows each broker in the cluster: where clients reach it, how many partitions it leads and holds, how much data it stores, and which settings it runs with. Use it to spot a broker that carries more than its share of the load, to check a broker's log directories, and to change a broker's dynamic settings without restarting it.

To open it, connect the cluster and click Brokers under its name in the Kafka Clusters pane.

1 2 3 4 5 6 7 8

1. The cluster at a glance: how many brokers, which node is the active controller, and the cluster's Kafka version (below).
2. Partitions: replication health and the controller type.
3. The brokers list, one row per broker (below).
4. The ID of the selected broker. Click it to copy it to the clipboard.
5. The selected broker's details: Overview, Log Dirs and Settings. They stay greyed out until you select a broker.
6. Refresh All Brokers (see Refreshing).
7. Undock window: move the detail tabs into a window of their own (Split panes and detail tabs).
8. Duplicate Tab: open Brokers in a tab of its own, as was done here for Consumers (Storefront Staging (Consumers)), so you can keep it open while you work elsewhere on the cluster (Duplicating a tab).

The summary cards

The cards describe the whole cluster, as on the Cluster Overview, whose page explains Brokers, Active Controller, Version, URP and In-sync Replicas and what to do when they look wrong. Three cards are only here:

The brokers list

Each row is a broker. Click a column header to sort by it, and right-click a row for Copy Value, Copy Summary and Copy All Summaries (Copying to the clipboard). Select a row to see that broker's details underneath.

1 2 3 4 5

1. Disk Usage: how much data the broker stores, and how many partition replicas that is.
2. Replicas Skew: how far the broker's replica count is above the cluster average.
3. Leaders: how many partitions the broker leads.
4. Leaders Skew: how far the broker's leader count is above the cluster average.
5. Replicas: how many partition replicas the broker holds.

Columns

Leaders, replicas, skew and disk usage count every partition KafkIO can see, internal topics included (such as __consumer_offsets, with its 50 partitions).

Leader skew and replica skew

Every partition has one leader, and producers and consumers talk to the leader. A broker that leads more partitions than the others therefore handles more requests, and is the first to run short of CPU or network. Replicas are about storage and replication traffic: a broker that holds more replicas stores more data and copies more from the leaders. Ideally both are spread evenly. Skew tells you how uneven they are.

KafkIO shows each broker's skew as a percentage above the cluster average (without a % sign): its count, minus the average count per broker, divided by that average. In the screenshot above, 115 partitions over three brokers is about 38.3 leaders per broker, so brokers 1 and 3, with 44 leaders each, show a Leaders Skew of 14.8. A value of 0 means exactly average.

A few percent of skew is normal. When it's large:

• Leader skew often follows a broker restart: its partitions moved their leadership elsewhere and haven't moved back. Brokers do this themselves when auto.leader.rebalance.enable is on (the default, checked every 5 minutes); otherwise, run a preferred leader election with Kafka's kafka-leader-election tool.
• Replica skew comes from where partitions were placed, for example topics created while a broker was down, or brokers added later. Moving replicas takes a partition reassignment (kafka-reassign-partitions, or a tool such as Cruise Control).

KafkIO doesn't move leaders or replicas itself. Bear in mind that skew counts partitions, not bytes or requests: a broker with average counts can still be the busiest if it leads the busiest partitions. The question-mark pop-ups on the two column headers describe skew in terms of messages and data volume, but both figures count partitions.

Broker details: Overview and Log Dirs

Select a broker in the list to fill the tabs underneath. Drag the divider above them to give the tabs more room.

Overview tab

1 2 3 4 5

1. Overview, the tab that opens first.
2. Segment Size: the size of the data the broker stores, as in the Disk Usage column.
3. Segment Count: the number of partition replicas in its log directories. Like the count in Disk Usage, this counts replicas, not segment files, whatever its question-mark pop-up says.
4. Host: the host name the broker advertises, as in the list.
5. Port: the port it advertises.

Log Dirs tab

A broker stores its data in one or more log directories (its log.dirs setting, often one per disk). The Log Dirs tab lists them, so you can check that every disk is in use and none has failed.

1 2 3 4 5

1. Log Dirs, one row per log directory.
2. Name: the directory's path on the broker's machine.
3. Error: empty while the directory is healthy. If the broker has taken the directory offline (usually a failed or full disk), Kafka's error appears here, and the partitions stored in it are unavailable on this broker.
4. Topics: how many topics have at least one partition replica in the directory.
5. Partitions: how many partition replicas the directory holds.

For the amount of data, see Segment Size on the Overview tab (the total across all of the broker's directories). An empty tab means the broker didn't report its log directories, for the same reasons Disk Usage can show N/A.

Broker settings

The Settings tab lists the selected broker's configuration as the broker reports it: every setting, with its current value and where that value comes from. Brokers can differ, so check the broker you mean (its ID is next to the Brokers title).

1 2 3 4 5

1. Filter settings...: matches names and values, as plain text or, with the .* toggle on, as a regular expression (Filtering). Here a regular expression picks out a few settings by name.
2. Source: where the value comes from (table below).
3. Read Only: Yes for settings that can't be changed while the broker runs.
4. Search: look the setting up on the web, in your browser.
5. Edit: change the setting (below); greyed out for read-only settings.

Reading broker settings

When several sources set the same setting, the one higher in this table wins:

Read Only comes from the broker: No means Kafka accepts a new value at runtime (a dynamic setting), Yes means it only takes effect from server.properties after a restart. An empty Value means the setting has no value, or that it's sensitive (passwords, keys): Kafka never sends those to clients.

Right-click a row for Edit, Copy Value, Copy Summary and Copy All Summaries. If the table says No broker settings (maybe due to permissions)..., your user isn't allowed to describe the cluster's configuration (it needs DescribeConfigs on the cluster), or the service doesn't offer broker settings at all, as with Azure Event Hubs and Amazon MSK Serverless.

Updating dynamic settings

You can change any setting whose Read Only is No, on a running broker, without a restart. The change applies to the selected broker only.

1. Open the setting

   Select the broker, open Settings, find the setting and click its pencil in the Edit column (or right-click it and choose Edit). The Broker Setting Update dialog opens.

2. Enter the new value

   1 2 3 4

   1. Name, filled in from the row you clicked. You can type another setting's name here to change that one instead.
   2. Value: the new value, in the form Kafka expects (for sizes and times, a plain number of bytes or milliseconds).
   3. Update Setting sends the new value.
   4. Clear Setting removes it instead (below).

3. Update and confirm

   Click Update Setting and confirm with Yes. When the broker accepts the change, KafkIO says The setting was updated (Kafka reported no error)., waits a second and refreshes the brokers, and the setting's Source becomes DYNAMIC_BROKER_CONFIG. The dialog stays open for another change; click Close when you're done.

If the broker refuses, KafkIO shows Error updating broker settings. with Kafka's reason, for example Invalid value abc for configuration min.insync.replicas: Not a number of type INT, and nothing changes. Changing broker settings needs AlterConfigs permission on the cluster.

KafkIO sets a value for one broker at a time. To change a setting for the whole cluster, either update it on every broker in turn, or set a cluster-wide default with kafka-configs --bootstrap-server ... --entity-type brokers --entity-default --alter --add-config name=value; KafkIO then shows it as DYNAMIC_DEFAULT_BROKER_CONFIG.

Clearing a setting

Clear Setting, in the same dialog, removes the broker's own value after you confirm (Are you sure you want to clear this broker setting? Kafka will apply a default if cleared.). The broker then falls back to the next source down: the cluster-wide dynamic default if there is one, then server.properties, then Kafka's default. Clearing the log.cleaner.threads value set above, for example, takes it back to 1 from DEFAULT_CONFIG.

Refreshing

The Brokers section reads the brokers, their log directories and their settings when you open it, and doesn't update by itself after that. Click 6 Refresh All Brokers to read them again (KafkIO also does this after updating or clearing a setting).

The cards, and the leader, replica, skew and disk usage figures in the list, come from the statistics KafkIO collects for each connected cluster every 30 seconds (the same ones behind the Overview), so they can be up to half a minute old even right after a refresh. If the brokers can't be read, KafkIO shows Error retrieving brokers list. with the cause; see Troubleshooting Connections.

Related

• Cluster Overview
• Topic Overview, Consumers & Settings
• Troubleshooting Connections
• The KafkIO Workspace