KafkIO | Glossary

The Kafka terms these docs use, and the names KafkIO gives its own features. Terms are grouped by area and sorted alphabetically within each group; where another page explains a term in detail, the definition links to it. Names in code are Kafka settings or values as Kafka spells them.

Cluster and brokers

Active controller: The broker (or KRaft controller) currently in charge of the cluster's metadata: which brokers are alive, which replica leads each partition. The Active Controller card shows its ID; -1 means KafkIO couldn't get the cluster's metadata (Cluster Overview).
Advertised listener: The address a broker tells clients to use once they've connected to a bootstrap server (advertised.listeners). It must be reachable from your machine, which is the usual problem with Kafka in Docker (Listeners and advertised listeners).
Bootstrap servers: The host:port of one or more brokers that KafkIO contacts first, to learn about the rest of the cluster (Bootstrap servers).
Broker: One Kafka server. A cluster is a set of brokers that share the work of storing partitions and answering producers and consumers (Brokers).
Cluster: A set of brokers that together hold a group of topics. In KafkIO, each cluster you add has its own configuration, tab and colour.
Controller: The part of Kafka that manages the cluster's metadata and elects partition leaders. A cluster has one active controller at a time. The Controller Type card on Brokers says whether it's KRaft or ZooKeeper.
Dynamic setting: A broker setting that can be changed on a running broker, without a restart: those whose Read Only column says No (Updating dynamic settings).
ISR (in-sync replicas): The replicas of a partition that are fully caught up with its leader, the leader included. Only an in-sync replica can take over as leader without losing messages. The In-sync Replicas card counts them over the whole cluster (Cluster Overview).
KRaft: Kafka's built-in way of managing cluster metadata with a quorum of controllers, which replaced ZooKeeper. New clusters use KRaft; Kafka 4.0 and later support nothing else.
Leader: The one replica of a partition that producers write to and consumers read from; the other replicas copy it. The Leader ID column shows which broker holds it, and N/A means the partition has no leader and can't be used (The partitions table).
Leader skew: How unevenly partition leadership is spread over the brokers. A broker that leads more partitions handles more requests. KafkIO shows each broker's skew as a percentage above the cluster average, and N/A below the average or on clusters with fewer than 50 partitions (Leader skew and replica skew).
Listener: A port a broker accepts connections on, with its own security type. A cluster often has several, for example plaintext on 9092 and SASL on 9094 (Choosing a security type).
Log directory: A folder on a broker's disk where it stores partition data (log.dirs), often one per disk. The Log Dirs tab lists them.
Min in-sync replicas: min.insync.replicas: how many in-sync replicas must have a message before Kafka accepts it from a producer that waits for all replicas (acks=all). Below that, those producers' writes fail (Min in-sync replicas).
Offline partition: A partition with no leader, usually because every broker that holds a replica is down. It can't be written to or read from. The Partitions card (Online on Brokers) shows fewer online than in total (Cluster Overview).
Partition: One of the ordered logs a topic is split into, numbered from 0. Messages with the same key go to the same partition and stay in order; there's no ordering across partitions. The partition count also caps how many consumers in one group can read in parallel (Name, partitions and replication factor).
Preferred leader: The first broker in a partition's replica list: the one Kafka moves leadership back to after a restart, when auto.leader.rebalance.enable is on or a preferred leader election is run.
Replica: One copy of a partition, on one broker. A partition has as many replicas as its replication factor: one leader and the rest followers. A replica that has fallen behind its leader is out of sync.
Replica skew: How unevenly replicas are spread over the brokers. A broker with more replicas stores more data and copies more from the leaders. Shown like leader skew (Leader skew and replica skew).
Replication factor: How many brokers keep a copy of each of a topic's partitions; 3 is usual in production. It can't be more than the number of brokers (Changing the replication factor).
Under-replicated partition (URP): A partition with fewer in-sync replicas than replicas: some copy has fallen behind or its broker is down. The data is still readable, but with less redundancy than intended. The URP card should be 0 (Cluster Overview, per topic).
ZooKeeper: The separate service older Kafka clusters use to store their metadata, replaced by KRaft. KafkIO talks to the brokers only, never to ZooKeeper.

Topics and messages

Cleanup policy: cleanup.policy: what Kafka does with old messages. delete removes them after the retention limits, compact keeps the latest message per key, and compact,delete does both (Cleanup policy).
Compaction: Removing older messages that have the same key as a newer one, so a compacted topic keeps at least the latest value for each key, however old. KafkIO can't clear a compacted topic with Clear Messages (Compacted topics).
Compression: Producers can compress batches of messages (gzip, Snappy, LZ4, zstd). Consumers, KafkIO included, decompress them before decoding (Compression).
End offset: The offset the next message written to a partition will get, one above its newest message. KafkIO's Last Offset column shows it (The partitions table).
Header: A key–value pair attached to a message besides its key and value, often used for tracing IDs or content types. A message can have any number of them (The Headers tab, producing headers).
Internal topic: A topic Kafka or one of its tools creates for itself, such as __consumer_offsets, __transaction_state or _schemas. KafkIO lists them with the others (Internal topics).
Key: The optional part of a message that decides its partition: messages with the same key always go to the same partition, which keeps them in order. Compaction also works per key. A message with no key is spread over the partitions.
Log segment: One of the files a partition is stored in. Kafka deletes whole segments, not single messages, so a topic can hold somewhat more, and older, messages than its retention suggests.
Message (record): One entry in a partition: an optional key, a value, any headers, a timestamp and its offset (Viewing Messages).
Message count: In KafkIO, a partition's end offset minus its first offset. Compaction and transactions can make it higher than the number of messages you can actually read (Message counts).
Null (tombstone): A key or value that is absent, not an empty string. A message with a key and a null value is a tombstone: on a compacted topic it deletes that key (Sending true nulls).
Offset: A message's position in its partition, counting up from 0. Offsets are never reused, so after retention or compaction they have gaps and the first offset is above 0. To read from a given offset, see From an offset.
Producer: An application that writes messages to topics. KafkIO is one when you produce messages.
Retention: How long (retention.ms) and how much per partition (retention.bytes) a topic keeps before Kafka deletes its oldest messages, whichever limit comes first (Retention time and size).
Serde, serializer, deserializer: A serializer turns what you type into the bytes of a key or value; a deserializer turns those bytes back into readable text. A serde is the pair. KafkIO picks one for each topic and lets you change it (Deserialization & Formats, Serializers & Deserializers).
Timestamp: The time stored with each message: when the producer created it, or when the broker appended it, depending on the topic's message.timestamp.type. Searches by date and time use it (From a date and time).
Topic: A named stream of messages, such as orders, split into partitions (The Topics List).
Value: The main content of a message: the JSON, Avro record, text or bytes it carries. It can also be null.

Consumers

Committed offset: The offset a consumer group has saved for a partition: where it will carry on reading. Kafka keeps it in __consumer_offsets, and you can move it with Offsets Tweak.
Consumer: An application that reads messages from topics. KafkIO reads topics too, but without joining a group or committing offsets, so browsing a topic never moves anyone's position.
Consumer group: A set of consumers sharing one group.id. Kafka shares the partitions of the topics they read between them, each partition going to one member, and keeps the group's committed offsets. A group stays listed after its consumers stop, for as long as Kafka keeps those offsets (Consumers).
Consumer lag: How far a group is behind: for each partition, the end offset minus the group's committed offset. KafkIO adds this up over a group's partitions. How it changes over time matters more than its size (Understanding consumer lag).
Coordinator: The broker that manages a consumer group: it tracks the members, runs rebalances and stores the group's committed offsets. Shown in the Coordinator column (Columns).
Group state: What a group is doing: STABLE (consuming), PREPARING_REBALANCE or COMPLETING_REBALANCE, EMPTY (no members, offsets kept) or DEAD (Group states).
Member: One consumer instance in a group, with the partitions assigned to it (Group details).
Offsets Tweak: KafkIO's name for resetting a group's committed offsets: to the earliest or latest message, a point in time or exact offsets. Only possible while the group has no members (Tweaking offsets).
Rebalance: The broker reassigning a group's partitions among its members, when one joins or leaves. Consumption pauses briefly; a group that keeps rebalancing usually has a consumer that takes too long between polls.

Schemas

Compatibility level: The rule Schema Registry applies when a new version of a subject is registered, such as BACKWARD (new consumers can read old data) or NONE. A subject has its own level or follows the registry's default (Compatibility levels).
Schema: A description of the structure of a key or value, in Avro, JSON Schema or Protobuf, that producers and consumers agree on (Schema types).
Schema ID: The registry-wide number of a schema. Producers using Schema Registry write it into each message, and consumers use it to find the schema to decode with (Schema IDs).
Schema reference: A schema that uses a type defined in another subject, such as an Address record shared by several schemas (Schema references).
Schema Registry: A service next to the Kafka cluster that stores schemas and hands them out by ID (Confluent Schema Registry, or a compatible one such as Karapace) (connecting, browsing and managing).
Soft and hard delete: A soft delete hides schema versions but keeps them, so messages written with them still decode; a hard delete removes them for good (Soft and hard deletes).
Subject: The name a schema and all its versions are registered under, usually the topic name plus -key or -value, such as orders-value (The schemas list).
Subject naming strategy: How a producer names subjects: after the topic (TopicNameStrategy, the default), the record type (RecordNameStrategy) or both (TopicRecordNameStrategy). KafkIO uses it to decide which subjects to offer for a topic (Subject naming strategies).
Version: Each schema registered under a subject gets the next version number, from 1. The registry keeps the old versions (All versions).

Kafka Connect

Connect cluster: A group of Kafka Connect workers that share connectors, reached through one REST URL. A Kafka cluster can have several; KafkIO lists their connectors together (One or several Connect clusters).
Connector: A configured job that copies data between Kafka and another system: a source connector reads from that system into topics, a sink connector writes topics out to it (Kafka Connect).
Connector offsets: How far a connector has got: for a sink, its consumer group's committed offsets; for a source, its position in the other system. Resetting them makes the connector start again from the beginning (Resetting connector offsets).
Connector and task states: RUNNING, PAUSED (stopped processing but still assigned), STOPPED (shut down, configuration kept), FAILED, RESTARTING or UNASSIGNED. A connector can be RUNNING while all its tasks have failed (Connector lifecycle).
Kafka Connect: Kafka's framework for moving data in and out of Kafka with ready-made connectors, run as a separate service with a REST API (Kafka Connect Connections).
Task: A unit of a connector's work, spread over the workers. tasks.max sets how many a connector may have; a sink's tasks share its topics' partitions. Tasks fail and restart individually (Status and tasks).
Worker: One Kafka Connect process. The workers of a Connect cluster share its connectors and tasks between them.

ksqlDB

ksqlDB: A service that runs SQL-like statements (KSQL) over Kafka topics, continuously (ksqlDB Connections, ksqlDB).
Persistent query: A query ksqlDB keeps running on the server and writes to a topic, started by CREATE STREAM ... AS SELECT, CREATE TABLE ... AS SELECT or INSERT INTO ... SELECT. It runs until terminated (Persistent queries).
Processing log: KSQL_PROCESSING_LOG, ksqlDB's own stream of records it couldn't process. Query it when a stream or table is missing rows (Streams and tables).
Pull query: A SELECT without EMIT: ksqlDB answers once, from the current state, like a database query (Running a statement).
Push query: A SELECT ... EMIT CHANGES (or EMIT FINAL): it keeps running and sends new rows as they happen, until you stop it (Push queries).
Stream: A ksqlDB collection over a topic in which every message is a separate event, like a log of orders (Streams and tables).
Table: A ksqlDB collection that keeps the latest value for each key, like a customer's current address, often built by aggregating a stream. A windowed table aggregates per time window (Streams and tables).

Security

ACL (access control list): An entry that allows or denies one principal one operation (such as READ) on one resource or set of resources (such as a topic). A matching DENY always wins (ACL concepts).
JAAS configuration: sasl.jaas.config: the line that names the SASL login module and its options, such as user name and password (JAAS configuration).
Kerberos (GSSAPI): A SASL mechanism common in corporate networks, logging in with a keytab or ticket cache. It also needs a krb5 configuration (Kerberos (GSSAPI), Kerberos preferences).
Keystore: A file holding your own certificate and private key, which KafkIO presents with mutual TLS. Formats include JKS, PKCS12 and PEM (Supported certificate and key formats).
Mutual TLS (mTLS): TLS in which the client proves who it is too, with a client certificate from a keystore, and the broker uses that certificate's name as its identity (Mutual TLS).
Principal: The identity a client authenticates as, written type:name, nearly always User: plus the SASL user name or the client certificate's distinguished name. ACLs are granted to principals (ACL concepts).
SASL: The framework Kafka uses for logging in. The mechanism says how: PLAIN, SCRAM-SHA-256/512, GSSAPI (Kerberos), OAUTHBEARER or AWS_MSK_IAM (SASL mechanisms, Security & Authentication Matrix).
Security type: KafkIO's name for security.protocol: Plaintext, SSL, SASL plaintext or SASL SSL. It must match the listener behind the port you connect to (Choosing a security type).
Super user: A principal listed in the brokers' super.users setting, which bypasses ACLs. It doesn't appear in KafkIO's ACL list.
TLS (SSL): Encryption of the connection between KafkIO and the brokers (or a REST service). Kafka still calls it SSL in its settings (SSL / TLS).
Truststore: A file of certificate authorities (CAs) you trust; KafkIO accepts a server whose certificate one of them signed. Not needed for services with public certificates (Truststores, JKS Converter).

KafkIO terms

Cluster configuration: Everything KafkIO knows about one cluster: name, bootstrap servers, colour, security, and any Schema Registry, Connect and ksqlDB connections. All of them are kept in clusters.json (Managing Cluster Configurations).
Cluster colour: A colour you give a cluster, shown on its row in the Kafka Clusters list, on its tabs and in the status bar, so a production cluster stands out (Colour).
Cluster tab and duplicate tab: Each connected cluster opens in a tab that shows one section at a time. A duplicate tab, such as Storefront Dev (Consumers), opens a second view of the same cluster, so you can watch two sections or topics side by side (Duplicating a tab).
Data folder: The folder in your user profile where KafkIO keeps its configuration, preferences and logs (The data folder).
Dump: Writing a topic's messages to a delimited text file, in the foreground or in the background (Dumping a Topic to a File).
Event Log: The list of every notification, warning and error KafkIO has shown, under Help ▸ View Event Log (The Event Log).
Favourites: Topics you've starred in a cluster's topics list, so you can narrow the list to them in one click. They belong to the cluster (Favourites).
Health Log: A pane that checks every connected cluster every 30 seconds and records its status and last error (The Health Log).
Memory Stick Mode: Keeping KafkIO's configuration next to the application instead of in your user profile, so it can run portably (Memory Stick Mode).
Message template: A saved key, value and headers in the Produce dialog, shared by every cluster and topic (Message templates).
Network Tool: Checks that host names resolve and ports are reachable from your machine, to tell a network problem from a settings problem (Network Tool).
Search type: Where a message search starts and in which direction it reads: Newest, Oldest, from an offset, from a date and time, or Stream Live (Search types).
Stream Live: The search type that shows new messages as they're written, until you stop it (Streaming Messages Live). Not the same as a ksqlDB stream.
Substitution variable: A ${…} placeholder in a produced message, such as ${counter} or ${random.uuid}, replaced with a fresh value for each message (Substitution Variables).
Test Connection: The button in a cluster's settings dialog that tries the settings as they are, without saving them (Testing a configuration).