Creating Topics

A new topic needs only a name and a number of partitions; everything else can be left to the cluster's defaults. When the defaults don't suit (a longer retention, larger messages, a compacted topic), set those properties in the same dialog, so the topic is right from its first message.

Opening the Create Topic dialog

Open the cluster's Topics section and click Create Topic, the last of the topic buttons above the topics list (see Topic actions). It doesn't matter which topic is selected, or whether one is.

The dialog doesn't block the main window: you can look up another topic's settings while you fill it in. Clicking Create Topic again brings the open dialog to the front rather than opening a second one. Closing it with Close discards what you entered.

1 2 3 4 5 6 7 8

1. Name: required.
2. Partitions: required; starts at 2.
3. Replication factor: blank uses the cluster's default.
4. Common properties: five frequently changed settings, each blank by default.
5. Shortcuts that fill in a value for you (here 2w, two weeks).
6. A property added from Other properties.
7. Define another property adds another row.
8. Create Topic: greyed out until the name and partitions are filled in.

The Advanced tab stays greyed out here: it's for changing the partitions and replication factor of an existing topic, from Edit Settings (see Updating a topic).

Name, partitions and replication factor

Name

Letters, digits, periods (.), underscores (_) and hyphens (-) only, up to 249 characters; no spaces. KafkIO doesn't check the name itself: the cluster rejects a bad one when you click Create Topic (see If creation fails). A topic can't be renamed later, so it's worth settling on a naming scheme (orders, orders.v2, order-returns-dlq) before you create many. Kafka warns against mixing periods and underscores, as order.returns and order_returns clash in its metrics.

Partitions

How many ways the topic is split, from 1 to 10,000. Partitions are what let several consumers in one group share the work, so the count caps how many of them can read in parallel. Messages with the same key always go to the same partition, which keeps them in order; across partitions there's no ordering. You can add partitions later but never remove them, and adding them changes which partition a key goes to, so start with what you expect to need.

Replication factor

How many brokers keep a copy of each partition. Leave it blank to use the cluster's default (default.replication.factor, which you can look up in a broker's Settings tab). It can't be more than the number of brokers. In production, 3 is the usual choice: the topic survives one broker going down while still having two copies. A local, single-broker cluster can only have 1.

Common properties

These five are the topic settings people change most. Leave any of them blank (or <broker default>) and the topic follows the cluster's default for it, now and if the default changes later; fill it in and the topic keeps that value whatever the cluster's default. The information pop-ups beside the number fields name the Kafka setting, describe it and give Kafka's usual default.

Min in-sync replicas

min.insync.replicas: how many copies of a partition must have a message before Kafka accepts it from a producer that waits for all replicas (acks=all). With a replication factor of 3, setting it to 2 lets the topic keep accepting writes with one broker down without risking a message being held by only one broker. It must not be more than the replication factor, or those producers can't write at all. Kafka's usual default is 1.

Cleanup policy

1

1. Cleanup policy (cleanup.policy): what happens to old messages.

Compacting relies on keys, so producers to a compacted topic must set one. In KafkIO, a compacted topic can't be cleared with Clear Messages (see Compacted topics).

Retention time and size

How long, and how much, the topic keeps before Kafka deletes the oldest messages (with a cleanup policy that includes delete). Messages go when either limit is reached.

• Retention time (ms) (retention.ms) is in milliseconds. Click a shortcut (12h, 1d, 2d, 7d, 2w, 4w, 8w) rather than counting zeros; the duration you've entered is shown beside the field (14d). Kafka's usual default is 7 days.
• Retention size (bytes) (retention.bytes) is the limit per partition, not for the whole topic: 1 GB on a six-partition topic lets it grow to about 6 GB (per copy). Shortcuts go from 128mb to 100gb, and the size is shown beside the field. -1, Kafka's usual default, means no size limit.

Kafka deletes whole log segments, not single messages, so a topic can hold somewhat more, and somewhat older, messages than these limits suggest. The message counts show this too.

Max message size

Max message size (bytes) (max.message.bytes) is the largest batch of messages, after compression, that the topic accepts from a producer: in practice, the largest message. Kafka's default is about 1 MB. Shortcuts: 128kb, 1Mb, 5mb, 10mb, 50mb (binary units: 5mb is 5,242,880 bytes).

Raising it on the topic is only half the job: producers (max.request.size) and consumers (max.partition.fetch.bytes, fetch.max.bytes) have their own limits, and followers need replica.fetch.max.bytes on the brokers to be large enough to copy the messages.

Other properties

For any other topic setting, click Define another property. A row appears with a list of settings and a value field; click the list and type part of a name to narrow it down.

1 2

1. Type to filter the list (here comp).
2. Pick a setting: its value field fills with Kafka's usual default, and the field's information pop-up describes it.

Change the value, then add more rows as needed; remove takes a row away. The list offers 23 settings, among them compression.type, delete.retention.ms, min.compaction.lag.ms and max.compaction.lag.ms, min.cleanable.dirty.ratio, segment.bytes and segment.ms, message.timestamp.type, unclean.leader.election.enable and, for tiered storage, remote.storage.enable with local.retention.ms and local.retention.bytes. The five common properties aren't in it. Settings that aren't listed (a vendor's own, for example) can be set once the topic exists, in its Settings tab.

If creation fails

Kafka checks the request when you click Create Topic. If it refuses, an Error creating topic dialog gives Kafka's reason, and the Create Topic dialog keeps everything you entered so you can fix it and try again.

Creating several topics at once

The dialog stays open after each topic is created, with all its fields as they were. To create a set of similar topics (an order-returns topic and its order-returns-dlq, say), fill the dialog in once, click Create Topic, confirm the Topic created message with OK, then change just the name (and anything else that differs) and click Create Topic again. Every topic gets the partitions and properties the dialog shows at the time. Click Close when you're done.

To copy an existing topic's settings to a new one instead, open the existing topic's Settings tab alongside the dialog (it doesn't block the main window) and enter the values that differ from the defaults.

After creating

The new topic appears in the topics list straight away, empty, ready to select.

1 2 3

1. Create Topic, where the dialog came from.
2. order-returns, selected, with no messages yet.
3. order-returns-dlq, created next with the same settings.

From here you might:

• Check what the topic ended up with: partitions and replicas on its Overview tab, every setting (yours and the defaults) on its Settings tab.
• Send it some test messages (see Producing Messages).
• Star it as a favourite, so it's easy to find among many topics (see Favourites).
• Change its properties, or add partitions, later with Edit Settings (see Updating a topic).

Related

• The Topics List
• Topic Overview, Consumers & Settings
• Updating, Clearing & Deleting Topics
• Producing Messages
• Brokers