Producing Messages

The Produce Message(s) dialog writes messages to a topic: one message to test a consumer, a copy of one you found while searching, or thousands of varied test messages with counters, random values and dates filled in for each one. You choose the partition, how the key and value are turned into bytes (plain text, numbers, hex or a Schema Registry schema), and any headers, and you can save what you typed as a template for next time.

Opening the Produce dialog

Select a topic in the topics list, then either click 1 Produce Message(s) in the topic toolbar, or right-click the topic and choose Produce Message(s). To send a copy of an existing message, right-click it in the message table and choose Re-produce Message (Re-producing an existing message).

1

1. Produce Message(s): opens the dialog for the selected topic.

The dialog doesn't block the rest of KafkIO, and it stays open after each send, so you can change the message and send again, or search the topic to see what arrived. You can open several at once, for the same topic or different ones, on one cluster or several: each dialog keeps the topic it was opened for (shown at its top), whatever you select afterwards. Together with duplicated cluster tabs, that lets you feed one topic while you watch another, for example a live stream of the topic your consumer writes its results to.

The dialog at a glance

1 2 3 4 5 6 7 8 9 10 11

1. Partition: Auto, or a partition number (Choosing a partition).
2. Key serializer: how the key becomes bytes.
3. Value serializer: how the message (the value) becomes bytes (Key and value serializers).
4. The Key, Message and Headers tabs, where you type what to send (Key, value and headers).
5. Send null message (and Send null key on the Key tab) (Sending true nulls).
6. Template: load, delete or save templates (Message templates).
7. Where the last message went (Results and errors). Shown after the first send.
8. '${variable}' substitution, on by default, and its Examples (Variable substitution).
9. How many (max 500k): how many messages to send.
10. Delay between messages, in milliseconds (Producing many messages).
11. The send button. Its label says how many messages it will send and, with a delay, how long that takes.

Partition and serializers

Choosing a partition

Leave Partition on Auto to let Kafka's producer decide, as your applications' producers usually do: a message with a key goes to the partition its key hashes to, so messages with the same key always land in the same partition, in order. Messages without a key (a null key) are spread across partitions by the producer, a batch at a time, so a quick run of keyless messages often lands in a single partition.

Pick a partition number to send everything there, whatever the key: useful to test how a consumer handles one particular partition, or to put a message next to the ones it relates to. Note that a key normally lands in one partition only, so writing it elsewhere by hand can break the ordering your consumers rely on.

Key and value serializers

A Kafka message is only bytes. The serializers decide how the text you type becomes those bytes, separately for the key and the value. Choose the format your consumers expect: the same one you'd pick as the deserializer to read the topic.

The number serializers need a plain number with no spaces or separators; anything else, or a number out of range, fails when you send (Results and errors). Protobuf (raw) and __consumer_offsets, which you may see when reading messages, can only read, so they aren't offered here. Exactly what each serializer accepts is listed in the Serializers & Deserializers reference.

When the dialog opens, both serializers start on String (raw), except that the value starts on Schema Registry when the cluster has a Schema Registry connection and the topic has a schema subject of its own (such as orders-avro-value). Schema Registry appears in the list only when there's at least one subject it could use.

Using a Schema Registry subject

When you choose Schema Registry, a Subject box appears next to it: the subject whose latest schema version is used to encode what you type. Pick one before sending (the send button stays disabled until you do). Type in the box to filter long lists.

1 2

1. Schema Registry chosen as the value serializer.
2. Subject, open: each subject is labelled with how it relates to this topic.

Subjects are listed in this order, so the likeliest ones come first:

• topic subject: this topic's own subjects, orders-avro-value (or -key for the key) and any orders-avro-RecordName subjects. The first of these is preselected.
• record subject: subjects named after a record, such as com.storefront.OrderPlaced, which any topic can use.
• other topic's subject: another topic's topic-RecordName subjects.
• other subject: anything else that might fit.

Other topics' plain -key and -value subjects aren't offered. How KafkIO tells these naming strategies apart is explained under Subject naming strategies.

Type the message as JSON, whatever the schema type: KafkIO checks it against the schema and encodes it as Avro, Protobuf or JSON Schema data. For Avro, write a field whose type is a union other than plain null with the type as a key, for example "note": {"string": "gift wrap"} (and "note": null for null), which is how KafkIO displays such fields when reading. For Protobuf, when the subject is named after a message type (a record or topic-record subject), that type is used; otherwise, the schema's first message type. KafkIO never registers schemas when producing: register or change them in the Schema Registry section.

Key, value and headers

Type or paste the key on the Key tab and the value on the Message tab (the one shown when the dialog opens). Both are full editors with line numbers, undo and redo, and a search box (Search key..., Search message...) that works like the one for searching within a message. The text is sent exactly as typed, including line breaks and indentation: KafkIO doesn't reformat JSON.

Leave the key empty if your topic doesn't use keys, but note that an empty key isn't the same as no key: see below.

Sending true nulls

An empty key or value is sent as zero bytes, which is a value. Send null key (on the Key tab) and Send null message (on the Message tab) send a real null instead, and grey out the editor. The difference matters to consumers and to Kafka itself:

• A message with a null value is a tombstone. On a compacted topic it tells Kafka to delete every earlier message with that key: send one with the key filled in to remove that key from the topic.
• A null key leaves the partition to the producer (see Choosing a partition), while an empty key always hashes to the same partition.
• Consumers often test for null, not for empty, so it's worth checking they cope with both.

After sending, Serialized key size (or message size) in the result shows -1 for a null and 0 for an empty one.

Headers

Headers carry metadata next to the message: a trace ID, the event type, the producing service. On the Headers tab, click 1 Add header for each one, and fill in its Header key and Header value. Header values are sent as UTF-8 text, and can use variables too.

1 2 3

1. Add header: adds an empty row.
2. Null value: sends the header with a null value rather than an empty one.
3. remove: deletes that row.

Each header key is sent once: if you add the same key twice, only one of the rows is sent (Kafka allows repeated header keys, but this dialog doesn't produce them).

Producing many messages

Set How many (max 500k) to send the same message up to 500,000 times in one go, to load-test a consumer, fill a topic for a demo or check how lag builds up. With variable substitution on, each copy gets its own values; without it, every copy is identical.

Delay between messages waits that many milliseconds between one message and the next, up to 300,000 (5 minutes); the links under it set common values, and the figure beside it shows the delay in seconds. Use it to trickle messages in at a realistic rate, for example one a second while you watch a live stream or a consumer's dashboard. The delay is ignored, and greyed out, when sending a single message. The send button's label shows the total time, e.g. Produce 10 Messages (in 1.00s).

While a batch is being sent, the dialog shows Sending N messages... with a Cancel link. Cancelling (or closing the dialog) stops the batch; the messages already sent stay in the topic. The rest of KafkIO stays usable meanwhile, so you can watch the topic fill up.

All messages in a batch go through one producer connection, and a batch stops at the first message Kafka rejects (Results and errors).

Variable substitution

With '${variable}' substitution ticked (the default), KafkIO replaces ${…} placeholders in the key, the value and header values with a fresh value for every message it sends. That's how one dialog produces a thousand different orders, each with its own number, customer and timestamp, as in the dialog above:

{
  "orderId": ${counter},
  "customer": "customer-${random.int:1:20}",
  "amount": ${random.int:5:500},
  "status": "NEW",
  "placedAt": "${current.datetime:yyyy-MM-dd'T'HH:mm:ss}"
}

A placeholder KafkIO doesn't recognise is left as it is. Untick the option to send text that contains ${ literally, such as a template for another tool.

Counters and random values

${counter} starts at 1 each time you click the send button, and goes up by one every time it's used, not once per message. The key is filled in first, then the value, then the headers. So with ${counter} only in the value, ten messages get 1 to 10; with ORD-${counter} as the key and "orderId": "ORD-${counter}" in the value, the first message gets key ORD-1 and order ID ORD-2. Use the counter in one place per message when the numbers need to match.

Dates and epoch times

Dates and times are your computer's local time, taken as each message is built. Formats use Java's date pattern letters: yyyy year, MM month number, MMM short month name, dd day, EEEE day name, HH hour (00-23), mm minutes, ss seconds, SSS milliseconds. Put literal letters in single quotes, as in yyyy-MM-dd'T'HH:mm:ss. There's no time zone variable: for a UTC timestamp, use ${epoch.millis}.

Every variable, with more examples, is listed in the Substitution Variables reference.

The examples window

Click Examples next to the option to open Variables Substitution Examples: a scratch pad listing every variable, with the result of substituting them below. Edit the top box to try out your own text or date formats, click 1 Test to see the result, and copy what works into your message. 2 Reset brings back the original list.

1 2

1. Test: substitutes the variables in the top box again.
2. Reset: restores the original examples.

Message templates

A template saves the key, message and headers you've typed, so you can send the same kind of message again later without retyping it: a sample order, a test event with all its headers. Templates are shared by every cluster and topic, so one template can feed the same topic on your development, staging and production clusters.

A template holds only the text: the key, the message, and the header keys and values (variables included, as typed). It doesn't keep the partition, the serializers, how many to send, or the null options; a header saved with Null value ticked comes back with an empty value.

Saving a template

Click Save as template and enter a name. The suggested name is the template selected in the list, if any, otherwise my_template_ followed by the date and time. If a template of that name exists, KafkIO asks before overwriting it.

Loading a template

Choose the template in the Select template... list and click Load. Its key, message and headers replace what's in the dialog; anything the template doesn't have (no key, say) is cleared. Check the serializers and partition afterwards, since the template doesn't set them.

Deleting a template

Choose it in the list and click Delete, then confirm.

Templates are plain text files in the message_templates folder of KafkIO's user data folder (see What's left behind for where that is): key.name.txt, value.name.txt and headers.name.txt (one key=value per line). You can copy them to another machine, or keep them in version control with your project.

Re-producing an existing message

To send a message again, as it is or changed, right-click it in the message table and choose Re-produce Message. The dialog opens filled in from that message:

• its key, value and headers (with the null options ticked where they were null);
• its partition, rather than Auto: set it back to Auto if you'd rather let the key decide;
• as serializers, the deserializers the message was read with; for a Schema Registry message, the Subject that matches the message's schema ID.

It's handy for replaying a message a consumer failed on, after fixing the consumer, or for sending a corrected version of a bad one: change the value, then send. With Avro, the value appears as the JSON KafkIO displayed, which is the form it expects back.

Results and errors

When a send succeeds, the dialog shows Last message sent at the date and time, with details of the last message in the batch:

The topic's message count in the topics list is refreshed after each send. To see the messages themselves, search the topic (Newest shows them first), or stream it while you send.

If something goes wrong, an Error sending message(s) dialog says why:

The usual causes:

• The text doesn't fit the serializer: letters for an Int (32), invalid JSON for MessagePack, or a value that doesn't match the schema, as above. Nothing is sent: fix the text and try again.
• Kafka refuses the message: not authorised to write to the topic (an ACL), a message bigger than the topic's max.message.bytes, a partition without a leader. KafkIO's producer doesn't retry, so you see Kafka's own error straight away rather than a timeout later.
• The cluster can't be reached: see Troubleshooting Connections.

When producing many messages, the batch stops at the first failure; the messages before it were sent. Click Show Details for the full error, or View Logs to open KafkIO's log.

Related

• Searching Messages
• Viewing Messages
• Streaming Messages Live
• Deserialization & Formats
• Substitution Variables
• Serializers & Deserializers
• Schema Registry Connection