Searching Messages

The Messages tab is where you read a topic. Nothing is fetched until you search: the search form at the top says where to start reading (the newest or oldest messages, an offset or a point in time), which partitions to read, how many messages to bring back and, optionally, a text that every message must contain. This page covers the form and how a search behaves; reading the results is covered in Viewing Messages.

The search form

Here the newest 20 messages of the orders topic have been read and the first one selected:

1 2 3 4 5 6 7 8 9 10 11 12

1. Search type: where reading starts and in which direction (Search types).
2. Partition(s): all partitions, or the ones you tick (Choosing partitions).
3. Text filter, with the parts of the message it looks in below it (Filtering by text).
4. Key deserializer and
5. Value deserializer: how the raw bytes are turned into text (Deserializers).
6. Max page/poll size: how many messages one search brings back (Page size).
7. Key/message trim: the most of each key and value kept in memory (Trimming large messages).
8. Search runs the search; its arrow holds Dump Topic To File (below).
9. Collapse after search: fold the form away once results arrive (Collapsing the search form).
10. Refresh Partitions and Serdes (below).
11. What the last search read (Reading the search statistics).
12. Next: the next page of results, shown while there are more (Paging through results).

A new Messages tab starts at Newest, all partitions, a page size of 5,000, a trim of 500 KB, no text filter, and the deserializers KafkIO prefers for the topic. Click Search, or press Enter anywhere on the tab. Before the first search the table also has a Search button of its own.

Running a search

While a search runs, KafkIO shows Retrieving messages... over the tab, and results are added to the table as they arrive rather than all at the end, which matters for slow, filtered searches over a big topic. When it finishes, a notification says Retrieved N messages. Every search starts again from the criteria in the form; it doesn't add to the previous results.

Search is greyed out while the form has an error (an offset of 0, for example) and while a live stream is running.

Reading the search statistics

The badge at the bottom left of the table reads, for example, Showing: 20 | Received: 21 | Received size: 2.4 KB:

• Showing: the messages in the table.
• Received: the messages KafkIO read from Kafka to get them, and Received size their total size.

The two counts often differ, and that's normal: a text filter drops the messages that don't match, and a read over several partitions can fetch a few more messages than fit on the page (Page size). Received is the number to look at when you want to know how much of the topic a search covered.

Search types

All of them stop when the page is full (Page size) or the partitions run out. With several partitions, the table merges them by timestamp, so messages from different partitions are interleaved and each partition's messages stay in offset order. Click a column header to sort the results another way.

From an offset

Offset start and Offset end add an offset field. Offsets are numbered per partition, so the same offset is used in every selected partition; to read around one message, tick its partition first. An offset past the end of a partition reads nothing from it, and one below its first offset starts at the first message.

1 2 3 4

1. Offset start reads forwards from the offset.
2. Start offset: digits only.
3. Only partition 1 is ticked.
4. Six messages: offsets 25 to 30, the end of the partition.

The offset must be greater than 0: with 0 (or an empty field) the field shows an error and Search stays greyed out. To read from the very beginning, use Oldest. An offset taken from the topic's partitions table or from a consumer group's lag is a good starting point.

From a date and time

Occurring after and Occurring before add a date and a time (hours and minutes). Use them to look at what happened around an incident without knowing any offsets.

1 2 3 4

1. Occurring after reads forwards from the time.
2. The date (calendar button) and the time (clock button, or click the hours or minutes and use the arrow keys).
3. All partitions: each one starts at its own first message at or after 01:26.
4. The 20 messages written after that time.

The date starts as today and the time as when the tab was opened. The time is in your computer's time zone, whatever zone the table's Time column is set to (Viewing Messages). Kafka finds the position from the messages' timestamps, so on topics whose producers set their own timestamps the result follows those, not the time the message reached the broker.

Choosing partitions

Partition(s) shows ALL until you tick partitions in its list; it then shows the ones ticked (1, or 0, 2). Untick them all to go back to all partitions. Reading one partition is faster and, with offsets, the only way to get exactly the messages you mean. Messages with the same key always go to the same partition, so if you know which partition a key lands on, tick it and search for the key.

The list is read when the tab opens. If partitions are added to the topic while it's open, click Refresh Partitions and Serdes (below).

Filtering by text

Type into Text filter to keep only the messages that contain that text. Here only the value is searched, with a regular expression that matches cancelled and new orders:

1 2 3 4 5

1. The filter text. The regex button at its right end turns regular expressions on (orange, as here) and off; the × beside it clears the filter.
2. Where to look: Key, Value and Headers (names and values). A message matches if any ticked part contains the text. They're greyed out while the filter is empty.
3. The page size: with a filter, it counts matches, not messages read.
4. 61 of the 100 messages read matched.
5. The page-size hint (Filter performance).

• Plain text matches anywhere and is case-sensitive: cancelled finds nothing in these messages, CANCELLED finds them.
• Regular expressions use Java syntax and also match anywhere (no need for .* at either end). Start with (?i) to ignore case, as in (?i)cancelled. An invalid expression stops the search with Error retrieving messages and the reason (for example Unclosed group).
• The filter looks at the deserialized text, as the table shows it, so it only finds text in Avro or Protobuf messages when the matching deserializer is selected. JSON is matched as it was sent, not as the detail pane pretty-prints it: "status": "NEW" may be "status":"NEW" in your messages.
• With Key, Value and Headers all unticked, the filter is ignored and every message is shown.

The filter text is also copied into the search boxes of the Key and Message panes below the table (for the parts ticked), so the match is highlighted when you select a message (Viewing Messages).

Filter performance

Kafka can't filter messages for a client: KafkIO reads every message in the range, decodes it and checks it itself. A filtered search keeps reading until it has a page of matches or reaches the end of the partitions, so on a big topic with few matches it can read the whole topic. While it runs, the progress shows Matched and Checked counts and the data read so far.

KafkIO reads in batches of the page size, so a larger page size finds sparse matches much faster. That's why a filtered search with a page size of 10,000 or less shows the hint For performance, use a larger page/poll size when using text filters, with buttons that set 50,000, 100,000 or 200,000. It appears once per tab, or on every search with a page size of 500 or less. Narrowing the search first (one partition, a date) helps as much.

Page size

Max page/poll size is the most messages one search (or one Next) puts in the table: 20, 200, 500, 1,000, 2,000, 5,000 (the default), 10,000, 50,000, 100,000 or 200,000. Use a small page to glance at a topic, and a large one to filter or to have more results to sort.

It also sets how KafkIO reads: each round takes the page size divided by the number of partitions from every partition. So a page of Newest over three partitions holds roughly the newest third of a page from each, not strictly the newest messages of the whole topic, and Received can be a little higher than Showing. Every message on the page is kept in memory (up to the trim size each), so very large pages of large messages need memory to match. For a live stream the page size is the most rows the table keeps (Streaming Messages Live).

Trimming large messages

Key/message trim limits how much of each key and value KafkIO keeps: 20 KB, 50 KB, 100 KB, 200 KB, 500 KB (the default), 1 MB, 5 MB, 10 MB or 20 MB. A longer key or value is cut at that size and ends with ...<Trimmed by KafkIO>; its raw bytes are cut at the same size. This keeps memory and display time down on topics with very large messages, especially when streaming on a modest machine.

Trimming happens after the text filter, so a match beyond the cut still counts. To read a trimmed message in full, pick a larger trim and search again.

Deserializers

Key deserializer and Value deserializer choose how the bytes Kafka stores are turned into what you read: String (raw) for text and JSON, Schema Registry for Avro, JSON Schema and Protobuf messages, and others for numbers, hex, MessagePack and raw Protobuf. KafkIO preselects the one it thinks fits the topic. If the first search after you open a topic's messages reads Schema-Registry-encoded data with another deserializer, a small warning triangle appears at the corner of the drop-down; hover it for the advice to use the Schema Registry deserializer. Later searches of that topic don't repeat the check (and clear the triangle), so its absence proves nothing. The choices, and what to do when decoding fails, are in Deserialization & Formats.

The choice applies to the next search: change it and search again to see the same messages decoded another way.

Paging through results

When a search stops because the page is full, Next appears at the bottom right of the table. It reads the following page with the same criteria, carrying on where the last page stopped, in the same direction (further back for Newest, further on for Oldest), and replaces the table with it. When the range is used up there's no Next. To go back, search again.

Collapsing the search form

The form takes a lot of the tab. Once a search returns more than 10 messages, a collapse search link appears beside the Search title; click it, or the title's arrow, to fold the form away. Tick Collapse after search to have it fold up by itself whenever a search returns results.

Enter still runs the search while the form is collapsed, with the same criteria.

Refreshing partitions and deserializers

The partition list and the deserializer lists are read when the tab opens. The Refresh Partitions and Serdes button at the right of the form's title reads both again: use it after adding partitions to the topic, or after registering a schema for it, without closing the tab. Notifications say how many key and value deserializers were found.

Dumping the results to a file

The arrow beside Search opens Dump Topic To File, which writes the messages matching the current form (search type, partitions, text filter and deserializers) to a file instead of the table, without the page limit. It isn't available for Stream Live. See Dumping a Topic to a File.

Related

• Viewing Messages
• Streaming Messages Live
• Deserialization & Formats
• Dumping a Topic to a File
• Topic Overview, Consumers & Settings
• Topics & Messages features