Deserialization & Formats

Kafka stores keys and values as plain bytes; it doesn't know whether they hold text, a number or an Avro record. Before KafkIO can show a message, a deserializer turns those bytes into something readable. You choose one for the key and one for the value in the search form of a topic's Messages tab, and KafkIO preselects the one that fits the topic best. This page explains that choice, what each deserializer does, and what to check when a message comes out as gibberish.

Choosing a deserializer

The Key deserializer and Value deserializer drop-downs list every deserializer that can be used on the topic, with the one KafkIO prefers at the top. Here the shipments topic has an Avro schema for its values, so Schema Registry is preselected for the value, while the key, a plain string, gets String (raw):

1 2 3

1. Key deserializer: how message keys are decoded.
2. Value deserializer: how message values are decoded.
3. Refresh Partitions and Serdes: reads the lists again, for example after a schema has been registered for the topic (see Refreshing partitions and deserializers).

KafkIO preselects, separately for key and value:

1. __consumer_offsets on Kafka's internal __consumer_offsets topic (below).
2. Schema Registry when the cluster has a Schema Registry set up and the topic has a subject of its own for that part of the message (see Subject naming strategies).
3. String (raw) otherwise.

Your choice applies to the next search, so to read the same messages another way, change the deserializer and search again. The choice stays with the topic: switch to another topic and back, and it's still selected.

Built-in deserializers

These work on any cluster, with or without a Schema Registry.

The number deserializers don't check the length: Int (32) on an 8-byte value reads only its first four bytes, and gives a wrong (but plausible-looking) number. If numbers look odd, check the Size column (4 bytes for a 32-bit number, 8 for a 64-bit one). Protobuf (raw) shows field numbers because, without the schema, the names aren't in the data; for names, register the schema and use Schema Registry.

When producing, most of these are also offered as serializers; Protobuf (raw) and __consumer_offsets are read-only (see Producing Messages). Each one's exact behaviour and limits are in the Serializers & Deserializers reference.

Consumer offsets

Kafka keeps consumer groups' committed offsets in its internal __consumer_offsets topic, in a binary format of its own. When you open that topic, KafkIO preselects its __consumer_offsets deserializer for both key and value, and shows each record as JSON. It isn't offered on any other topic.

1 2

1. The key deserializer and
2. the value deserializer, both preselected as __consumer_offsets.

There are two kinds of record:

• Offset commits: the key is {"group", "topic", "partition"} and the value {"offset", "leader_epoch", "metadata", "commit_timestamp"} (older brokers write slightly different fields). Each one is a group committing its position on one partition.
• Group metadata: the key is {"group"} and the value describes the group: protocol_type, generation, leader and its members (client ID, host, timeouts).

A text filter on the group name is a quick way to see a group's commit history. For the current offsets and lag, the Consumers section is easier to read.

Schema Registry formats

Producers that use a Schema Registry (Confluent's serializers, and compatible ones) don't write the schema into each message. They write a zero byte, the 4-byte ID of the schema in the registry, then the encoded data. The Schema Registry deserializer reads that ID, fetches the schema from the registry, and decodes the data with it. It supports all three schema types, and shows each message as JSON:

The message detail pane shows the schema ID and type of the message you select (see Schema IDs). Because the ID travels with every message, messages written with different versions of a schema on the same topic each decode with their own version.

The Schema Registry deserializer is available only when the cluster has a Schema Registry switched on in its settings, and the registry has a subject for that key or value, or subjects named after records (which could apply to any topic: see below). If it's missing from a list after you register a schema, use Refresh Partitions and Serdes.

Subject naming strategies

Schemas are registered under subjects, and producers choose how subjects are named. Decoding doesn't depend on the subject (it uses the schema ID in the message), so every strategy decodes the same way. The subjects only decide whether KafkIO offers and preselects Schema Registry for a topic:

KafkIO finds out which strategy a subject follows from the record names in its latest schema. Producing with a chosen subject is covered in Using a Schema Registry subject.

Avro display options

Two preferences change how decoded Avro messages look. Here a Shipment record with the defaults (both on): trackingNumber is a nullable string and destination a nullable Address record.

1 2

1. Schema ID 2, type AVRO: decoded by the Schema Registry deserializer.
2. Shown as JSON.

A union field with a value is wrapped in an object naming the branch that was written, as in Avro's own JSON encoding: "trackingNumber": {"string": "JD014600006399"}. A union field that is null is just null.

Change the options under File ▸ Preferences ▸ Kafka, in Schema Registry properties:

1 2 3

1. The Kafka page.
2. Avro: show null fields in decoded messages (on by default): null fields appear as "field": null. Off, they're left out, which makes records with many optional fields shorter, but hides which fields exist.
3. Avro: show fully qualified names in decoded messages (on by default): union branches are named in full, com.storefront.Address. Off, they get the short name, Address, unless two types in the union share a short name.

With both off, the same message reads:

{"orderId":104,"carrier":"DHL","trackingNumber":{"string":"JD014600006399"},"destination":{"Address":{"city":"Berlin","country":"DE"}}}

and a shipment without a tracking number has no trackingNumber field at all. The options apply from the next search, without reconnecting. They only affect Avro: JSON Schema and Protobuf messages look the same either way. See also Kafka & Timeouts.

Compression

There's nothing to choose for compression. Messages compressed by the producer or the topic with gzip, snappy, lz4 or zstd are decompressed by the Kafka client before the deserializer sees them, so they read like any other message. The Size column shows the uncompressed size of the key and value.

When decoding fails

If the chosen deserializer can't decode a key or value (a 4-byte value read with Int (64), data without a schema ID read with Schema Registry, a schema ID that isn't in the registry), KafkIO doesn't stop or show an error: it shows that key or value as String (raw) instead. Messages that decode are unaffected, so one bad message doesn't spoil a search.

The flip side is that a wrong deserializer is easy to miss. Binary data shown as text looks like this: the Avro topic orders-avro read with String (raw):

1

1. Value deserializer set to String (raw) on a topic written with a Schema Registry.

Signs that the deserializer is wrong, and what to do:

• Boxes, � and fragments of text: the data is binary. Select the message and look at its bytes with Display as HEX (pre-deserialization) or the Raw tab, which always show what Kafka stored, whatever the deserializer.
• The bytes start with 00 and then four bytes (here 00 00 00 01, schema ID 1): the message was written with a Schema Registry. Choose Schema Registry; if it isn't in the list, set up the registry for the cluster first. On the first search after you open a topic's messages, KafkIO also spots this itself and puts a small warning triangle on the drop-down (hover it for the advice), but later searches don't check again, so don't wait for it.
• You chose Schema Registry but the detail pane says Schema ID: none: that message couldn't be decoded with it and is shown as text. Either it wasn't written with a Schema Registry, or its schema isn't in the registry KafkIO is connected to (another environment's registry, or a deleted schema).
• Numbers that look wrong: compare the Size with the deserializer (4 or 8 bytes), and try the signed or unsigned variant.

The deserializer also matters beyond display: a text filter matches the decoded text, and a dump to file writes it.

Related

• Searching Messages
• Viewing Messages
• Producing Messages
• Connecting to Schema Registry
• Serializers & Deserializers