Serializers & Deserializers

A serde (serializer/deserializer) turns a message's bytes into text you can read, and the text you type into bytes. This page describes each of KafkIO's serdes in turn: exactly what it reads and writes, when it's preselected, and where it stops working. How to pick one is covered where you pick it: the Key deserializer and Value deserializer in Deserialization & Formats, and the key and value serializers in Producing Messages.

At a glance

The drop-downs list the preselected serde first, then the others that apply to the topic in this order: Schema Registry, String (raw), Int (32), Int (64), Unsigned Int (32), Unsigned Int (64), Hex, MessagePack and Protobuf (raw). The Produce dialog leaves out the read-only ones. There are no settings per serde; the only options are the two Avro display options in Preferences.

Rules for every serde

• Nulls skip the serde. A null key or value is never passed to a deserializer: it shows as <null> whatever you chose. When producing, Send null key and Send null message send a null without serializing anything (see Sending true nulls).
• A failed read falls back to text. When a deserializer can't decode a key or value, that key or value is shown with String (raw) instead, without an error (see When decoding fails). Several serdes below can also "succeed" on data that isn't theirs and show a plausible but meaningless result: their sections say when.
• A failed write sends nothing. When a serializer can't use what you typed, an Exception Details window says Error sending message(s), followed by the reason, and the message isn't sent. The reasons quoted on this page are the ones KafkIO shows.
• Variables come first. Substitution variables are replaced before the serializer sees the text, so ${counter} works with Int (32), and the result must still suit the serializer.

String (raw)

Reads the bytes as UTF-8 text, unchanged. Bytes that aren't valid UTF-8 show as � or as boxes; that's the sign of binary data, which another serde (or Hex) reads properly. Writes what you type as UTF-8, exactly: nothing is trimmed or reformatted, so a trailing space or line break is part of the message.

Preselected for keys and values unless one of the rules for __consumer_offsets or Schema Registry applies, and used for every key or value another deserializer fails on. It's the right choice for plain text, JSON, XML and CSV. Only UTF-8 is supported: text in another character set, such as Windows-1252, shows its accented characters as �.

Int and Unsigned Int

The four number serdes read and write integers in big-endian byte order, as Kafka's own serializers do:

Reading:

• Only the first 4 (or 8) bytes are read; any further bytes are ignored without a warning. Text read as a number gives a number too: the key customer-9 read with Int (32) shows 1668641652, which is just the bytes of cust. Check the Size column: 4 or 8 bytes is what a real number takes.
• A value shorter than that can't be read, and is shown as text.
• Signed and unsigned read the same bytes differently: FF FF FF F9 is -7 with Int (32) and 4294967289 with Unsigned Int (32). Numbers can't tell you which one the producer meant; ask, or use the one that gives sensible values.

Writing: type a whole number in decimal, with an optional + or - (no - for the unsigned ones). Anything else fails with a NumberFormatException, including a space or line break before or after the number (For input string: " 42"), a decimal point (42.0), thousands separators, and a number out of range (2147483648 with Int (32)).

There's no serde for 16-bit integers, floating-point numbers (Kafka's FloatSerializer and DoubleSerializer) or little-endian data. Read those with Hex and work the value out from the bytes; send them by typing the bytes in hex.

Hex

Reads any data: every byte becomes two upper-case hex digits, separated by spaces (CA FE 00 01). Nothing can fail, so use it to see exactly what a message holds, or when no other serde fits. Unlike Display as HEX (pre-deserialization) in the message detail pane, it puts the hex in the table, so you can filter on byte sequences and dump them to a file.

Writes the bytes you type in hex, which is how to send binary data you can't type as text: a Protobuf payload, a value in another serializer's format, a deliberately malformed message for testing.

• Upper or lower case, with or without spaces, colons and # anywhere: CA FE 00 01, ca:fe:00:01 and #cafe0001 all send the same four bytes.
• Spaces and line breaks at the start and end are ignored. An empty key or message sends zero bytes.
• Not accepted: line breaks or tabs between the digits, so a multi-line hex dump has to be joined onto one line first; 0x prefixes; commas; and an odd number of digits (IllegalArgumentException: string length not even: 3).

MessagePack

MessagePack is a compact binary form of JSON-like data.

Reads one MessagePack value from the start of the bytes and shows it as compact JSON, for example {"sku":"A-1","qty":3,"price":1.5,"tags":["new",null],"ok":true}.

Anything after that first value is ignored, which makes this serde easy to misread on the wrong data: many single bytes are complete MessagePack values. JSON text read with MessagePack shows 123 for every message (the code of its opening {), and a 4-byte integer 00 00 00 2A shows 0. If every row shows the same small number, the topic isn't MessagePack.

Writes JSON as MessagePack. Any JSON value works: an object, an array, a number, true, false, null, or a string in double quotes ("hello"; unquoted text fails with JsonParseException: Unrecognized token 'hello'). Whole numbers are sent as MessagePack integers and numbers with a decimal point as 64-bit floats. MessagePack features JSON can't express, such as binary data, extension types and non-string map keys, can't be typed.

Protobuf (raw)

Reads a Protobuf message without its schema, in Protobuf's text format. Without the .proto file there are no field names or types, only what's on the wire: field numbers, and values by their wire type. For example:

1: 150
2: "testing"
3: {
  1: 1
}

• Varint fields show as unsigned numbers, so negative int32/int64 values appear as very large numbers, sint values as their zig-zag encoding, and booleans and enums as 0/1/2.
• Fixed-width fields (fixed32, float, double and so on) show as hex, such as 0x3ff8000000000000.
• A length-delimited field is shown as a nested message in { } when its bytes happen to parse as one, and as a quoted string otherwise. Strings, bytes and nested messages look the same on the wire, so this is a guess.

Data that isn't valid Protobuf is shown as text, but random data occasionally parses: the MessagePack string "hello" reads as 1668: 0x6f6c6c65. Messages written with a Schema Registry don't parse (their first byte, 00, isn't a valid Protobuf start) and are shown as text: use Schema Registry for those, which also gives the field names. Read only: to send Protobuf without a registry, encode it elsewhere and send the bytes with Hex.

__consumer_offsets

Reads Kafka's internal __consumer_offsets topic, where brokers record consumer groups' committed offsets and membership, as JSON. It's offered, and preselected for both key and value, on that topic only, and it's read only. What the records contain is explained under Consumer offsets.

It understands offset commits (key versions 0 and 1) and classic group metadata (key version 2), with value versions 0 to 4. Records of any other kind, which newer brokers may write (for example for groups using the new consumer rebalance protocol), aren't decoded and are shown as text. For a group's current offsets and lag, Consumers is easier to read than this topic.

Schema Registry

Reads and writes data in the Confluent Schema Registry wire format: a 00 byte, the 4-byte ID of the schema in the registry, then the encoded data (for Protobuf, also the index of the message type in the .proto file). It handles all three schema types; the sections below give the details of each.

When it's offered. Only on clusters with Schema Registry switched on. On such a cluster, it's listed for a topic's key (or value) when the registry has the subject topic-key (or topic-value), or has any subject that doesn't end in -key or -value, since a record-named subject could belong to any topic. So on a registry with only -key/-value subjects, topics without one of their own don't offer it at all. It's preselected when the topic has a subject of its own (see Subject naming strategies). After you register a schema, click Refresh Partitions and Serdes to update the lists.

Reading doesn't depend on subjects: KafkIO takes the schema ID from each message, fetches that schema and decodes with it, so every message on a topic decodes with its own schema version. The result is always JSON. A message without the 00 byte and ID, or whose ID isn't in the registry KafkIO is connected to, is shown as text, with Schema ID: none in the detail pane (see Schema IDs).

Writing uses the Subject you pick next to the serializer, and that subject's latest schema version at the moment you send. You always type JSON; KafkIO checks it against the schema and encodes it. It never registers or changes schemas: do that in the Schema Registry section. Choosing a subject is described in Using a Schema Registry subject.

Avro

Decoded Avro is shown in Avro's JSON encoding, with logical types made readable:

Whether null fields appear and how union branches are named depends on the Avro display options. When producing:

• A field you leave out, or set to null, is written as null. That's fine for a field whose type is a union with null; for any other field it fails, even if the schema gives it a default: defaults aren't filled in. The error doesn't name the field (for a string field it's NullPointerException: Cannot invoke "java.lang.CharSequence.toString()" because "charSequence" is null), so check that every non-nullable field is there.
• Fields that aren't in the schema are ignored without a warning, so a misspelt field name looks like a missing field.
• Text that isn't JSON fails with String is not a valid json.

JSON Schema

Reads the JSON as stored, shown compactly. Writes what you type after validating it against the schema; whitespace and line breaks are removed before sending. A message that doesn't fit names the problem, for example does not fit schema: [#: required key [event] not found] or [#/count: expected type: Integer, found: String].

Protobuf

Reads into JSON using Protobuf's standard JSON mapping, with three things to know:

• Field names are as written in the .proto file (tracking_code), not converted to camel case.
• Every field is shown, including those left at their default value (0, "", false, the first enum value), which Protobuf doesn't actually store.
• 64-bit integers (int64, uint64 and similar) are shown as strings, "parcel_id":"12345678901", enums by name, and bytes in Base64.

Writes JSON in the same mapping. Field names can be the .proto names or their camel-case form (tracking_code or trackingCode); enums by name or number; 64-bit integers as numbers or strings. A field that isn't in the message type fails (Cannot find field: colour in message com.storefront.Parcel), and fields you leave out are sent at their default value. The message type is the one a record-named subject is named after; for a topic-value (or -key) subject, it's the first message type in the file. Imported .proto files are resolved through the registry, as Confluent's serializer does.

Related

• Deserialization & Formats
• Producing Messages
• Viewing Messages
• Schema Registry Connection
• Schema Registry
• Substitution Variables