Schema Registry

The Schema Registry section shows the schemas in the cluster's Confluent-compatible Schema Registry, one row per subject, and lets you read every version of them, register new subjects and versions, check a change for compatibility before you make it, set compatibility levels, and delete versions or whole subjects.

To open it, connect the cluster and click Schema Registry under its name in the Kafka Clusters pane. The entry is greyed out until the cluster has a registry switched on in its settings (in the screenshot below, Storefront Prod (eu-west-1) has none).

1 2 3 4 5 6 7 8 9 10 11 12 13 14

1. Filter schemas...: show only the subjects whose name matches, as plain text or, with the .* toggle on, as a regular expression (Filtering).
2. Edit Schema: register a new version of the selected subject, or change its compatibility level (below).
3. Delete Schema: delete the selected subject's latest version, or all of it (below).
4. Refresh All Schemas: read every subject, and the default compatibility level, again.
5. Refresh this subject only.
6. Default compatibility level: the registry-wide level (below).
7. Create Schema: register a new subject (below).
8. The selected subject. Click it to copy its name to the clipboard.
9. Latest: the subject's latest version.
10. All Versions: every version of the subject (below).
11. The Schema ID and Version of the schema shown.
12. Search schema...: find text in the schema, with arrows to step through the matches and a count.
13. View: open the schema in a window of its own.
14. Duplicate Tab: open the section in a tab of its own, as was done here for Topics (Storefront Dev (Topics)), so you can keep a topic's messages and its schema side by side (Duplicating a tab).

The schemas list

Schemas are registered under subjects. A subject is usually named after the topic it's for and whether it describes the key or the value (customers-key, customers-value), but it can also be named after a record type, such as com.storefront.OrderPlaced (see Subject naming strategies). Each time a subject's schema changes, the registry keeps the old one and adds a new version. The list has one row per subject, showing its latest version:

Click a column header to sort by it (Sorting and refreshing); the count at the bottom of the list says how many subjects there are. Right-click a row for Refresh, Edit Schema and Delete Schema, and for copying the row or the whole list as text (Copying to the clipboard).

The list is read when you open the section. It isn't refreshed by itself when someone else changes the registry: use Refresh All Schemas 4 or the refresh button on a row 5. Changes you make in KafkIO show straight away. Subjects that have been soft-deleted aren't listed. If the list says No schemas to display; is a schema registry configured?, the registry has no subjects yet, or couldn't be read (an error says why).

Viewing schemas

Select a subject to see it in the tabs below the list. They stay greyed out until you select one. Like the detail tabs in other sections, you can drag the divider above them to give them more room, or undock them into a window of their own (Split panes and detail tabs).

Latest version

The Latest tab 9 shows the latest version's schema with syntax highlighting: Avro and JSON Schema as formatted JSON, Protobuf as .proto source. It's read-only; to change it, register a new version. Type in Search schema... 12 to find a field or type: matches are highlighted and the arrows move between them.

All versions

The All Versions tab lists every version of the subject that hasn't been deleted, oldest first. KafkIO reads them the first time you select a subject.

1 2 3

1. All Versions.
2. View: open that version in a window of its own.
3. Delete This Version (below; also on the row's right-click menu).

The Compatibility column shows the subject's current level on every row: the registry keeps one level per subject, not one per version.

Opening a schema in its own window

The View button on the Latest tab 13, and the eye on each row of All Versions, open the schema in a separate window titled Subject: name. These windows don't block the rest of KafkIO, so you can open several (two versions side by side, say) and keep working.

1 2

1. Search the schema, as on the Latest tab.
2. Always on top: on by default, so the window stays above the main window while you work there. Click it to let the window go behind.

The coloured corner is the cluster's colour, so you can tell which cluster a window belongs to.

Creating a schema

Click Create Schema 7 to register a schema under a new subject.

1 2 3 4 5

1. Subject: the name to register the schema under. Required.
2. Schema type: AVRO (the default), JSON (JSON Schema) or PROTOBUF.
3. Schema examples: replace the editor's content with a complete example of that type.
4. JSON validation: whether the text is well-formed JSON (Avro and JSON Schema only).
5. Create Schema: register it.

Subject naming

Name the subject the way your producers and consumers will look for it. With Confluent's serializers and their default naming strategy, that's the topic name plus -key or -value: a schema for the values of topic returns goes under returns-value. Other strategies name the subject after the record type (com.storefront.Return) or both (returns-com.storefront.Return). KafkIO uses the same names to decide which subjects to offer for a topic when producing and decoding messages.

Schema types

• AVRO: a compact binary format described by a JSON schema (records, enums, arrays, maps, unions and logical types such as dates and decimals).
• JSON: JSON Schema, for messages that are plain JSON.
• PROTOBUF: Protocol Buffers, written as .proto source.

The editor highlights the syntax of the type you choose. Pick the type before you paste in a schema, or the registry will try to read it as the wrong kind.

Starting from an example

The AVRO, Protobuf and JSON links 3 fill the editor with an example of that type, and set Schema type to match. Each describes the same ComplexEvent record, using most of what the format offers (nested records, enums, arrays, maps, optional fields, timestamps, decimals), so it doubles as a syntax reference: delete what you don't need and rename the rest. The link replaces whatever was in the editor.

JSON validation

For Avro and JSON Schema, KafkIO checks that the text is well-formed JSON shortly after you stop typing. JSON validation 4 then says success, or failed @ line n, with an info icon whose pop-up gives the line, column and parser message. This only checks the JSON, not the schema: the registry checks the schema itself when you create it, and you can still click Create Schema while validation fails (the registry will then refuse it). Protobuf schemas aren't validated in the dialog.

After creating

When the registry accepts the schema, a message gives its ID, version (1 for a new subject) and compatibility level, the details also appear in the dialog, and the subject appears in the list. The dialog stays open and switches to updating that subject, so you can carry on with a second version or set its compatibility level. If the registry refuses the schema (invalid syntax, a type that doesn't exist), an error says why and nothing is registered.

Schema references

KafkIO registers schemas without references: there's no way in the dialog to say that a schema uses a type defined under another subject (an Avro record type registered elsewhere, or a Protobuf import of another subject's file). Such a schema is refused, because the registry can't resolve the type. Register it with your build tooling or the registry's REST API instead. Subjects that use references are listed and viewed like any other, and messages written with them decode normally.

Registering a new version

To change a subject's schema, select it and click Edit Schema 2 (or right-click it). The Update Schema dialog opens with the latest version's schema in the editor. Edit it, then click Update Schema: the registry adds it as the subject's next version, as long as it's compatible with the earlier versions under the subject's compatibility level. Older versions are kept, so messages already written with them still decode.

1 2 3 4 5 6 7

1. Subject: fixed when updating.
2. Schema type: see Changing the schema type.
3. Compatibility level: the subject's current level, for reference.
4. Set Compatibility Level: the level to give the subject (below).
5. Set: apply that level.
6. Check Compatibility: test the schema in the editor without registering it.
7. Update Schema: register it as a new version.

The dialog doesn't block the main window, and there's one per subject: clicking Edit Schema again for the same subject brings its dialog to the front. When a version is registered, a message gives its schema ID and version number, and the list and tabs update.

Checking compatibility first

Check Compatibility 6 sends the schema in the editor to the registry, which tests it against the subject's existing versions under the subject's compatibility level, without registering anything. It answers Schema is compatible!, or lists the problems the registry found (a field added without a default, a type changed, and so on). Fix them and check again.

You don't have to check first: Update Schema runs the same test, and if the schema isn't compatible, nothing is registered and you get an error instead:

Here, giving the field a default ("type" : [ "null", "string" ], "default" : null) makes the change compatible, and it registers as version 4. Check Compatibility tells you which rule was broken, which this error doesn't.

Changing the schema type

Schema type can be changed when updating, for example to move a subject from JSON Schema to Avro. The registry treats a schema of another type as incompatible with the earlier versions, so it refuses the change unless the subject's compatibility level is NONE. If you really mean to switch formats, consider a new topic and subject instead: consumers would otherwise meet both formats on one topic.

Compatibility levels

A compatibility level is the rule the registry applies when a new version is registered, so that consumers and producers using different versions can still read each other's data:

The examples are for Avro; JSON Schema and Protobuf have their own rules for what counts as compatible, and the registry applies them. The info icon beside each compatibility drop-down in KafkIO summarises the levels too.

The default level

Default compatibility level 6, at the top of the section, is the registry-wide level. It applies to every subject that has no level of its own, including subjects created later (Confluent Schema Registry starts out with BACKWARD).

1

1. Default compatibility level.

Choose a level and confirm with Yes; Cancel puts the drop-down back. The Compatibility column then changes for every subject that follows the default.

Per subject

To give one subject its own level, select it, click Edit Schema, choose the level in Set Compatibility Level 4 and click Set 5. The level applies straight away and doesn't create a new version (the editor's content is ignored). Compatibility level 3 and the list show the new level.

Set Compatibility Level always starts at NONE, whatever the subject's current level, so pick the level you want before clicking Set. KafkIO can't remove a subject's own level to make it follow the default again; to match the default, set the same level.

Deleting schemas

Select a subject and click Delete Schema 3 (or right-click it).

1 2 3

1. Version: Latest Version removes only the latest version, so the one before it becomes the latest; All Versions removes the whole subject.
2. Soft delete: ticked by default (below).
3. Delete, then Yes to confirm.

Deleting the latest version of a subject that has only one version removes the subject from the list.

Soft and hard deletes

• Soft delete (the default) marks the versions as deleted. They disappear from KafkIO and from the subject's version list, and new versions are no longer checked against them, but the registry keeps them: their schema IDs still resolve, so messages already written with them still decode, and a registry administrator can still see them.
• Hard delete (untick Soft delete) removes them for good. The registry only allows a hard delete of something already soft-deleted, so KafkIO does both in one go.

Because KafkIO doesn't list soft-deleted subjects and versions, you can't hard-delete them from KafkIO later: untick Soft delete at the time if you want them gone for good.

Deleting one version

To delete a version other than the latest, open the subject's All Versions tab and click Delete This Version on its row (or right-click the row). The same dialog opens with that version fixed in Version; choose soft or hard delete as above. Remaining versions keep their numbers.

Schemas when producing and consuming

Schemas registered here are what KafkIO uses elsewhere:

• Reading messages: the Schema Registry deserializer reads the schema ID in each message and decodes it with that schema, whichever version it is (Schema Registry formats). The message detail pane shows the ID (Schema IDs), which you can look up in the Schema ID column here or on All Versions.
• Producing messages: choose Schema Registry as the serializer and pick a subject; KafkIO encodes your JSON with the subject's latest version (Using a Schema Registry subject). Producing never registers or changes schemas: register a new version here first.
• After you create a subject for a topic that's already open, use Refresh Partitions and Serdes on the topic for Schema Registry to be offered there.

Related

• Schema Registry Connection (URLs, authentication and TLS)
• Deserialization & Formats
• Producing Messages
• Viewing Messages
• The KafkIO Workspace (tabs, filtering, split panes)