Managing Cluster Configurations

Every cluster in the Kafka Clusters list is a saved configuration: its name, colour and bootstrap servers, its security settings, and optionally its Schema Registry, Kafka Connect and ksqlDB endpoints. This page covers the life of a configuration: adding, editing, testing, duplicating and deleting it, connecting with it, and the file KafkIO keeps it in. The individual settings are described in the rest of this part, starting with General Connection Settings.

Where the cluster actions are

Every action on this page works on the cluster selected in the Kafka Clusters list, and is in three places: the Cluster menu, the toolbar, and the cluster's own ⋮ menu at the right of its row. Which ones are available depends on whether the cluster is connected.

1 2 3 4 5

1. Disconnect (and Connect above it, greyed out here because the cluster is already connected).
2. Edit Cluster: greyed out while the cluster is connected.
3. Duplicate Cluster: available whether or not it's connected.
4. Delete Cluster: also available either way.
5. The connection icon: green bars when connected. Click it to connect or disconnect.

In the toolbar, the same actions are the buttons after Add Cluster: edit (pencil), duplicate, delete, then connect and disconnect. The KafkIO Workspace labels each one.

Adding a cluster

Click Add Cluster in the toolbar, or choose File ▸ Add Cluster. (While the list is empty, an Add a cluster button in it does the same.) The Cluster Properties dialog opens with five tabs:

The minimum is a Name and the Bootstrap servers, plus whatever security the cluster requires. Test the settings (see below), then click Save: the cluster is added to the end of the list, not connected. For a managed service, Provider Recipes lists the settings to use; for a first cluster step by step, see the Quick Start.

Names don't have to be unique, but give each cluster a distinct one that includes its environment (Payments Prod, not Kafka), together with a colour: it's how you tell clusters apart in the list, on tabs and in the status bar.

Editing a cluster

Select the cluster and click Edit Cluster (the pencil in the toolbar, the Cluster menu or the row's ⋮ menu). The same dialog opens, titled name - Cluster Properties, with the saved settings filled in. Change what you need and click Save; KafkIO confirms with Saved cluster config. Cancel leaves the saved configuration as it was.

Edit Cluster is only available while the cluster is disconnected: a connection is set up from the settings when you connect, so changing them underneath an open connection wouldn't take effect. To change a connected cluster, disconnect it first (its tabs close), edit, save, and connect again. To look at a connected cluster's settings without disconnecting, open its Overview: KafkIO Cluster Configuration at the bottom shows them as JSON (see Cluster Overview).

Renaming a cluster or changing its colour keeps everything else attached to it, including its favourite topics: KafkIO identifies a configuration by an internal id, not by its name.

Testing a configuration

The buttons at the bottom left of the dialog try the settings as they are in the dialog, without saving them, so you can fix a problem before it's stored. Which ones show depends on the tab you're on:

1 2 3

1. Test Connection tests the Kafka cluster itself (bootstrap servers and security). It's there on every tab.
2. The tab you're on decides the second button.
3. Test Schema Registry tests the registry settings on this tab. On the Kafka Connect and ksqlDB tabs it's Test Kafka Connect and Test ksqlDB.

A service's test button is greyed out until that service is switched on (for example Use Schema Registry) and its settings are valid. When a test succeeds, a Connection success window says A connection was successfully made!; when it fails, an error dialog shows what went wrong, with the full exception under its details. A timeout is the most common failure and often a misleading one: it can mean the host or port is unreachable, but also that the security settings don't match what the broker expects. Troubleshooting Connections goes through the usual causes, and Check network reachability under a host field opens the Network Tool to tell them apart.

Saving and validation errors

Save stays greyed out while anything in the dialog is incomplete or invalid. The field at fault has a small red mark at its bottom left; hover over the mark to see the problem.

1 2

1. The field with the problem, its red mark, and the message shown when you hover over the mark.
2. Save, greyed out until every problem is fixed.

The checks, by tab:

These checks are about form only: KafkIO doesn't contact the cluster when you save, so a configuration with a wrong host or password saves happily and fails when you connect. That's what the test buttons are for. If the file itself can't be written (a read-only folder, a full disk), an error dialog says Error saving config file. and nothing is changed.

Duplicating a configuration

Duplicating is the quickest way to set up a cluster that shares most of its settings with one you have, such as Staging from Dev, or the same cluster with a different user. Select the cluster, click Duplicate Cluster and confirm with Duplicate. KafkIO saves a copy called name (copy) at the end of the list:

1 2

1. The copy: every setting of the original, including its colour, security settings, Schema Registry, Kafka Connect and ksqlDB endpoints, and its favourite topics. It isn't connected.
2. Edit Cluster, to rename and adjust the copy.

You can duplicate a connected cluster; the copy is a separate configuration and connects on its own. Edit the copy straight away: give it a name and colour of its own, then change what differs (bootstrap servers, credentials). The copy keeps the original's passwords and certificate paths, so a copy of a production cluster can still reach production until you change them.

Deleting a configuration

Select the cluster, click Delete Cluster (the bin) and confirm with Delete in the Delete name window. KafkIO removes the configuration from its list and its file, with everything stored in it: security settings and passwords, service endpoints and favourite topics. You can delete a connected cluster too; its tabs close.

Nothing changes on the Kafka cluster itself; you can add it again at any time. Files the configuration pointed to, such as truststores and keystores, stay where they are. There's no undo, but until KafkIO next saves a cluster change, the previous version of the file is still in clusters.json.bak (see below).

Connecting and disconnecting

Saving a cluster doesn't connect to it, and KafkIO doesn't reconnect anything when it starts: you connect when you need a cluster. To connect, double-click the cluster in the list, click its connection icon, select it and press →, or use Connect in the toolbar or the Cluster menu. You can have several clusters connected at once.

While connecting, KafkIO covers the window with its progress (Testing connection..., then Getting topic and broker info...) and a Cancel link. It sets up fresh clients from the saved configuration and your current Preferences (timeouts, proxy), checks that the cluster answers, and reads its brokers and topics. When it's done, Connected to name appears, the status bar says name: connected, and the cluster expands into its sections; click one to open the cluster's tab. If connecting fails, the error dialog explains why and the cluster stays disconnected.

To disconnect, use Disconnect (toolbar, Cluster menu, the row's ⋮ menu or a cluster tab's right-click menu), click the connection icon, or select the cluster and press ←. KafkIO asks first, with Are you sure you want to disconnect; any retrieved data will be lost?:

Disconnecting and connecting again is also how you apply changes that only take effect on connect: an edit to the cluster's settings, or new timeouts or proxy settings in Preferences.

Where configurations are stored

The clusters.json file

All your cluster configurations are kept in one file, clusters.json, in the config folder of KafkIO's data folder:

You don't need to remember these: Tools ▸ Show Config In opens the file's folder in a Terminal or File Browser, and Text Editor opens the file itself, wherever it is.

1 2

1. Tools ▸ Show Config In.
2. Text Editor opens clusters.json itself; see Config & Log Files.

KafkIO rewrites the whole file every time you add, edit, duplicate, delete or reorder a cluster, or star a favourite topic. Before each write it copies the previous version to clusters.json.bak in the same folder, which is your one step of undo.

Editing the file by hand

The file is plain, indented JSON: a version (the KafkIO version that last wrote it) and a clusters list, one entry per cluster in list order. Each entry has an id and a clusterProperties object holding what the dialog shows:

{
  "version" : "2.1.18",
  "clusters" : [ {
    "id" : "02a2f7b3-4156-5290-989a-10f448c7977d",
    "clusterProperties" : {
      "name" : "Storefront Dev",
      "bootstrapServers" : "localhost:9092",
      "color" : "#2E7D32FF",
      "security" : { "securityType" : "PLAINTEXT", ... },
      "favourites" : { "topics" : [ "orders", "payments" ] },
      ...
    }
  } ]
}

The dialog is the safer way to change a cluster, but a text editor is handy for bulk changes, such as replacing a hostname in twenty clusters. If you do:

• Quit KafkIO first. KafkIO reads the file when it starts and writes its own copy of the list back on many actions (reordering, starring a topic), so edits made while it runs can be overwritten, and won't show until a restart anyway.
• Keep a copy of the file before you start.
• Give every cluster its own id. If you copy an entry to make a new cluster, change the copy's id (any unique text works; a new UUID is best). Two entries with the same id confuse editing and deleting.
• Keep it valid JSON: a missing comma or quote is enough to break the whole file.

Upgrading from older versions

Versions of KafkIO before 2.1.7 kept the cluster list in clusters.yml in the same folder. The first time a newer version starts, if there's a clusters.yml and no clusters.json, KafkIO copies it across and converts any older layouts to the current one; you don't need to do anything. The old clusters.yml is left in place, unused, and you can delete it once you're happy everything came across. Note that it is only read when clusters.json doesn't exist yet: editing it later has no effect.

Sharing configurations between machines

KafkIO has no import or export for clusters; the file is the unit of sharing. To move your clusters to another computer, or hand a set of team clusters to a colleague:

1. Quit KafkIO on both machines.
2. Copy clusters.json into the other machine's config folder (see the table above). This replaces its whole cluster list, so to merge, copy individual entries from the clusters list instead, keeping each id unique.
3. Start KafkIO there and check each cluster: certificate and keystore files are referenced by their full path, so copy those files too and fix the paths with Edit Cluster if they differ (the dialog won't save while a referenced file doesn't exist).

Remove passwords from a copy you share with other people, and let each person enter their own. To carry your clusters with you rather than copy them, run KafkIO in Memory Stick Mode, which keeps clusters.json (and your truststore) next to the program, for example on a USB stick; turning it on offers to copy your current configuration across.

Related

• Quick Start
• The KafkIO Workspace
• General Connection Settings
• Security & Authentication
• Troubleshooting Connections
• Memory Stick Mode
• File Locations