Kafka Connect Connections

To see and manage your connectors in KafkIO (their status and tasks, restarting, pausing, editing, creating them), KafkIO needs the REST API of the Kafka Connect cluster that runs them. You set it up per Kafka cluster, on the Kafka Connect tab of the Cluster Properties dialog (Add Cluster, or Edit Cluster on a disconnected cluster; see Managing Cluster Configurations).

Like Schema Registry, Kafka Connect is a separate service with its own URL, login and TLS settings, which may or may not match the brokers'. Nothing on the Kafka Security tab applies to it. If several Connect clusters work with the same Kafka cluster (one per team, or one for sources and one for sinks), you can add them all, and KafkIO shows their connectors side by side.

Enabling Kafka Connect

Switch on Use Kafka Connect 1. A Kafka Connect Cluster Connection box appears below it, and two fields in it are required: Name 2 and URL 3. Until both are filled in, Save and Test Kafka Connect stay greyed out, with a red mark on the field that's missing. Everything else depends on how your Connect cluster is secured.

1 2 3 4 5 6 7 8

1. Use Kafka Connect turns Kafka Connect on for this cluster.
2. Name: the name KafkIO shows for this Connect cluster; see Naming a Connect cluster.
3. URL: the Connect REST API; see Endpoint URL.
4. Check network reachability opens the Network Tool with the host and port from the URL.
5. Use proxy (if configured); see Using a proxy.
6. User:password for Connect clusters that use basic authentication.
7. Add header adds a row of Additional HTTP headers, sent with every request.
8. The SSL/TLS settings, for an https:// URL; see TLS for Kafka Connect.

Once you've saved and connected the cluster, its Kafka Connect section (greyed out in the cluster tree otherwise) lists the connectors of every Connect cluster you've added, and the cluster's Overview shows Kafka Connect as available. Managing the connectors themselves is covered in Kafka Connect.

KafkIO doesn't contact Kafka Connect when it connects to the cluster, so a wrong setting here doesn't stop the cluster connecting: it shows up later, as an error when you open the Kafka Connect section. Run the test before you save. Switching Use Kafka Connect off later keeps every Connect configuration you entered (they're still saved), so you can switch it back on without retyping them.

One endpoint per Connect cluster

Each Kafka Connect Cluster Connection box takes exactly one URL. That's enough because a Connect cluster is a group of workers sharing one configuration: any worker's REST API answers for the whole cluster (it lists every connector, and forwards changes to the leader). So give one worker's URL, or better, the load balancer or DNS name in front of the workers if there is one, so KafkIO keeps working when a particular worker is down.

What you shouldn't do is add two workers of the same Connect cluster as two configurations: KafkIO would then list every connector twice, once for each. Add a second configuration only for a separate Connect cluster, one with its own group.id and its own connectors.

Adding more Connect clusters

Click Add Kafka Connect config 6, below the last box, for another empty Kafka Connect Cluster Connection. Each box has its own URL, login, headers and TLS settings, so Connect clusters run by different teams, or secured differently, can sit side by side.

1 2 3 4 5 6 7

1. Name of the second Connect cluster: it must differ from the first's.
2. User:password: this cluster's own login.
3. A header row: Header key, Header value, and remove to delete the row.
4. test config tests this box only.
5. remove deletes this box.
6. Add Kafka Connect config adds another box.
7. Test Kafka Connect tests every box in turn. It only shows while this tab is open.

Naming a Connect cluster

The Name is only a label for you: it doesn't have to match anything on the Connect side (the field's hint says "the value does not matter"). Choose something that tells your Connect clusters apart at a glance, such as storefront-connect or analytics-connect. KafkIO shows it in the Connect Cluster column of the connectors list, and you pick it in the Connect Cluster list when you create a connector, to say which cluster runs it.

Each name must be unique within the cluster, ignoring case: Storefront-Connect next to storefront-connect gets a red mark (Name is a duplicate of another configuration.) and the dialog won't save.

1 2

1. Connect Clusters: how many Connect configurations the cluster has (configured, not necessarily reachable).
2. Connect Cluster: which configuration, by name, each connector belongs to.

If one of several Connect clusters can't be reached, KafkIO still lists the connectors of the others and shows an error for the one that failed. In the current version that error appears as an Uncaught exception dialog rather than a clear message; the real cause (a timeout, an unknown host) is in the logs (View Logs).

Endpoint URL

Enter the base URL of the Connect REST API, starting with http:// or https:// and including the port: http://localhost:8083 for a local Connect worker (8083 is Kafka Connect's default REST port; the grey example in the field shows 8081, which is Schema Registry's), or the endpoint your platform gives you. Leave out paths such as /connectors: KafkIO adds them. With an https:// URL, the SSL/TLS settings further down apply.

Using a proxy

Tick Use proxy (if configured) when the Connect cluster can only be reached through the HTTP proxy set up under File ▸ Preferences ▸ Network (see Network, Proxy & Trust), for example a Connect cluster in the cloud seen from a corporate network. Leave it unticked for one on your own network or machine: KafkIO then connects directly, even if a proxy is configured. The box only takes effect while the proxy is switched on in the preferences, and the proxy's username and password, if it has them, are used too.

Authentication

A Connect cluster on a development machine is usually open. Secured ones typically sit behind basic authentication (Kafka Connect's own rest.extension.classes with a JAAS login, or a reverse proxy), a gateway that wants an API key header, or mutual TLS (see below).

Basic authentication

Type the username and password into User:password, separated by a colon: kafkio-reader:s3cret-from-your-vault. KafkIO sends them with every request. Leave the field empty if the Connect cluster has no login.

Additional HTTP headers

Add header adds a Header key / Header value row, and remove next to a row deletes it. KafkIO adds these headers to every request to this Connect cluster: use them for gateways or managed platforms that expect an API key, a tenant ID or a bearer token in a header (an Authorization header with a token, for example, when basic authentication isn't an option). A token pasted here is sent as it is until you replace it.

TLS for Kafka Connect

The SSL/TLS section of each box applies when its URL starts with https://. It has the same fields as the Schema Registry tab's: Truststore file, Truststore type and Truststore password for the certificates to trust, SSL validation, and Two-way SSL/TLS? with the keystore holding your client certificate. They're separate from the brokers' and the registry's settings, and from each other box's, so if several need the same files, choose them in each.

• Certificate signed by a public certificate authority: fill in nothing. Leave Truststore file blank.
• Signed by your organisation's own CA, or self-signed: choose a truststore that holds the CA certificate, or add the certificate once under File ▸ Preferences ▸ Network (Trusted certificates), which then covers the Schema Registry and Kafka Connect endpoints of every cluster (not ksqlDB).
• Mutual TLS: also switch on Two-way SSL/TLS? and choose the keystore with your client certificate and key, as for brokers.

SSL validation is unticked in a new box, so KafkIO checks that the certificate is trusted but not that it was issued for the host in the URL. Tick it for anything beyond local testing (see Hostname validation). A truststore or keystore file you choose must exist, or the dialog won't save.

Removing a Connect configuration

With two or more boxes, each has a remove link 5 (in the second screenshot) that deletes it straight away, without asking. The change only becomes permanent when you click Save, so Cancel brings a box back if you removed the wrong one.

The last remaining box has no remove link. To stop using Kafka Connect for the cluster, switch Use Kafka Connect off and save: the settings stay in the configuration (but aren't used), and the Kafka Connect section is greyed out again.

Testing the connection

Test Kafka Connect 7 builds a client for each box from the tab as it is now (nothing needs saving first) and asks each Connect cluster for its installed connector plugins, a request any working Connect REST API and login can answer. If every one answers, you see A connection was successfully made! (or All connections were successfully made! for several). It stops at the first one that fails and shows an Error connecting dialog with the underlying error, and the full exception under Show Details.

The error doesn't say which box failed, so with several Connect clusters use each box's test config link 4 instead: it tests that box alone. A test on a box whose name or URL is missing or invalid is greyed out, like the button.

Errors you're likely to see, and what they usually mean:

A successful test covers the connection only: whether your login may also create, change or delete particular connectors is decided by Connect when you do it. More on reading errors in Troubleshooting Connections.

Related

• Kafka Connect (managing connectors and tasks)
• Managing Cluster Configurations
• Schema Registry Connection
• ksqlDB Connections
• Provider Recipes
• Troubleshooting Connections
• Network, Proxy & Trust
• Kafka & Timeouts