ksqlDB Connections

To run KSQL and browse ksqlDB's streams, tables, persistent queries, topics and connectors from KafkIO, KafkIO needs the REST API of the ksqlDB server (or cluster of servers) that works with your Kafka cluster. You set it up per Kafka cluster, on the ksqlDB tab of the Cluster Properties dialog (Add Cluster, or Edit Cluster on a disconnected cluster; see Managing Cluster Configurations).

ksqlDB is a separate service with its own URL, login and TLS settings, which may or may not match the brokers'. Nothing on the Kafka Security tab applies to it. If several ksqlDB clusters work with the same Kafka cluster (one per team or application, say), you can add them all and switch between them in KafkIO.

Enabling ksqlDB

Switch on Use ksqlDB 1. A ksqlDB Cluster Connection box appears below it, and two fields in it are required: Name 2 and URL 3. Until both are filled in, Save and Test ksqlDB stay greyed out, with a red mark on the field that's missing (an empty name's message says "Please enter a Kafka Connect cluster name.": it means the ksqlDB name). Everything else depends on how your ksqlDB server is secured.

1 2 3 4 5 6 7 8

1. Use ksqlDB turns ksqlDB on for this cluster.
2. Name: the name KafkIO shows for this ksqlDB cluster; see Naming a ksqlDB cluster.
3. URL: the ksqlDB REST API; see Endpoint URL.
4. Check network reachability opens the Network Tool with the host and port from the URL.
5. Use proxy (if configured); see Using a proxy.
6. User:password for ksqlDB servers that use basic authentication.
7. Add header adds a row of Additional HTTP headers, sent with every request.
8. The SSL/TLS settings, for an https:// URL; see TLS for ksqlDB.

Once you've saved and connected the cluster, its ksqlDB section (greyed out in the cluster tree otherwise) opens on the KSQL editor of one of the ksqlDB clusters you added, and the cluster's Overview shows ksqlDB as available. Using it is covered in ksqlDB.

KafkIO doesn't contact ksqlDB when it connects to the cluster, so an unreachable ksqlDB server doesn't stop the cluster connecting: it shows up later, as an error when you run KSQL or open one of the ksqlDB tabs. Run the test before you save. (One exception: a URL KafkIO can't parse, such as one ending in /, does stop the cluster connecting; see Endpoint URL.) Switching Use ksqlDB off later keeps every ksqlDB configuration you entered (they're still saved), so you can switch it back on without retyping them.

One endpoint per ksqlDB cluster

Each ksqlDB Cluster Connection box takes exactly one URL, and the tab's own hint says so: "Only define ONE endpoint in any ksqlDB cluster." A ksqlDB cluster is a group of servers sharing one ksql.service.id: they share the same streams, tables and persistent queries, so any one server's REST API shows the whole cluster. Give one server's URL, or better, the load balancer or DNS name in front of the servers if there is one, so KafkIO keeps working when a particular server is down.

Don't add two servers of the same ksqlDB cluster as two configurations: you'd only see the same streams and tables twice under two names. Add a second configuration only for a separate ksqlDB cluster, one with its own ksql.service.id.

Adding more ksqlDB clusters

Click Add ksqlDB config 6, below the last box, for another empty ksqlDB Cluster Connection. Each box has its own URL, login, headers and TLS settings, so a local ksqlDB server and a managed one in the cloud, for example, can sit side by side.

1 2 3 4 5 6 7

1. Name of the second ksqlDB cluster: it must differ from the first's.
2. URL: https:// with no port, so KafkIO uses 443.
3. User:password: this cluster's own login, here an API key and secret.
4. test config tests this box only.
5. remove deletes this box.
6. Add ksqlDB config adds another box.
7. Test ksqlDB tests every box in turn. It only shows while this tab is open.

Naming a ksqlDB cluster

The Name is only a label for you: it doesn't have to match anything on the ksqlDB side (the field's hint says "the value does not matter"). Choose something that tells your ksqlDB clusters apart at a glance, such as storefront-ksqldb or analytics-ksqldb: it's what you pick in the cluster's ksqlDB section.

Each name must be unique within the cluster, ignoring case: Storefront-KSQLDB next to storefront-ksqldb gets a red mark (Name is a duplicate of another configuration.) and the dialog won't save.

Switching between ksqlDB clusters

Unlike Kafka Connect, where the connectors of every Connect cluster are listed together, the ksqlDB section works with one ksqlDB cluster at a time. Pick it in the ksqlDB cluster list at the top right 1; every tab below (KSQL Editor, Streams, Tables and the rest) then shows that cluster, and KafkIO keeps a separate editor and views for each, so switching back doesn't lose your KSQL. To work with two at once, duplicate the cluster's tab and pick a different ksqlDB cluster in each.

1 2

1. ksqlDB cluster: one entry per configuration, by name.
2. Refresh ksqlDB Cluster List.

Endpoint URL

Enter the base URL of the ksqlDB REST API: http:// or https://, the host name and, usually, the port. http://localhost:8088 is a local ksqlDB server (8088 is ksqlDB's default listener port; the grey example in the field shows 8081, which is Schema Registry's). Without a port, KafkIO uses 80 for http:// and 443 for https://, so a managed endpoint such as https://ksqldb.analytics.example.com works as your provider gives it. With an https:// URL, the SSL/TLS settings further down apply.

KafkIO talks to ksqlDB with ksqlDB's own client, which uses HTTP/2. A load balancer, gateway or proxy between you and ksqlDB must pass HTTP/2 through; a URL that points at something that only speaks HTTP/1.1 (another service, or the wrong port) fails with an Http2Exception rather than an HTTP error.

Using a proxy

Tick Use proxy (if configured) when the ksqlDB server can only be reached through the proxy set up under File ▸ Preferences ▸ Network (see Network, Proxy & Trust), for example a managed ksqlDB in the cloud seen from a corporate network. Leave it unticked for one on your own network or machine: KafkIO then connects directly, even if a proxy is configured. The box only takes effect while the proxy is switched on in the preferences, and the proxy's username and password, if it has them, are used too. KafkIO connects to the proxy as an HTTP proxy.

Authentication

A ksqlDB server on a development machine is usually open. Secured ones typically use basic authentication (ksqlDB's own authentication.method=BASIC with a JAAS realm, or a managed service's API key and secret), a gateway that wants a header, or mutual TLS (see below).

Basic authentication

Type the username and password into User:password, separated by a colon: kafkio-reader:s3cret-from-your-vault, or for a managed ksqlDB such as Confluent Cloud's, the ksqlDB API key and secret as KEY:SECRET. KafkIO sends them with every request. Leave the field empty if the ksqlDB server has no login.

Additional HTTP headers

Add header adds a Header key / Header value row, and remove next to a row deletes it. KafkIO adds these headers to every request to this ksqlDB cluster: use them for gateways or platforms that expect an API key, a tenant ID or a token in a header. A token pasted here is sent as it is until you replace it.

TLS for ksqlDB

The SSL/TLS section of each box applies when its URL starts with https://. It has the same fields as the Schema Registry tab's: Truststore file, Truststore type and Truststore password for the certificates to trust, SSL validation, and Two-way SSL/TLS? with the keystore holding your client certificate. They're separate from the brokers' settings and from each other box's, so if several need the same files, choose them in each.

• Certificate signed by a public certificate authority (managed services): fill in nothing. Leave Truststore file blank.
• Signed by your organisation's own CA, or self-signed: choose a truststore (JKS, P12 or PEM) that holds the CA certificate in Truststore file.
• Mutual TLS: also switch on Two-way SSL/TLS? and choose the keystore with your client certificate and key, as for brokers.

SSL validation is unticked in a new box, so KafkIO checks that the certificate is trusted but not that it was issued for the host in the URL. Tick it for anything beyond local testing (see Hostname validation). If Accept non-trusted certificates automatically is ticked in the network preferences, KafkIO trusts any ksqlDB certificate, whatever the truststore says. A truststore or keystore file you choose must exist, or the dialog won't save.

Removing a ksqlDB configuration

With two or more boxes, each has a remove link 5 (in the second screenshot) that deletes it straight away, without asking. The change only becomes permanent when you click Save, so Cancel brings a box back if you removed the wrong one.

The last remaining box has no remove link. To stop using ksqlDB for the cluster, switch Use ksqlDB off and save: the settings stay in the configuration (but aren't used), and the ksqlDB section is greyed out again.

Testing the connection

Test ksqlDB 7 builds a client for each box from the tab as it is now (nothing needs saving first) and sends each ksqlDB cluster a harmless statement (SHOW VARIABLES;), which any working ksqlDB REST API and login can answer. It waits up to 10 seconds for the answer. If every cluster answers, you see A connection was successfully made! (or All connections were successfully made! for several). It stops at the first one that fails and shows an Error connecting dialog with the underlying error, and the full exception under Show Details.

The error doesn't always say which box failed, so with several ksqlDB clusters use each box's test config link 4 instead: it tests that box alone. A test on a box whose name or URL is missing or invalid is greyed out, like the button.

Errors you're likely to see, and what they usually mean:

A successful test covers the connection only: whether your login may also create streams, run queries or drop tables is decided by ksqlDB when you do it. More on reading errors in Troubleshooting Connections.

Related

• ksqlDB (the KSQL editor, streams, tables and queries)
• Managing Cluster Configurations
• Kafka Connect Connections
• Schema Registry Connection
• Provider Recipes
• Troubleshooting Connections
• Network, Proxy & Trust
• Kafka & Timeouts