Provider Recipes

Each recipe below lists what to put in the Cluster Properties dialog for one kind of Kafka service: the bootstrap servers, the Kafka Security settings, and the Schema Registry, Kafka Connect and ksqlDB tabs where the service has them. The recipes are short on purpose; each field is explained on its own page (General Connection Settings, Security & Authentication, Schema Registry Connection, Kafka Connect Connections and ksqlDB Connections), and the recipes link there.

Values in CAPITALS or with xxxxx in them are placeholders: copy the real ones from your provider's console. After filling in a tab, click Test Connection (and the tab's own test button) before you save; if it fails, see Troubleshooting Connections.

At a glance

Truststore Blank means the service's certificates are signed by a public certificate authority, which KafkIO trusts already (see Truststores). With TLS, tick SSL validation for every managed service: their certificates match their host names, and the box is unticked in a new cluster (see Hostname validation).

Local Kafka in Docker

For a broker running in Docker on your own machine, with KafkIO outside Docker:

• Bootstrap servers: localhost:9092, or whichever host port the broker's port is published on.
• Security type: Plaintext, unless you've set up a SASL or TLS listener.
• Schema Registry, Kafka Connect and ksqlDB, if you run them too: http://localhost:8081, http://localhost:8083 and http://localhost:8088 (their default ports), with the ports published the same way.

What makes or breaks it is the broker's advertised listener: it must say localhost:9092 (the address KafkIO can reach), not the container's name. A wrong one connects and then times out. See Listeners and advertised listeners, and for a file that's set up correctly, the Sample Docker Compose file (Tools ▸ Sample Docker Compose). The REST services (Schema Registry, Connect, ksqlDB) have no such setting: a published port is enough.

Confluent Cloud

Confluent Cloud issues a separate API key for each resource: one for the Kafka cluster, one for Schema Registry, one for each ksqlDB cluster. A key for one doesn't work on another, so create the ones you need in the Confluent Cloud console (or with confluent api-key create --resource ...) and keep them apart.

Kafka

• Bootstrap servers: the cluster's bootstrap server from Cluster settings, for example pkc-xxxxx.eu-west-1.aws.confluent.cloud:9092.
• On Kafka Security, as below: SASL SSL, the PLAIN example, with the Kafka API key as the user name and its secret as the password. Leave the truststore blank.

1 2 3 4 5

1. Security type: SASL SSL.
2. SSL validation: ticked. The truststore fields above it stay blank.
3. SASL type: PLAIN, set by the example.
4. JAAS config: the Kafka API key as username, its secret as password.
5. The PLAIN example: click it first, then replace the placeholders.

Schema Registry

• URL(s): the environment's Schema Registry endpoint, for example https://psrc-xxxxx.eu-west-1.aws.confluent.cloud (no port, no path).
• User:password: the Schema Registry API key and secret, as KEY:SECRET. Not the Kafka key.
• Or, with OAuth instead of an API key: switch on Oauth and fill in Logical cluster (lsrc-...) and Identity pool ID (pool-...) as well as your identity provider's details. See Confluent Cloud: logical cluster and identity pool.
• Leave the TLS fields blank.

Kafka Connect and ksqlDB

ksqlDB: on the ksqlDB tab, set URL to the ksqlDB cluster's endpoint, for example https://pksqlc-xxxxx.eu-west-1.aws.confluent.cloud:443 (without :443 works too: it's the default for https://), and User:password to that ksqlDB cluster's API key and secret. KafkIO talks HTTP/2 over TLS to it, which Confluent Cloud supports. See ksqlDB endpoint URL.

Kafka Connect: if you run your own Connect workers against Confluent Cloud, add their REST URL as for any Connect cluster. Confluent Cloud's fully managed connectors are different: they have no worker URL, and are managed through the Confluent Cloud API at https://api.confluent.cloud/connect/v1/environments/env-xxxxx/clusters/lkc-xxxxx, which follows the Kafka Connect REST API and takes a Cloud API key (not a cluster key) as basic auth. KafkIO adds /connectors and the other paths to whatever base URL you give it, but this has not been checked against the Confluent Cloud API, so expect some actions not to work.

Confluent Platform

Self-managed Confluent Platform is Apache Kafka with Confluent's Schema Registry, Kafka Connect and ksqlDB alongside, so there's nothing provider-specific to set: use the listener, security and REST endpoints your administrators set up. KafkIO supports Confluent Platform 8 (KRaft only) from version 2.1.0, as well as earlier versions; the screenshots in these docs are of Confluent Platform 7.9.

• Bootstrap servers and Security type: the listener meant for clients like KafkIO. Clusters often have several (plaintext inside the network, SASL or TLS from outside) on different ports, so ask which is which (see Choosing a security type).
• TLS: an in-house cluster's certificates are usually signed by your organisation's CA, so you need its truststore (the same file your Java applications use as ssl.truststore.location).
• SASL: SCRAM-SHA-512, PLAIN, Kerberos (GSSAPI) and OAUTHBEARER all have examples (see Configuration examples).
• REST services: Schema Registry, Connect and ksqlDB default to ports 8081, 8083 and 8088. With https:// endpoints signed by your own CA, set the truststore on each tab, or add the CA once under Preferences ▸ Network (Trusted certificates) for Schema Registry and Connect (ksqlDB still needs the truststore on its tab).

Amazon MSK

An MSK cluster has a different bootstrap string, with a different port, for each way of authenticating. Copy the one for your method from View client information on the cluster's page in the AWS console:

MSK brokers live in a VPC. Unless the cluster has public access switched on, KafkIO has to run somewhere that can reach the brokers' private addresses (on a VPN into the VPC, for example). An SSH tunnel to one broker isn't enough: the client is told the brokers' own addresses and connects to those next (see Timeouts and misconfiguration). MSK's broker certificates are signed by a public CA, so the truststore stays blank in every case; tick SSL validation.

IAM

With IAM access control, KafkIO signs in with your AWS credentials, using the AWS MSK IAM library that ships with it. Choose SASL SSL, then click the AWS IAM example:

1 2 3 4

1. SASL type: AWS_MSK_IAM.
2. SASL client callback handler: the MSK IAM handler, which signs each login with your AWS credentials.
3. JAAS config: awsProfileName names the AWS profile to use. Remove the option (keeping the ;) to use the AWS default credentials instead.
4. The AWS IAM example sets all three.

• Bootstrap servers: the IAM string, port 9098 (9198 with public access).
• Credentials: a profile in ~/.aws/credentials and ~/.aws/config (from aws configure or aws sso login), or, without awsProfileName, the default chain: the AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY environment variables, then the default profile. Environment variables are read when KafkIO starts, so set them first.
• The IAM identity needs a policy allowing kafka-cluster:Connect and the kafka-cluster:* actions for what you want to do (describing topics, reading data, and so on). A missing permission shows up as an authorisation error on that action, not at login.

The same library also offers an OAUTHBEARER variant: SASL type OAUTHBEARER, SASL login callback handler software.amazon.msk.auth.iam.IAMOAuthBearerLoginCallbackHandler and the JAAS config org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule required;. Use it only if your cluster's administrators ask you to; the AWS IAM example is the usual setup. More on the fields in Amazon MSK IAM.

SCRAM

• Bootstrap servers: the SASL/SCRAM string, port 9096 (9196 with public access).
• SASL SSL, then the SCRAM example, and change SASL type to SCRAM-SHA-512: MSK only supports SHA-512, and the example picks SHA-256.
• In the JAAS config, the user name and password from the AWS Secrets Manager secret associated with the cluster.

For TLS client authentication instead (port 9094 / 9194), choose SSL, switch on Two-way SSL/TLS? and choose a keystore with the client certificate issued by the cluster's AWS Private CA (see Mutual TLS).

MSK Serverless supports IAM only, so use the IAM recipe. It doesn't let clients read broker configuration, so a Serverless cluster's broker settings are empty in KafkIO. AWS Glue Schema Registry isn't compatible with Confluent's Schema Registry API, which is what KafkIO's Schema Registry support speaks, so leave Use Schema Registry off for it. MSK Connect has no Kafka Connect REST endpoint to point KafkIO at; your own Connect workers do.

Azure Event Hubs

Event Hubs has a Kafka endpoint on its Standard, Premium and Dedicated tiers (not Basic). Each event hub appears as a topic. The settings for every method:

• Bootstrap servers: NAMESPACE.servicebus.windows.net:9093.
• Security type: SASL SSL; tick SSL validation; leave the truststore blank.

Connection string (SAS)

Click the Azure Event Hubs (SAS) example. Leave the user name as $ConnectionString, exactly, and replace the password with the connection string of a shared access policy (in the Azure portal: the namespace's or event hub's Shared access policies, then Connection string–primary key). A namespace-level policy gives access to every event hub in the namespace; one on a single event hub, only to that event hub. Details in Azure Event Hubs: connection string.

Microsoft Entra ID

Click the Azure Event Hubs (Entra/OAuth) example and leave its JAAS config as it is: KafkIO's own callback handler gets a token with whatever Azure sign-in is available on your machine (the Azure CLI's az login, or a service principal in the AZURE_CLIENT_ID, AZURE_TENANT_ID and AZURE_CLIENT_SECRET environment variables, among others). The identity needs an Event Hubs data role on the namespace, such as Azure Event Hubs Data Owner, or Data Receiver for read-only use. Give exactly one bootstrap server: the handler takes the token's audience from it. Details in Azure Event Hubs: Microsoft Entra ID.

Event Hubs implements the Kafka protocol, not every Kafka feature behind it. It doesn't let clients read broker configuration (KafkIO shows broker settings as empty), and features with no Event Hubs equivalent, such as ACLs, aren't available. Azure Schema Registry isn't compatible with Confluent's API, so leave Use Schema Registry off; for Kafka Connect, point KafkIO at your own Connect workers.

Aiven

Aiven for Apache Kafka authenticates clients with a client certificate by default, and can also accept SASL on a separate port. Either way, the brokers' certificates are signed by your Aiven project's own CA, not a public one, so download the CA certificate (ca.pem) from the service's Overview page in the Aiven console and use it as the truststore: Truststore file ca.pem, Truststore type CRT / PEM / X.509 (no password). Tick SSL validation.

Client certificate

• Bootstrap servers: the service's host and port from Overview (authentication method Client certificate), for example kafka-xxxxx-myproject.aivencloud.com:12345.
• Security type: SSL, truststore as above, and Two-way SSL/TLS? on.
• Keystore: Aiven gives you the Access key (service.key) and Access certificate (service.cert) as separate files, and a keystore needs both in one file. The simplest is to combine them into a PKCS12 file, which asks for a password:

  openssl pkcs12 -export -inkey service.key -in service.cert -out aiven-client.p12

  Then choose aiven-client.p12 as the Keystore file, type P12 / PFX / PKCS12, with that password as the Keystore password. Leave Keystore private key password blank (see Mutual TLS).

SASL

Once SASL is enabled on the service, Overview shows a second port for authentication method SASL. Use that port, SASL SSL with the same ca.pem truststore, the SCRAM example (SCRAM-SHA-256, as in Aiven's own examples), and the service user's name (for example avnadmin) and password.

Schema Registry (Karapace)

Aiven's Schema Registry is Karapace, which speaks the Confluent Schema Registry API. Enable it on the service, then use its Schema Registry URL from the console (https://, host and port, no path) and, in User:password, the service user and password (avnadmin:PASSWORD). If Test Schema Registry reports a certificate error (PKIX path building failed), set the tab's truststore to ca.pem as well, or add it once under Preferences ▸ Network (Trusted certificates).

Aiven for Apache Kafka Connect, if you use it, is another REST endpoint with the same kind of user and password: add it on the Kafka Connect tab.

Strimzi on Kubernetes

Strimzi runs Kafka in Kubernetes, and KafkIO usually runs outside it, so the cluster needs an external listener (type loadbalancer, nodeport, route or ingress in the Kafka resource). kubectl port-forward to a broker or the bootstrap service doesn't work for Kafka: the brokers tell KafkIO their in-cluster addresses, which it can't reach, so it connects and then times out (see Timeouts and misconfiguration).

Bootstrap and TLS

• Bootstrap servers: the external listener's address, which Strimzi publishes in the resource's status:

  kubectl get kafka my-cluster -o jsonpath='{.status.listeners}'

• With tls: true on the listener (the usual setup), the brokers' certificates come from the cluster CA. Extract it and use it as a PEM truststore (CRT / PEM / X.509):

  kubectl get secret my-cluster-cluster-ca-cert \n  -o jsonpath='{.data.ca\.crt}' | base64 -d > strimzi-ca.crt

Authentication

The listener's authentication type decides the rest. Strimzi's User Operator puts each KafkaUser's credentials in a secret with the user's name:

• scram-sha-512: SASL SSL (or SASL plaintext without TLS), the SCRAM example with SASL type changed to SCRAM-SHA-512. The user's secret has the password under password, and a ready-made JAAS config under sasl.jaas.config that you can paste as it is.
• tls: SSL with Two-way SSL/TLS? on. The user's secret has user.p12 and its password user.password: save the file and use it as a P12 / PFX / PKCS12 keystore.
• oauth: SASL SSL and the Strimzi example, with your identity provider's token endpoint, client ID and secret. See Strimzi OAuth.

Kafka Connect

A Strimzi KafkaConnect cluster's REST API is a service called CONNECT-NAME-connect-api on port 8083. Unlike Kafka, it works through a port-forward (kubectl port-forward svc/my-connect-connect-api 8083:8083, then http://localhost:8083). If the cluster manages connectors as KafkaConnector resources, the operator puts back whatever the resource says, so changes made in KafkIO (pausing, editing the config) don't last: change the resource instead.

WarpStream

WarpStream's agents speak the Kafka protocol, and an agent can also serve WarpStream's Schema Registry, which speaks the Confluent API (supported since KafkIO 2.1.15).

• Bootstrap servers: your agents' address, port 9092 by default (often a load balancer or DNS name in front of them).
• Security: as the agents are configured. Without SASL or TLS, Plaintext. With SASL required, SASL plaintext (or SASL SSL if the agents also have TLS) and the credentials created for the virtual cluster in the WarpStream console, with the mechanism they were created for (usually PLAIN).
• Schema Registry: http://AGENT-HOST:9094 (9094 is the agent's default Schema Registry port), with the same credentials in User:password if authentication is required.

Agents advertise their own address to clients, as Kafka brokers do, so a local agent in Docker has to advertise one KafkIO can reach. For the playground, for example:

docker run -p 9092:9092 -p 9094:9094 public.ecr.aws/warpstream-labs/warpstream_agent:latest \
  playground -advertiseHostnameStrategy custom -advertiseHostnameCustom localhost

then localhost:9092 with Plaintext, and Schema Registry http://localhost:9094. Some things look different from Apache Kafka: each agent appears as one broker on the Brokers page, with the version shown as Unknown and no meaningful leader counts (ignore the Controller Type card for WarpStream), and the topics list can include WarpStream's internal __ws_... topics.

Related

• General Connection Settings
• Security & Authentication
• Schema Registry Connection
• Kafka Connect Connections
• ksqlDB Connections
• Troubleshooting Connections
• Security & Authentication Matrix