Apache Kafka / Confluent Connector

Connect SAP S/4HANA and ECC to Apache Kafka clusters – self-hosted or Confluent Cloud – for high-throughput event streaming. Supports Avro and JSON Schema via Confluent Schema Registry, exactly-once producer semantics, and consumer group management.

Overview

The ASAPIO Kafka connector supports:

• Kafka versions: Apache Kafka 2.6 and higher, Confluent Platform 6.0 and higher, Confluent Cloud
• Outbound: SAP acts as a Kafka producer, publishing events to one or more topics
• Inbound: SAP acts as a Kafka consumer, polling topics via a consumer group
• Message formats: JSON (CloudEvents 1.0 default), Avro (with Confluent Schema Registry), JSON Schema (with Confluent Schema Registry)
• Security: SASL/PLAIN, SASL/SCRAM-SHA-256, SASL/SCRAM-SHA-512, mTLS (SSL with client certificate)
• Exactly-once: Idempotent producer enabled by default for Kafka 2.6+ (enables exactly-once outbound delivery)

Prerequisites

• Kafka broker addresses (one or more, comma-separated for bootstrap)
• Topic name (must already exist, or auto-create enabled on the broker)
• SASL credentials or client certificate, depending on broker security configuration
• Schema Registry URL and credentials (optional – only if using Avro or JSON Schema)
• For inbound: consumer group ID (coordinate with the team owning the topic)
• TCP connectivity from SAP to Kafka broker(s): port 9092 (SASL_PLAINTEXT), 9093 (SASL_SSL), or custom

Configuration Parameters

Confluent Cloud Setup

In ASAPIO Event Studio, configure the Kafka connector with:

• Bootstrap Servers: pkc-xxxxx.region.provider.confluent.cloud:9092
• Security Protocol: SASL_SSL
• SASL Mechanism: PLAIN (Confluent Cloud uses SASL/PLAIN over TLS)
• SASL Username: your API key
• SASL Password: your API secret

Self-Hosted Kafka Setup

For self-hosted Apache Kafka, SASL/SCRAM-SHA-256 provides a good balance of security and simplicity. For highest security, use mTLS (Security Protocol = SSL) with client certificates managed in SAP's STRUST certificate store.

SASL/SCRAM-SHA-256 broker configuration

# server.properties (relevant settings)
listeners=SASL_SSL://:9093
advertised.listeners=SASL_SSL://broker.example.com:9093
ssl.keystore.location=/etc/kafka/certs/broker.keystore.jks
ssl.truststore.location=/etc/kafka/certs/broker.truststore.jks
sasl.enabled.mechanisms=SCRAM-SHA-256
sasl.mechanism.inter.broker.protocol=SCRAM-SHA-256
security.inter.broker.protocol=SASL_SSL

Schema Registry Integration

When Schema Registry is configured, ASAPIO uses Avro serialization instead of JSON. The schema ID is embedded in the first 5 bytes of the Kafka message value (Confluent wire format). ASAPIO resolves the schema from the Schema Registry automatically at runtime.

Example Avro Schema – SAP Material Goods Movement

{
  "type": "record",
  "name": "GoodsMovement",
  "namespace": "com.asapio.sap.s4hana.mm",
  "doc": "SAP S/4HANA goods movement event",
  "fields": [
    {"name": "materialDocument",  "type": "string",  "doc": "SAP material document number"},
    {"name": "postingDate",       "type": "string",  "doc": "ISO 8601 date"},
    {"name": "plant",             "type": "string"},
    {"name": "storageLocation",   "type": "string"},
    {"name": "items", "type": {
      "type": "array",
      "items": {
        "type": "record",
        "name": "GoodsMovementItem",
        "fields": [
          {"name": "material",     "type": "string"},
          {"name": "quantity",     "type": "double"},
          {"name": "unit",         "type": "string"},
          {"name": "movementType", "type": "string"}
        ]
      }
    }}
  ]
}

Schema evolution is supported with BACKWARD compatibility (default). New optional fields with defaults can be added to the schema without breaking existing consumers.

Outbound Example

Scenario: SAP S/4HANA inventory update → Kafka topic sap.mm.goods-movements → downstream data platform (Apache Flink, Databricks, etc.)

Producer configuration applied automatically by ASAPIO:

• acks=all – wait for all in-sync replicas to acknowledge
• enable.idempotence=true – exactly-once producer semantics (Kafka 2.6+)
• retries=2147483647 – effectively unlimited retries at the Kafka client level (ASAPIO retry policy applies on top)
• max.in.flight.requests.per.connection=5 – optimal for idempotent producer

Inbound Example

Scenario: External order management system publishes purchase orders to Kafka topic external.orders.new → ASAPIO consumer group reads messages → creates SAP Purchase Order via BAPI_PO_CREATE1.

Key inbound configuration:

• Consumer Group ID: sap-s4h-prd-po-creator (unique per SAP system)
• Auto Offset Reset: latest (process only new messages from flow activation)
• Batch Size: 10 (process 10 messages per polling cycle)
• Offset commit: ASAPIO commits offsets only after successful BAPI call. On BAPI error, the offset is not committed and the message is retried.
• Delivery guarantee: at-least-once (implement idempotency in ABAP handler using message key as deduplication ID)

Performance Tuning

Recommended settings for high-throughput scenarios:

Expected throughput: 10,000+ events/minute per connector instance with default settings. With batch mode and lz4 compression: 100,000+ events/minute. Actual throughput depends on SAP server capacity, Kafka broker capacity, and message payload size.