Getting Started Kafka Producer
OAS is both a consumer and producer of Kafka messages. If you want to receive data from Kafka please see the Getting Started Kafka Consumer guide.
You can view the Getting Started with Kafka Video to familiarize yourself with the following steps to setup OAS as a producer and consumer of Kafka.
- 00:00 – Introduction
- 01:20 – Kafka Producer
- 09:07 – Kafka Consumer
- 10:20 – Save Configuration
- 11:00 – Transaction Logging
- 11:45 – Publish 100,000 Values per Second
- 15:50 – Contact Us
Configure a Kafka Driver
OAS tag values from both local and remote OAS Engines can be published to local or cloud Kafka brokers. The following steps can be used to setup direct communications with a Kafka Broker.
Step 1
Start Configure OAS application from the program group Open Automation Software.
Step 2
Select Configure-License and verify that Kafka is one of the available Drivers in the lower left of the form. If you do not see the Kafka driver available contact support@oasiot.com to update your license.
Note: You will need to be running Open Automation Software Version 18.0.0.4 or greater to support Kafka communications. You can download the latest version at www.openautomationsoftware.com/downloads/open-automation-software/
Step 3
Select Configure-Drivers.
Step 4
Select localhost or the remote service you wish to modify with the Select button to the right of the Network Node list.
Note: Optionally select the Live Data Cloud node if you are hosting data over the Internet with a standard Internet connection.
Step 5
Enter a meaningful Driver Interface Name that you will refer to this connection and define the properties.
Set the driver to Kafka. Set the Bootstrap Servers of the brokers you wish to publish to.
If sending data to the cloud set the appropriate Security Protocol.
When connecting with Confluent Cloud set the SASL Username with the provided API Key and the SASL Password with the Secret Key.
Step 6
Select the Add Driver button in the lower part of the form to add the Driver Interface.
Note: If you need to define several Driver Interfaces you can use the CSV Export and CSV Import on the toolbar in the upper right together with Microsoft Excel.
Publish Selected Tags to Kafka Broker
There are 2 ways to publish data from OAS to a Kafka Broker. Both require Tags to be setup first for the data sources you want to transfer.
View the following video for a complete demonstration of how to publish data to Kafka Brokers, MQTT Brokers, Azure IoT, and AWS IoT Gateway.
Step 1
Enable Publish Selected Tags at the bottom of the Driver configuration.
Step 2
Select to publish data continuously at a specified interval, based on event, or at a specific time of day.
If Event Driven browse for a local or remote OAS tag that will trigger the publish. Select a Boolean tag that will change state from false to true, true to false, or both. Or choose an Integer tag that trigger a publish anytime the value changes other than 0.
Step 3
Enable Publish Latest Value Only to send only the latest value of each tag when published or disable to send all value changes since the last time a publish occurred.
Enabled Include All Tags Each Publish to send at least the latest value of each tag when published or disable to only send the tags that have changed since the last publish.
Enable Publish All Tags As One Topic to publish all tag values as one topic or disable to send each tag as its own topic.
See examples in Step 6 below for each selectable option.
Step 4
Specify the Publish Topic Id if choosing to Publish All Tags As One Topic.
When publishing as one topic to optimize communications with Kafka multiple messages are sent for each publish when the number of tag value changes exceeds the Max Tags Per Publish limit.
Specify the Tag Id, Value Id, an optional Quality Id, and Timestamp Id for each tag value that is sent.
When including the Timestamp Id also specify the timestamp format, use Custom to specify your own date and time format.
Step 5
Specify local and remote OAS tag variables to include in each publish and specify the Id. Value is the current value from the data source or you can select any of the over 600 tag variables of each tag to publish.
Optionally use CSV Export and CSV Import buttons to set up additional tags to publish using Microsoft Excel.
When selecting remote tags use Basic Networking syntax or Live Data Cloud syntax in the tag path.
Step 6
Select Apply Changes to begin publishing to the Kafka brokers. Select Save to save the new driver configuration within the tag file.
Examples:
Examples of publishing every 2 seconds with each tag value changing every second:
Use Data Route to Send Data to Kafka
Step 1
See Getting Started – Data Route to define a Target tag in any source tag.
Reading Data from Kafka Broker
Step 1
Select Configure-Tags.
Select localhost or the remote service you wish to modify with the Select button to the right of the Network Node list.
Note: Optionally select the Live Data Cloud node if you are hosting data over the Internet with a standard Internet connection.
Step 2
Select to Add a Tag.
Note: You can also add organizational Groups as many levels deep as you prefer and add tags to groups. To do this first add a Group to Tags Group at the root level, then right click on the Group in the right window to add additional Groups or Tags.
Step 3
Change the Data Type property to String or JSON.
Change the Data Source Tag property to Kafka.
Step 4
Select the correct Driver Interface from the Driver Interface pull down list.
Step 5
Enter the topic.
Step 6
Select Apply Changes to begin monitoring the topic from the Kafka Broker.
Step 7
To setup automated data transfer of any OAS Tag to the Kafka topic see Getting-Started – Data Route and set the Target Destination to Tag with the newly created tag Value as the destination.
Step 8
To define multiple tags use the CSV Export and CSV Import on the toolbar in the upper right together with Microsoft Excel.
Note: You can also programmatically define Tags using the free to use OASConfig .NET Standard assembly, OPCSystems .NET Framework 4 assembly, or REST API.
Step 9
Select the Save button on the toolbar at the top.
Step 10
Create a directory on the local C: drive with the name OASDemo.
Save the file DemoTags.tags in the directory C:\OASDemo or the correct path for Linux.
Step 11
Under Configure – Options set the Default Tag Configuration File so when the computer restarts the tag file will automatically be loaded.