Kafka Cookbook

Checkout Kafka In Hindsight

NOTE: the hindsight series are based on old HowTos and are being ported to the more recent versions. Some of this info may be presently out of date until this porting effort is finished.

Installation

1. Install Apache Zookeeper. NOTE: For production, the zookeeper ensemble’s (i.e. cluster) ideal size is 5 nodes.
i) Install Zookeeper:
MacOSX:

$ brew install zookeeper
$ zkServer start
$ zkServer status

Ubuntu:

$ sudo apt-get install zookeeper
$ sudo apt-get install zookeeperd
# should start automatically however can be controlled manually
$ sudo service zookeeper start

Fedora:

$ sudo dnf install zookeeper
$ sudo cp /etc/zookeeper/zoo_sample.cfg /etc/zookeeper/zoo.cfg
$ sudo zkServer.sh start

ii) Verify the version. Start telnet and type srvr to get zookeeper to print out its server version. e.g.

$ telnet localhost 2181<cr>
srvr<cr>

The version should be at least Zookeeper version: 3.4.8

2. Install Apache Kafka.
i) Prepare installation directory

$ mkdir /usr/local/kafka
$ cd /usr/local/kafka

ii) Get the latest version using Scala 2.11 by going to https://kafka.apache.org/downloads
e.g.

$ curl -o kafka_2.11.tgz "http://apache.mirrors.lucidnetworks.net/kafka/0.10.2.1/kafka_2.11-0.10.2.1.tgz"

iii) untar the tar file in /usr/local/kafka directory and setup links

$ tar xf kafka_2.11.tgz
$ ln -s ${install-subfolder e.g. kafka_2.11-0.10.2.1} kafka_latest
$ ln -s kafka_latest/* .

When installing a new version of kafka, place the actual folder under /usr/local/kafka and delete the old link kafka_latest and make a new one pointing to the newly installed folder e.g. if the new folder is kafka_2.11-0.10.2.2 and the old one was kafka_2.11-0.10.2.1,

$ rm kafka_latest
$ ln -s kafka_2.11-0.10.2.2 kafka_latest

iv) Start kafka

$ /usr/local/kafka/bin/kafka-server-start.sh -daemon /usr/local/kafka/config/server.properties

3. Install the Debezium MySQL Apacke Kafke Connector
See: http://debezium.io/docs/connectors/mysql/
i) goto https://repo1.maven.org/maven2/io/debezium/debezium-connector-mysql/ and download the latest (0.5.x) debezium-connector-mysql-xxx-plugin.tar.gz to get the tar package containing the jars. See Also: https://mvnrepository.com/artifact/io.debezium/debezium-connector-mysql/0.5.0

ii) Configure mysql to use binlog. Edit /etc/mysql/my.cnf to have the following:

log_bin = mysql-bin
binlog_format = row
expire_logs_days = 10
gtid_mode = on
enforce_gtid_consistency = on

iii) setup a mysql user ${connector-user} that has the following permissions:

CREATE USER '${connector-user}'@'localhost' IDENTIFIED BY '${connector-user-password}';
GRANT SELECT, RELOAD, SHOW DATABASES, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO '${connector-user}'@'localhost';

vi) Optional Get the source for the debezium connectors. Clone the repo and build the package

$ git clone https://github.com/debezium/debezium.git
$ cd debezium
$ mvn clean install -DskipITs

Appendix: Partition calculations

partitions: 360
with 360 partitions, the following # servers will be able to handle equal # of partitions
2,3,4,5,6,8,9,10,12,15,18,20,24 (13)

60: 2*2*3*5 - divisors 2,3,4,5,6,-,-,-,10, -,12, -, -,15, -, -, -, -,20, -, -, -, -, - (9)
120: 2*2*2*3*5 - divisors 2,3,4,5,6,-,8,-,10, -,12, -, -,15, -, -, -, -,20, -, -, -,24, - (11)
180: 2*2*3*3*5 - divisors 2,3,4,5,6,-,8,9,10, -,12, -, -,15, -, -,18, -,20, -, -, -, -, - (12)
240: 2*2*2*2*3*5 - divisors 2,3,4,5,6,-,8,-,10, -,12, -, -,15,16, -, -, -,20, -, -, -,24, - (12)
300: 2*2*3*5*5 - divisors 2,3,4,5,6,-,-,-,10, -,12, -, -,15, -, -, -, -,20, -, -, -, -,25 (10)
360: 2*2*2*3*3*5 - divisors 2,3,4,5,6,-,8,9,10, -,12, -, -,15, -, -,18, -,20, -, -, -,24, - (13)
420: 2*2*3*5*7 - divisors 2,3,4,5,6,7,-,-,10, -,12, -,14,15, -, -, -, -,20,21, -, -, -, - (12)
600: 2*2*2*3*5*5 - divisors 2,3,4,5,6,-,8,-,10, -,12, -, -,15, -, -, -, -,20, -, -, -, -,25 (11)
720: 2*2*2*2*3*3*5 - divisors 2,3,4,5,6,-,8,9,10, -,12, -, -,15,16, -,18, -,20, -, -, -,24, - (14)
840: 2*2*2*3*5*7 - divisors 2,3,4,5,6,7,-,-,10, -,12, -,14,15, -, -, -, -,20,21, -, -,24, - (13)
900: 2*2*3*3*5*5 - divisors 2,3,4,5,6,-,-,9,10, -,12, -, -,15, -, -,18, -,20, -, -, -, -,25 (12)
1200: 2*2*2*2*3*5*5 - divisors 2,3,4,5,6,-,8,-,10, -,12, -, -,15,16, -, -, -,20, -, -, -,24,25 (13)
1680: 2*2*2*2*3*5*7 - divisors 2,3,4,5,6,7,-,-,10, -,12, -,14,15,16, -, -, -,20,21, -, -,24, - (14)
2520: 2*2*2*3*3*5*7 - divisors 2,3,4,5,6,7,-,9,10, -,12, -,14,15, -, -,18, -,20,21, -, -,24, - (15)
22 3 5 - 60
222 3 5 - 120
2222 3 5 - 240
22 33 5 - 180
222 33 5 - 360
2222 33 5 - 720
22 3 55 - 300
222 3 55 - 600
2222 3 55 - 1200
22 3 5 7 - 420
222 3 5 7 - 840
2222 3 5 7 - 1680
22 33 5 7 - 1260
222 33 5 7 - 2520
2222 33 5 7 - 5040