Checkout Kafka In Hindsight

Installation

1. Install Apache Zookeeper. NOTE: For production, the zookeeper ensemble’s (i.e. cluster) ideal size is 5 nodes.
i) Install Zookeeper:
MacOSX:

$ brew install zookeeper
$ zkServer start
$ zkServer status

Ubuntu:

$ sudo apt-get install zookeeper
$ sudo apt-get install zookeeperd
# should start automatically however can be controlled manually
$ sudo service zookeeper start

Fedora:

$ sudo dnf install zookeeper
$ sudo cp /etc/zookeeper/zoo_sample.cfg /etc/zookeeper/zoo.cfg
$ sudo zkServer.sh start

ii) Verify the version. Start telnet and type srvr to get zookeeper to print out its server version. e.g.

$ telnet localhost 2181<cr>
srvr<cr>

The version should be at least Zookeeper version: 3.4.8

2. Install Apache Kafka.
i) Prepare installation directory

$ mkdir /usr/local/kafka
$ cd /usr/local/kafka

ii) Get the latest version using Scala 2.11 by going to https://kafka.apache.org/downloads
e.g.

$ curl -o kafka_2.11.tgz "http://apache.mirrors.lucidnetworks.net/kafka/0.10.2.1/kafka_2.11-0.10.2.1.tgz"

iii) untar the tar file in /usr/local/kafka directory and setup links

$ tar xf kafka_2.11.tgz
$ ln -s ${install-subfolder e.g. kafka_2.11-0.10.2.1} kafka_latest
$ ln -s kafka_latest/* .

When installing a new version of kafka, place the actual folder under /usr/local/kafka and delete the old link kafka_latest and make a new one pointing to the newly installed folder e.g. if the new folder is kafka_2.11-0.10.2.2 and the old one was kafka_2.11-0.10.2.1,

$ rm kafka_latest
$ ln -s kafka_2.11-0.10.2.2 kafka_latest

iv) Start kafka

$ /usr/local/kafka/bin/kafka-server-start.sh -daemon /usr/local/kafka/config/server.properties

3. Install the Debezium MySQL Apacke Kafke Connector
See: http://debezium.io/docs/connectors/mysql/
i) goto https://repo1.maven.org/maven2/io/debezium/debezium-connector-mysql/ and download the latest (0.5.x) debezium-connector-mysql-xxx-plugin.tar.gz to get the tar package containing the jars. See Also: https://mvnrepository.com/artifact/io.debezium/debezium-connector-mysql/0.5.0

ii) Configure mysql to use binlog. Edit /etc/mysql/my.cnf to have the following:

log_bin = mysql-bin
binlog_format = row
expire_logs_days = 10
gtid_mode = on
enforce_gtid_consistency = on

iii) setup a mysql user ${connector-user} that has the following permissions:

CREATE USER '${connector-user}'@'localhost' IDENTIFIED BY '${connector-user-password}';
GRANT SELECT, RELOAD, SHOW DATABASES, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO '${connector-user}'@'localhost';

vi) Optional Get the source for the debezium connectors. Clone the repo and build the package

$ git clone https://github.com/debezium/debezium.git
$ cd debezium
$ mvn clean install -DskipITs

Appendix: Partition calculations

partitions: 360
with 360 partitions, the following # servers will be able to handle equal # of partitions
2,3,4,5,6,8,9,10,12,15,18,20,24 (13)

60: 2*2*3*5 - divisors 2,3,4,5,6,-,-,-,10, -,12, -, -,15, -, -, -, -,20, -, -, -, -, - (9)
120: 2*2*2*3*5 - divisors 2,3,4,5,6,-,8,-,10, -,12, -, -,15, -, -, -, -,20, -, -, -,24, - (11)
180: 2*2*3*3*5 - divisors 2,3,4,5,6,-,8,9,10, -,12, -, -,15, -, -,18, -,20, -, -, -, -, - (12)
240: 2*2*2*2*3*5 - divisors 2,3,4,5,6,-,8,-,10, -,12, -, -,15,16, -, -, -,20, -, -, -,24, - (12)
300: 2*2*3*5*5 - divisors 2,3,4,5,6,-,-,-,10, -,12, -, -,15, -, -, -, -,20, -, -, -, -,25 (10)
360: 2*2*2*3*3*5 - divisors 2,3,4,5,6,-,8,9,10, -,12, -, -,15, -, -,18, -,20, -, -, -,24, - (13)
420: 2*2*3*5*7 - divisors 2,3,4,5,6,7,-,-,10, -,12, -,14,15, -, -, -, -,20,21, -, -, -, - (12)
600: 2*2*2*3*5*5 - divisors 2,3,4,5,6,-,8,-,10, -,12, -, -,15, -, -, -, -,20, -, -, -, -,25 (11)
720: 2*2*2*2*3*3*5 - divisors 2,3,4,5,6,-,8,9,10, -,12, -, -,15,16, -,18, -,20, -, -, -,24, - (14)
840: 2*2*2*3*5*7 - divisors 2,3,4,5,6,7,-,-,10, -,12, -,14,15, -, -, -, -,20,21, -, -,24, - (13)
900: 2*2*3*3*5*5 - divisors 2,3,4,5,6,-,-,9,10, -,12, -, -,15, -, -,18, -,20, -, -, -, -,25 (12)
1200: 2*2*2*2*3*5*5 - divisors 2,3,4,5,6,-,8,-,10, -,12, -, -,15,16, -, -, -,20, -, -, -,24,25 (13)
1680: 2*2*2*2*3*5*7 - divisors 2,3,4,5,6,7,-,-,10, -,12, -,14,15,16, -, -, -,20,21, -, -,24, - (14)
2520: 2*2*2*3*3*5*7 - divisors 2,3,4,5,6,7,-,9,10, -,12, -,14,15, -, -,18, -,20,21, -, -,24, - (15)

22 3 5 - 60
222 3 5 - 120
2222 3 5 - 240

22 33 5 - 180
222 33 5 - 360
2222 33 5 - 720

22 3 55 - 300
222 3 55 - 600
2222 3 55 - 1200

22 3 5 7 - 420
222 3 5 7 - 840
2222 3 5 7 - 1680

22 33 5 7 - 1260
222 33 5 7 - 2520
2222 33 5 7 - 5040