Kafka in Hindsight

A 20/20 on what was being installed by Kafka Cookbook

NOTE: the hindsight series are based on old HowTos and are being ported to the more recent versions. Some of this info may be presently out of date until this porting effort is finished.

Apache Kafka is data mover framework for moving large amounts of data throughout your data centers. It also has support for extracting data from sources using its producer or connector framework, transforming this data as its moved, and tieing it to consumers using its consumer or connector API.

The use cases for Kakfa have been broaden since its inception, however at its start and what its best at is being used as a generalized replication log capable of moving data from one source, maintaining a full copy for any clients, and making the data readily available to those clients in a very efficient manner.

Misc – Partition Calculation

partitions: 360
with 360 partitions, the following # servers will be able to handle equal # of partitions
2,3,4,5,6,8,9,10,12,15,18,20,24 (13)

60: 2*2*3*5 - divisors 2,3,4,5,6,-,-,-,10, -,12, -, -,15, -, -, -, -,20, -, -, -, -, - (9)
120: 2*2*2*3*5 - divisors 2,3,4,5,6,-,8,-,10, -,12, -, -,15, -, -, -, -,20, -, -, -,24, - (11)
180: 2*2*3*3*5 - divisors 2,3,4,5,6,-,8,9,10, -,12, -, -,15, -, -,18, -,20, -, -, -, -, - (12)
240: 2*2*2*2*3*5 - divisors 2,3,4,5,6,-,8,-,10, -,12, -, -,15,16, -, -, -,20, -, -, -,24, - (12)
300: 2*2*3*5*5 - divisors 2,3,4,5,6,-,-,-,10, -,12, -, -,15, -, -, -, -,20, -, -, -, -,25 (10)
360: 2*2*2*3*3*5 - divisors 2,3,4,5,6,-,8,9,10, -,12, -, -,15, -, -,18, -,20, -, -, -,24, - (13)
420: 2*2*3*5*7 - divisors 2,3,4,5,6,7,-,-,10, -,12, -,14,15, -, -, -, -,20,21, -, -, -, - (12)
600: 2*2*2*3*5*5 - divisors 2,3,4,5,6,-,8,-,10, -,12, -, -,15, -, -, -, -,20, -, -, -, -,25 (11)
720: 2*2*2*2*3*3*5 - divisors 2,3,4,5,6,-,8,9,10, -,12, -, -,15,16, -,18, -,20, -, -, -,24, - (14)
840: 2*2*2*3*5*7 - divisors 2,3,4,5,6,7,-,-,10, -,12, -,14,15, -, -, -, -,20,21, -, -,24, - (13)
900: 2*2*3*3*5*5 - divisors 2,3,4,5,6,-,-,9,10, -,12, -, -,15, -, -,18, -,20, -, -, -, -,25 (12)
1200: 2*2*2*2*3*5*5 - divisors 2,3,4,5,6,-,8,-,10, -,12, -, -,15,16, -, -, -,20, -, -, -,24,25 (13)
1680: 2*2*2*2*3*5*7 - divisors 2,3,4,5,6,7,-,-,10, -,12, -,14,15,16, -, -, -,20,21, -, -,24, - (14)
2520: 2*2*2*3*3*5*7 - divisors 2,3,4,5,6,7,-,9,10, -,12, -,14,15, -, -,18, -,20,21, -, -,24, - (15)
22 3 5 - 60
222 3 5 - 120
2222 3 5 - 240
22 33 5 - 180
222 33 5 - 360
2222 33 5 - 720
22 3 55 - 300
222 3 55 - 600
2222 3 55 - 1200
22 3 5 7 - 420
222 3 5 7 - 840
2222 3 5 7 - 1680
22 33 5 7 - 1260
222 33 5 7 - 2520
2222 33 5 7 - 5040