kafka 기본 설치

kafka 설치

기본 설치 디렉토리 : /usr/local/bin

zookeeper 실행

./zookeeper-server-start ../etc/kafka/zookeeper.properties

kafka-server 실행

./kafka-server-start ../etc/kafka/server.properties

등록된 토픽 조회

./kafka-topics –bootstrap-server localhost:9092 –list

토픽 생성

./kafka-topics –bootstrap-server localhost:9092 –create –topic pip-events –partitions 1

토픽 구독

./kafka-topics –bootstrap-server localhost:9092 –describe –topic pip-events

consumer 실행

./kafka-console-consumer –bootstrap-server localhost:9092 –topic pip-events –from-beginning

producer 실행

./kafka-console-producer –broker-list localhost:9092 –topic quickstart-events

2. Kafka connector 설치

https://www.confluent.io/installation

      or 

kafak confluent-community 설치

curl -O http://packages.confluent.io/archive/6.1/confluent-community-7.0.1.tar.gz

connector 실행

./connect-distributed /usr/local/etc/kafka/connect-distributed.properties

3. JDBC connector 설치

JDBC Connector (Source and Sink)

https://www.confluent.io/installation

confluentinc-kafka-connect-jdbc-10.3.3.zip

/usr/local/etc/kafka/connect-distributed.properties 에 추가

plugin.path=~/confluentinc-kafka-connect-jdbc-10.3.3/lib

mysql-connector-java-8.0.25.jar 이동

~/confluentinc-kafka-connect-jdbc-10.3.3/share/java/kafka/mysql-connector-java-8.0.25.jar

1. postman 전송

Kafka source connector 등록

POST http://localhost:8083/connectors

{
"name": "my-source-connect",
"config": {
"connector.class": "io.confluent.connect.jdbc.JdbcSourceConnector",
"connection.url": "jdbc:mysql://localhost:3306/pip-member",
"connection.user":"pickpie",
"connection.password":"pickpie1234!!",
"mode":"incrementing",
"incrementing.column.name" : "member_seq",
"table.whitelist" : "member_mst",
"topic.prefix" : "example_topic_",
"tasks.max" : "1"
}
}

Kafka connector 조회

GET http://localhost:8083/connectors Kafka sink connector 등록

POST http://localhost:8083/connectors

{
"name": "my-sink-connect",
"config": {
"connector.class": "io.confluent.connect.jdbc.JdbcSinkConnector",
"connection.url": "jdbc:mysql://localhost:3306/pip-member",
"connection.user":"pickpie",
"connection.password":"pickpie1234!!",
"auto.create":"false",
"auto.evolve":"false",
"insert.mode":"upsert",
"tasks.max":"1",
"pk.mode": "record_value",
"pk.fields": "member_seq",
"topics":"example_topic_member_mst"
}
}

Kafka connector 삭제

http://localhost:8083/connectors/my-source-connect