Saturday , January 18 2020
Home / Uncategorized / How to install Apache Kafka Distributed Streaming Platform on Ubuntu

How to install Apache Kafka Distributed Streaming Platform on Ubuntu



Apache Kafka is a distributed streaming platform developed by the Apache Software Foundation and written in Java and Scala. Apache Kafka was originally developed by LinkedIn and was opened in 2011.

Apache Kafka is used to create a real-time streaming data pipeline that reliably collects data between system and applications. Provides unified, high-throughput and low-latency data processing in real time.

In this tutorial, we'll show you how to install and configure Apache Kafka step-by-step on Ubuntu 18.04. This guide will cover the installation and configuration of Apache Kafka and Apache Zookeeper.

Prerequisites

  • Ubuntu 18.04
  • Root privileges

What we will do?

  1. Install Java OpenJDK 8
  2. Install Apache Zookeeper
  3. Download and configure Apache Kafka
  4. Configure Apache Kafka and Zookeeper as a service
  5. analysis

Step 1: Install Java OpenJDK 8

Apache Kafka was written in Java and Scala, so you need to install java on the server.

Before installing any package, update the repository and update all packages.

sudo apt update
sudo apt upgrade

Now install Java OpenJDK 8 from the Ubuntu repository using the apt command below.

sudo apt installs openjdk-8-jdk -y

At the end of the installation, check the installed java version.

java -version

Now you will see Java OpenJDK 8 installed on Ubuntu 18.04.

Install Java

Step 2: Install Apache Zookeeper

Apache Kafka uses zookeeper for the election controller, cluster membership and argument configuration. Zookeeper is a distributed configuration and synchronization service.

In this step, we will install Zookeeper from the Ubuntu repository.

Run the apt command below.

sudo apt install zookeeperd -y

Wait until the installation is complete.

Install Apache Zookeeper

Step 3: download and configure Apache Kafka

In this step we will install Apache Kafka using the binary files that can be downloaded from the Kafka website. We will install and configure apache Kafka and run as a non-root user.

Add a new user called & # 39; kafka & # 39 ;.

useradd -d / opt / kafka -s / bin / bash kafka
passwd kafka

Now go to the & # 39; / opt & # 39; directory and download the Apache Kafka binaries using wget.

cd / opt
wget http://www-eu.apache.org/dist/kafka/2.0.0/kafka_2.11-2.0.0.tgz

Now create a new kafka directory.

mkdir -p / opt / kafka

Extract the kafka _ * file. Tar.gz in the & # 39; kafka & # 39; directory and change the owner of the directory in the user and in the & # 39; kafka & # 39; group.

tar -xf kafka_2.11-2.0.0.tgz -C / opt / kafka –strip-components = 1
sudo chown -R kafka: kafka / opt / kafka

Now log in to the "kafka" user and change the server.properties configuration.

su – kafka
vim config / server.properties

Paste the following configuration to the end of the line.

delete.topic.enable = true

Save and exit.

Launch Apache Kafka

The Apache Kafka configuration has been completed.

Step 4: Configure Apache Kafka and Zookeeper as services

In this step, we will configure Apache Kafka as a service and configure the customs service configuration for the zoo keeper.

Go to the & # 39; / lib / systemd / system & # 39; directory and create a new service file & # 39; zookeeper.service & # 39 ;.

cd / lib / systemd / system /
vim zookeeper.service

Paste the configuration below.

[Unit]



Requires = network.target remote-fs.target
After = network.target remote-fs.target

[Service]
Type = simple
User = kafka
ExecStart = / opt / kafka / bin / zookeeper-server-start.sh /opt/kafka/config/zookeeper.properties
ExecStop = / opt / Kafka / bin / zookeeper-server-stop.sh
Restart = on-abnormal

[Install]
WantedBy = multi-user.target

Save and exit.

Now create the Apache Kafka service file & # 39; kafka.service & # 39 ;.

vim kafka.service

Paste the configuration below.

[Unit]



Requires = zookeeper.service
Next = zookeeper.service

[Service]
Type = simple
User = kafka
ExecStart = / bin / sh -c & # 39; /opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties'
ExecStop = / opt / Kafka / bin / kafka-server-stop.sh
Restart = on-abnormal

[Install]
WantedBy = multi-user.target

Save and exit.

Reload the system manager configuration.

demon systemctl-reload

Now it starts the Apache Zookeeper and Apache Kafka services.

systemctl start zookeeper
systemctl enable zookeeper

systemctl starts kafka
systemctl enables kafka

Start Zookeeper and Kafka

The apache zookeeper and Kafka are active and functioning.

Zookeeper that runs under the door & # 39; 2181 & # 39; and Kafka on port '9092', check it using the netstat command below.

netstat -plntu

The software is listening to its default ports

Step 5: Apache Kafka test

Log in to the user & # 39; kafka & # 39; and go to the & # 39; bin / & # 39; directory.

su – kafka
Trash can /

Now create a new topic called & # 39; HakaseTesting & # 39; using the executable file & # 39; kafka-topics.sh & # 39 ;.

./kafka-topics.sh –create –zookeeper localhost: 2181
–replication-factor 1 –partitions 1
–topic HakaseTesting

Run "kafka-console-producer.sh" with the "HakaseTesting" topic.

./kafka-console-producer.sh –broker-list localhost: 9092
–topic HakaseTesting

Now open a new terminal and log in to the server, then log in to the user & # 39; kafka & # 39 ;.

Run "kafka-console-consumer.sh" for the "HakaseTesting" topic.

./kafka-console-consumer.sh –bootstrap-server localhost: 9092
–topic HakaseTesting – from the beginning

And when you type any input from the shell & # 39; kafka-console-producer.sh & # 39 ;, you'll get the same result on the shell & # 39; kafka-console-consumer.sh & # 39 ;.

Apache Kafka test

The installation and configuration of Apache Kafka on Ubuntu 18.04 have been completed successfully.

Reference

About Muhammad Arul

Muhammad Arul is a freelance system administrator and a technical writer. He has worked with Linux Environments for more than 5 years, passionate about Open Source and strongly motivated in the installation and troubleshooting of Linux problems. Mainly working with RedHat / CentOS Linux and Ubuntu / Debian, Nginx and Apache web servers, Proxmox, Zimbra Administration and Website Optimization. I am currently learning about OpenStack and Container Technology.


Source link

52 comments

  1. WONDERFUL Post.thanks for share..extra wait .. …

  2. Hey I know this is off topic but I was wondering if you
    knew of any widgets I could add to my blog that automatically tweet my newest twitter updates.
    I’ve been looking for a plug-in like this for
    quite some time and was hoping maybe you would have some experience with something like this.
    Please let me know if you run into anything. I truly
    enjoy reading your blog and I look forward to
    your new updates.

  3. Its such as you read my thoughts! You seem to understand so much
    about this, like you wrote the guide in it or something.
    I believe that you just can do with a few % to pressure the message house a bit, however other than that,
    that is excellent blog. A fantastic read. I will definitely be back.

  4. There’s certainly a lot to learn about this issue.

    I love all of the points you’ve made.

  5. Piece of writing writing is also a excitement, if you know after that you can write or else it is complex to write.

  6. I’m gone to tell my little brother, that he should also pay a quick
    visit this webpage on regular basis to obtain updated from latest reports.

  7. Thank you for the good writeup. It in reality used to be a enjoyment account it.
    Look complicated to far added agreeable from
    you! However, how could we keep in touch?

  8. Greetings, I think your site could be having
    browser compatibility problems. Whenever I look at your
    site in Safari, it looks fine however, if opening in IE,
    it has some overlapping issues. I just wanted to give you
    a quick heads up! Apart from that, great website!

  9. The other day, while I was at work, my sister stole my iphone and tested
    to see if it can survive a thirty foot drop, just so she can be a youtube sensation. My iPad is
    now broken and she has 83 views. I know this is completely off topic but I had to share it with someone!

  10. Hello, Neat post. There is a problem together with your site in web explorer,
    may check this? IE nonetheless is the marketplace chief and a good section of
    people will omit your magnificent writing due to this problem.

  11. Saved as a favorite, I really like your site!

  12. Wow that was odd. I just wrote an incredibly long
    comment but after I clicked submit my comment didn’t appear.
    Grrrr… well I’m not writing all that over again. Regardless, just wanted to say superb blog!

  13. Thank you for the auspicious writeup. It in reality was a enjoyment account it.
    Glance complex to far brought agreeable from you!
    However, how can we be in contact?

  14. Neat blog! Is your theme custom made or did you download
    it from somewhere? A theme like yours with a few simple tweeks would really make my blog stand out.
    Please let me know where you got your design. Thank
    you

  15. Hi there, its nice article regarding media print, we all understand media is a enormous source of
    facts.

  16. Quality posts is the crucial to invite the users to pay a visit the web site, that’s what this website
    is providing.

  17. I all the time used to read piece of writing in news papers but now as I am a
    user of net thus from now I am using net for articles or reviews, thanks to web.

  18. Cool blog! Is your theme custom made or did you download
    it from somewhere? A design like yours with a few simple adjustements would really make my blog stand out.
    Please let me know where you got your design. Appreciate it

  19. Your means of describing all in this article is genuinely nice, all be capable of effortlessly be aware of it, Thanks a lot.

  20. What’s up friends, nice paragraph and pleasant arguments commented at this place,
    I am truly enjoying by these.

  21. Hi, I would like to subscribe for this weblog to take most up-to-date updates,
    therefore where can i do it please help.

  22. What a information of un-ambiguity and preserveness of valuable know-how concerning unexpected emotions.

  23. Very good article! We are linking to this great content on our website.
    Keep up the good writing.

  24. Hi! I just want to give you a huge thumbs up for your excellent information you’ve got
    here on this post. I’ll be returning to your blog for
    more soon.

  25. What’s up mates, its enormous piece of writing on the topic of tutoringand completely explained, keep it up all the time.

  26. Amazing blog! Is your theme custom made or did you download
    it from somewhere? A theme like yours with a few simple tweeks would really
    make my blog shine. Please let me know where you got your theme.
    Thanks

  27. Hello, i think that i saw you visited my web site thus i
    came to “return the favor”.I am attempting to find things to enhance my
    site!I suppose its ok to use some of your ideas!!

  28. Very quickly this site will be famous among all blogging
    and site-building users, due to it’s nice content

  29. That is really interesting, You are a very professional blogger.
    I’ve joined your feed and sit up for in search of more of your magnificent post.

    Also, I’ve shared your web site in my social networks

  30. Helpful information. Fortunate me I discovered your site by chance, and I am shocked why this accident didn’t happened
    earlier! I bookmarked it.

  31. Great article, exactly what I was looking for.

  32. What’s up, its pleasant paragraph on the topic of media print, we all be familiar with media is
    a enormous source of data.

  33. An interesting discussion is definitely worth comment. I think that
    you ought to write more on this topic, it may not be a taboo matter but
    typically people don’t speak about these subjects.
    To the next! All the best!!

  34. If you want to take a good deal from this paragraph then you have to apply
    these strategies to your won webpage.

  35. Hi there, I enjoy reading all of your post. I like to write a little comment to support you.

  36. Hey There. I found your weblog the use of msn. This is a really smartly written article.
    I will be sure to bookmark it and come back to learn more of your helpful information. Thanks for the post.
    I’ll definitely return.

  37. I think this is among the most significant info for me.
    And i am glad reading your article. But want to remark on some
    general things, The website style is great, the articles is really nice
    : D. Good job, cheers

  38. I blog quite often and I seriously appreciate your content.
    Your article has truly peaked my interest. I will bookmark your website and
    keep checking for new details about once per week. I subscribed to your
    Feed too.

  39. Hi, this weekend is good in favor of me, since this time i am reading this impressive informative post here at
    my house.

  40. Appreciate this post. Let me try it out.

  41. I’m really enjoying the theme/design of your weblog.

    Do you ever run into any browser compatibility issues? A small number of
    my blog visitors have complained about my site not operating correctly
    in Explorer but looks great in Firefox. Do you have any advice to help fix this issue?

  42. If some one wants expert view regarding blogging and site-building then i
    propose him/her to go to see this weblog, Keep
    up the nice work.

  43. We are a group of volunteers and starting a new scheme in our
    community. Your website offered us with valuable info to work on. You have
    done a formidable job and our whole community will be thankful to you.

  44. Having read this I thought it was very informative.

    I appreciate you finding the time and effort to put this short article together.
    I once again find myself personally spending a significant amount of
    time both reading and posting comments. But so what, it
    was still worthwhile!

  45. If you wish for to grow your knowledge only keep visiting this website and be
    updated with the most up-to-date information posted here.

  46. Outstanding story there. What occurred after? Take care!

  47. What’s up colleagues, nice post and good urging commented at this place, I am really enjoying by
    these.

  48. You really make it appear really easy with your presentation but I in finding this topic to be actually something that I believe I’d
    by no means understand. It kind of feels too complicated and very wide for me.
    I am looking forward to your subsequent put up, I will attempt to get the hold of it!

  49. I am genuinely grateful to the holder of this web site who has
    shared this fantastic piece of writing at at this time.

  50. It’s not my first time to pay a quick visit this web page, i
    am visiting this website dailly and obtain good facts from here all the
    time.

  51. Hi! I just wish to give you a big thumbs up for your excellent info you have
    here on this post. I will be coming back to your web site for more soon.

  52. Hello i am kavin, its my first occasion to commenting anywhere, when i read this paragraph i thought i could also create comment due to this brilliant article.

Leave a Reply

Your email address will not be published.