Use of ELK with Python

Question

The project that I am working on is a bit confidential, but I will try to explain my issues and be as clear as possible because I need your opinion.

Project:

They asked me to set up a local ELK environment , and to use Python scripts to communicate with this stack (ELK), to store data, retrieve it, analyse it and visualise it thanks to Kibana, and finally there is a decision making based on that data(AI). So as you can see, it is a Data Engineering project with some AI for the decision making process. The issues that I am facing are:

I don't know how to use Python to communicate with the stack, I didn't find resources about it
Since the data is confidential, how can I assure a high security?
How many instances to use?
I am lost because I am new to ELK and my team is not Dev oriented

I am new to ELK, so please any advice would be really helpful!

Isaac Keleher · Accepted Answer · 2022-03-02 23:43:33Z

2

I don't know how to use Python to communicate with the stack, I didn't find resources about it

For learning how to interact with your stack use the python library:

You can install using pip3 install elasticsearch and the following links contain a wealth of tutorials on almost anything you would need to be doing.

https://kb.objectrocket.com/category/elasticsearch?filter=python

Suggest you start with these two:

https://kb.objectrocket.com/elasticsearch/how-to-parse-lines-in-a-text-file-and-index-as-elasticsearch-documents-using-python-641

https://kb.objectrocket.com/elasticsearch/how-to-query-elasticsearch-documents-in-python-268

Since the data is confidential, how can I assure a high security? You can mask the data or restrict index access.

https://www.elastic.co/guide/en/elasticsearch/reference/current/authorization.html

https://nl.devoteam.com/expert-view/field-level-security-and-data-masking-in-elasticsearch/

How many instances to use? I am lost because I am new to ELK and my team is not Dev oriented

I suggest you start with 1 Elasticsearch node, if you're on AWS use a t3a.large or equivalent and run Elasticsearch, Kibana and Logstash all on the same machine.

For setting it up: https://www.elastic.co/guide/en/elastic-stack-get-started/current/get-started-stack-docker.html#run-docker-secure

answered Mar 2, 2022 at 23:43

Isaac Keleher

836 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

bechirjamoussi Over a year ago

thank you very much, the ressources are really helpful!!

uenokazuma · Accepted Answer · 2022-03-02 10:19:51Z

If you want to use phyton as your integration tools to Elasticsearch you can use elasticsearch phyton client.

The other options you can use python to create the result and save it in log file or insert to database than Logstash will get your data.

For the security ELK have good security from API authorization user authentication to cluster security. you can see in here Secure the Elastic Stack

I just use 1 instance, but feel free if you think you will need to separate between Kibana and Elasticsearch and Logstash (if you use it) or you can use docker to separate it. Based on my experience, if you are going to load a lot of data in a short time it will be wise If you separate it so the processes don't interfere with each other.

Collectives™ on Stack Overflow

Use of ELK with Python

2 Answers 2

1 Comment

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Related