I know how frustrating it is to set up the ELK Stack for the time. If you have tried it before you know what I am talking about. The ELK Stack can be really tricky to set up. You might get version mismatches or the services might fail or the dashboards might be broken or the logs might just not appear.
This is my fourth time trying to build a working ELK Stack.. After taking it down and reinstalling it many times I finally have a setup that works. I can now say that I am analyzing logs from my Kali Linux machine and I have even set up alerts for suspicious activity.
This time I did not just install the ELK Stack. I actually understood how the ELK Stack works.
Why I’m Sharing This
Because if you are a beginner in cybersecurity or detection engineering or just someone trying to build your lab you need to hear this:
You are not alone.
The ELK Stack is not supposed to be easy to set up.
Most importantly:
“It’ll get easier day by day, but you have to do this every day.”
What Changed This Time?
I Had a Plan, Not Just Hope
Before I started I asked myself:
- What logs do I want to analyze with the ELK Stack?
- Where do I want the logs to go in the ELK Stack?
- How do I want to search and alert on the logs in the ELK Stack?
I mapped out my architecture:
- I used Kali Linux as the source of logs for the ELK Stack.
- I used Filebeat to ship logs to Logstash in the ELK Stack.
- I used Elasticsearch for storage. Searching in the ELK Stack.
- I used Kibana for dashboards and alerts in the ELK Stack.
This made all the difference.
Parsing Kali Logs
I focused on:
- Syslog (/var/log/syslog)
- Authentication logs (/var/log/auth.log)
- Command history and bash activity
- Custom log files for script-based monitoring
I used Filebeat modules and custom Logstash pipelines to parse the logs and ensure all fields aligned with ECS, which’s Elastic Common Schema for compatibility and consistency in the ELK Stack.
Writing Basic Alerts
Once logs started flowing, I began small:
- Multiple failed SSH attempts from the same IP
- Privilege escalation (sudo usage)
- Unexpected user logins
- Script execution in odd hours
Each alert includes:
- Rule logic
- Context (user, IP, command)
- Severity level
- MITRE ATT&CK mapping (where applicable)
Yes I made sure these were not noisy alerts in the ELK Stack.
Key Lessons I Learned
Repetition Builds Clarity
Reinstalling the ELK Stack times was not a waste it was how I learned where everything breaks in the ELK Stack.
Learn the Internals, Not Just the Commands
Do not just copy-paste, learn what each config file, each module each pipeline does in the ELK Stack.
That understanding is the key to owning the system the ELK Stack.
Document Everything
This time I kept a changelog and a GitHub repo for:
- Configurations
- Detection rules
- Lessons learned
Now I can iterate, not rebuild the ELK Stack.
For Beginners: My Advice
If you are starting your ELK Stack journey remember:
- It will frustrate you.
- It will test your patience.
- But it will also make you better.
Whether you are a cybersecurity student, an analyst or just ELK-curious stick with the ELK Stack.
You do not have to master everything at just master one piece at a time with the ELK Stack. Remember:
“It’ll get easier day by day, but you have to do this every day.”
What’s Next?
- I will be sharing some of my rule templates on GitHub soon for the ELK Stack.
- I am planning to integrate Sigma rules. Maybe even push alerts into TheHive or Slack for the ELK Stack.
- Eventually I will be building a threat hunting dashboard based on MITRE ATT&CK for the ELK Stack.
Connect with Me
If you are struggling with your setup or want to share your experience drop a comment or DM me on LinkedIn or GitHub.
We are all learning, at different speeds with the ELK Stack.
Thanks, for reading.
If you have made it this far you have already got the persistence you need with the ELK Stack.
Stay curious. Stay consistent. ELK will become your friend, I promise.
— Manish Rawat
From Frustration to Triumph: How I Finally Conquered the ELK Stack was originally published in OSINT Team on Medium, where people are continuing the conversation by highlighting and responding to this story.