Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ossec-analysisd high memory usage over time. #1502

Open
Lunakrypt opened this issue Aug 14, 2018 · 12 comments
Open

Ossec-analysisd high memory usage over time. #1502

Lunakrypt opened this issue Aug 14, 2018 · 12 comments

Comments

@Lunakrypt
Copy link

Hello everyone. I am currently running OSSEC 3.0.0. I am having an issue where the ossec-analysisd service eventually uses all the memory. Currently we have 4GB of ram for the vm that OSSEC is running on. Over the course of approximately 2 days ossec-analysisd's memory usage slowly raises until maximum memory is used, then ossec-analysisd stops. Restarting OSSEC does reset the memory usage, but it just slowly rises again.

For a little bit of history. On this server we were running 2.8.3 for over a year, and it worked perfectly. We decided to upgrade to 2.9.4. That is when we first ran into the memory issue. When version 3.0.0 came out we did a clean install of that to attempt to fix the memory issue, but the issue still occurs.

Is there recommended hardware requirements for OSSEC? Or is there a new setting that I need to change to alleviate this issue? I have searched and could not find anything. I am sorry if this does turn out to be something obvious that I missed.

@ddpbsd
Copy link
Member

ddpbsd commented Aug 17, 2018

How many agents do you have? I'm not seeing this issue, but I have very few agents connected to my systems.
I have been going through some of the coverity output looking for things like memory leaks, but I can't remember if I adjusted anything in analysisd specifically.

@Lunakrypt
Copy link
Author

Lunakrypt commented Aug 17, 2018

There is approximately 40 active agents currently running. Though we have approximately 76 added, some have been retired.

One of the things I have been wondering about was in the /var/ossec/etc/internal_options.conf

The new internal_options.conf file has this:

Analysisd stats maximum diff.
analysisd.stats_maxdiff=999000

The one from 2.8.3 had this:

Analysisd stats maximum diff.
analysisd.stats_maxdiff=25000

Would changing this to a lower number cause any issues?

Thank you for your help so far.

@ddpbsd
Copy link
Member

ddpbsd commented Aug 20, 2018

You could try it, I don't think it would cause any issues. I can't remember why that was changed off hand, but see if it helps.

@Lunakrypt
Copy link
Author

I apologize for a late response. I did change the setting to about a tenth of what it was. I am just waiting to see if the issue still occurs. I changed it on Monday, and the ossec-analysisd is currently at 76.2% memory usage. I will wait to see if the service does self terminate again. I will keep you updated. But thank you for all of your help so far.

@Lunakrypt
Copy link
Author

Good Morning. Changing the analysisd.stats_maxdiff setting to a lower number has had no effect i'm afraid.

@Lunakrypt
Copy link
Author

Well I didn't get the issue fixed i'm afraid. But I did get around it by just making a cron job to restart the OSSEC services every 24 hours.

@TheoPoc
Copy link

TheoPoc commented Oct 4, 2018

Hi,

I got exactly the same problem, ossec-analysisd increase memory over time. Maybe you can help me if i give the configuration for server and agent. I don't use use fully the fonctionnality OSSEC, i use it just for the integrity of files. Hope you can help me.

Best regards
internal_options.conf.txt
ossec.conf.agent.txt
ossec.conf.txt

@bigwoody2253
Copy link

I on a new OSSEC server with version 3.0.0-5505.el6.art from the Atomic repository. Downgrading to version 2.9.3-2833.el6.art did not help.

I didn't have any previous versions available to try, so I cloned another OSSEC VM that had version 1:2.9.0-1700.el6.art installed and that did the trick. So it seems this issue started sometime after version 2.9.0 and on or before version 2.9.3.

@bigwoody2253
Copy link

Whoops, mangled that first sentence. Should read "I had this issue on a new OSSEC server..."

@atomicturtle
Copy link
Member

Can you retest on the 2.9.1 branch and see if it pops up there?

@bigwoody2253
Copy link

Is there a 2.9.1 RPM available for CentOS 6 (or similar)? Or would I need to compile 2.9.1 from source?

@karlgrz
Copy link

karlgrz commented Jun 11, 2019

We're experiencing this unless we restart the service on a cron.

It's very easy to replicate on a small instance (t3.micro). Anything I can provide to help diagnose this please let me know.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

6 participants