-
-
Notifications
You must be signed in to change notification settings - Fork 1.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Create crowdsec.conf #3407
base: master
Are you sure you want to change the base?
Create crowdsec.conf #3407
Conversation
Hi, thanks for the PR. Can you clarify please the field of application for that (necessity to report IP address to Crowdsec)... Regarding the action, I don't think it is justified to use whois in that manner:
Especially 1st and 2nd are annoying, since it'd be invoked synchronously, so the jail would not really serve any further bans until the action returns. Moreover it could involve extra risky consequences because the filter thread doesn't stop to find failures in that time, so it can be like a snowball - the longer delays action has, the more failures are found by filter, the more bans it could cause, the more load the action may get (which closes the vicious circle). This may look like a hostility against competitors, but... I'm not a friend of potentially long playing synchronous actions. |
I'm indeed using two IPS. But not on the same machine. Crowdsec consists of agents detecting ips whiich then give this information enriched with other information to the Crowdsec local API. The local APi does not need to be running on the same machine. The local API uploads detected ips to Crowdsec central API from which it also retrieves a list of checked ips detected by other users. The blocking is done by bouncers getting the list of ips to block from the local API. In my case I have a local API running, a bouncer on my edge router and agents on two Nginx server. Due to missing availablility of a recent agent implementation for Debian stable I still have fail2ban running on my reverse proxy. I see this a bit like sending the information about blocked ips to abuseipdb (which is also a fail2ban action). At the moment I have about 18k ips in the bouncer blocklist retrieved form central API from which 80 - 300 really get blocked by the bouncer and this only on port 80, 443. I also have around 30 ips not blocked by Crowdsec bouncer but detected by fail2ban because they are blocked on the reverse proxy. Those are definitly crawlers, spider or whatever you call em. So I felt it to be good idea to report them to Crowdsec central API. Using fail2ban as an agent for Crowdsec does possibly not follow the itentional idea of fail2ban nor Crowdsec. But in my case it closes a gap so I consider it a valid use case. I'm sure the action can be optimized. |
Isn't this what e.g. the mail action does? Calling whois and put the info into the mail body for each blocked ip. Look to me very similar. But for working with Crowdsec local API the enrichment with whois data is not necessary. It works with ip only. But the display on the Crowdsec / Metabase metrics look much better. |
Not all mails actions, you mean Regarding the crowdsec and suggested approach, it is a bit different instead of informing (what mail actions do), I though the crowdsec action should also have a protection function (if I understand it correctly). Anyway I'd like to avoid errors made previously... (at least to inform about the cons of usage of such actions). |
Actually yes. I believe for the use case I have (add ips detected by fail2ban to Crowdsec local api for blocking) the whois is not needed, it will work with ip only. And yes: Crowdsec does it's own blocking. This is what I want to make use of. Crowdsec is designed to take information from multiple sources to create blocklists for bouncers which may run on firewall devices or e.g. an Nginx. The sharing is the part fail2ban does not allow. I now like to use fail2ban as a Crowdsec agent to feed the information about the detected ip to the Crowdsec system. |
Dunno. Alternatively one can do the whois lookup as an option.
At the moment, yes. There is an RFE #881 for that. |
I'll suggest to split this into two agents: one simple for usage without whois plus a second one with whois (similar to mail agents). |
No. I don't like it for many reasons, especially because of code duplication and necessity to support/maintain 2 actions hereafter... Let alone the way with option is pretty simple here: [Definition]
...
# Option `lookup` (default `none`) - supply `whois` to perform a lookup for country and ASN using whois command
lookup = none
actionban = <lookup-<lookup>/lookupinfo>
TIME=$(LC_ALL=C date -d "@<time>" +"%%Y-%%m-%%d, %%H:%%M:%%S")
JSON_STRING=...
...
[lookup-none]
lookupinfo = WHOIS_COUNTRY=''
WHOIS_ASN=''
WHOIS_ASNAME=''
[lookup-whois]
lookupinfo = WHOIS_RESULT=$(whois <ip>)
WHOIS_COUNTRY=$(echo "$WHOIS_RESULT" | grep -a -i -m 1 Country | sed -n -e 's/^.*[Cc]ountry:\s*//p')
WHOIS_ASN=$(echo "$WHOIS_RESULT" | grep -a -i Origin | tail -1 | sed -n -e 's/^.*[Oo]rigin.*:\s*AS//p')
WHOIS_ASNAME=$(echo "$WHOIS_RESULT" | grep -a netname | tail -1 | sed -n -e 's/^.*netname:\s*//p') Moreover it makes possible to extend the action with new ways implementing lookup using other facilities (e.g. GeoIP databases or SaaS's, etc). And configuration for that would be also very simple (since one needs to specify other parameters anyway): action = %(known/action)s
crowdsec[lookup=whois, other_params=...] (or even set Although as already said if I were you, I'd still prefer pythonic action... |
Didn't really get this. I'm no experienced in phyton. |
I changed the action so that call of whois is optional (and off by default). In terms of Crowdsec the Crowdsec agents install geoip db with em to do the resolution. Crowdsec considers this information as relevant for finding decission on ban or not to ban. The info is also forwarded to the Crowdsec local API for statistics and display. I'll test if I find a way to at least put the crowdsec calling part into background. I do agree with your statements made in #3422 about fail2ban to be as lightweight as possible. But for DOS or DDOS situations with lots of ips to ban I beleive this is something that is not fail2ban to handle. Usually fail2ban is acting on log entries which usually is after a request is processed. For heavy requests situations rate limiting on a reverse proxy seems much more suitable. |
Hi @sebres
This is a draft for the crowdsec agent. Any feedback welcome.