Hi, thanks for the PR.

Can you clarify please the field of application for that (necessity to report IP address to Crowdsec)...
Are you using two IPS systems together? Why?

Regarding the action, I don't think it is justified to use whois in that manner:

• it makes external communication with all that that implies - connectivity issues, long delays or timeouts (service under high load, busy or restricted), etc
• it may be restricted by quotas (e. g. maximal 1000 lookups in a day), so it could fail hereafter or still worse enter a certain delay if your host gets blocked (which can even cause a timeout);
• its response is not really standardized (depending on many factors), so can at the end cause parsing mistakes or errors

Especially 1st and 2nd are annoying, since it'd be invoked synchronously, so the jail would not really serve any further bans until the action returns. Moreover it could involve extra risky consequences because the filter thread doesn't stop to find failures in that time, so it can be like a snowball - the longer delays action has, the more failures are found by filter, the more bans it could cause, the more load the action may get (which closes the vicious circle).

This may look like a hostility against competitors, but... I'm not a friend of potentially long playing synchronous actions.
Maybe a pythonic action capturing whois info (but also notifying crowdsec) completely asynchronous, with some cache of response (by a subnets) and a fallback mechanisms could be better and at least fewer error-prone.

Can you clarify please the field of application for that (necessity to report IP address to Crowdsec)... Are you using two IPS systems together? Why?

I'm indeed using two IPS. But not on the same machine.

Crowdsec consists of agents detecting ips whiich then give this information enriched with other information to the Crowdsec local API. The local APi does not need to be running on the same machine. The local API uploads detected ips to Crowdsec central API from which it also retrieves a list of checked ips detected by other users. The blocking is done by bouncers getting the list of ips to block from the local API.

In my case I have a local API running, a bouncer on my edge router and agents on two Nginx server. Due to missing availablility of a recent agent implementation for Debian stable I still have fail2ban running on my reverse proxy.
Also, I feel the Crowdsec agents a bit heavy (espacially with the enriching). So I figured this out to send the information about ips detected by fail2ban to my local API, which then updates the blocklist on the router (and therefor blocks the ips for all open systems in the network) and also uploads the info to Crowdsec central API for sharing.

I see this a bit like sending the information about blocked ips to abuseipdb (which is also a fail2ban action).

At the moment I have about 18k ips in the bouncer blocklist retrieved form central API from which 80 - 300 really get blocked by the bouncer and this only on port 80, 443. I also have around 30 ips not blocked by Crowdsec bouncer but detected by fail2ban because they are blocked on the reverse proxy. Those are definitly crawlers, spider or whatever you call em. So I felt it to be good idea to report them to Crowdsec central API.

Using fail2ban as an agent for Crowdsec does possibly not follow the itentional idea of fail2ban nor Crowdsec. But in my case it closes a gap so I consider it a valid use case.

I'm sure the action can be optimized.

Regarding the action, I don't think it is justified to use whois in that manner:

Isn't this what e.g. the mail action does? Calling whois and put the info into the mail body for each blocked ip. Look to me very similar.

But for working with Crowdsec local API the enrichment with whois data is not necessary. It works with ip only. But the display on the Crowdsec / Metabase metrics look much better.

Isn't this what e.g. the mail action does?

Not all mails actions, you mean mail-whois action surely, but...
Yes. And that is why it is not recommended to use such actions at all (let alone I never understand why someone would need that - thousands mails about some bans, hmm...)

Regarding the crowdsec and suggested approach, it is a bit different instead of informing (what mail actions do), I though the crowdsec action should also have a protection function (if I understand it correctly).

Anyway I'd like to avoid errors made previously... (at least to inform about the cons of usage of such actions).

Actually yes. I believe for the use case I have (add ips detected by fail2ban to Crowdsec local api for blocking) the whois is not needed, it will work with ip only.
It's just not showing the country information in Crowdsec metrics or dashboard.
Maybe this is a way to go for this agent?

And yes: Crowdsec does it's own blocking. This is what I want to make use of. Crowdsec is designed to take information from multiple sources to create blocklists for bouncers which may run on firewall devices or e.g. an Nginx. The sharing is the part fail2ban does not allow.
So with Crowdsec I can run mutiple agents on different services/servers I have and share the bad ip detection and blocking decission by blocking on my edge router for all services all together. And also share the information with the Crowdsec community.

I now like to use fail2ban as a Crowdsec agent to feed the information about the detected ip to the Crowdsec system.

Maybe this is a way to go for this agent?

Dunno. Alternatively one can do the whois lookup as an option.

The sharing is the part fail2ban does not allow.

At the moment, yes. There is an RFE #881 for that.

I'll suggest to split this into two agents: one simple for usage without whois plus a second one with whois (similar to mail agents).
I feel a need for a way to use fail2ban with Crowdsec. To outnumber the bad I (as a user) appriciate if I have the option to use both systems working together to give best results.

I'll suggest to split this into two agents: one simple for usage without whois plus a second one with whois (similar to mail agents).

No. I don't like it for many reasons, especially because of code duplication and necessity to support/maintain 2 actions hereafter...
Sure we have many mail-actions but it is historical and I don't think it is justified to continue this handling further.

Let alone the way with option is pretty simple here:

[Definition]

...

# Option `lookup` (default `none`) - supply `whois` to perform a lookup for country and ASN using whois command
lookup = none

actionban = <lookup-<lookup>/lookupinfo>
            TIME=$(LC_ALL=C date -d "@<time>" +"%%Y-%%m-%%d, %%H:%%M:%%S")
            JSON_STRING=...
            ...


[lookup-none]
lookupinfo = WHOIS_COUNTRY=''
            WHOIS_ASN=''
            WHOIS_ASNAME=''

[lookup-whois]
lookupinfo = WHOIS_RESULT=$(whois <ip>)
            WHOIS_COUNTRY=$(echo "$WHOIS_RESULT" | grep -a -i -m 1 Country | sed -n -e 's/^.*[Cc]ountry:\s*//p')
            WHOIS_ASN=$(echo "$WHOIS_RESULT" | grep -a -i Origin | tail -1 | sed -n -e 's/^.*[Oo]rigin.*:\s*AS//p')
            WHOIS_ASNAME=$(echo "$WHOIS_RESULT" | grep -a netname | tail -1 | sed -n -e 's/^.*netname:\s*//p')

Moreover it makes possible to extend the action with new ways implementing lookup using other facilities (e.g. GeoIP databases or SaaS's, etc).

And configuration for that would be also very simple (since one needs to specify other parameters anyway):

action = %(known/action)s
         crowdsec[lookup=whois, other_params=...]

(or even set lookup in action.d/crowdsec.local)

Although as already said if I were you, I'd still prefer pythonic action...

Although as already said if I were you, I'd still prefer pythonic action...

Didn't really get this. I'm no experienced in phyton.

I changed the action so that call of whois is optional (and off by default).

In terms of Crowdsec the Crowdsec agents install geoip db with em to do the resolution. Crowdsec considers this information as relevant for finding decission on ban or not to ban. The info is also forwarded to the Crowdsec local API for statistics and display.
As for fail2ban the decission to ban is already done when calling the action, the geo information is not needed. Also, Crowdsec seems to work an enhancements on local API side to take away such tasks as geoip lookup if not needed on the agent in future.

I'll test if I find a way to at least put the crowdsec calling part into background. I do agree with your statements made in #3422 about fail2ban to be as lightweight as possible. But for DOS or DDOS situations with lots of ips to ban I beleive this is something that is not fail2ban to handle. Usually fail2ban is acting on log entries which usually is after a request is processed. For heavy requests situations rate limiting on a reverse proxy seems much more suitable.