The goal here is not to learn how to use fail2ban (though... at least minimally how to adapt), but solely to enable its functionality.
Although available in pkgsrc, it requires some small adjustments before it can work on illumos.
I'm basing this on OmniOSce, but the same kind of adjustments must be made on OpenIndiana.

For the example, we will configure a simple jail for SSH and ban users for a few minutes after 5 failed connection attempts.
This is just an example, of course, but it demonstrates the type of adaptations that are necessary.

1) Installing the pkgsrc repository
(switch to root)
The commands block below is a summary of the guide available here :
[https://pkgsrc.joyent.com/install-on-illumos/]
Copy/paste the following commands block into a shell:

cd /tmp && BOOTSTRAP_TAR="bootstrap-trunk-x86_64-20240116.tar.gz" && BOOTSTRAP_SHA="4d92a333587d9dcc669ff64264451ca65da701b7" && curl -O https://pkgsrc.smartos.org/packages/SmartOS/bootstrap/${BOOTSTRAP_TAR} && tar -zxpf ${BOOTSTRAP_TAR} -C / && sed -i 's/PATH=\(.*\)/PATH=\1:\/opt\/local\/sbin:\/opt\/local\/bin/' /etc/default/login && cd /tmp && UPGRADE_TAR="bootstrap-trunk-x86_64-20240116-upgrade.tar.gz" && UPGRADE_SHA="3099211460e84c34b9f558c8f7354a871187be41" && curl -O https://pkgsrc.smartos.org/packages/SmartOS/bootstrap-upgrade/${UPGRADE_TAR} && tar -zxpf ${UPGRADE_TAR} -C / && echo 'PATH=/opt/local/sbin:/opt/local/bin:$PATH' >> /etc/profile && echo 'MANPATH=/opt/local/man:$MANPATH' >> /etc/profile 
alias pkg_add=/opt/local/sbin/pkg_add ; alias pkgin=/opt/local/bin/pkgin
pkg_add -U pkg_install pkgin libarchive && pkgin clean && pkgin -y upgrade

Close your root session and reconnect as root to reload the PATH

2) Installing fail2ban

pkgin -y in fail2ban

A few other things need to be created :
mkdir /opt/local/etc/fail2ban/filter.d/ignorecommands
touch /opt/local/etc/fail2ban/filter.d/ignorecommands/apache-fakegooglebot
chmod 0644 /opt/local/etc/fail2ban/filter.d/ignorecommands/apache-fakegooglebot

3) Configure IPFilter action
One of the main issues with fail2ban on illumos distributions is its interaction with the IPFilter firewall.
Indeed, the fail2ban action files are found in /opt/local/etc/fail2ban/action.d, including the one for our firewall: ipfilter.conf...
... However, the content of this file is tailored for NetBSD, and illumos distributions do not use the same commands or paths to operate IPF.
So, we need to change that.

Install nano if you are not familiar with vi :
pkg install nano

Change the path of the IPF command:
vi /opt/local/etc/fail2ban/action.d/ipfilter.conf

First, here's what needs to be changed in ipfilter.conf : actionstart.
Replace the default actionstart by this :
actionstart = /usr/sbin/ipf -E

Then, change the actionban and actionunban.
Change actionban to this:
actionban = echo "block in quick from <ip>/32" | /usr/sbin/ipf -f -
And actionunban to that:
actionunban = echo "block in quick from <ip>/32" | /usr/sbin/ipf -r -f -

4) Configure the IPFilter firewall
I won't explain here how to configure IPF.
IPF should be configured with a file containing rules (such as /etc/ipf/ipf.conf).
I'll write a small article if needed.

We will just enable ipfilter for now :
svcadm enable svc:/network/ipfilter:default

Check if the firewall is enabled (you must wait a few seconds after enabling ipfilter to see it running with svcs)
svcs svc:/network/ipfilter:default

5) Configure a simple jail for SSH with TCP Wrappers
First, we will disable jail.conf and rather use our custom jail.local
mv /opt/local/etc/fail2ban/jail.conf /opt/local/etc/fail2ban/jail.conf.BAK

vi /opt/local/etc/fail2ban/jail.local

[DEFAULT]
bantime = 10m
findtime = 10m
maxretry = 5
maxmatches = %(maxretry)s
backend = auto
usedns = warn
logencoding = auto

#
# ACTIONS
#

port = 0:65535
fail2ban_agent = Fail2Ban/%(fail2ban_version)s
banaction = ipfilter
banaction_allports = ipfilter
action_ = %(banaction)s[port="%(port)s", protocol="%(protocol)s", chain="%(chain)s"]
action_mw = %(action_)s
        %(mta)s-whois[sender="%(sender)s", dest="%(destemail)s", protocol="%(protocol)s", chain="%(chain)s"]
action_mwl = %(action_)s
             %(mta)s-whois-lines[sender="%(sender)s", dest="%(destemail)s", logpath="%(logpath)s", chain="%(chain)s"]
action_xarf = %(action_)s
             xarf-login-attack[service=%(__name__)s, sender="%(sender)s", logpath="%(logpath)s", port="%(port)s"]
action_cf_mwl = cloudflare[cfuser="%(cfemail)s", cftoken="%(cfapikey)s"]
                %(mta)s-whois-lines[sender="%(sender)s", dest="%(destemail)s", logpath="%(logpath)s", chain="%(chain)s"]
action_blocklist_de  = blocklist_de[email="%(sender)s", service="%(__name__)s", apikey="%(blocklist_de_apikey)s", agent="%(fail2ban_agent)s"]
action_badips = badips.py[category="%(__name__)s", banaction="%(banaction)s", agent="%(fail2ban_agent)s"]
action_badips_report = badips[category="%(__name__)s", agent="%(fail2ban_agent)s"]
action_abuseipdb = abuseipdb

#
# JAILS
#

[ssh-tcpwrapper]
enabled     = true
filter      = sshd
action      = hostsdeny[daemon_list=sshd]
#action    = ipfilter
logpath     = /var/adm/auth.log

You will notice that for my [ssh-tcpwrapper] jail, I did not choose IPFilter to block the intruder's IP address.
I preferred to use the action directly using TCP-wrappers in order to demonstrate that when not using IPF to block an IP, you generally also need to modify the file of the desired action, as its default syntax probably won't work as it.
So, here we will see a bit further how to adapt the action file /opt/local/etc/fail2ban/action.d/hostsdeny.conf.
However, now that our ipfilter.conf action file is properly configured to work with IPF on illumos, you can perfectly delete the line 'action = hostsdeny[daemon_list=sshd]' and replace it by 'action = ipfilter' since the banning action stated in [DEFAULT] is ipfilter.

So we go on with TCP-wrappers action.
In order to make this jail work, two things need to be done: The first one is mandatory, even for IPFilter action !

First, ensure that openssh connection logs are sent to /var/adm/auth.log because syslog does not do it by default.
So, add the following directive at the bottom of /etc/syslog.conf
auth.info /var/adm/auth.log
WARNING! It's not a space between auth.info and /var/adm... but a tabulation!
If you use a space, it won't work!

Then create the file in question :
touch /var/adm/auth.log

Finally, (re)start syslog:
svcadm disable svc:/system/system-log:default && sleep 5 && svcadm enable svc:/system/system-log:default

Secondly, make a modification in the action file /opt/local/etc/fail2ban/action.d/hostsdeny.conf because the command to unban an IP is not functional.
(As I already told it, you will often encounter this issue with fail2ban on illumos. There are adjustments of this kind to make... But we can do it!)
Replace the default command with this one:
actionunban = IP=$(echo "<ip_value>") && sed -i "/^<daemon_list>: $IP$/d" <file>

Now, we can start fail2ban:
fail2ban-server start
Apart from a small warning about IPv6, "Server ready" should appear.

When starting fail2ban, a log file appears: /var/log/fail2ban.log

You can view it live while testing more than 5 failed SSH connections to your server to observe the effective (or ineffective) banning of the client's IP address:
tail -f /var/log/fail2ban.log

During banning, the directive 'sshd: CLIENT-IP' will be directly added to the /etc/hosts.deny file.
During unbanning, the directive should disappear. If it doesn't, it means the syntax of the unban directive in the action file is incorrect. (that is why I show here how to change it)

If you use ipfilter banaction instead of hostsdeny, then you can see the IPF banning rule added with :
ipfstat -io

Some additional commands:

Show banned IPs:
fail2ban-client banned
Unban all IPs: (just for the test !)
fail2ban-client unban --all

Good luck !