I’m currently going through a similar situation at the moment (OPNSense firewall, Traefik reverse proxy). For my solution, I’m going to be trial running the Crowdsec bouncer as a Traefik middleware, but that shouldn’t discourage you from using Fail2Ban.
Fail2Ban: you set policies (or use presets) to tempban IPs that match certain heuristic or basic checks.
Crowdsec Bouncer: does fail2ban checks if allowed. Sends anonymous bad behavior reports to their servers and will also ban/captcha check IPs that are found in the aggregate list of current bad actors. Claims to be able to perform more advanced behavior checks and blacklists locally.
If you can help it, I don’t necessarily recommend having OPNSense apply the firewall rules via API access from your server. It is technically a vulnerability vector unless you can only allow for creating a certain subset of deny rules. The solution you choose probably shouldn’t be allowed to create allow rules on WAN for instance. In most cases, let the reverse proxy perform the traffic filtering if possible.
If you’re running an email server for more than a handful of persistent users, I’d probably agree. However, there are self-host solutions that do a decent job of being ‘all-in-one’ (MailU, Mailcow, Docker-Mailserver) that can help perform a lot of input filtering.
If your small org just needs automation emails (summaries, password resets), it’s definitely feasible to do actually, as long as you have port 25 available in addition to 465, 587 and you can assign PTR records on reverse DNS. Optionally you should use a common TLD for your domain as it will be less likely to be flagged via SpamAssassin. MXToolbox and Mail-Tester together offer free services to help test the reliability of your email functionality.