Firewall rule saved in panel but not applied

I added an allow rule for 5432 in the built-in firewall manager, it shows in the list, but the port is still refused from outside. Had to open it with ufw on the box directly. Panel firewall seems out of sync with what's live.