I want to reject/refuse a tcp connection
There are a number of program types that can do this at different levels:

• BPF_PROG_TYPE_XDP allows you to drop packets (including TCP SYN packets) at the driver level. But only for incoming packets on a physical network interface
• BPF_PROG_TYPE_SCHED_CLS allows you to block incoming and outgoing packets at the TC level, both on physical and virtual interfaces of your choice.
• BPF_PROG_TYPE_CGROUP_SKB allows you to block incoming and outgoing packets at the cGroup level.
• BPF_PROG_TYPE_CGROUP_SOCK_ADDR at the BPF_CGROUP_INET4_CONNECT or BPF_CGROUP_INET6_CONNECT attach point allows you to audit connect calls or modify the parameters at a cGroup level.
• BPF_PROG_TYPE_LSM allows you to audit connect(...) syscalls, so to block outgoing connections on the syscall level.
• BPF_PROG_TYPE_KPROBE in combination with the bpf_override_return helper (if enabled by your kernel) can do the same as the LSM program, but on older kernels.

So plenty of options, depending on you exact usecase.

If it has exceeded some thresholds

BPF_PROG_TYPE_SOCK_OPS programs are great for observation but not enforcement. A sock ops program would be great to track the amount of open TCP sockets and storing that info in a map which one of the above mentioned programs can then used to start blocking new connections.

I’d like to refuse it when it connects(or at lease before the connection has been established)

If you decide to go for a packet-level apprach, you should make sure to send back an RST or ICMP reply to avoid the client retrying.

If you decide to go for a syscall level approach (for outgoing traffic) that shouldn’t be needed.