Is a 5 gallon tank big enough to breed betta fish in?

No, a 5 gallon tank is not an appropriate size for breeding betta fish. Breeding pairs of the Betta splendens species and any hybrids they spawn should occupy about a 10 gallon aquarium as a bare minimum. Smaller betta tanks and other types of container setups do not afford enough swimming space for adults and their developing fry.