my gaming group

[Tsotha-lanti]

one of the antagonists is my brother who never takes the game sriously and wastess everbody's time by giving joke answers, replying stupidly or taking a stupid action which by houserule has been outlawed. my houserules are not evil alignments, no killing random people just for fun, no betraying temmates and no stealing items from your allies. that is all so it shouldn't be that much of a restriction. and they're free to kill innocent monsters just not innocent townspeople.

He's clearly not actually interested in playing. Tell him he's "free" to no longer play.

the other two are my cousins who can't seem to stop tackling each other no matter what I do. they come over once a week and if I kicked them out they'd have nobodyto play with so what can I do to convince them to actually sit down and behave?

Tackling each other? Can you explain?

Seriously, your OP answered your question already: without these three people, the game goes smoothly. Therefore, remove these three people.

Also, you're all teens and/or pre-teens, right? Back at that age, I had the same problem DMing anything whenever the group was bigger than 4-5 people, no matter who was involved. You can only cram so many kids around a table and get any productivity.