So... you just said you think he's beyond help, yet you're trying to figure out "what to tell him to do" and "how to help him."
WHY do you need to tell him what to do? Does he not have the knowledge or intelligence or resources to figure out what he should do if he really wanted a job?
Maybe ask yourself why you feel the need to correct, control, or "fix" your brother... and your mother's relationship with him? How does their relationship reflect on you?
It sounds like you just need to step back and get a little distance, set some boundaries and focus on your own actions and your own immediate family (husband, yourself, and children if any), and to let your mother and brother deal with the consequences of their choices on their own.
I know it may sound and feel "cold" to do this, but doing the "nice" thing isn't always what is most loving or best thing for the other person. Sometimes, backing off and letting them feel their consequences is the best, and/or only, way to help in the long run.
You have to believe that your brother and mom are capable of figuring it out and learning to change without you dragging them along--and if they are actually *unable* to do that, then all your advice and attempts to change them won't make any difference, anyway, right?
You cannot force anyone to get a job--or keep one. No matter how much you want to see your brother succeed, no matter how much potential you feel he has, you cannot live his life FOR him. And you can't nag, bully, or debate him into a different mindset, either.
I know it's hard to stay out of it. I used to try to "help" family with well-meaning (usually unwanted) advice, too, but they rarely took it, and somehow, they've survived without me. I've learned that we can't control other people, no matter how logical our reasoning... we can only control ourselves and our own response to others.