Alliances with mainstream parties are the kiss of death for these sorts of parties. The only viable alternative to an outright victory is to form an alliance with socialists or some other party that is ideologically closer to them.

Yeah, that's why I'd suggest the Bernie movement as natural allies. The left-wing students who make up the bulk of it naturally bridge the gap. Organized labor is a little harder to incorporate, they tend to ally with environmentalists on concerns that impact employee health but otherwise there's always been tension between the two groups. But, with today's service economy and most new industrial jobs being "green tech" (high-speed rail), the tension lessens.

There's no socialist party that's large enough, on their own, to bolster the Greens and shed the hippie baggage. Socialist Alternative are growing extensively in the northern part of the West Coast, but they're focusing on city politics and piggy-backing the Greens presidentially. PSL are Stalinoids who bring a lot of baggage of their own.

Just forming a coalition with the Democrats like Bernie himself did, first off, is mostly impossible on a party-wide scale. Unlike countries like Germany we don't have proportional representation, so the usual response to the rise of a third party is either its swift collapse or it replacing one of the two major ones. Our system is set up in such a way that the only real way to avoid throwing the election to the House of Representatives is two parties.

Second, as you mentioned, it's never led to anything more than providing grassroots cover for the major party they're aligning with and watering themselves down in compromise to get one or two small pieces of their agenda on the table.