The question about Malcolm Turnbull’s political future evokes the refrain: will he or won’t he? A more appropriate query is: should he? To which the sensible response is: definitely not.

It made sense for Turnbull to replace Brendan Nelson as opposition leader in late 2008. Nelson had been a competent minister in the Howard government. However, as Liberal leader he proved to be long on emotion but short on substance.

Turnbull’s real support in the party room on December 1 last year was not demonstrated in the leadership ballot – which he lost to Tony Abbott by a mere vote. Turnbull’s position came under challenge only due to his determination to support the broad outlines of the Rudd government’s emissions trading scheme. After the change in leadership, there was a secret ballot on the scheme. The motion not to pass this legislation in the Senate was carried by 54 to 29.

This means Turnbull staked his leadership to an issue on which he had the backing of only a third of his parliamentary colleagues. The support within the Coalition was even smaller, since the National Party opposed the government’s legislation. Most members of the press gallery favour an ETS. Turnbull’s convictions were widely praised. This overlooked the fact he could not convince his colleagues to support his position on the environment.

After losing the leadership, Turnbull acted properly in staying on as member for Wentworth. There was only about a year to the election and the Liberals did not want to endure the effort or cost of a byelection. Turnbull said he would consider his future. This month, he announced his retirement, to take effect at the coming election.

Writing in the Herald, he described losing the leadership as heartbreaking but said he had not changed his position on the need to introduce an ETS, irrespective of what other nations – including the United States – intend to do.

There was a suggestion his decision to quit was sparked by Abbott’s decision not to offer him the position of shadow finance minster after Senator Nick Minchin’s retirement. Again, this demonstrates Turnbull’s political naivety. Abbott could hardly give him one of the top positions in the opposition, knowing Turnbull had indicated an intention to cross the floor and support the ETS legislation.

Some commentators have suggested the Liberal Party forced Turnbull out and reflected it could ill afford to lose a person of his intelligence and ability. The fact is, Turnbull lacked support in the party room.

In democracies, the smartest politician in the party room has just as many votes as the not-so-smart. That is, one each. In December, Turnbull was unable to get a majority of his colleagues to back him on emissions trading. It is unlikely this situation will change soon.

Of course, politicians make comebacks. Robert Menzies was forced to quit as United Australia Party prime minister in 1941. He re-emerged as leader of the new Liberal Party just over three years later. But Menzies was never at odds with a clear majority of his colleagues over a key policy issue. Moreover, he acknowledged he had been a poor manager of people and resolved to temper his arrogance and to limit self-indulgent displays of cleverness. So far, Turnbull has not demonstrated such self-awareness or discipline.

Then there is the matter of social policy. It makes no sense for the Coalition leader to be outflanked by Labor on social issues. Yet this is precisely what occurred in 2008 when Bill Henson’s photography of nude prepubescent children became a matter of controversy. As David Marr wrote in The Henson Case, Rudd went further than any politician in condemning the pictures while Turnbull “was the first politician of any stature to question Henson’s ordeal and put in a word for freedom”.

In a sense, Turnbull emerged as the political hero of Marr’s book. That might have won him support in Wentworth. But Rudd’s condemnation would have had majority support in suburban and regional Australia where most marginal seats are. There is no reason to question Rudd’s sincerity on this issue. It’s just that his stand also made political sense.

The other candidate in last year’s leadership ballot was Joe Hockey, the member for North Sydney. In a speech to the Grattan Institute in March, Hockey expressed concern about the anti-terrorism laws introduced by the Howard government and supported by the Labor opposition under Kim Beazley. The legislation has led to a number of convictions in both Melbourne and Sydney. As the Finance Minister, Lindsay Tanner, has pointed out, it’s unusual when a senior Liberal politician criticises Labor from the left. On the anti-terrorism legislation, Hockey is closer to the Greens.

Once again, Hockey’s position may have appeal on the lower north shore. Yet it is unlikely to engender support in the outer suburbs and regional centres. The same is true of Hockey’s criticism of the attempt by the Communication Minister, Stephen Conroy, to stop child pornography on the internet.

Turnbull and Hockey are admirers of Menzies. But Menzies was more socially conservative than Labor – and the Liberals have never won an election on a libertarian agenda.

Turnbull has had an impact on Australian politics – in both government and opposition. He made his own decision to resign and he would be well advised to stick by his initial judgment.