The world's energy system seems to have come unhinged. Oil is trading at record high prices because demand keeps rising even as supplies become unreliable. Oil exporters from Iran to Russia and Venezuela are using their petrocash to pursue agendas that undercut western security and interests. Supplies of natural gas also seem less secure than ever.

The root cause of these troubles is dysfunctional energy politics. The countries with the strongest incentives to cut their vulnerability to volatile energy markets - notably America - are unable to act because influential politicians view all serious policies as politically radioactive. Efforts to boost supply have little leverage because the most attractive geological riches are found mainly in countries where state-owned companies control the resources and outsiders have little clout. Thus, the current energy debates are generating a volcano of proposals that have no positive impact on tight markets.

Yet these structural barriers to serious policy remain hidden because the debate labours under the meaningless umbrella of "energy security". Proper policy on oil and gas must start with the distinct uses for these fuels - each requiring its own political strategy.

The effort on oil must focus on transportation. Vehicles and aircraft work best with liquid fuels that can store large quantities of energy in a compact space and flow easily through pipes to engines. Searching for a better substitute is worthwhile, but the effort faces an uphill battle. With today's technologies, no other energy liquid can reliably beat petroleum. Liquids can be made from coal, as South Africa and China are doing. But that approach is costly and has unattractive environmental implications. Brazil and the US have focused on ethanol, which they distill from sugar or grain from crops. However, those programmes, which account for less than 0.5 per cent of the world's energy liquids, have a negligible impact on the oil market. Yet, America is redoubling its ethanol effort because it is politically unbeatable to reward corn growers and grain handlers who are a formidable force in US politics. Indeed, requirements for ethanol in America have created a more rigid fuel supply system that actually raises the price of oil products, although ethanol's backers originally claimed they would cut energy costs. That same political force also blocks imports of cheaper Brazilian ethanol. In principle, a better approach is so-called "cellulosic ethanol", which promises lower costs as it converts whole plants into ethanol rather than just the grain. But like most messiahs, its attraction lies in the future. So far, nobody has made the system work at the scale of a commercial refinery.

The best way to temper oil demand today is by lifting efficiency. Even this economic winner is politically difficult to implement. The US, which consumes one-quarter of the world's oil, has not changed fuel efficiency standards for new cars in 16 years. Every big economy - even China's - has stricter fuel economy rules than America's. Political gridlock has stymied even modest proposals to allow trading of efficiency credits. A trading scheme is politically inconvenient as it could force US carmakers (which make generally inefficient cars) to buy valuable credits from foreign brands. No politican wants to multiply Detroit's problems.

Even better ideas - such as a stiffer petrol tax - stay stuck on opinion pages of newspapers and in academic journals. Despite what is increasingly termed today's "energy crisis", these ideas barely cross the lips of politicians who want to remain viable among the thicket of anti-tax conservatives and pro-Detroit lobbyists.

The approaches needed for natural gas are quite different. In western Europe, which has long depended on imported gas from Russia, Algeria and a few smaller suppliers, the vulnerabilities are particularly stark. In principle, though, gas dependencies are easier to manage than oil because gas has rivals for each of its major uses. In electric power generation, countries must preserve diversity - ensuring, for example, that advanced coal and nuclear technologies remain viable. While "diversity" is motherhood in energy policy, in reality it requires difficult choices. In continental Europe, for example, policy-­makers have not seriously confronted the conflict between the need for diversity while, at the same time, opening the power sector to morecompetition. Historically, companies in competitive power markets have invested heavily in gas because gas plants are smaller and require less capital than coal or nuclear plants.

Gas suppliers who dream of extending their powers forget that it is harder to corner gas markets when users have a choice. Algeria learnt that lesson in 1981 when it left a key pipeline empty in a pricing dispute with Italy - extracting a better price at the time but losing billions of dollars for the future by destroying its reputation as a reliable supplier.

That lesson should be sobering for Russia today. In December, Gazprom, Russia's giant state gas company, cut deliveries to Ukraine, which then siphoned supplies that flow on to Europe. The company rattled its pipes again last month - threatening retaliation if Europe dared try to wean itself from Russia's gas. While Gazprom's management must pander to Russian nationalism (where pipe-rattling is welcome), the company's long-term viability rests on its reliability as a supplier to lucrative west European markets. Similarly, the recent decision by Evo Morales, Bolivia's president, to nationalise his country's gas fields will give him a boost domestically and might generate some instant extra revenue, but it will also encourage his customers in Brazil and Argentina to look elsewhere for energy.

"Resource nationalism" is back in vogue. But for gas suppliers in particular, it usually ends badly - not least because the infrastructure is costly to build and buyers can afford to be choosy. Gas users can further subdue Russia's rattling by multiplying sources of supply. A robust market for liquefied natural gas will help.

The tendency for gridlock in energy politics means that policymakers must focus where tough decisions matter most, such as efficiency in the use of oil and diversity in the application of gas. Yet, prospects for serious policy are poor - not least because the US, which should be a leader, is the most hamstrung. Luckily, the markets are responding on their own - albeit slowly and patchily. Costly oil is encouraging conservation and new supplies; LNG is accelerating, and gas buyers are more wary of Russian gas than they were a decade ago when Russia was seen as a reliable supplier. If the political structure remains dysfunctional on matters of energy, then the best second is perhaps no policy at all.