Last November, a sharp-eyed Israeli woman named Niva Ben-Harush was alarmed to notice a young man attaching something that looked suspiciously like a bomb to the underside of a car in a quiet street near Tel Aviv port. When police arrested him, he claimed to be an agent of the Mossad secret service taking part in a training exercise: his story turned out to be true – though the bomb was a fake.

No comment was forthcoming from the Israeli prime minister's office, which formally speaks for – but invariably says nothing about – the country's world-famous espionage organisation. The bungling bomber was just a brief item on that evening's local TV news.

There was, however, a far bigger story – one that echoed across the globe – two years ago this week, when a bomb in a Pajero jeep in Damascus decapitated a man named Imad Mughniyeh. Mughniyeh was the military leader of Lebanon's Shia movement Hizbullah, an ally of Iran, and was wanted by the US, France and half a dozen other countries. Israel never went beyond cryptic nodding and winking about that killing in the heart of the Syrian capital, but it is widely believed to have been one of its most daring and sophisticated clandestine operations.

The Mossad, like other intelligence services, tends to attract attention only when something goes wrong, or when it boasts a spectacular success and wants to send a warning signal to its enemies. Last month's assassination of a senior Hamas official in Dubai, now at the centre of a white-hot diplomatic row between Israel and Britain, is a curious mixture of both.

With its cloned foreign passports, multiple disguises, state-of-the-art communications and the murder of alleged arms smuggler Mahmoud al-Mabhouh – one of the few elements of the plot that was not captured on the emirate's CCTV cameras – it is a riveting tale of professional chutzpah, violence and cold calculation. And with the Palestinian Islamist movement now vowing to take revenge, it seems grimly certain that it will bring more bloodshed in its wake.

The images from Dubai follow the biblical injunction (and the Mossad's old motto):"By way of deception thou shalt make war." The agency's job, its website explains more prosaically, is to "collect information, analyse intelligence and perform special covert operations beyond [Israel's] borders."

Founded in 1948 along with the new Jewish state, the Mossad largely stayed in the shadows in its early years. Yitzhak Shamir, a former Stern Gang terrorist and future prime minister, ran operations targeting German scientists who were helping Nasser's Egypt build rockets – foreshadowing later Israeli campaigns to disrupt Iraqi and (continuing) Iranian attempts to acquire nuclear and other weapons.

The Mossad's most celebrated exploits included the abduction of the fugitive Nazi war criminal Adolf Eichmann, who was later tried and hanged in Israel. Others were organising the defection of an Iraqi pilot who flew his MiG-21 to Israel, and support for Iraqi Kurdish rebels against Baghdad. Military secrets acquired by Elie Cohen, the infamous spy who penetrated the Syrian leadership, helped Israel conquer the Golan Heights in the 1967 Middle East war.

It was after that that the service's role expanded to fight the Palestinians, who had been galvanised under Yasser Arafat into resisting Israel in the newly occupied West Bank and Gaza Strip. The 1970s saw the so-called "war of the spooks" with Mossad officers, operating under diplomatic cover abroad, recruiting and running informants in Fatah and other Palestinian groups. Baruch Cohen, an Arabic speaker on loan to the Mossad from the Shin Bet internal security service, was shot in a Madrid cafe by his own agent. Bassam Abu Sharif, of the Marxist Popular Front for the Liberation of Palestine, was badly disfigured by a Mossad parcel bomb sent to him in Beirut.

Steven Spielberg's 2006 film Munich helped mythologise the Mossad's hunt for the Black September terrorists who massacred 11 Israeli athletes at the 1972 Olympics. Eleven of them were eliminated in killings across Europe, culminating in the small Norwegian town of Lillehammer, where a Moroccan waiter was mistaken for Ali Hassan Salameh, the Munich plot's mastermind. Salameh was eventually killed by a car bomb in Beirut in 1979 – the sort of incident that made Lebanese and Palestinians sit up and notice last year's botched training episode in Tel Aviv.

Some details of the assassination of Mabhouh last month echo elements of the campaign against Black September – which ended with the catastrophic arrest of five Mossad agents. Sylvia Raphael, a South African-born Christian with a Jewish father, spent five years in a Norwegian prison; she may have been among the young Europeans in Israel who were discreetly asked, in nondescript offices in Tel Aviv, if they wished to volunteer for sensitive work involving Israel's security. Other agents who had been exposed had to be recalled, safe houses abandoned, phone numbers changed and operational methods modified.

Over the years, the Mossad's image has been badly tarnished at home as well as abroad. It was blamed in part for failing to get wind of Egyptian-Syrian plans for the devastating attack that launched the 1973 Yom Kippur war. Critics wondered whether the spies had got their priorities right by focusing on hunting down Palestinian gunmen in the back alleys of European cities, when they should have been stealing secrets in Cairo and Damascus. The Mossad also played a significant, though still little-known, role in the covert supply of arms to Ayatollah Khomeini's Iran to help fight Saddam Hussein's Iraq, as part of the Iran-Contra scandal during Ronald Reagan's presidency.

It has, in addition, suffered occasional blows from its own disgruntled employees. In 1990, a Canadian-born former officer called Victor Ostrovsky blew the whistle on its internal organisation, training and methods, revealing codenames including "Kidon" (bayonet), the unit in charge of assassinations. An official smear campaign failed to stop Ostrovsky's book, so the agency kept quiet when another ostensibly inside account came out in 2007. It described the use of shortwave radios for sending encoded transmissions, operations in Iran for collecting soil samples, and joint operations with the CIA against Hezbollah.

But the worst own goal came in 1997, during Binyamin Netanyahu's first term as prime minister. Mossad agents tried but failed to assassinate Khaled Mash'al – the same Hamas leader who is now warning of retaliation for Mabhouh's murder – by injecting poison into his ear in Amman, Jordan. Using forged Canadian passports, they fled to the Israeli embassy, triggering outrage and a huge diplomatic crisis with Jordan. Danny Yatom, the then Mossad chief, was forced to quit. Ephraim Halevy, a quietly spoken former Londoner, was brought back from retirement to clear up the mess.

The Dubai assassination, however, may yet turn out to be far more damaging – not least because the political and diplomatic context has changed in the last decade. Israel's reputation has suffered an unprecedented battering, reaching a new low during last year's Operation Cast Lead in the Gaza Strip. "In the current climate, the traces left behind in Dubai are likely to lead to very serious harm to Israel's international standing," the former diplomat Alon Liel commented yesterday.

Even though Israel is maintaining its traditional policy of "ambiguity" about clandestine operations, refusing to confirm or deny any involvement in Dubai, nobody in the world seems to seriously question it. That includes almost all Israeli commentators, who are bound by the rules of military censorship in a small and talkative country where secrets are often quite widely known.

It would be surprising if a key part of this extraordinary story did not turn out to be the role played by Palestinians. It is still Mossad practice to recruit double agents, just as it was with the PLO back in the 1970s. News of the arrest in Damascus of another senior Hamas operative – though denied by Mash'al – seems to point in this direction. Two other Palestinians extradited from Jordan to Dubai are members of the Hamas armed wing, the Izzedine al-Qassam brigades, suggesting treachery may indeed have been involved. Previous assassinations have involved a Palestinian agent identifying the target.

Yossi Melman, the expert on intelligence for Israel's Haaretz newspaper, worries that, as before the 1973 war, the Israeli government may be getting it wrong by focusing on the wrong enemy – the Palestinians – instead of prioritising Iran and Hizbullah.

"The Mossad is not Murder Inc, like the Mafia; its goal is not to take vengeance on its enemies," he wrote this week. "'Special operations' like the assassination in Dubai – if this indeed was a Mossad operation – have always accounted for a relatively small proportion of its overall activity. Nevertheless, these are the operations that give the organisation its halo, its shining image. This is ultimately liable to blind its own ranks, cause them to become intoxicated by their own success, and thus divert their attention from their primary mission."

From an official Israeli point of view, the Mossad has an important job to do. Its reputation for ruthlessness and cunning remains a powerful asset, prompting what sometimes sounds like grudging admiration as well as loathing in the Arab world – where a predisposition for conspiracy theories boosts the effect of the disinformation and psychological warfare at which the Israelis are said to excel.

The government's official narrative, of course, is that Hamas is a terrorist organisation that pioneered horrific suicide bombings, fired thousands of rockets at Israeli civilian targets and – despite occasional signs of pragmatism or readiness for a temporary truce or prisoner swap – remains dedicated to the destruction of the Jewish state. It refuses to admit that its ever-expanding West Bank settlements remains a significant barrier to peace.

In western countries, including Britain, there was widespread anger at the 1,400 Palestinian casualties of the Gaza war. Barack Obama has declared the occupation "intolerable". Netanyahu heads the most rightwing coalition in Israel's history; his famous quip that the Middle East is a "tough neighbourhood" no longer seems to justify playing dirty.

Yet Israelis, and not just those on the right, worry that their very existence as an independent state is being de-legitimised. And, judging by the jobs section of the Mossad website, there are still plenty of opportunities for Israel's wannabe spies: challenging positions are available for researchers, analysts, security officers, codebreakers and other technical work. Speakers of Arabic and Persian are invited to apply to be intelligence officers.The work involves travel abroad and a "young and unconventional" environment.

It is a novelty of this episode that ordinary Israeli citizens are angry that their identities appear to have been stolen by their own government's secret servants – one reason why the Mossad chief Meir Dagan may find his days are numbered. But it is hard not to detect an undercurrent of popular admiration for the killers of Mabhouh. The day after the sensational CCTV images and passport photos were shown, the Israeli tennis champion Shahar Pe'er reached the quarter-finals of a major international competition in the emirate. "Another successful operation in Dubai," the Ynet website headlined its story.

Or Kashti, Haaretz's education correspondent, did not have his passport cloned, but he does bear a striking resemblance to the hit-squad member named as Kevin Daveron. "My mum rang and asked gently if I'd been abroad recently," he wrote. "Friends asked me why I hadn't brought back any cigarettes from the duty free shop in Dubai. I thought I sensed admiring glances in the street. 'Well done,' said an elderly woman who came up to me in the supermarket and clapped me the shoulder. 'You showed those Arabs.'"

• This article was amended on 24 February 2010. The original misspelled Haaretz's education correspondent as Ofer Kasti. This has been corrected.