In 1936, he was appointed as the head of the Kriminalpolizei (Kripo), which would later become the Criminal Police of Department V in the Reichssicherheitshauptamt). He also commanded Einsatzgruppe B, which was one of four large murder squads sent out by SS-ObergruppenführerReinhard Heydrich (the head of the RSHA) to eliminate what the Nazis euphemistically termed 'undesirables' in the New Order.
He was believed to have been involved in various plots including the July 20, 1944 bomb plot against Adolf Hitler, and after the failure of the assassination attempt, went into hiding on an isle in the Wannsee but was later arrested after a rejected mistress informed on him. According to Nazi records, he was executed on March 21, 1945, but there have been several alleged eyewitness sightings of Nebe after the war in Ireland and Italy.