I'm going to talk in terms of a murder mystery, but that's just a useful shorthand. Most types of mystery can be developed in much the same way.
Step One: Create the Characters First
More than most other types of adventures, mysteries are all about the characters, both living and dead. If there has been a murder, it is important to know who did it, how they did it, and why they did it. If the murderer discovered that his wife was having an affair and killed her lover in a rage, that's going to lead to a very different set of clues than if the wife was a long-suffering victim of domestic abuse and so quietly poisoned her husband over some months.
So, create the characters first.
Start with the murderer and the victim. Presumably, they knew one another, but what sequence of events brought them to a violent end? Was it the culmination of a long-standing hate, or was it a sudden reversal?
And then, create the characters around the central pairing. What friends and relatives did they have? What about co-workers? How did the local police interact with the two? (This last is particularly important, since if the PCs are investigating the murder, sooner or later they're going to have to interact with the local police. So, you'll want to know who they are.)
Step Two: Fill in the Details of the Mystery
Okay, you know who committed the crime, and who was the victim. Now, fill in the details of how the murder occurred, the set of motives that led to this most heinous of deeds, and the set of clues that immediately resulted.
So, where did the murder occur? And how did the murder occur? Was there anyone else there?
Trace the key events leading up to the murder, starting either a day before the murder took place or the point where the murderer decided to act (whichever came first), through the preparations for the murder, to the point where the two were together, the murder, and then the next day or so.
Who saw the victim last (other than the muderer)? Who knew where he was going? Who might have seen the murderer acting strangely, either before or after the deed?
Consider the nature of the murder itself. All murders generate evidence of some sort, even if that is only the dead body itself. What did the murderer do with the corpse? Did he have clothes to clean or destroy? A murder weapon?
At this point, you don't need to fix specifics. All you need to do is generate a whole bunch of potential clues for the PCs to find.
Step Three: Prepare the Locations
It may seem a bit odd, but it is not until this point that you want to lock down the locations in the adventure. You'll want some nice, mundane locations: the hostel where the PCs are staying, the police station, the local church. You'll want some 'neutral' locations keyed to the victim and the murderer: places of work, their homes, the homes of their ex-wives or mistresses. And you'll want the locations tied directly to the murder: the lock-up where the deed occurred, the pit where the body is buried...
Again, you don't need to tie these down with specifics too tightly at this place. In many cases, these will just be 'stock' locations: the church may well just be a church, same as any other. You'll need a handful of 'flavour' details to bring the place to life, but not much else.
Step Four: Build the Web
Now, we get to put things together.
Start with the conclusion that the PCs need to reach. "Col. Mustard, in the Study, with the crowbar."
Now, determine the clues that they need to find to come to each of these conclusions. Here, you want to establish the minimal set of investigations to get from the start of the adventure to the finish.
But at that point, you might be tempted to stop. Don't!
You need at least three clues for each conclusion the PCs need to reach, and these shouldn't all be in the same place. Furthermore, you need various 'entry points' to the network of characters you have - knowing the Bob has been killed may well point the PCs to speak to his wife, but probably won't direct them to his mistress automatically. But if Bob's wife comments bitterly on "that woman", that may direct them to the mistress, and the clues she has.
And so, you get an adventure that looks like a web. It starts at a single point (the PCs arrive), and ends at a single point ("Col. Mustard, in the Study, with the crowbar."), but in between there are lots of different points, joined by lots of little lines, all marked with clues.
(And when you run it, the PCs are likely to jump all across the web, probably skipping off the lines quite frequently. That's fine - you just need to make sure you have enough to ensure they don't get stuck.)
Step Five: Take a Break
You've forgotten something. No, really, you have. And if you just leave it at that, either the PCs will get stuck, or they'll immediately short-circuit the thing and ruin your adventure.
So, take a break from your planning, for at least an hour and preferably for a day or more. Then come back to it, check the web again, check the characters, and try to think "what if..." And then plug the holes you find.
Step Six: Play!
Some more thoughts:
- Remember, your players aren't Sherlock Holmes. They won't pick up on fine details unless you specifically point them out, and if you point them out then they're not really details so much as "big honking clues". So, make your clues fairly obvious and easy to find.
- Everyone lies. In a mystery adventure, it seems to be quite common to have a whole bunch of "nice guys", and then one "shifty guy". That is, the regular folk are nice and helpful, while the murderer glows brightly under detect evil. Gosh, that's a tough mystery... So remember, everyone lies, and everyone has something to hide. The grocer maybe didn't kill anyone, but if he was also having an affair with the murderer's wife, suddenly he might seem a tad evasive. The local police are being awkward - did they have something to do with it, or do they just not want amateurs getting in their way?
- Don't forget to include some "floating clues". Chances are that, despite your best efforts, the PCs will get stuck. If you have some clues that can be dropped in when they just "ask around", or do something you didn't expect, you can easily get them going again.
- No matter what you do, there's still a real good chance that the mystery will fail. Either the players simply won't be able to figure it out, or they'll solve it in an instant, quite possibly by accident, and you're stuck for the evening. Ultimately, though, that's not that big a deal. Have the bad guy make a run for it, and turn the adventure into a big chase instead. Or just accept that sometimes things fail.
And that's it.