Isn't refusing to let the police search me an admission of guilt?
No. If a police officer asks your permission to search, you are under no obligation to consent. The main reason why officers ask is because they don't have enough evidence to search without your consent. If you consent to a search request you give up one of the most important constitutional rights you have -- your Fourth Amendment protection against unreasonable searches and seizures.
A majority of avoidable police searches occur because citizens naively waive their Fourth Amendment rights by consenting to warrantless searches. As a general rule, if a person consents to a warrantless search, the search automatically becomes legal. Consequently, whatever an officer finds during such a search can be used to convict the person.
Don't expect a police officer to tell you about your right not to consent. Police officers are not required by law to inform you of your rights before asking you to consent to a search. In addition, police are prepared to use their authority to get people to consent to searches, and most people are predisposed to comply with any request an officer makes. For example, the average motorist stopped by an officer who asks them, "Would you mind if I search your vehicle, please?" will probably consent to the officer's search without realizing that they have every right to deny the request.
If for any reason you don't want the officer digging through your belongings, you should refuse to consent by saying something like, "Yes, I do mind. I have private, personal items in my [car, backpack, etc.] and do not want you looking through them." If the officer still proceeds to search you and find illegal contraband, your attorney can argue that the contraband was discovered through an illegal search and should be thrown out of court.
You should never hesitate to assert your constitutional rights. Just say "NO!"