I would not worry at all. I've had my DSLR camera with me on all my longer bicycle tours (10,000+ kms) and never had any problems. I had a Nikon D70 for about 12 years. Now I have used a Nikon Df for the last 5 years. I would not expect other cameras to be generally easier to break.
For easy access, I usually have the camera in my handlebar hardcase bag and even though I usually have other stuff in the bag as well, I don't think I have ever used something to on purpose pad or polster the camera. I am not really doing any off-road cycling, but have cycled my fair share on gravel roads and cobblestone. I have no count of how often my bicycle has fallen over, loaded bicycles are often quite unstable when only using a kickstand, and even if the camera has experienced a bit of beating, it has never been a problem.
The only thing I usually try to avoid, not only when bicycling, but whenever I carry the camera around, is to have a long and/or heavy lens attached. Most commonly, I am only using Nikon's small and light 35mm f2 lens.
