There are grand highs and depressing lows when it comes to religion.
Nobody could argue that the Arabic peoples at the time of Muhammad were a depraved, despotic peoples. Then look at the next few centuries after Muhammad. Their architecture became astounding, they constructed the first universities, produced the most glories poetry, literature and maths, established great reforms of justice, elevated the status of women from less than a dog to a semblance of legal recognition. They may not have risen to an ideal state but it was a staggering rise nonetheless. They became the envy of the world and greatly astounded the European crusaders who imagined them little better than a barbaric rabble. Today they are 4 steps back.
A similar story can be presented for Christianity.
If we remove religion completely there is not another single educating, coordinating body to lead humans into any kind of moral, cohesive state. We don't move forward that 1 step, from 5 forward and 4 back. Today most countries have a legal system, some kind of bill of rights etc. These came via religion not via political or social awareness.
Humanity is an unwieldy, partly savage animal with a big and dangerous brain and a gargantuan ego. It is an extraordinarily difficult journey to bring this beast to heel. If you remove religion then there is no other utility that will work away tirelessly to elevate humankind from self absorbed, ego driven hedonism to philanthropic, charitable humanity loving grace.
As for the Genesis story and other such ancient religious tales.
People who despise religion should despise real religion and not the idiocy of religious fools who read these works as literal. Why always attack the lowest common denominator and throw all the rest in with them? Why always pick on the lowest hanging fruit. There are so many religious people who have married science, logic and reason into their religious beliefs and placed humanity before any dogmatic religious orthodoxy.
I tell you, religion is our only hope. I say this when like you I'm not religious because I will never join these dead religions just to feel warm and fuzzy. Nothing else can or will lead humanity into this next era of planethood without us tearing ourselves apart and destroying the planet. What that religion will be I'm not sure but it will most assuredly not be any of the establish old world religions.
'Personally, I think we would be better off without it.' You are probably right, certainly at this moment in time. However, we are responsible for all our actions. Nobody can blame their religion for their actions. The is no way of comparing what the state of humanity would be like without religion. I believe that in spite of the present state of the world we are much better off because of the influence of religion.
