Open source solutions sometimes have more known security holes, because hackers can review the code to look for vulnerabilities. Look for a package with a longtime reputation and lots of users, and if possible, something that is not open source.
No cart is free from allowing fraudulent transactions to go through. People steal credit card numbers all the time, and until the owner of the card finds out and reports it stolen, the card may be accepted. And if you don't use a realtime gateway, you'll have to find that out manually.
In general, check the AVS (address verification system) results of each transaction to make sure the billing address provided by the customer matches the address to which the credit card statement is sent. Watch out for overseas shipments, shipments with expedited shipping methods, and domestic shipments going to a freight forwarded. Although all of these can be used legitimately, you should review them more carefully before deciding to ship.