I think it's a factor to consider but it depends on your threat model. A few people have linked an article about a Bitcoin wallet that was on one of these drives that was cracked. I imagine replicating the process would be difficult but with a big enough group going after you who knows?