Initial Thoughts:
During an AWS seminar, a person mentioned Wasabi and how they use it for their S3 storage. I took a look and they sound pretty good for a variety of reasons. They are one fifth the cost of Amazon and do not have any ingress or egress charges. Their pricing is $5.99 per TB per month ($0.0059 GB per month) for a hot tier of S3 storage. They tout eleven 9’s of reliability among their Datacenters in US-West, US-East, and EU.
Theory:
Is it possible to use cloud storage, such as S3, as local storage for user and departmental shares? This would remove the need for dedicated backups and locally tiered storage while increasing resiliency. Also, would it be possible with an S3 hot tier to rethink our storage strategy to work more like snapshots for user shares with shadow copy? What if we did a full backup once a month and then incrementals between those periods. Would it save us a lot of money and give us faster access to data?
Backup Costs:
So I did some thinking. My employer was performing full backups each week and incrementals each day for our block and file storage. For the month of February, 1.5 PB of data to S3 Glacier cost us $8,019.22.
The last full backup report showed our upload size was roughly 400 TB for a full backup. Given that number and a projected incremental growth of 5% per week and that amounts to 20 TB per week or under 500 TB of storage for the month. For 500 TB of hot storage that would cost $2,995 or about one third of what our current monthly Glacier backup costs were. Even if you round up all the way to 1 PB it would be $6,133.76 per month and still save 25% of the current monthly backup costs while providing access to faster storage.
As we use Commvault, I also took a look at the Commvault partners and saw that they were listed as partners with Wasabi. In fact, here is the step by step guide on how to setup the object, API key, and backup.
From a cost perspective that sounds great, but what about the performance? What would end users think?
Testing Configuration:
Using my own personal account I went ahead and built out storage for Amazon S3 and Wasabi S3. I set up a fresh VM with 4 cores and 8 GB of memory to simulate what typical desktop users might get. I installed using Windows 10 LTSC, joined the domain, provisioned my account to have access, added to VMware Horizon, and began my testing.
I started with a program called Raidrive. It will allow you to add multiple cloud storage providers as local drives. I signed up using my email and got the Education Tier. It is very similar to the Professional tier in that you can write to cloud drives from Google, Microsoft, etc except it does not allow you to write to S3 storage locally. That requires the Professional license which meant that I had to upload files to S3 manually. I am tempted to sign up to complete my testing with write speeds using File Explorer, but I wanted to start testing to get some data to analyze.
Here are some screenshots to show the options for the storage possibilities.
For my initial test I added Amazon S3, Wasabi S3, Google Workspace, and Box as local drive letters inside Windows 10 LTSC. I did some testing for functionality and speed. Here are some of my early results.
Testing:
Wasabi S3 – Fast. Many Datacenters (US-West, US-East, US-Central, EU-Central, AP-NorthEast aka Tokyo). The closest Datacenter is located in Plano, TX at 260 miles. Ping times were sub 10 ms with transfers hitting 1 GB per second on my test VM.
Amazon S3 – Decent. Many Datacenters (US-West, US-East, plus Edge locations and other continents). The closest Datacenter is located in North Virginia at 1,400 miles. Transfers hitting 66 MB per second on my test VM.
Google WorkSpace – Decent. Many Datacenters (US-West, US-East, US-Central, plus other continents). The closest Datacenter is located in Iowa at 1,000 miles. Transfers hitting 110 MB per second on my test VM.
Box – Decent. Many Datacenters. Transfers hitting 36 MB per second on my test VM.
Summary and Closing Thoughts:
Regarding backups, changing backup increments s is something to consider when looking at the totality of a storage solution. In our case, it makes sense given the increased speed that compensates for having more incremental backups.
It is possible to use software to mount object storage so that it can be used as file storage. One interesting benefit using Raidrive was that it was even possible to transfer files from one storage location to another. This means you can transfer data from one cloud provider to the next seamlessly. It is not the most efficient or speediest option available, but with more time and optimizations it is possible to tweak the performance metric further. In conjunction with backups, it is possible to look at a storage at a high level and adjust for better growth, resiliency, and cost.