If you’re one of those people that look at typical storage offerings for backup and recovery and say, “I can’t afford this,” you’re not alone. A lot of ready-to-go storage solutions can get very expensive very quickly. Our guest this week (Erik Ableson from Infrageeks.com) ran into this a lot with his small-government customers and SMBs in France, and knew he had to get creative. He wanted to build a hardened Linux repository for Veeam backups, and he also wanted an S3 object storage system to serve as the second copy. His customers couldn’t easily buy cloud services, so he needed something they could own and manage themselves. (He explains the unique reason they can’t buy cloud services.) He built the Linux repository using a Synology box to run both the storage and a Linux VM, and he built the object storage system out of the free version of MinIO and what I will call a very unique build of hardware. Learn the details in this fun episode of a real practitioner’s solution to a unique and challenging problem.
Video
Transcript
[00:00:00] W. Curtis Preston: hi, and welcome to Backup Central’s Restore it All podcast. I’m your host W Curtis Preston, AKA Mr. Backup, and I have with me, my USB connection consultant, Prasanna Malaiyandi
[00:00:55] Prasanna Malaiyandi: I’m good Curtis. Um, it shouldn’t be that difficult with USBs. Come on.
[00:01:00] W. Curtis Preston: It shouldn’t be, but in yet, and yet you were able to like, cuz it was like, why don’t we connect the thing to the thing. But then if the things connected to the thing, you get the thing, you get, the problem, you get the,
[00:01:12] Prasanna Malaiyandi: Maybe you should tell the listeners kind of what you were struggling with.
[00:01:15] W. Curtis Preston: Yeah. Well, I, I, you know, it’s funny is I, I I’ve recently downgraded my monitor. I, I, I say that because I, I had a very nice apple display that those of you, the apple purists are like, oh no, he got rid of his apple display. It was a 10 year old apple display. And it was a really nice apple display, but I got tired of converting everything.
Right. I, I have a, a more modern laptop. And so I was having to convert the Thunderbolt, uh, display port thing as well as also converting the power. Uh, it was the mag, the old mag safe style, power connector. To, to a USB C style power connector, cuz I have the MacBook pro now and, and that was always just a source of, of
[00:02:02] Prasanna Malaiyandi: Dongles and dongles and more dongles
[00:02:05] W. Curtis Preston: way more dongles than I wanted to deal with.
And now I have, I’m very happy to say I have a single dongle. Um, Although, because of various peripherals that had to do with this podcast, I have more peripherals that can be handled by said dongle. So there is technically a dongle off of the dongle there is,
[00:02:25] Prasanna Malaiyandi: Oh,
[00:02:26] W. Curtis Preston: is a, a USB, uh, hub that, by the way, I, I did figure this out, uh, Prasanna.
I, I actually used some double sided tape to stick the, the USB hub to the back of my monitor.
[00:02:39] Prasanna Malaiyandi: Yeah. Then it’s out of the way. You don’t have to worry about it. You don’t see it.
[00:02:43] W. Curtis Preston: exactly, but you may, you may recall the various generations of cable mess that we went through trying to, trying to get my, my desktop to, and I think it’s my, the only one that’s not where I wanted is the keyboard that I don’t have a wireless keyboard.
I have a USB keyboard and the USB keyboard doesn’t seem to want to plug into the USB hub that is plugged into the USB. Dongle that is plugged into the USB port. It only wants to be plugged into the, to the dongle, but so, you know, just things
[00:03:17] Prasanna Malaiyandi: so, so, so here’s so. So here’s something right. Um, so now Apple, right. Or I should say the EU is trying to force everyone to use USBC as the standard. Right? So soon I think in 2024, your iPhones will probably more than likely have USBC.
[00:03:37] W. Curtis Preston: Right. Yeah. Which is good. I mean, but at this point, I mean, I, I’m almost always wirelessly charging those, so that’s not, that’s not as big of a deal, but, uh, and anybody, all of, and the other thing that happened is complete office reorg. For those of you that, for the, for the 13 of you that watched this via video, Um, you’ll notice that this wall is no longer peach.
This wall is, is, uh, agreeable gray is the, is the official color that we could all agree on. That’s actually the name of the Sherwin Williams color that we chose, uh, not a sponsor. Uh and, uh, uh, so I paid, uh, four painters, uh, for two and a half days, that that’s how long it took to paint, uh, my downstairs, uh, world.
But now
[00:04:25] Prasanna Malaiyandi: I was gonna say if they spent two and a half days, just painting your office. Good Lord.
[00:04:30] W. Curtis Preston: Good Lord 19 coats. I don’t know. Uh, they said, they told me, they told me that’s what I needed. I needed 19 coats and I believed them. We should probably bring on our guest at this point and not bore people with my painting, uh, woes. Uh, he has been in it for over 35 years currently focusing on infrastructure and data protection.
Backup wise, he goes all the way back to AIT the Sony drive. Uh, for those of you that might remember that I’d love to hear about that a little bit. He is a Canadian living in France. Welcome to the podcast, Eric Ableson.
[00:05:03] Erik Ableson: Hello there. Good to be here. And, uh, nice to meet both of you for the first time. After many, many hours of podcast listening.
[00:05:09] W. Curtis Preston: Long long longtime listener. First time
[00:05:11] Erik Ableson: First time
[00:05:13] W. Curtis Preston: Yeah.
[00:05:13] Erik Ableson: guest, uh, caller
[00:05:15] W. Curtis Preston: It’s it’s, it’s always nice to hear from a fan. I, I think we, we reached out to you because we saw, and I don’t remember where or how I saw it, but, um, I see way too many things. You wrote a, a blog post, which now I’m trying to pull it up in my
[00:05:32] Prasanna Malaiyandi: It’s on the using NAS as a Veeam hardened repository
[00:05:37] W. Curtis Preston: Yeah, I, I knew that was the, the title, but I was trying to pull up the the page so that I could look at it. And I’ve only got 37 tabs open on my browser. So, um,
[00:05:48] Prasanna Malaiyandi: actually hosted on your blog, right? Eric?
[00:05:50] Erik Ableson: That’s correct? Yes. Switches over at infrageeks.com. Um, and yeah, it was, uh, a cry of despair and, uh, needing something to help people that are trying to do backups on a shoestring. Because one of the problems I run into very frequently is a lot of clients. I mean, I’ve got big clients too, that will happily throw lots of money at the problem.
And I’ve got lots of other clients that are like, please, oh please, oh, please. I have no money. What can I do?
[00:06:17] W. Curtis Preston: Right. Right. And so they, and so this idea of using the Synology box, that’s definitely a more budget friendly device. Would you.
[00:06:28] Erik Ableson: Oh completely. That’s been, my big problem is a lot of these companies. And like, I deal with like small city halls in a lot of cases and they just don’t have the budget. And for some strange reason, Synology manages to sell a decent CPU. With a reasonable amount of memory and a bunch of disks for considerably less than you would find the, an equivalent server, even from the better players, like super micro that are not as offensive about charging you three times the cost of the same commodity disc as everyone else, because it’s got their special firmware
[00:06:55] W. Curtis Preston: Right. I, I I’ve had experience with the Synology box as well here, actually, there’s, there’s one sitting behind me down on the floor. It’s it’s now a, it’s now a, it’s a dead ology. Uh, but
[00:07:06] Erik Ableson: that.
[00:07:06] W. Curtis Preston: I, I, I have for the record, it didn’t die. I just, uh, retired it. Uh, but yeah, I’m a fan of the Synology and what, so you, it, it can do both SMB and NFS as well as, uh, iSCSI apparently.
[00:07:23] Erik Ableson: That’s correct. Well, the trick is that none of those are great protections if you want to use something with, uh, Veeam’s immutable repository, which is becoming a priority for all of my clients and anyone in the backup business,
[00:07:35] W. Curtis Preston: Uh, yeah. And, and, and you know what, before we jump in and by the way, I’m glad to hear that, right. Because you know, it’s something, as you know, that we’ve been preaching from, from this particular, uh, pulpit, if you will, let’s go back just a little bit. Uh, so you, you just to talk about man, so many AIT, I haven’t heard that phrase in a long time. Uh, this is, this is. Like mid nineties, late nineties.
[00:08:03] Erik Ableson: Yeah, well, late nineties because the AIT-1 sucked
the AIT-2 was actually pretty nice, but the golden solution was the AI two cat four tape auto loader. Sorry, I’m gonna switch into French there from time to time. And I say cat, BDA, maybe. So it’s, uh,
it
[00:08:23] W. Curtis Preston: your word for.
[00:08:24] Erik Ableson: That’s right. It was it’s. It was a four tape auto loader and it changed the equation for me because all of a sudden we could get something like, I think half a terabyte of compressed data, little over that into a tiny little box
[00:08:35] W. Curtis Preston: Right, right.
[00:08:36] Erik Ableson: and.
It worked with Dantz retrospect, which I was playing with at the time, because I built a backup solution for my, my division working at Nortel. And we were looking at the LTOs and it was like, yeah. Okay. And then we looked at the AITs and it was like, Ooh, I can build a business out of this. Cuz if I lease these, I can probably work out the economy.
The economies to go. Nortel will pay me X per machine backed up and we can lease these and the two CX, uh, computers to drive it.
[00:09:03] W. Curtis Preston: Yeah. For, for those of you that were not aware of the AITs they were, I’m pretty sure they’re the last. They’re the last helical scan drive that was manufactured for backup. Right? So
[00:09:18] Erik Ableson: Pretty much.
[00:09:19] W. Curtis Preston: LTO and, um, you know, the, the, the IBM drives and the, the, the StorageTek drives. Those are all, uh, linear drives, we had, um, who was. Prasanna that we had, that
[00:09:34] Prasanna Malaiyandi: Yeah, we had Mark lance.
[00:09:37] W. Curtis Preston: And he actually gave us a technical reason why helical scan essentially went by the by, but AIT was the last of those.
[00:09:47] Prasanna Malaiyandi: the way
[00:09:48] W. Curtis Preston: go.
[00:09:49] Prasanna Malaiyandi: should throw out our disclaimer.
[00:09:50] W. Curtis Preston: Yeah, I, I, I have forgotten to throw out our disclaimer, Prasanna and I, uh, work for different companies. He works for Zoom. I work for Druva. This is not a podcast of either company and the opinions that you hear are ours. And also be sure to rate us at ratethispodcast.com/restore.
And if you, like Eric, are a long time listener and you know, or if you just joined us and you’re like, I know about this stuff and I’m interested in backup and archive and security and all of those things. And just reach out to me @wcpreston on Twitter, I take all DMs and, or you can send, uh, an email to wcurtispreston@gmail.com.
So, so let let’s move forward to the, why has it become such a demand from your customers to, to, to secure their, their backup server?
[00:10:41] Erik Ableson: well, the biggest one obviously is you guys have been going on for a while is you have to find the problem that people and specifically we’re not dealing with the same kind of threats anymore. In the old days, your disaster recovery plan was all about the building blew up. Functionally in one way or another, that was the big disaster recovery plan, but it never included malicious intent where someone would go, ha ha not only am I gonna take out your production system because I want to make your life miserable.
I’m also gonna go looking for your backup system. And that has changed the entire equation around how we have to protect our data these days. And just as an anecdote that didn’t, don’t think I included that in my little note to you guys, but I spent about a year working with a company I got called in after a massive ransomware attack to rebuild their backup infrastructure, because it was a backup account that had admin rights everywhere that had been used to take everything out.
And so of course, since they started with the backups and then they ran somewhere to everything else. Uh, it took a year for several hundred sites for me to rebuild all of these branch offices all over the place. Some of them that were still using believe it or not the old tape rotation routine. And it was the secretary’s responsibility to swap the tapes out on Friday.
And guess how often that happened? Yeah. Well so nowadays the key thing is we need to find ways to, well, functionally air gap. I will not go into the technical nitty gritty of it, but we need some way to say the data I put here can’t be modified. Now one obvious way is we’ve seen the cloud is a great solution for that.
Uh, you can use just about every backup tool out. There has some means of sending off to an S3 with, uh, the option to lock your objects so that they can’t be modified, which is great. But there’s a lot of companies that despite everything that gets expensive over time, the more data you keep, if you have long term data retention plans or policies that gets expensive.
And so they’re looking for, how can I do this onsite. Uh, on a shoestring.
[00:12:38] Prasanna Malaiyandi: Yeah.
[00:12:39] Erik Ableson: So the trick was coming up with, well, a lot of my customers are very I’m right into the VMware world with Veeam as the, uh, the most predominant backup tool there. And they built this really nice immutable backup model, as long as you do it right.
[00:12:53] Prasanna Malaiyandi: We actually had Dave on back up Dave on the podcast a while ago, talking about immutable repositories and things like that.
[00:13:01] W. Curtis Preston: It does really stink that, you know, the, the, the thing that you said that is absolutely the case that they’re, you know, that they’re coming for your backups. And in many cases, you know, I was just looking at, there’s actually another tool, uh, that we should probably talk about, uh, which I don’t, I don’t know if you’re aware.
Have you heard of, uh, blocky.
[00:13:21] Erik Ableson: Blocky. That’s new to me.
[00:13:23] W. Curtis Preston: So, um, so this, you can, you can add this to your list. So it’s, it’s literally called blocky for Veeam. So it, it seems like it’s a separate company, but it’s, it looks like it’s and, and I know. This, I’m gonna literally tell you the entire thing that I know. Right. So I haven’t, I haven’t researched it that much and we should probably have them on, but what it appears to be is a file system driver for windows that locks the, uh, either, uh, an NTFS or a, uh, refs file system from any rights, except for Veeam.
[00:13:59] Erik Ableson: Ah,
[00:14:00] W. Curtis Preston: Um, and theoretically it could be used for other applications and other backup applications, and I’m guessing that they will do that, but they saw Veeam as the, you know, the big one there. So they went after that. And so, so essentially it’s a way to do what you are talking about doing on the Linux hardened repository, but doing it in a Windows.
So that’s, um, you know,
[00:14:22] Erik Ableson: That’s actually
[00:14:22] W. Curtis Preston: not a sponsor , but.
[00:14:24] Erik Ableson: No, no, that’s actually interesting take, because it’s true that there’s a lot of places where. They’re Windows shops and bringing Linux is like one of these. Oh my God, this is, this is one of those strange incantation things from, uh, the, uh, the, it gurus are doing things and I don’t understand it.
And I’m very, very scared.
[00:14:39] W. Curtis Preston: Yeah, what is a command line? Right?
[00:14:42] Prasanna Malaiyandi: Yeah. And how do I secure it in all the rest? Right. It’s like even managing it.
[00:14:47] W. Curtis Preston: how does that go over? Like Eric, cuz you’ve been dealing directly with these folks. How does that go? Or there, if an all they’re an all Windows shop and you’re bringing in a Linux box just for this.
[00:15:01] Erik Ableson: Surprisingly low friction lately. Uh, the two, the two pressures are number one. Oh, my God, we need a solution.
[00:15:09] W. Curtis Preston: Right.
[00:15:11] Erik Ableson: there. There’s nothing like the fear of losing everything to, uh, put, uh, people into a more open mindset for that kind of stuff. And the second one, and this is one of the nice things I like about this Synology approach is Synology is an appliance.
It’s a box. You plug it in the network. You do whatever configuration need needs doing. And then you basically forget about it. And that is actually what we’re able to do with these companies that have limited, but just say, look, it’s just a sinology I’m gonna take advantage of some of the native features to use it, to install a virtual machine.
On the synology. And that will be the only point of access to the storage on the system. So we actually cut off all access to the DSM and the administration stuff. So the only part that’s listening on this box is the one from the virtual machine listening for Veeam. So when you get it there and you just go, it’s a box, it’s an appliance.
You go in your Veeam console and you say, what’s the state of my backups and says, here’s the state of your backups? And they’re like, great. Do I need to restore? Okay, you do a restore and everything works fine.
[00:16:09] Prasanna Malaiyandi: treat it like a black box, right. They shouldn’t care. What’s running in it, how it works, any of that.
[00:16:14] Erik Ableson: Exactly.
[00:16:17] W. Curtis Preston: well, let me, let me ask you about that. You know, uh, push back just a little bit because it, it’s not really a black box, right? it is, it is a Linux kernel that’s running that needs to be administered and needs to be like, there needs to be security, patches and stuff put on it. How do you handle that? If you’re at, if you’re an all Windows shop?
[00:16:40] Erik Ableson: Well, again, one of the nice parts of Synology is that they have actually a really decent user interface that even a Windows person will go, oh, it’s just a web interface for my appliance. And I go in, and there’s a little thing that says there’s an upgrade to be installed. So I click on the upgrade button and it does its thing and I go for coffee and I come back and it’s done.
Um, So that aspect of it is actually fairly easy for most people, but it has to be, like I said, cuz what we do is in this design because the only thing I want actually on the network on a day to day basis is that Veeam repository. We actually disconnect. And I tell ’em this. No, no physically seriously unplug the management port of the server.
[00:17:20] W. Curtis Preston: cause I remember, I remember you saying that, uh, somewhere in the blog that you said, the one thing that you didn’t like was you couldn’t turn that port off.
[00:17:28] Erik Ableson: There’s gonna be other network interfaces. Cause I separate the backup, uh, flow on hopefully a 10 gig network and the admin flow just on a regular administrative VLAN somewhere. But I, I, since that time and I have to update the wall critical I’ve discovered there are firewall rules that I could do to say, even if you come in through this interface, And you use the D H C P MDNS 1 69, 2 50, for whatever type address we can actually block access to that management
[00:17:54] W. Curtis Preston: so you can, okay.
[00:17:55] Erik Ableson: you can do it. Yep. That’s uh, I have to go back and update.
[00:18:00] W. Curtis Preston: So why don’t you walk us through be because there, there were, so you, you talked about a couple of different things that you didn’t do, right? You didn’t do just an SMB and by the way, thank you for saying SMB and not CIFS.
[00:18:15] Erik Ableson: I’m I’m preaching to the choir here,
[00:18:18] Prasanna Malaiyandi: Yeah,
[00:18:19] Erik Ableson: to all my NetApp brethren. No, no. CIFS is in the documentation. It’s in the screenshots, but it is SMB. The protocol you’re actually using,
[00:18:27] W. Curtis Preston: Yeah. SMB, not CIFS. Um,
[00:18:30] Prasanna Malaiyandi: pet peeves.
[00:18:31] Erik Ableson: uh, mine as well.
[00:18:32] W. Curtis Preston: thank you very much for Foskett for, for hammering that into me. The, um, and, and, and, and why not use just SMB or
[00:18:42] Erik Ableson: Well, the first problem with SMB is just especially with Veeam, it’s horribly inefficient. Uh, it’s just a catastrophe because when you’re doing the way, the incrementals work there’s commit stages where it’s gonna have to read a whole bunch of data and rewrite it back into big files. And you’re just chewing up network bandwidth in dis IO left.
And right. It’s just really inefficient aside from the fact that SMB is, I would not say horribly insecure. But probably not the most secure protocol you have out there. And I don’t recall that NFS is actually any better with Veeam from a practical standpoint, except that it’s the only difference you have to front it with a Linux machine anyway.
So you’re not gaining anything by using NFS in that context.
[00:19:26] W. Curtis Preston: Okay.
[00:19:27] Erik Ableson: So the key thing there was just say, get rid of all your NAS protocols. Then you have the option, like you pointed out there is iSCSI available on this. And what you, I used to see people doing with this was they’d take their Synology box or their NAS box box, whatever they had.
And they’d say present an iSCSI volume as a raw device map to a Linux virtual machine. And that will be my hardened repository. I will configure it properly and do all that stuff. And we’re good to go. The hiccup as I’ve run into is I had the pleasure of helping a client out of a ransomware attack over the holidays this year and their attack vector.
Was not just the system layer. They actually attacked the ESX servers directly.
[00:20:08] Prasanna Malaiyandi: And there’s a lot of zero days out there now coming out right around against VMware. So.
[00:20:13] Erik Ableson: most of it’s against the desktop stuff. There’s less on the, uh, the actual ESX level, but the problem is, all it takes is someone that gets an admin account off one of those other systems and that admin account is admin on your vCenter. And then from there, you just turn on ssh on all the boxes you go in and you run those little ransom or, uh, well not ransomware the encryption tool and what do they do?
Well, they just look at all the available discs. where all VMs live.
[00:20:38] W. Curtis Preston: So I, I don’t think you mentioned when you said, when you said they build a, a V a Linux. I don’t think you mentioned that it was a VM, but
maybe you did.
[00:20:47] Erik Ableson: no, I, I
[00:20:48] W. Curtis Preston: so, so that’s what you meant. You meant a Linux VM. And so then it, it got attacked via ESX. So that’s a bad idea.
[00:20:54] Erik Ableson: that’s right.
[00:20:55] W. Curtis Preston: trying to eliminate catch 22s,
[00:20:57] Erik Ableson: Exactly. So the other option is if you’ve got the money, you could just throw a physical box in front of your Synology and treat it as cheap storage that works well.
[00:21:06] Prasanna Malaiyandi: but then these are probably shops, which may not want to be managing a Linux box. And now you have a physical system you gotta procure during supply chain issues. What’s the right equipment, all the rest of that, right.
[00:21:17] Erik Ableson: are exactly the points to keep coming up. It’s like, well, why am I throwing extra money to buy a network card and CPU and memory? When I just bought a box that has a network card and CPU and memory in it, that’s kind of silly.
[00:21:28] W. Curtis Preston: So I think the surprise and, uh, the surprise for me was this idea that you could run the Linux on the Synology box itself.
[00:21:38] Erik Ableson: Yeah, that. That was the one thing I discovered a few, I guess about a year ago was that the, the machines are getting more and more capable and they all, basically everybody, everybody’s a hypervisor now. I mean, you’re gonna
[00:21:51] W. Curtis Preston: I’m a
hypervisor. I mean,
[00:21:52] Erik Ableson: That’s right.
[00:21:53] W. Curtis Preston: look at me.
[00:21:57] Erik Ableson: You’ve got KVM. QMU you’ve got everything you could possibly want on just about any OS. So the synology, well, we’ll just put a nice, pretty wrapper on it so that it’s easy to approach and you can just deploy a VM inside. And then it’s like, oh, and it’s inside the machine and I can give it access to all the storage.
And then I don’t need the DSM other than just to do the basic configuration to sort of bootstrap myself to a point where I’ve got a Linux box talking to all my storage.
[00:22:26] W. Curtis Preston: By the way, usually, by the way, Prasanna, you, you didn’t ask him what a DSM is. Normally that’s your job.
[00:22:32] Prasanna Malaiyandi: oh yeah. So what’s a DSM Eric
[00:22:36] Erik Ableson: Oh, whoops. Yes. Acronym city. We are always living in it is the disc station manager, I think is the Synology term for it, which is basically their proprietary layer or interface layer on top of their Linux, uh, kernel. So it’s the web UI that drives all the stuff. Uh, when you’re playing with Synology boxes.
G boxes.
[00:22:56] Prasanna Malaiyandi: Gotcha. And it’s kind of cool that they’re giving you this capability. It’s if I think about it, it’s kind of like how for vCenter, right? You have your ESXi host, right. And then you’re doing whatever you want. Now. It’s like, oh, here’s storage. And now you can run some additional compute against it. Right.
And take advantage of it. The. One question I would have though is in these environments where you’re deploying Synology plus a Veeam hardened repository together. I’m assuming you’re only dedicating that for backup storage. Correct. And you don’t necessarily want it for being used for primary NAS storage as well.
Right?
[00:23:30] Erik Ableson: A exactly. No, that’s precisely the objective here is like, see people are coming and going. I’m either. I, I need to revamp my backup storage. I need to do something. And the, my objective in these cases is we’ll buy you a box. It will do this. It will do nothing else, but this, you can buy another Synology if you wanna do file shares and all that other fun stuff, uh, which is.
But this box should live and breathe only your backup storage. So the idea is it just cut off on the rest of the world. It does one thing. And one thing only it’s the strange part is just the economics of it. In that you can buy a Synology with 12 decent sized hard drives in or 16 decent sized hydrogen for considerably less than you can buy just about any off the shelf brand name server right now, that’s, that’s where the kicker comes in, which is why.
[00:24:15] W. Curtis Preston: a server with a bunch of storage.
[00:24:18] Erik Ableson: Yes. Yes. Oh no, it’s still cheaper than a storage bay. Uh, I just did a couple quotes the other day and I was like, I want a storage bay with this kind of capacity. And they’re like, oh, that’ll be 40,000 euros.
And I said, okay, well I want a server with the same amount of discs in it, and I’m gonna throw true NAS or something on there just for something else. And they’re like, oh, well that’ll be $14,000, 14,000 euros. And I’m like, um, Okay. I know you’ve got dual controllers, but I could go to IX systems and probably get myself the same thing for the same price.
So yeah, it’s, there’s some really strange economics that are in play. As soon as you start talking to brand name, uh, enterprise providers in the storage front.
[00:24:57] W. Curtis Preston: Interesting. And, and so I, I guess the, the part, I’m the part I’m not fully grasping and, and mainly, mainly it’s because I didn’t even know that Synology could do this, this VM. And again, I’m, I’m, I’m gonna poke you again, just because I wanna make sure I understand what we’re talking about here. You talked about Synology ability to update. , the kernel or, or at least that’s what I, what I was understanding you. So, so help me understand that there is, is this Linux box not, is the VM, the kernel inside that VM, not something that will, that will get outta date, or is that something that the customer can manage without having to know too much about Linux.
[00:25:43] Erik Ableson: that one actually does require a little bit more handholding and some documentation, cuz there’s two steps. Cuz if I do this properly, this machine is cut off from the network. and I have two things that I gonna potentially need to upgrade. You’re absolutely correct. I have to potentially upgrade the DSM.
Although on the practical front, I’m not so much worried about the security problems on the Synology interface, because most of the time it’s disconnected. So it should be one of those things that they sort of, they get the mail from, Synology going Woohoo, new update here. And it’s like, oh, okay. I better go connect my machine and do that update thing on the Linux side.
Again. You’re doing off the shelf, the most standard vanilla clean install you can possibly do. So the only thing we need to get the, in the documentation to say is if you’re using an Ubuntu system, it’s just gonna be going app, get update app, get upgrade, reboot, and you’re done. So that should be the sum total of the user’s interaction with that side of things.
[00:26:41] Prasanna Malaiyandi: What about the Veeam specific pieces?
[00:26:45] Erik Ableson: That is a little bit more complicated. Good call, Prasanna. Um,
[00:26:48] W. Curtis Preston: But that’s, that’s something you’re only gonna have to do once though, right?
[00:26:52] Erik Ableson: Until there’s a Veeam update.
[00:26:54] Prasanna Malaiyandi: Yeah. yeah.
[00:26:55] W. Curtis Preston: okay. All right.
[00:26:57] Erik Ableson: Because Veeam is actually fairly finicky about the notion of different components in its system, not running exactly the same version. So what it’s got is I think what’s called the data mover is what they install on the repository component. You’re talking to it over the backup network. So it’s already on the network. You don’t need to touch the Synology layer at this point, but what you do need, because it’s an immutable repository and this applies to all the Linux hardened repository configurations is you need to go to the console.
You need to enable SSH. Then you go to the Veeam console and you say, do my update thing, use this account and password, but don’t remember it, whatever you do. And that’s one of the things that makes it a hardened repository is that Veeam has no trace and no memory of the accounts and the passwords are used for connecting it.
Cause once they’ve done the first stage, basically what they do is a key exchange. So any other further communication is done over that certificate exchange process to go. Hi, I’m your Veeam server and hi, I’m I’m a repository. Are you really the Veeam server? Okay. You’ve got the right key. Are you really the right repository?
Yes, you’ve got the right key. We’re good. We’re golden. No passwords were exchanged.
[00:28:03] Prasanna Malaiyandi: Gotcha. And so the only time you need it is for the upgrade part where you need enable SSH. You do. Upgrade, which looks like it’s automated from the Veeam side, as long as you can connect. And then once that update is done, then I’m guessing you go back and you disable SSH, and you’re all good to go.
[00:28:20] Erik Ableson: That’s right. So it’s basically, uh, there’s a again, so I you’re absolutely right. There’s a few more commands. There’s the app get update, upgrade, reboot. And then there’s SSH, no system control server, no system control start S I can’t remember which one of the, the order, those on system control. SSH start. Do your upgrade, system control. SSH stop. Should do the
[00:28:41] W. Curtis Preston: job
And you’re doing this from the console that you’re accessing via the, the DSM.
[00:28:47] Erik Ableson: That’s correct. So you will have to plug it back into the network for those
[00:28:50] W. Curtis Preston: So you plug that big. So you then load the DSM, which is over a, a browser, right? I’m gonna access this via browser. So I gotta plug in the management port, access DSM, look at the terminal, the console via that interface, turn on SSH, do my update and then turn it off. So it it’s, it’s not it’s. Nothing, but it’s not, it’s not crazy either.
[00:29:17] Erik Ableson: no. And it’s the sort of thing that actually companies that like procedures, they like it because it’s really straightforward. It’s like, oh, You tell me I do this, this, this, and I get the results I want. And they’re like, oh, I can follow procedures and I can give this to, you know, any of my tech sys admins.
And they’ll go, oh yeah, I get it. No problem. It’s like I said, it’s not rocket science, but it
[00:29:38] W. Curtis Preston: then once, once you’ve done that, the protocol that is used to transfer backups to the Veeam hardened Linux repository is the Veeam protocols. It’s not using NFS or SMB or anything like that.
[00:29:51] Erik Ableson: Exactly. It’s the Veeam data mover protocol, which is up until now has not been subject to any CVEs. So I’m just keeping my fingers crossed that nobody starts attacking that one seriously.
[00:30:03] W. Curtis Preston: On July 6th, 2022 As,
As of this recording
[00:30:10] Erik Ableson: so what is, hold on persona now. It’s my turn. CVE is common vulnerability, something. I don’t remember what the last one is.
[00:30:17] Prasanna Malaiyandi: Vulnerabilities and exposures
[00:30:19] Erik Ableson: and exposures. Okay. So yes, that’s the nomenclature for security flaws that we have found and discovered and decided how bad they are.
[00:30:31] Prasanna Malaiyandi: You have this one Synology box, right? With your Veeam hardened backup repository for these customers on a shoestring budget. I know Curtis loves, I’m gonna steal your line, Curtis. He loves a 3, 2, 1 rule. So what do you do with that backup? Once it lands on that immutable repository?
[00:30:49] Erik Ableson: Well, there’s basically the logical next step is to find some kind of S3 provider that will do immutable backups. Now I’ve got a number of companies that for whatever reason, the cloud stuff is just not an option. Uh, specifically government agencies and here in France, it’s actually very specific. It’s not always to do with the security.
It has a lot more to do with procurement processes.
[00:31:15] Prasanna Malaiyandi: Interesting.
[00:31:16] Erik Ableson: Yeah, I’ll give you the ten cent tour is that city halls, their taxes are split into two types, one which comes from property taxes. And I forget which the other one comes from, but these two tax streams are split into what they can spend on CapEx and OPEX. And the problem is the one that is for OPEX. Is just peanuts.
So the problem is they can’t actually get, even if they find a right way to buy it, they can’t buy a lot of cloud services because they’re considered, sorry. They’re considered operation expenses. So that they can buy hardware. That’s not a problem, but they can’t buy an ongoing service.
[00:31:54] Prasanna Malaiyandi: that’s hilarious, actually. Yeah. It’s like the exact opposite of what you see in the US.
[00:31:59] Erik Ableson: Exactly everyone says, oh, I just wanna, I wanna level my budget. So it’s always a continuous operating thing, but there’s this strange tax law tax laws here that make it, that city hall, uh, has lots of money to spend on hardware, but they’re not allowed to spend well, traditionally it was to stop abuse of things like having, uh, people on salary that really weren’t doing any work.
[00:32:21] Prasanna Malaiyandi: mm-hmm
[00:32:21] Erik Ableson: So I think it was really it’s really to keep a limit on the headcount and limit on those, anything that goes in that operations expense side of the house. But if you need to throw money at a solution, it has to be physical hardware. So what I’ve been doing, and I don’t know if you’ve gone through the next step of my blog, which is, well, can I build myself my own S3 immutable, uh, object storage system?
And it turns out there’s lots of good solutions that do that.
[00:32:48] W. Curtis Preston: Are there, are there affordable ones to do that?
[00:32:52] Erik Ableson: Oh completely. The most affordable is the fact that, uh, MinIO, uh, is available for free. Uh, they have an open source version, which does everything the main one does, but obviously no support. You can pay for support from them if you want that. But all it takes is, uh, and here we go back to a Linux box.
[00:33:12] Prasanna Malaiyandi: Uh,
[00:33:13] Erik Ableson: Or several Linux boxes, depending on how you want to roll the thing.
Uh, but yeah, you can use the MinIO solution and I’ve got actually a, well, I like to, I like to tinker with hardware. This is the thing, everyone, cuz it’s a software is eating the world, but I like to play with hardware. So I do stupid things with hardware. So I have a, a mobile NAS, um, which is basically a ninth generation Intel NUC that comes with Thunderbolt.
And so with Thunderbolt, I can connect all sorts of Daisy chained, strange disc systems on this. OWC makes some wonderful ones where I can get these little four, uh, four bay, two and a half inch drive systems for my high performance SSDs and great big eight bay, uh, three and a half inch drives.
And so. I use this for a lot of clients that are looking for swing space. And so I said, oh, let’s see how far I can push this with MinIO. So currently, uh, I took that nook, threw a MinIO instance on it and eight, uh, 12 terabyte, hard drives. And so I’ve got this, uh, 96 terabyte, uh, MinIO instance hanging out for a client right now.
That’s migrating a whole bunch of data.
[00:34:19] Prasanna Malaiyandi: Nice.
[00:34:21] W. Curtis Preston: And these are the kind of things you gotta do when you can’t get the cloud.
[00:34:25] Erik Ableson: Uh, yep. Yep. But it’s fun. And I had to 3d print a whole container to hold it all in and put a handle on it.
[00:34:31] W. Curtis Preston: 3d printed the container.
[00:34:34] Erik Ableson: I will send you photos
[00:34:35] W. Curtis Preston: You weren’t kidding. When you, when you said you liked the tinker with the, uh, tinker with the hardware,
[00:34:40] Erik Ableson: oh, no, no. That’s, uh, it’s lots of fun.
[00:34:42] Prasanna Malaiyandi: Yeah. So actually,
[00:34:44] Erik Ableson: comes in handy.
[00:34:45] Prasanna Malaiyandi: yeah. So speaking of Veeam and object store, I actually don’t know if you heard, but the former Veeam founders they’ve started their own company where they’re building a S3 repository or an object store repository just for Veeam.
[00:34:59] Erik Ableson: Oh, cool. Well, there’s a good business to be had there.
[00:35:01] Prasanna Malaiyandi: Yeah,
[00:35:02] W. Curtis Preston: Yeah.
[00:35:02] Erik Ableson: long as it’s cheaper than Amazon
[00:35:04] W. Curtis Preston: whether or not it’ll be, you know, in that affordable category that you’re,
that you’re looking for.
[00:35:09] Prasanna Malaiyandi: yeah.
[00:35:09] Erik Ableson: yeah, yeah, no, the, the big trick there is that let’s face it. Storage itself is cheap. And if you’re not looking at high performance stuff, throwing. 8 12, 14, 16 terabyte drives into a system that gets you a long way. So it’s, uh, being able to throw a, you know, a reasonable amount of CapEx at it, and the problem goes away for another five years.
And the way MinIO works, you can swap those drives out and it’ll lot of rebuild and do all its fancy stuff. So assuming CMR discs with larger capacities still exist because obviously throwing the SMR drives in these things is just catastrophe.
[00:35:48] Prasanna Malaiyandi: What are SMR and CMR
[00:35:50] W. Curtis Preston: Yeah.
[00:35:51] Erik Ableson: Oh,
[00:35:52] Prasanna Malaiyandi: and why or why?
[00:35:53] W. Curtis Preston: Explain you just said.
[00:35:54] Erik Ableson: I’ll start with the first part. Cause that’s the easy one. The MinIO backend uses erasure coating. To spread, uh, parity across the discs. So it is a reliable system and it goes fairly far in that front, similar to ZFS that it will actually go run down all the data that’s stored and make sure that everybody is agree in agreement on the check sums and stuff like that.
So it’s in this particular context, very close to ZFS from a bit rot reliability standpoint.
[00:36:21] W. Curtis Preston: Shingled magnetic recording. This is the one that’s basically made for, for long term storage. Right. Because you have to, it’s got like multiple layers,
[00:36:32] Erik Ableson: Yeah. There’s too many tracks. I mean, it’s like the tape drives. There’s too many tracks overlapping and close to each other.
[00:36:37] W. Curtis Preston: right, right.
And CMR is.
[00:36:40] Prasanna Malaiyandi: stuff. Right.
[00:36:41] Erik Ableson: the old stuff. Um, it is traditional. No traditional doesn’t have a C in it.
[00:36:46] Prasanna Malaiyandi: I think it’s like conventional.
[00:36:48] Erik Ableson: uh, that may be
[00:36:49] W. Curtis Preston: probably. Yeah.
[00:36:50] Erik Ableson: Conventional magnetic recording.
[00:36:51] W. Curtis Preston: with the sh the shingled me, it’s literally like layers on top of layers and to, and to, to change a layer that’s under it. I don’t even, I don’t understand how it works at all, but it, it sounds like you would only use it if you’re going to write and never modify, right.
Um, or hardly ever modify, which is not a backup configuration.
[00:37:16] Erik Ableson: Well, and the problem really is the software needs to be aware. I mean, it’s like any modern storage bay that’s built for SSDs. Like it’s gonna optimize for 4k writes. It’s gonna optimize for making erasure blocks in 4k chunks so that it uses the system. The way it’s meant to be used. And SMR is in the same kind of category that if your software knows that that’s the underlying storage and is designed for it.
Awesome. I mean, that’s Amazon, that’s a Dropbox. Uh, cuz Dropbox did a huge investment in custom code to handle those kind of drives. But for your average off the shelf enterprise software, they’re not there yet.
[00:37:51] W. Curtis Preston: right, right. And so, yeah, so you want to use conventional drives, not SMR drives. And that is definitely a mistake that a, that a budget conscious consumer
[00:38:03] Erik Ableson: Can make.
[00:38:03] W. Curtis Preston: con budget conscious business or government would do. Right.
[00:38:08] Erik Ableson: Uh, this well, I actually have one client that was caught up in the Western digital fiasco that they were wondering why their system was their NAS was just horrible performance. And it’s because Western digital at a moment sold a whole bunch of SMR drives with the same part number as the CMR drive.
[00:38:28] W. Curtis Preston: this was a mistake
[00:38:29] Erik Ableson: Uh, this was, this was confusing for everybody involved.
[00:38:35] W. Curtis Preston: Yeah, did, I mean, did they find out if this was intentional or not?
[00:38:39] Erik Ableson: I don’t know what the final upshot on the story is, but I do all I know is that right now, there are specific drive serial, not serial numbers, but actual model numbers. You can go if you have one of these, take it out and put a real disc in
[00:38:51] Prasanna Malaiyandi: Yeah.
[00:38:51] W. Curtis Preston: Interesting. Interesting. Yeah. So that, so that’s interesting. Um, we should probably get from you a couple of links of, of the, the object object projects that you, that you’ve been playing with.
[00:39:06] Erik Ableson: Yeah, the big one I’ve
[00:39:07] W. Curtis Preston: I’m aware of are the ones that are too expensive that, you
[00:39:09] Erik Ableson: well. Yeah, I mean, there’s the Dell ECS, there’s the storage grid from NetApp there’s, uh, again, but as soon as we, as soon as I cite those names, all of a sudden the, uh, enterprise price tag just showed up.
[00:39:19] Prasanna Malaiyandi: Yep.
[00:39:20] Erik Ableson: And also the, the biggest problem I have with a lot of them is just the minimum point of entry.
Uh, because of the way they’re designed, I think the smallest DCS instance they’ll sell you is something like 60 terabytes, but it’s got to be with three nodes and those three nodes each have, you know, dual 10 or 25 gig can, I mean, just the price
[00:39:41] Prasanna Malaiyandi: are crazy. Yeah.
[00:39:42] Erik Ableson: Yeah. And it’s like, I’m going, look, you can buy an Intel NUC with Thunderbolt and throw eight drives in the back of it and have, uh, a hundred terabytes of usable storage right away.
Uh, and it’ll cost you. I don’t know. I. Well, the disc are expensive, but I don’t think I paid more than about 4,000 euros for the whole kit. And it does the job,
[00:40:02] W. Curtis Preston: By the way I, I, I just, I just wanna have a little fun with you. Um, I don’t think we needed to tell people that you were Canadian, cuz your
[00:40:10] Erik Ableson: the.
[00:40:11] W. Curtis Preston: definitely gives you a whatever you say about or, or process or O OPEX, when you said I’ve actually never heard anybody say OPEX before,
[00:40:22] Erik Ableson: Yeah, OPIC, uh, tomato, tomato.
[00:40:25] W. Curtis Preston: Yeah. Um, love, I love my friends to the North. Um,
One concern that I have just slightly is just a, a slight bone that I pick with the, with the folks, with the, you know, the hardened Linux repository. A lot of people call it immutable because they’re using the immutable flag. And, and I like to point out that, well, as long as no one has root, right. Because you can unset all those immutable, uh, flags,
[00:40:58] Erik Ableson: I, I can do better than that. I can just go in and hose the drives
[00:41:01] W. Curtis Preston: And, and just do
[00:41:04] Erik Ableson: DD input file equals dev uh, output file equals slash SD SD one SD two, go to town.
[00:41:11] Prasanna Malaiyandi: please do not try that
[00:41:13] W. Curtis Preston: a lot easier.
[00:41:15] Prasanna Malaiyandi: repositories.
[00:41:16] W. Curtis Preston: Yeah. Don’t
[00:41:16] Prasanna Malaiyandi: We li.
[00:41:17] Erik Ableson: don’t
try this at at
[00:41:19] W. Curtis Preston: Um, yeah, so it it’s, uh, I just, I just wanna make that point, uh, and this is, this is one of those differences between those enterprise level things, right. Is that they can produce an appliance that you don’t have that kind of access to.
Right. Um,
[00:41:37] Prasanna Malaiyandi: I think though, for your point though, Curtis, I think with the proposal that Eric’s recommending, right? The fact that your management port is disconnected, the fact that SSH is disconnected for all times, other than the period, when you have to do the updates, right. At least reduces some amount of of risk.
It doesn’t completely eliminate it, but it
does reduce it significant.
[00:41:59] W. Curtis Preston: think it’s a great design. It’s as good as this design can get. I guess my only point is you’re still not protected against Eric going nuts. Right?
[00:42:10] Erik Ableson: Yep. Yeah.
[00:42:11] W. Curtis Preston: If Eric has root and he decides to wipe out your data, there’s nothing. That’s gonna stop him.
[00:42:15] Erik Ableson: Exactly. Although if we go to the logical extreme, which is physically disconnecting that cable for the management port, it means I need to have physical access. So that does
[00:42:27] W. Curtis Preston: is a good, that is a very good point, Eric. And of course we all know that physical access, all bets are off
[00:42:32] Erik Ableson: it, trumps everything. It trumps everything.
[00:42:34] W. Curtis Preston: Um, the, um, I mean, physical access, all I need is a torch. I don’t.
[00:42:42] Erik Ableson: Exactly. I mean, it doesn’t have to be sophisticated, a couple of wire bolt cutters, and, uh, or just a big, heavy magnet. And I just run that across all those hard drives and yeah. I mean,
[00:42:52] Prasanna Malaiyandi: Another thing, do not try that at home.
[00:42:54] Erik Ableson: Yeah. Sorry. Sorry. No, that’s I was gonna say who’s who’s uh, yeah, there’s, there’s a special person who loves thermite Thermi and his hard drives.
[00:43:02] W. Curtis Preston: um, yeah. Um, actually we we’ve, we’ve had a, a thermite lover on the podcast. Um, Howard Marks
[00:43:10] Erik Ableson: Yes.
[00:43:10] W. Curtis Preston: works by the way at, at, at vast data who, one of the, one of those. Places that have a way too big of a, you know, that your, the initial block that you have to buy is way too big for anything you’re interested in, but I’ve seen him.
He actually made a video where he got, he got hired by a company to put thermite on some servers, uh, to destroy them and have the whole ha process kick in. And yeah. So,
yeah.
[00:43:37] Erik Ableson: I mean, again, if, if you’re looking for lots of S3 storage, Vast Data will do the job. Um, my only issue is that I believe the current entry point is about a half a petabyte, uh,
[00:43:49] W. Curtis Preston: Yeah, something like that. Yeah. Well, it Vast Data. It’s not called slightly large
[00:43:54] Prasanna Malaiyandi: I, think that they might have recently just come out with smaller nodes, entry level
nodes.
[00:43:59] W. Curtis Preston: He did say that, right. It might only be
[00:44:02] Prasanna Malaiyandi: Yeah, it might still be 200 terabytes, which is probably still way too large. Right.
[00:44:07] W. Curtis Preston: For for, for, for many environments. Yeah, it’s probably, yeah, but,
[00:44:11] Erik Ableson: I mean, that’s the interesting part about the way the market’s been going is that you’ve got this. I mean, the traditional thing is what’s the backbone of most countries, businesses, it’s small business,
[00:44:21] W. Curtis Preston: mm-hmm,
[00:44:21] Erik Ableson: and we’ve got all these companies spending tons and tons of time and money and effort on building solutions for the big enterprises.
and the problem is what I’m seeing more and more is that the small enterprise or the poor guys that are getting the short end of the stick going? I mean, Vast Data, I will be first one to say you I’ve been talking to mark. Uh, I have seen the tech field. I was present at some of the tech field day presentations.
It’s amazing technology. It just doesn’t scale down. And the problem is that we have this huge, massive companies out there that are everything from mom and pop shops to even like a hundred person shop. But their total data footprint is measured in 10, 20, 30, 40 terabytes.
[00:44:59] W. Curtis Preston: I say it a lot. That for every one enterprise that needs a vast data, there are 99, um, you know, smaller companies that also have a need to protect their data. And I think that’s one of the things I like about working about Druva is that we do scale up, but we scale down all the way down. Right. The only thing that limits us in a small, small environment is just the, the business processes.
Right. So we don’t, we don’t sell direct to consumer. We don’t sell to like a, a two person business kind of thing. Right. Just because of the, the paperwork issues, but, uh, tech technologically speaking, we can back up a single computer. Right. And we don’t, and we don’t need that on premises hardware and all that.
So I like the way that we, that we do scale down and, and we, we can handle those SMBs just fine. Um,
[00:45:45] Erik Ableson: But that’s always one of the things that, like I said, Druva has been one of those solutions on my radar. But again, like I explained with, for a lot of these, uh, city halls, they just can’t take advantage of
[00:45:56] W. Curtis Preston: Yeah, that that part of your conversation hurt hurt me. When I heard that, I was like, uh, well, that explains why we’re not big in France. I actually have no idea if we’re big in France or not, but, um,
[00:46:07] Erik Ableson: It’s not the same for regular companies. It’s strictly a, uh, city hall type
[00:46:12] W. Curtis Preston: oh, it’s a city hall. Okay.
[00:46:13] Erik Ableson: Yeah. So it’s it’s at the city level that, uh, those rules are in place.
[00:46:17] W. Curtis Preston: Interesting. Okay. All right. Yeah. So probably what we’re doing just fine in France. I know that, um, I know that we do, we do deal with, uh, data sovereignty problems really well because we’re, we basically run in all of the, you know, the, the AWS regions, right. So people that are concerned back in the day, I remember.
You remember, you know, you were in the, you were in Canada when we came out with the Patriot Act in the us and that, and that made storing data in the US verboten, for anybody outside of the us. Right. So
we, we, we don’t, we don’t have that issue because we’ve got, you know, we’re running in your local, AWS, um,
[00:46:58] Erik Ableson: Yeah. The da data doesn’t go across regions unless there’s a BGP routing Ms. App, but, oh, well, what can you do
[00:47:04] W. Curtis Preston: Yeah. Yeah. Don’t, don’t do that. Uh, any, any final, um, you know, thoughts you have Erik on this, uh, blog post
[00:47:14] Erik Ableson: uh, just
general.
[00:47:15] W. Curtis Preston: Veeam backups to that in
general.
[00:47:17] Erik Ableson: vem. Veeam has always been fun. Uh, I’m always on the lookout for new and interesting ways to, to do stuff. And as you guys have now motivated me, it’s time for me to write up, uh, some more detail on the MinIO TrueNAS because there’s some new stuff on that front, uh, for the S3, uh, backup repositories.
And the one thing is once you get hooked on this idea of protecting data is important. It never leaves you. I mean, it’s been 30 plus years that I had my first . We put up a new server that everybody thought was the greatest thing since slice bread. It was a copy of was called Meeting Manager at the time.
And basically it was the precursor to Outlook with shared calendars. And it was on a single machine and all the executives loved it. And every month in the monthly meeting, I came and said, we don’t have a backup of this machine until the day of the hard drive died. And then I got the budget to buy my tape drive.
And ever since that time, it’s been my hobby horse looking around and going, is that protected? And that’s now open up to the security aspect of things, but it’s always like, is your data safe. And yeah, once you get hooked, uh, you can’t get rid of it.
[00:48:23] W. Curtis Preston: You’ve probably, if you’ve listened to the podcast, you’ve probably heard me say the, we used to have this guy, his name was Joe Fitzpatrick and he was always the guy in the meeting that he would raise his hand. He’d say, are we getting this on tape? Right. That was, that was the phrase back in the day.
Right. Um, and you know, that is one part that hasn’t changed about backups. Since I joined the industry back in 93, Is that, you know, no one wants to be the backup person, no one wants to pay for the backups. Uh, it’s the last thing that anybody thinks about when they’re deploying a new system. When in reality it should be the first thing they think about, right.
If they start protecting it as the system is being developed in house, it’ll just be automatically protected. I, I, I think about we, we used to, we had, um, we had, um, A very boring naming convention back at MBNA, which is the bank that I worked at. It was, um, you had either had HP Fs, zero one, right?
HP file server zero one or H P DB zero one. Right. And, um, I remember. And so what I started seeing was that new systems were getting deployed and I, the person in charge of backups, wasn’t being told that they existed. And so I created this form that you had to fill out. This is pre-web right. So there was literally a physical form that you had to fill out and on the form, it said this form is not complete until you have a signed copy back from me that acknowledges that I received the form. You can’t just say, oh, I filled out the form. Right. Because what when I implemented this form was somebody asked me for a restore from hpdbsvk, right. HP database server. K. And I, I, this is the weird things that you remember. I remember that it was Kay and I go, so, Kay.
Like, yeah, I’ve never heard of K. In fact, the last server I remember hearing about was H so that means there’s, there’s an I and a J and a K none of which are being backed up. That was, that was the day that I implemented that form. Right. Because, uh, people, people just don’t think about backups, but we do.
[00:50:32] Erik Ableson: we do all the time.
[00:50:33] Prasanna Malaiyandi: then.
[00:50:34] W. Curtis Preston: You want us on that wall? You need us on that wall. I’m sorry. I, I, I, uh, went with a few good men there for a second. Well, anyway, thank thanks, Erik so much for, for coming on the podcast
[00:50:45] Erik Ableson: a pleasure.
[00:50:47] W. Curtis Preston: and Prasanna, as always great questions.
[00:50:50] Prasanna Malaiyandi: I try and thanks, Eric, and looking forward to seeing your, uh, object store article, and maybe we can have you back on the podcast.
[00:50:57] Erik Ableson: Sounds good
[00:50:59] W. Curtis Preston: And, uh, thanks again to our listeners. Um, you know, you’re why we’re here. Otherwise we’re just a couple of folks talking to each other.
[00:51:07] Prasanna Malaiyandi: I like talking to people.
[00:51:10] W. Curtis Preston: absolutely. And, uh, be sure to subscribe so that you can restore it all.
The post Really inexpensive AND secure storage systems for backup appeared first on Backup Central.