Remote Storage On Remote Storage

Let’s face it, the entire point of remote storage is technically just getting storage remotely. But at what point does your remote storage become…too remote? From Tales of the Department of Redundancy Department, I bring you this massive convoluted networking crime of NASing while NASing.

This story starts 4 years ago. I had been back to work after a major spinal injury for a few months and I was in desperate need of more storage space. Microcenter had a 3TB Seagate Personal Cloud thing on sale for a stupid cheap price. I figured if the NAS function was crap; I could at least yank the disk out of it. It also gave me an excuse to pick up some cheap GigE network switches to support this thing.

Things went well for 4 years. It sat on a shelf in a room with power and network making stuff available over SMB. Sure, the software was total shit and a real PITD to mess with; giving you very little control over how stuff was shared and seeming to put a large focus on bullshit like “cloud” connectivity, media sharing, downloads, and other stuff I don’t need a box putting storage on my network to do.

Personal stuff found me within the last week acquiring a Dell PowerEdge R610 for some amateur radio related activities. This is where things began to change. For a while I had happily had things serving data to the internet while simultaneously pulling off the NAS; afterall I had a gigabit to play with locally. This changes when your internet connection also becomes gigabit. Now serving data to the internet while simultaneously pulling from the NAS caused the network topology to be a choke point. I had wanted for a while to pick up a PCCard ethernet adapter for the laptop this was primarily an issue on; the idea was to just bridge the adapters and give the “server” dedicated paths to the NAS and network. I never got around to doing that and in fact, that laptop died during the lightning strike. But this R610 has 4 network ports and in my non data-center environment; that gives me three to play with. Plugging the Seagate in to one of those and bridging seems like the local choice. It would have been the easiest.

But that’s not what I did; because Seagate’s SMB sucks, I wanted to have finer control over data shares (so family can easily access stuff like movies), and I seem to recall trying to share a path over smbd that itself is a smbd mount just results in things breaking. So my first step was to remove Seagate’s firmware and install Debian on to the “PersonalCloud” thing.

I also created a “NAS” VM on the server. I’m not running an actual NAS distribution, though I thought about loading FreeNAS. Instead, it’s just the same basic installation of Ubuntu server I’ve used for all my VMs with software to perform whatever NAS duties are necessary. The only non-NAS utility I put on is dhcpd. I envisioned myself at some point in the future getting angry at me for setting a static IP on an entirely different subnet; making it a PITA to get in to should my setup change. Since the NAS is basically being plugged directly into an ethernet port, attaching dhcpd to that interface seemed like the best compromise. I can configure it to offer a single IP to the single device it will ever talk to in its lonely existence while maintaining DHCP ability for whatever network I need.

The NAS VM also has the most number of used network interfaces of any of the VMs at three. eth0 is attached to my physical lan, eth1 is attached to the Seagate on a 10.0.0.0/24 network, and eth2 is attached to an internal Xen network on 10.1.1.0/24. I planned on having other VMs make use of “remote” storage, so giving them an internal static network seemed like a good idea.

My original un-educated idea was to set the now hacked Seagate appliance up as an iSCSI target, set up a VM to act as a nas with the iscsi initator, and then just share the iscsi mount since it would appear as a regular drive. The problem is the drive performance apparently really sucks in this configuration.

So I went with my next idea. I removed all the stuff I did to make an iSCSI volume and reconfigured the thing to work as an NFS mount. I knew that in most cases NFS to NFS between two Linux machines was a hell of a lot faster than SMB; so this was the logical choice on this path. From the VM doing NAS duty, I would just configure a share in Samba for where the NFS share is mounted.

It works, at least up to this point. Speeds are not great; though I’ll admit I don’t remember how bad writing data was as I rarely did it all in one go. The read speeds were a little slower than before; but they’re actually a lot more stable. Transfers tend to level out around 64 to 72 MB/s and stay there. Now I could have figured out a way to attach the Seagate up to the internal Xen network and have all the VMs access it that way; in fact there’s a chance that’s probably a hell of a lot more efficient and something I’ll do.

But, for the time being; I just set an NFS mount on the nas VM to the path the nas has the Seagate’s NFS share mounted to.

But I mean…it all works. At least until we got to the second chapter…

PERMISSION HELL

When I was in the process of testing my rtorrent + rutorrent installation, I had no problem reading and writing to the remote storage from any of the VMs. I also had no problem writing to and reading from Windows via Samba. But I hit a snag when I tried reading files written through Samba by Windows with Linux, or reading files written with Linux on Windows….it didn’t work. I found myself stuck in a mild form of permissions hell over this. Files I put in autowatch never appeared; existing data copied from the desktop couldn’t be used for seeding.

Thankfully, the permissions hell wound up looking like it could be more complicated than it was; and the only change I really had to make was to samba.

The problem turned out to be that Samba was creating files on disc belonging to the ‘nobody’ user and group; so literally nobody can read/write to them. That’s perfectly fine since smbd is ‘nobody’; but that’s not the case for the NFS side of things. I use the exact same username on all the VMs, and as far as I can tell that’s all that matters when dealing with the files. If there’s a UUID associated it’s either generating the exact same one between all the installs (I don’t have deployment images or anything)…or it just sees the username ‘dewdude’ from all the machines. Since I have client access IP’s coded in to the NFS servers, that may have something to do with it.

But in the case of samba, samba is nobody. Nobody can read nobody’s files, but they can’t read dewdude’s files; just like dewdude can’t read nobody’s files. It’s probably possible for me to give nobody read access to files; but it was easier to just tell samba to be dewdude.

TS;DL (too silly, didn’t read): just add force user and force group to smbd share configuration.

At the end of the day, as long as all the VMs can read/write to the remote-squared storage, I can read the data they write over samba, and they can read what I write over samba; it works. File permissions on the Linux filesystem I can have root deal with at a later date if necessary.

Now maybe this time I’ll organize the data as I put it on the NAS.