Modernizing Your File Systems

In August 2008, my boss came to me with a flaw in our infrastructure that had the potential to become a larger issue, and possibly disastrous to our organization if not addressed. The problem involved our main file server, we had a setup that had all clients connected to a windows server 2003 box to serve roaming profiles, user home directories and workgroup folders, as well as serve network apps, software installation files, music/video files, amoung other things. With so much load on this one server the issue that appeared was the server slowing down to a crawl, as well after upgrading the machines ram we had it crash several times randomly so we reverted to the old RAM. To compound these issues we had no server that could take its place if anything went wrong, system redundancy was placed on the RAID configuration which came with the Dell 745N PowerVault server. So we began assessing the following weaknesses in our file system:

• Software RAID configuration
• Single server
• Heavy, high bandwidth load (Upwards of 200 active clients, and hundreds of thousands of active files)
• No backup/rescue scenarios
• Outdated OS (Running windows server 2003 storage edition, Dell customized, this became an issue as it could not be updated and was severely lacking, feature and performance wise)
• Reliability to the clients
• Capacity limitations (1 Terabyte total, with over 600 gigs in use)

To deal with these weaknesses, the whole file server setup needed to be revamped, while the existing setup ran in parallel. It was apparent that a new server was needed, Dell no longer supplied 745N PowerVault file servers, and were instead pushing expensive SAN systems, we wanted a low cost solution that had a great deal of redundancy and performance. So we looked at our existing server, and decided we would get the same server again, but where would be find the same model? Well ebay.

Ebay had a large listing of Dell 745N PowerVaults, all for under $1000, the server we finally choose had the fastest P4 possible 3.2ghz, 2 GB of ram,  4 x 250GB hard drives, and a hardware raid card (PERC 5\i). This solved one major problem, taking the raid management away from the OS and putting it on a dedicated RAID controller, as well we upgraded the RAM to 4GB, and increased hard drive space to 4 x 750GB. After receiving the server, we did the hardware upgrades, then came the next problem, what do we do with the old server 2003 storage edition OS? Well its 2008, and Microsoft had updated their OS, we wanted to use server 2008. However, if you have ever worked with a Dell PowerVault they lack a CD-ROM drive, and do not support any OS besides a USB installed version of server 2003. In the months before working on the storage issue, I was working with Windows Deployment, and utilizing its PXE boot feature, we could boot into a Vista PE environment. Then I copied the contents of our server 2008 installation disc onto a large flash drive, and from a command prompt I loaded up the setup.exe from the installation files. This is a fairly easy way to install modern versions of windows, you could even make a bootable usb drive to do the installation. Once into the installation wizard, I got as far as the disk configuration, then a road block, the raid setup was not detected. It took a while but we finally found the adaptec driver from a german Dell ftp. So now we had server 2008 installed, a machine with larger capacity then before, now 2 terabytes in hardware RAID 5.

After running our new server for a month, doing testing with the new OS, as well as load testing, we were ready for production. Then came the painful situation of moving all the data from the old file server to the new, without disruption. Using robocopy we copied each home directory, one by one, preserving the file attributes including time stamps, then updating each user AD object to the new server. Finally we moved the workgroups, and other miscellaneous files, leaving us with a fully operational file system, that was both up to date and redundant. Looking at our initial assessment we had crossed a lot off our list:

• Software RAID configuration
• Single server
• Heavy, high bandwidth load (Upwards of 200 active clients, and hundreds of thousands of active files)
• No backup/rescue scenarios
• Outdated OS (Running windows server 2003 storage edition, Dell customized, this became an issue as it could not be updated and was severely lacking, feature and performance wise)
• Reliability to the clients
• Capacity limitations (1 Terabyte total, with over 600 gigs in use)

Still, we had a few more items to accomplish, and with our offices being spread across a university, the idea of a multisite file system soon took root. To build this system, we needed to upgrade the old server, find a load balancing/file replication technology, and find an alternate location. In upgrading the old server, we added a hardware RAID card which was special ordered from Dell for a 745N, apparently they still have 5 year old RAID cards ready for order. As well we upgraded the RAM, hard drives, and fully cleaned the server to be  similar to the Ebay purchased 745N. Next we updated the OS to server 2008, using the same method, and placed it in our main IT support office building, in a storage room. At that point we had two servers with similar specs, configurations, and separate locations, so all we needed was a piece of software that works with Windows, and can handle file replication/load balancing.

The software we decided to go with, was something that I had worked with a year before, it was built into Windows, Distributed File System (DFS). DFS is the underlying technology that is already used to replicate data/files between domain controllers, and has been integrated into Windows server for quite sometime. In server 2003, DFS was not as feature full as I would have liked, but it did show promise, for that reason I was willing to give it another try in server 2008. After initial testing, I had concluded that DFS had made many steps forward since server 2003, it had an easier to understand setup, added namespace and replication group features. So using DFS, I configured a domain namespace for user directories, workgroups, and profiles, these namespaces mean that a client would connect to the following type of share and it would then take that user and connect them to the regionally closest file server, all transparently. Namespace links can be used the same as any share, and look like this:

• \\domainname\users$
• \\domainname\profiles$
• \\domainname\workgroupname

You can also use existing share names, which will act the same as a DFS namespace, pointing to one of the servers based on region, shares like:

• \\server1\users$
  • Points to which ever server is closest, or if no sites configured it will choose randomly:
    • Server1
    • Server2
• \\server2\users$

This made for a great setup, mainly if one server went offline, it would failover to the other server, and we can easily change or add new servers without changing the share name. With the shares taken care of, the next step before they could be enabled would be to replicate files so that they are the same on both servers, even if they are being constantly edited and opened. Using the file replication section of DFS, I separated each OU’s folders into replicating folders, and once completed we had a fully replicating group of servers. Of course there was issues, which are still being weeded out but they are minor, many include how some programs place locks on certain files. So far we have alleviated any issues quickly and they have not caused disruption.

In conclusion, if you are a small to medium sized organization, and are running an aging file system, you should consider looking at the options I have outlined. The older Dell Powervault line may seem out of date for modern operating systems, but in the correct setup they can work perfectly to quickly serve files, and can be stacked with cheap hard drives. When looking at storage systems, always have hardware RAID, and if possible have offsite backup or even multisite replication. Server 2008, is a necessary upgrade if you choose windows for your file systems, although it is based on Vista, it lacks the issues you encounter with that desktop OS. In future posts, I will get into further detail and provide greater instruction on configuring file systems.