My Own Cloud Server
What I Did
I built a server. I use it to provide myself with remote access to various programs and files and to host this website. As I add more functionality to it, I hope it will be a complete replacement for virtually any cloud service available. Moreover, it will be secure, free (as in cost and as in freedom), easy to use, and 100% under my control.
Why I Did It
I have a desktop with two operating systems that I switch between fairly regularly (can't be helped if you're a computer gamer, no matter how little use you have for Windows otherwise), a laptop, a cell phone, and a work computer. Wouldn't it be nice if, no matter which one of these devices I was on, I could, with the click of a button, have access to a constant set of files and applications? Having a homemade server allows for this. For instance, at work I was stuck using Outlook for calendaring. Outlook's fine, for the most part, but accessing it from home was a pain. Now I use calcurse, a simple and wonderful calendar program, which I run on my server and can access from any one of my computers with total ease. Text files are another example: I have a lot of little text files that are simply lists, that I need to edit often (think shopping lists, say). With my server, now I can create and edit text-files from anywhere, even on my phone, and have them stored on one always-on system.
Now, I realize that there's something called "the Cloud." But my computer does much of what various Cloud services provide; what's more, for all that my computer does, it isn't really doing anything that similar computers weren't doing 30 years ago (just switch out SSH for telnet). The "Cloud," then, is a haze of hyperbole that makes Gmail into something Hotmail wasn't, makes Dropbox into something that Geocities couldn't have been, and makes remote access to computers a stunning innovation rather than a sensible return to the way computers worked 40 years ago.
Worse by far is the public's blind faith in cloud service providers to do the right thing with their personal information. Consider: what happens if Omnicloud Corp. uses your data to target you with ads? What if Omnicloud Corp. is hacked? What if some rogue Omnicloud Corp. employee released your data or, looked at it out of some prurient interest? What if they lose your data and don't care enough to have backups? What if Omnicloud Corp. decides to charge for access to the data you just gave it?
Fortunately, these are just hypothetical questions, right? No. Let's look at Google's track record, not because they're especially bad, but because they generally get things right. Even so, look at these problems they've had:
- In 2010, Google fired one of its engineers for accessing the accounts of several teenagers and digging through their personal information.
- As is well known, Google's software effectively reads your email, scanning it for keywords in order to direct advertising at you. Google also saves your search history for advertising and for tailoring your search results to your supposed interests. And now Google has a social networking component, ostensibly for your amusement and edification, but really just another way for Google to track you. Recently Google unified the terms of services for their various services, allowing them to aggregate a tremendous amount of previously data about their users.
- Google is cavalier about restoring any data it loses, as attested by this accountin the Atlantic. The author's wife, through not fault of her own, lost over six years of email and was told that it could not be recovered. When the author emailed a personal acquaintance high up in Google the emails were restored. Moral? Google doesn't really care about your data if it's going to take some effort to recover and if you don't have some clout.
- Even Google doesn't hit a home run every time. Google has discontinued a lot of cloud products. To Google's credit, they seemingly do a fine job of providing users with some access to these defunct offerings (I say this as a onetime user of Google Notebooks). Will other companies, especially when the defunct service was their only service and they have no funds left to draw on?
- Google and others collaborate with the U.S. government to make your data available to secretive government agencies.
- Finally, even a giant company isn't invulnerable to hackers. On your own you're probably not any less vulnerable than a giant company, but at least you're probably not going to be a target yourself. For instance, if you were hosting your website with your own machine, you wouldn't have had your credentials exposed the way they were for Dreamhost subscribers in January 2012.
(One could add to the list of cloud problems the decision of many one-click uploading sites [Filesonic, Fileserve, etc.] to go offline in the wake of the closure of Megaupload in January 2012. Truly the only service you can control is the one your provide yourself.)
In the face of all this dreary news, I suspect that many people would respond, "but that's the price we have pay for convenience." No, it's the price you choose to pay. As I will show, using fairly inexpensive hardware and free, open source software, I can get the same benefits as those that trust the corporate cloud without the aforementioned risks.
How I Did It: Hardware (original 2011 build)
I didn't use any fancy hardware for the build:
- Motherboard and CPU: Intel BOXD525MW motherboard with a dual core Atom D525 processor running at 1.8 GHz.
- Memory: Corsair 4 GB DDR3 Laptop Memory
- Case: SilverStone Micro ATX GD04B
- Power supply: Xigmatek ACXTNRP-PC402 400W
- Hard Drive: Samsung Spinpoint 1TB 7200 RPM
- Network card: Intel PWLA8391GT
- Optical drive: Asus 24X DVD Burner
How I Did It: Hardware (updated 2018 build)
In 2018, I replaced the motherboard and what attaches to it. Over the prior few years I had replaced other bits and pieces, so that now only the case remains from my 2011 build. Here's what I've got now:
- Motherboard and CPU: ASRock J3455-ITX. Intel Celeron J3455 CPU with four cores running at 1.50 GHz.
- Memory: Kingston HyperX Impact 8GB DDR3L memory
- Case: SilverStone Micro ATX GD04B
- Power supply: Lepa G700-MA 700W power supply
- Hard Drive: Samsung SSD 850 Pro
- Network card: Intel 82574L
- Optical drive: none
The most important components, the motherboard and the attached processor, were chosen because they offer really good performance/power ratio. When the entire rig is assembled and plugged into the wall, it only uses around 20 watts of power. (This is also thanks to the reasonably efficient power supply.) It costs between one and two dollars a month to operate.
The case is easy to use and looks nice, but it is larger than I expected it to be. And yet somehow its interior is so cramped that I would be wary of putting a traditional, hot processor in there without having a modular power supply to improve airflow. Finally, one nice feature of the case is that the stock fans all have dust guards behind them, preventing dust from getting in the case.
Honestly, though, 99.999% of the time I'm using 70 out of 8192 megabytes of of RAM and only a slight fraction of the processing power. You could do what I'm doing by using an old laptop, or probably even a 10 year old PC as long as you don't mind the energy cost. (For a PC that uses 120 watts, you're looking at about 10 dollars per month for 24/7 usage.)
How I Did It: Operating System
Choosing the rig's operating system involved two choices.
First, I had to choose between BSD and Linux, the two categories of free, open source operating systems most suitable for server use. I chose BSD because I like that all of the BSDs make documentation a top priority. (We'll soon see what this means in practice.) I also like that they are regarded as having higher quality code than Linux, albeit at the cost of lagging behind Linux. More could be said of the differences between Linux and the BSDs -- and of the areas in which BSD excels -- but I'll leave that to this pithy document, BSD vs Linux. Finally, I choose a BSD because I just wanted to try something different.
My next choice was which of these three main BSDs to run. Making this choice requires you to think a little differently than you would when choosing a Linux distribution. Each distribution of Linux is just the Linux kernel -- maybe with a few different tweaks -- and some version of the GNU utilities, and some combination of applications. Something like Ubuntu might look and feel a lot different than, say, Crunchbang, but these differences, while numerous, are superficial. Each BSD, on the other hand, is a separate operating system, with its own kernel and utilities (which are developed together). The BSDs have a common ancestor, and they do borrow code when possible, but they really have been developed separately for over 15 years now. Moreover, each has its own focus or philosophy.
I settled on OpenBSD over NetBSD and FreeBSD for several reasons:
- OpenBSD focuses on security. The other BSDs are also secure, of course, but they worry about other things (portability in NetBSD's case and performance and features in FreeBSD's). Recent releases have had only a couple of security patches, for services I don't use. Of course, that's not surprising given that the project's motto is "two remote holes in the default install, in a heck of a long time." With the provision of remote access being the main objective of this machine, that motto is one I like to hear.
- Every aspect of the system seems carefully documented, either in the "FAQ" on the site's page or in the man pages that ship with the system. The manual pages receive serious attention; if you look at the commits for OpenBSD 5.1, there's a few that pertain just to documenting the history of various utilities at the end of various manuals. I also saw it said that the developers delay new feature's that are otherwise ready rather than releasing them without adequate documentation.
- On a related note, they've tried to make the documentation
accessible. When you boot your OpenBSD machine for the first time you're
told that you can type
helpfor, well, help. This help manual exists "to familiarize new users and system administrators with OpenBSD and, if necessary, UNIX in general." New installs also come with a mail from the project's leader, giving tips on what to do next and where to find help. This mail also points to the afterboot manual, which describes some of the steps you may want to take with a new installation.
- OpenBSD's developers prefer you to use the system without attempting to configure every last bit of it. Sane defaults is the watchword here. I find OpenBSD very easy to install and very quick to set-up. Moreover, compiling anything from source code is discouraged (the FAQ tells you that you will be laughed at if you compile a custom kernel and then your system fails to boot). Similarly, I like that they encourage reinstalls over upgrades and that binary packages are preferred. I think this is a pretty sound set of suggestions and one I'm familiar with from the Linux world.
- The development model seems sound. Instead of trying to backport features to several older releases, a la FreeBSD, or having different branches of development, there's just one single branch of development that's spun off to create new releases every six months. From what I understand, they even halt development altogether prior to a release so that every developer can do testing. You can read more about the OpenBSD release method here.
- The handy applications included in the base system are almost all you need for a good server (or even a minimalistic desktop). After a five minute install, you get tmux, X.org if you want it, along with some decent window managers, a nice shell, and so on. It also has the project's own lightweight web server. Since all of these tools are in the base of the system, you can be reasonably sure that their code quality is quite high and that they're secure.
Having lived with my choice for almost a year now, I have to say that OpenBSD is extremely stable (no unplanned downtime despite 24x7 use), simple, and performs well. I only wish that more packages were available and that disk encryption were easier. Regardless, an operating system also has to be judged by what you can do with the software it does make available, so let's move on.
How I did it: Applications
Let's start with the foundation pieces.
OpenSSH does the heavy lifting on this rig. I use it to remotely connect to this machine. Properly configured, this is very secure. Using it on OpenBSD is best, since this is the platform it's developed for. With SSH clients for pretty much every operating system under the sun, I can connect to this machine from Windows computer at work, my laptop and desktop here at home, and even my phone.
Once I connect to the machine thus, I simply see a command prompt. So long as I want to use more than one program at a time, I need a terminal multiplexer to allow me several virtual terminals (or something. Terminals and consoles and the related terminology and options still confuse me). tmux, which seems to be developed primarily as part of the OpenBSD base system, is similar to screen, a program many Linux users have heard of or used for years, but tmux seems better in every way.
And then once I'm connected and have several of those virtual terminals to work in, I have a whole suite of handy programs I can use. These programs all have textual interfaces, but don't be fooled -- these programs, once you familiarize yourself with them, are amazingly simple and quick. Here are some of the ones I use...
- abook, a text-interface contacts manager.
- calcurse, an excellent calendaring program. This program is really essential to organizing my life. Using the export to ical command, cron, and the included, secured webserver, I can easily host ical files that my phone will subscribe to. Learn more about using calcurse on a mobile device here.
- dict, a client for looking up definitions from the command line. If you've been on Merriam-Webster's website lately, you'll know how much time dict saves and how many annoyances it circumvents.
- newsboat, a really nice RSS reader. A perfect replacement for privacy-invading, ad-riddled RSS readers that might be shut down whenever their owners feel like it (cough google reader cough).
- rtorrent, a small torrent client.
- Taskwarrior, powerful, fast, and simple text todo list program.
- vi, the famous text editor. My philosophy of computing is that almost anything can be stored in a text file, and if it can be, it damn well should be. As such, there are a few dozen text files that I edit regularly on my server, from a list of books I've read to shopping lists. I probably have multiple more documents on this machine than I ever had on Google Docs, and I bet I edit them multiple times faster using vi.
- vimwiki, a personal wiki plugin for vim. I use this for recording notes on news and books that I read.
- vitetris, a swell clone of early versions of Tetris. This works pretty well over a network.
- weechat, a feature-rich IRC client.
- weather, a super handy command-line program for getting a forecast or the current weather.
Now, that's only a few programs, but they do most everything I would want from the cloud services I'm familiar with. I get a calendar, contacts list, and documents. How is this anything less than what a Google account provides? Moreover, I'm getting these services freely, securely, privately, and better yet, by way of simple little programs that are so much quicker and easier to use than anything I've seen available through a browser.
Let me give an example of all this works in practice. I use my server to keep a shopping list. It's just a text file I edit with vi. This is simple and orders of magnitude faster than trying to use something like Google Docs or to type it using my phone's on-screen keyboard. All I need to do is -- on any computer I'm using -- fire up ssh, log in to my server (and it's usually as simple as typing 's' -- everything is aliased), and then quickly open up the shopping list using the vi editor. So editing the shopping list is really easy. But what about displaying it?
Fortunately, OpenBSD includes in its base system a rather small and simple webserver, httpd. Basically, when httpd is running any file in the /var/www/htdocs directory can be accessed by a web browser pointed at that machine. Using the cron program, it is trivial to automatically copy my shopping list to the above directory every fifteen minutes. Thus, all I need to do to see my shopping list -- never more than 15 minutes out of date -- from any computer is to type in the file's URL.
But all of this is just the tip of the iceberg. If I want to stream music from this computer to a browser, I can use Zeya. If I wanted my own wiki, I'd just run dokuwiki, mobiki, oddmuse, or any of the other wiki platforms. And so on.
My attempt to create a simple server that would provide constant remote access to my documents and calendars has been a complete success. My solution isn't for everyone, of course: it uses a 200 dollar PC and a suite of command-line tools that many might find off-putting, at least before trying them and seeing just how simple and user-friendly they are. Put another way, there's a fairly high barrier to entry.
Fortunately, a new generation of hardware and software should make it possible for anyone, regardless of expertise or income, to have their own private cloud. NextCloud, for instances, offers a simple, attractive browser-based cloud platform, replete with features like document editing, calendaring, and music streaming. Hardware like the low-powered, low-cost Raspberry Pi make running that kind of cloud software both cheap to start using and cheap to keep going on a 24-7 basis.