TUAW Interview: Inco, simple sysadmin from your iPhone

The dream of remote system management from your handheld device -- like Dick Tracy's wrist radio, but for geeks -- has long enticed IT professionals. Who wouldn't rather be at the beach or the ballgame instead of chained to a laptop and network access in case something goes wrong? There are already RDC and SSH clients for Windows Mobile and Blackberry devices, but it sure seems like the iPhone would be great for this sort of thing... basic, simple system admin from wherever you happen to be, and on the world's coolest phone, what could be better?

That's what Virginia Tech sysadmin Josh Eckstein thought, and hence: Inco, the system management tool for iPhone. Inspired by a vacation (sans Internet access) where he needed to keep an eye on his servers, and complete with heads-up displays of processes and load, file/user controls and an SSH shell (all via MobileSafari, no need to jailbreak or install custom apps), Inco looks like a promising helper for the iPhone owner with rack-mounted gear back in the datacenter that needs monitoring.

The elevator pitch for Inco is "like a Green Beret for your computer: be able to get in, do your work, and get out, no hassle," says Josh. If that sounds good to you, the beta signup for Inco has begun over at getinco.com, and you can try it out for yourself; the eventual release will run you $39. Josh was kind enough to sit down with us for a quick chat about the current state of Inco and where he hopes to get the product in the near term.

What prompted you to create Inco? A family vacation...? A girlfriend's demand that you leave that *&!@#$ laptop at home?

Yeah, pretty much. My girlfriend's parents had invited me to visit their beachhouse in NC for a week. The problem: no Internet to speak of. I can't say no... at the same time, I have users at Virginia Tech, users on a few servers, various projects -- dial-up's a waste of $8 for a week, and trying to get down to the local Starbucks in August is like wading through one of the plagues. The locust one, specifically, not the rivers of blood.

And you couldn't rent or steal an EVDO card...

Righto. Anyway, so I'm one guy managing these systems. I don't have fancy notification systems in place, there's no lights-out-management, etc. The top 90% of what I do when I'm managing these systems is little manual maintenance. It's nothing terribly complex -- complex automata is automate-able, but for manual stuff, I need access. I'm the kind of guy who chose his most recent apartment based on how far it was from the [colocation] center.

And, undoubtedly, how easily you could bike to the colo if the streetlights were out...

(ha, yes, actually -- i managed to snag a nice place about two blocks away in VT's corporate research center so, I think i get major geek points there.)

If you have a picture of yourself standing in front of the XServe-based supercomputer, that'll get you over the top.

Gah! [System X/Terascale] is about 1,000 feet away, actually, in AISB. My boxes are in VTKW1, but it's close.

[Anyway,] all these fancy notification systems don't really help. What use is it to know your server's on fire if you can't do anything about it remotely? So Inco came out of that so I could keep an eye on things without being "that guy" -- you know, the guy in movies who's at the beach with his laptop.

I know that guy... in fact I am that guy.

Right! We all know that guy, a lot of us are that guy. Inco's for the people who don't have on-call staff.

What did you want Inco to be capable of; how many of those features are in v1?

First, i knew i wanted Inco to be intentionally minimalist. There are a lot of products that focus on being an interface to the entire system - cPanel, Plesk, those kinds of very thick and heavy control panels... but the problem is that if you don't conform to the idea of what those control panels think a system should look like, you're likely to be out of luck entirely, or have a broken system if you choose to stray from their vision.

Philosophically, I wanted inco to really be all about those core UNIX features and not try to enforce its own views of what administration should look like. So we have load, we have service monitoring, we have processes, users, files, and a shell. That's really what it boils down to is processes, users, files, and commands.

A lot of what I wanted to do is just have the ability to be well-informed about your system -- and that's in this version, but it's really only a small portion of what i want Inco to become, and there are some very neat features coming down the line. Even v1 users will see a lot of that value. While I'm eager to get inco out the door and have people get to know the product, users who purchase Inco should by no means think that we're done.

How much of Inco is wrapping F/OSS and how much is your own special sauce?

Inco is a full-fledged ruby on rails application, and there are a few reasons for that -- including just the act of not being a native [iPhone] app -- I expected the Apple hammer to come down sooner or later. We use webshell for the shell, and then there's some definite sauce to get it working. Webshell's got tremendous promise, but it's also not exactly friendly. We use plotr for the javascript graphing code itself, and we'll be using that a lot more -- that will show up more in individual process, service, and user load graphs that aren't present yet. (We use blueprint for some of the css, but, they'll probably be leaving soon since the only thing we use is their column CSS, in favor of something more lightweight and homegrown.)

So since it's built RoR, in theory the Inco Administrator console is portable to other host platforms besides the Mac?

Administrator is actually homegrown and is a universal binary, but is completely portable. Right now it's just on the Mac, because I think there's a strong intersection there between smartphone platforms that can use all features of Inco (population: 1, the iPhone) and obviously, Mac users. Soon it'll be on Linux. I wanted it to be there for Mac users so people wouldn't believe this kind of stereotype that's often put on Mac users, that technical users and Mac users have no intersection. I think the iPhone break process has shown there are just as many hardcore users who have a use for something like Inco, that use Macs, as there are on other OSs.

The rest of it is all us. It doesn't sound like a lot after that kind of an introduction, but there's actually a tremendous amount of data we're collecting on system state, in a portable way, that we plan to get into using. Not remotely, in a phone-home kinda way, but in a way that's available to the user.

Comparing Inco to 'maximized' remote control like Telekinesis/iPhone Remote... Did you consider using Telekinesis for this purpose? Or did you feel that something like that was too far away from the core functions of administration?

I think there's a different focus. I think Telekinesis is cool, for sure, but I was at the beach with definite administrative needs more than the need to, e.g., use iTunes. And there's a different direction we're going, too. One of the things I've kept alluding to was stuff that's coming down the road, and there are a few of these kinds of ideas that I'm testing.

We have the Great Three Hacks, as I call them, index.html, robots.txt, and favicon.ico. Anybody doing any sort of hosting or systems management is familiar with those files. I figure since we're cluttering up our root directories, we might start moving what has historically been the purview of local-scope-only type information -- what's my system load? What's taking up a more than average amount of memory? -- into the public realm. So in a few months Inco will, at the option of the user, push out a file called system.xml, which is really just very generic information about the system's state.

Very generic, non-identifiable, non-sensitive, to clarify. Now imagine what we can do when we have information that used to be of interest only to local system users and make it public, put it out there? Maybe I want to check up on not just one system with Inco, but 30. Maybe I can automate that process for uptime reporting. Maybe then, I can tag my servers with identifiers like "database" or "www" and see not just the 1, 5 and 15-minute averages and loadgraphs for a single system, but for all of your "www" tagged systems, and -all- of those systems, in comparison.

Think of what having that information would do for something like distributed computing, for example. So Inco isn't just about a single system, it's about going in a direction that really gives you a lot more power without being more complex.

For the current version of Inco to be useful, target machines have to be exposed to the Internet? Or are you VPNing from the iPhone?

Inco installs locally on each machine you want to monitor, so while it's a Rails app, it runs on the machine you want to monitor -- which is why it's important that it's very low load. So it runs on each machine I own or operate. That's obviously a hassle for if you have 300 machines, which is the reason system.xml will even exist, for automating the kinds of displays where you can see at-a-glance, that one machine is on fire, and then go straight to its Inco HUD.

I'll be more explicit in a vague sort of way -- there -will- be a service component for v1 users. It's just not anywhere near done yet. The neat thing about Rails is that I can say that now, and in three months demo the entire thing.

And that service component will be about monitoring and reporting and being able to look at groups of Incos, not just single Incos. If that makes sense.

From a security perspective, is Inco a risky install? Or is it as secure as your (already installed) Plesk, cpanel, SSH etc. that you presumably also use?

My straight up answer: Inco controls systems. It is an inherently dangerous tool, and there are no training wheels. But I find that a lot of times we worry about security in the wrong categories. So, we have this product that can kill processes and delete files. Right? Right. But killing processes and deleting files isn't something we couldn't do before, even over a network, now it just makes it easier. So it's not a question about Inco, it's a question about if the computer will protect us from ourselves. Inco makes every possible effort in its code to use preexisting conduits to system calls and not try to be too clever with introducing new magic, so if you're killing a process you're killing it the same way you would before; there's no extra surface area where something could go wrong.

I'm a big fan of the idea that you can't eliminate risk, you can only mitigate it. If you decide to only have 6000 hulls on the titanic, then you'll be out of luck vs. the guy who has 6001. :)

First: Inco runs as whatever user you start it as, and so it only has the permissions you give it. User 'josh' only has permissions for the items of user 'josh', and we always try to notify users in a lot of ways when there isn't something they're allowed to do. Webshell runs over HTTPS by default (which is no easy task -- we, for example, auto-generate certificates if users don't have one.) We use the bcrypt hashing libraries for user password storage, the same ones that openbsd uses -- rainbow dictionaries be gone. User auth permissions are set to no permissions, not even able to view the Inco logo, by default. And this is finally, one of the reasons Inco is an information-focused product vs. an action-focused product -- less magic equals fewer bugs.

The install is definitely non-risky. When Leopard's released, Ruby on Rails will be default on systems, so it won't even need to install that much.

Inco is not for sale yet, right? And what's your roadmap, in vague terms, looking like?

Inco will be released very shortly, hopefully by Thanksgiving. Right now, we have getInco.com up where I'd be glad to personally answer questions, and get a few users testing Inco -- free licenses for beta testers -- it's thirsty work. Immediately in v1, I'd like to find a better way to work with files -- permissions, groups of files, and so on. Secondly, the shell -- I mean the concept itself -- is good, but it's not great. At the end of the day, ssh is better to interface with a full keyboard, and that's just not the case with a mobile device. So I have a few novel ideas about how we can bring the full power of the shell to fruition, without having to toggle between four keyboards in webshell -- not getting rid of webshell, but making shell usage easier and more predictive and helpful.

Finally, Inco's fundamentally about information, as I said before -- so come spring, I want to draw out a lot of the information we're collecting and do some really interesting stuff with mobile visualization of systems you manage. And of course stamping the hundreds of bugs that come with a 1.0 release.

Our readers can head over to getInco.com and they can potentially sign up to beta test?

That would probably be really helpful. They can sign up and I promise a personal reply if they have a question.