A Cautionary Tale: Do not move or rename your iCloud folder
"As if it were a swarm of bees, you should stay away from the SyncServices folder.' -- Apple
The moral of the story that I am about to tell is that Apple's advice about the SyncServices folder also applies to your iCloud documents. Here is the summary:
"Do not move your iCloud folder. Do not touch your iCloud folder. Consider it the digital equivalent of a hand-grenade which has had the pin pulled and which is resting safely on its handle." -- Me
This is a cautionary tale for the geeks and the tinkerers in the audience. I'm going to share my story of something I did, so that others might avoid this same situation. Learn from my mistake.
Before I begin, I want to make a few things very clear: 1) This is not something which could have happened accidentally. It happened because of something I did, not realizing the consequences. 2) This is not Apple's fault, it's mine. You should not read this story as an indictment of Apple and/or iCloud (nor Dropbox). This was my fault.
How I Got Into This Mess
"The road to hell is paved with good intentions." -- Unknown
A few months ago when I first started using iCloud for documents, I didn't trust it. Unlike Dropbox, there is no way to access my files from the web, nor is there a 30-day revision history. However, more and more apps were starting to use iCloud, so I decided that I would start using it.
"But," I thought, "I'm going to be clever about it."
Now, in my defense, the method that I was attempting is one which many geeks/nerds/tinkerers have used with Dropbox. It goes like this:
"Well, I want to have [some folder] on Dropbox, but it's over here, and my Dropbox folder is over there, so I'll just move the folder and then link it back to its original spot.
If you somehow started reading this article in the middle, let me repeat: Do not move your iCloud folder.
Your iCloud documents are saved in ~/Library/Mobile Documents. I am telling you this for several reasons. First, it's not a well-kept secret. Second, it is important to be able to discuss specifics.
I went into Terminal.app and moved ~/Library/Mobile Documents/ to ~/Dropbox/Library/Mobile Documents/ and then created a link from ~/Dropbox/Library/Mobile Documents/ to ~/Library/Mobile Documents/. Then I launched some iCloud-enabled app on my Mac and saved a document to iCloud. The file saved, so I assumed that everything was fine, and went on my merry way.
What Happened Next
One important thing to realize is that when this whole story started (back in June) there were some iCloud-enabled apps, but there were not a lot of them, at least not a lot of them that I was using. I used (and continue to use) Dropbox whenever possible, so I was only using iCloud if Dropbox wasn't available.
Looking at this all in hindsight, the problem seems obvious, but as I experienced it, the following things seemed like a series of seemingly unrelated events.
I launched an iCloud-enabled 3rd party app on my iPhone, and all of the data was gone. Fortunately it wasn't all that crucial, so its loss wasn't that big of a deal. Unfortunately, because the loss wasn't that big of a deal, I didn't stop to think about it, or try to figure out what had happened. Instead I brushed it off as "proof" that iCloud wasn't to be trusted. (Note: I believe the technical term for this is Confirmation Bias.)
Other iCloud-enabled apps on my iPad or iPhone occasionally complained about not being able to save or retrieve data from iCloud. Here again, the good news and bad news is that most of these apps are designed to do as much as they can in the background, as transparently as possible. They are also designed to deal with the reality that sometimes you may not have a network connection, or that network connect may not be very good. Most of the error warnings said little more than there was a "Connection Error" or a "Timeout". Unfortunately, my home Internet connection is through satellite, so I see these sorts of errors all the time and so I largely ignored them.
My iCloud contacts, calendars, reminders, bookmarks, notes, and Photo Stream all continued to work just fine. I took this to mean that iCloud as a whole was working fine. What I failed to understand is that "Documents & Data" on iCloud are completely different than those "built-in" features from Apple. In fact, I didn't really start to pay attention to the problem until Pages on my iPad stopped connecting to iCloud.
I don't use Pages all that often. Like many geeks, I tend to prefer text files, which manage with nvALT on my Mac and Simplenote. But one day I was trying to make a Pages document on my iPad and couldn't. Every time I tapped "New Document" and then chose a template, a progress indicator would spin briefly, and then leave me at the Template Chooser screen. On my Mac, I went to Pages and was able to open documents from "iCloud" just fine, but my iPad couldn't. More "evidence" that iCloud was flaky and not to be trusted. (See "Confirmation Bias" link above.)
I tried many things to try to fix the problem. I went to iCloud.com, downloaded my Pages files, and then deleted them from "the cloud" before trying Pages on my iPad again. I disabled iCloud on my iPad, created a new document in Pages (which worked fine) and then turned iCloud back on. Pages on my iPad asked if I wanted to use iCloud. I said yes. The test document I had created while iCloud was disabled disappeared. Then Pages crashed. I relaunched it, and it crashed again. I deleted Pages from my iPad, re-downloaded it, launched it again, and it crashed again. I disabled iCloud, and Pages worked.
By this time it was well over a month after I had moved my iCloud documents folder, and quite frankly I had forgotten that I had even done anything to it. Because iCloud had (seemed to) work fine when I first made the change, I had never stopped to think about it again after that. Again, the "move a folder to Dropbox and then link it back to its original location" technique has been used for a long time. I had no reason to think it wouldn't work here too, but it didn't.
When I realized I was in trouble
After having these little annoyances with iCloud documents for awhile, I finally decided to try to dig in and fix it this past week. I tried every solution that I could think of or that I found via web searches. I disabled iCloud documents on my Mac, and on my iOS devices. I disabled and re-enabled iCloud support in all of the various applications that I use, on the Mac as well as on my iPad and iPhone. I deleted iOS applications and re-downloaded/re-installed them. Nothing worked.
Finally I launched Day One on my iPhone, which gave me the most detailed error message I had seen:
I'll admit that most of that message is completely lost on me, but I did recognize one thing: the NSFilePath included "Dropbox" in the path, and that should not be there.
I'm not sure exactly what happened, but my theory is that either one of two things happened: 1) I did not turn off iCloud document syncing when I moved the ~/Library/Mobile Documents folder, which meant that as I was moving it it was trying to sync to iCloud, and/or 2) some or all of the iCloud apps determined that the "real path" was ~/Dropbox/Library/Mobile Documents (that is, they recognized that ~/Library/Mobile Documents was a Unix-style link in the path, and saved the actual path instead of the linked path.
It appears that from there, the path information was sent from from my Mac up to iCloud, where it was stored. From there, iCloud sent the path information back to my iOS devices and said "OK, you need to put this in /private/var/mobile/Dropbox/Library/Documents/." However, the iOS apps -- which are "sandboxed" to prevent them from creating files or folders anywhere except where they are very strictly allowed to -- either tried to create that path and failed, or never assumed that they would ever have to deal with a different path.
Again, those two paragraphs are my guesses as to what happened, based on what I have seen. Several smart people have looked at this and said "Wait, that's... not supposed to happen. How did that happen?"
All I know is that I can't fix the problem from my Mac, and nothing, even as far as restoring my iPad to factory conditions, will fix it on the iOS side.
Finally, in a fit of frustration, I created a brand new Apple ID and used that with all of my iCloud apps. Result: Everything worked fine.
Time to call AppleCare
My primary Apple ID is a former Mobile Me account which is paid through the end of September, and it already has my photostream, contacts, calendars, reminders, and everything else on it, so the idea of throwing it away and starting with a new Apple ID for iCloud was not all that appealing.
My iPhone, iPad, and MacBook Air are all covered by AppleCare, so I figured it was time to try to see if Apple could fix this. I called yesterday and spent over three hours on the phone with 3 different levels of Mac, iOS, and iCloud support. Everyone was extremely knowledgeable and helpful. They had me try several different things to try to fix it, but in the end, nothing worked. My case has been escalated to some sort of "iCloud Technical Support Grand Master." (Note: that may not be his/her actual title, but by the tone of voice used by the 3rd-tier AppleCare support person I was talking to, that's how I'm going to think of them.
As I imagine it, this is a guy in a room full of Mac Pros and Xserves. He's surrounded by boxes, papers, computers, connectors, and a coffee pot. The office has an adjoining bathroom. It is located at the far end of a building that no one ever goes to, and where few people even know it exists. There is no phone. There are no windows. He's never seen coming or going. People wonder what he eats). I am supposed to hear back from Apple in the next 3-5 days, after the Grand Master calls the guy I was working with, who will then call me. (The Grand Master does not deal directly with end users, obviously. Would you?)
Conclusion
A 3+ hour phone call with tech support which ends with "Well, we'll see what he can do. You don't have any files you need in there, right?" might seem like a bad tech support experience. But the good news is that because this happened at the very beginning of my iCloud use, I don't have any files I need in there. I can't get angry at Apple for this, because this really isn't a situation that they could have foreseen. And, as one of the AppleCare folks said to me yesterday (in a very friendly and kind-hearted way): "Yeah, that Library folder is hidden for a reason." There are lots of things that you shouldn't touch in there, but of all of them, I suspect the most "dangerous" one of them may be the Mobile Documents folder, because *once that syncs to iCloud, you can't access the "official" version. The "official" version lives somewhere on Apple's iCloud servers.
My biggest frustration with iCloud documents is that there is no way to access the "canonical" version of these files, like there is with Dropbox. If I have a problem with my calendars or contacts, I can go to iCloud.com and access them directly. I can edit them, export copies of them, or delete them. But I can't do that with my files. (While iTunes Match is a completely separate service from iCloud, the problem remains the same: no direct access to my files.)
My distrust of iCloud is what made me try to tinker with it in the first place. By trying to make it "more reliable" (that is, by having my iCloud documents in my Dropbox, so I would have immediate backups and 30 days of revision history), I ended up making iCloud entirely non-functional. Apple took reasonable precautions to protect me from myself by making the Library folder "invisible". The "Mobile Documents" folder isn't protected by an Access Control List against moving or deletion, but if it had been, I would have simply overridden it anyway.
I didn't have any reason to distrust iCloud, other than my previous experience with iDisk, which was a completely different product. In my (admittedly limited) testing with a separate iCloud account, iCloud sync seems to work very well, even when I tried to fool it by editing the same document on my iPad and Mac at the same time.
Be that as it may, whenever I am given the choice between syncing with Dropbox or iCloud, I will choose Dropbox every time, because my experience with Dropbox shows that it works reliably and I know that if Something Bad happens, I can access my files on their Web site. I do worry, however, that Mac/iOS app developers will decide that supporting one method of sync is challenging enough, and will opt for iCloud instead of Dropbox. Only time will tell.
However, if you learn nothing else today, learn this: the Mobile Documents folder is not to be trifled with. Leave it alone. Don't move or rename any files or folders you find in there. You can probably safely copy files from there, but I wouldn't do anything more than that.
