We are as concerned as our users about the repeating Yodlee issues. I feel that an elaborate explanation is appropriate, explaining why the service is unreliable and perhaps more importantly - why Yodlee will always be unreliable (if it won't - we won't need it). I will seal with our own strategic dilemma at ClearBooks and why we are between a rock and a hard place.
The ubiquitous way for online systems to communicate between them is using an API (application programming interface) - a standard protocol allowing them to do so. ClearBooks offers an API so other systems (eg, ChannelGrabber) can interface with it; ClearBooks connects to other systems using their API - most notably and importantly for this topic: PayPal.
Correct at 1st Jan 2014 (and I believe until this very day), not a single bank in the UK offers an API allowing online systems like ClearBooks (or Yodlee) to retrieve bank records from their clients or businesses accounts. There are two main reasons for this:
First, banks are big cooperates with complex structures, nearly all had the bulk of their history well before online systems existed (in stark contrast to ClearBooks or PayPal). Like many other big companies whose principle operation is not internet-based, most of their online technology is the outcome of outsourcing and agency staff - they have limited internal expertise and vision when it comes to online technology; the people in charge are often managing executives. They care a great deal for their customers (and pay fortune to get their online systems tick), but they have little interest in companies like ClearBooks - we are only a bridge to a small and specialised section of their customers. (would anyone pick a bank because it offers API and seamless integration with online systems? Would any of you leave bank X for bank Y, because bank Y has seamless integration with ClearBooks?)
The second reason is that Banks are a bit of security-mad; the very notion of anyone but a human-being, who is the account holder accessing bank records seems to them like a madness of a sort. Needless to say that if PayPal offers an API (allowing a secure, limited and read-only access), this can't be such an issue as banks make of it. Ironically, the very fact banks do not offer an API, means their security is, in practice, compromised (although only a little, for the reasons described below).
Had any bank offered an API, ClearBooks would interface with that bank directly, cutting the middle man - Yodlee. But no bank does, and that's where Yodlee comes into play.
Since like ClearBooks Yodlee can't use an API, they employ a different strategy called Scraping. Being slightly more advance than the crawlers Google use to feed their search engine, a Scraper is a piece of software that simulates the actions of a real person using a web site - so essentially, the scraper pretends to be a human, doing what humans do. In simple terms, you program it like so: go to Natwest.com; Click the login button; in field X enter the customer number; in the next page read the first letter to the left of the first field; if it says '4th' type the 4th digit of the password... and so on. Scrapers are not intelligent, they only do what a human programs them to. So for scrapers to work, the site they visit needs to be identical each and every time they visit it. It is enough for an online chat message to suddenly pop up, or for the bank to add another step in the login sequence requiring the user to confirm they read a message, and the scraper can no longer execute the next action, and so it terminates. To combat this, each scraper logs (for a short period) each page it visits, so if it terminates, a human can look at the last page in the page log, see what has changed and program the scraper to respond to such change in the future (shall it happen again). Problem is, they have no way to test the new program unless the bank site behaves exactly the same way it did last time the scraper visited it (which is only sometimes the case). In practice, pretty much any change on an online banking site breaks the scraper and requires re-programming that often cannot be tested. I can only guess that Yodlee gets thousands of scraper terminations each hour, so no human looks into the issues without a support request. When the scraper terminates, Yodlee fails, and you guys see this in ClearBooks as a failed import.
I don't know about you guys, but my online NatWest business account is inaccessible about 3 out of 4 times I try it (but I'm a night owl). When I do manage to login, boy oh boy... their system makes BT phone boxes look modern. You first have to request a statement, then wait an unknown period of time before it shows in some link on the side, it's the most unusable thing ever. It's hard enough for a human to use the site, I pity the scraper. Many other banking systems have their occasional glitches and downtimes - all stop the scraper from working.
So any change or a glitch on the bank site means Yodlee doesn't work. ClearBooks integration with Yodlee was hardly ever the source of the real problem (we did, should and would improve where we can, like providing better feedback when possible, but if Yodlee fails its bit, there's nothing we can do about it). Nor is Yodlee to blame - they took upon themselves an ambitious and brave challenge, and they really do all they can, often as quick as they can, to get things fixed. Nor the banks are to blame - the last thing on their mind is to make their sites scraper-friendly, and if anything, they would surely prefer offering an API than having machines pretending to be humans on their sites (which is less secure than an API solution since a scraper can in theory perform much more than read actions; and since a 3rd party company holds the credentials to your online banking account, which an API wouldn't; yet it has to be emphasised that Yodlee is highly secure).
As far as ClearBooks goes, it is sometimes better not to offer users something, than offer something that annoys them. As a user experience expert, Yodlee would be thrown out of the window long ago unless: A) Doing so would mean that we take away a feature that does work sometimes, and people do find useful despite the glitches. B) Similar companies in the market would offer it (they all get from Yodlee what we do, and when Yodlee fails, it fails for them the same as for us).
We are also comfortable with scrapers, as we use them internally to gather data on our own system (like find what pages has a particular component, etc. - quicker than asking a human to search through all the pages the ClearBooks app has). But us trying to replace Yodlee in attempt to scrap online banking sites would only mean we would have to deal with exactly the same problems as them, being less experienced; that's pretty much what Yodlee does - writes and fixes scrapers; our developers can deliver better value to our customers doing other things. Although cutting the middleman may mean faster fix times, the problems will always be there.
So I hope that explains it. While we actively investigate if there is anything we can do better, neither us nor Yodlee can make this work seamlessly. If anyone can propose a way to make things better - please please do so. The only thing I can think of that may solve this issue once and for all is if the accounting community in the UK together with ClearBooks and similar companies sign a petition and send it to the banks, asking for an API.