The way you present is makes Twitter look cold towards their eco-sphere — which is surprising given how many partners and clients they have; maybe they've tried and are coming back from it, but their success is heavily wired to that strategy, and they appear not too ignorant of that fact. Therefore, I would let them the benefit of the doubt and imagine they have good reasons to do what they seem to have decided. Money appeared tight recently, and saving on the few rather wasteful services (albeit the only way to have Guy's new followers, you method is a real bleeding).
Regarding the specific features that you want, I would make a distinction between: * new followers (an information users can get by e-mail, and that they might agree to forward to you, with a simple IMAP filter); and * people who stopped following a certain user.
As pointed out by the extensive literature, interrupted relations or more generally most negative news impair greatly a social service. It's the first question I get asked about Facebook or Twitter, and I can give you examples of services whose usage dropped days after implementing a look-back feature. By empowering Guy, you remove my ability (and some would say 'right') to stop being fed his every move without him knowing. Guy is a great guy, and he knows he twits too much, and probably wouldn't be offended. Others might like that with less detachment, and drama will not only happen, but get front-page treatment. That is threatening to Twitter because, with services such as yours and your competitor's, they cannot guarantee the same freedom to browse feeds.
Twitter, or Facebook who recently banned the Whooper Sacrifice application, do not want the balance of information tilt that way. Given Facebook massive lack of personnel, any of they move is most likely triggered by users. I do not think they decided to remove such a popular app without good reasons, namely users offended either by the fact that they were un-friended, or who forgot they had the application active and un-friended someone who took offense. It could be that non-important friends, aka 'weak-ties' are essential to their success (they are) but I doubt ten of those were worth the ban.
It's a matter of opinion, and would be hard to prove, but the success of a technology such as yours demands that you balance the right to know with the right to remain discreet; a majority of experts would agree with me.
Finally, I would love to have an alert saying that the recent user that friended me is apparently a spambot — provided you give the people behind those the ability to challenge your call. Twitter would love to have the ability to list those, and find out which one are useless, or do not respect the terms of services to ban them, and save some money; maybe they already do.
tychoish
· 11 months ago
A couple of things:
1. XMPP. if they provided XMPP pub/sub nodes for users feeds and data.... data could be pushed, if people were smart about things, it would scale better than repeated HTTP polling, and it would be awesome.
2. Open Microblogging, Laconica is compatible with the twitter api, provides XMPP *and* services could build implementations of open microblogging that let them subscribe to their clients via that network, and not have to deal with polling limits.
Jesse Stay
· 11 months ago
None of this helps getting a user's friends and followers, which is the heart of this issue.
tychoish
· 11 months ago
If microblogging were implemented as an XMPP service, your friends/follwers would be the "roster," (or abstracted as such) or as members of a multi-user-chat (or abstracted as such), and getting a list of users, would be a single request/stream from the server.
OMB doesn't directly solve the API problem, but it does so indirectly: no OMB server needs to "hold" the entire microblogging world (like twitter needs to), so the scaling problems that twitter has with its API limits aren't likely to be as much of a sore point. Secondly, you can set up a local/private OMB instance that would allow you to write your App/site against a "local" platform/site, rather than against a third party site...
---
Having said that, these kinds of problem where "crowd sourced" platforms/sites end up moving things in a direction that helps their business, but hurts the crowd, are *very frustrating* but also not surprising. Alex is a great guy, twitter is a very inspiring platform (that secedes because of the number of users, and the tools that Alex has helped provide for), but it's still a business that really has to serve it's own interest, and if this doesn't illustrate the worth and utility of open source, and the software freedom approach, then I don't know what would.
The heart of twitter's issue, is that when they thought they were providing a "status message service" they never really expected that they'd be creating what amounts to a public messaging and communications service; combined with the fact that non-niche web-services are incredibly difficult to monetize/collect subscriber fee's for...
Developing for closed platforms isn't a bad thing at all--often--but it does put developers completely at the mercy of another party, and that's pretty tough to stomach.
candrewswpi
· 4 months ago
I don't understand how you can make 2 completely contradictory statements in the *same sentence.* How can "Developing for closed platforms isn't a bad thing at all" be true, while "developers completely at the mercy of another party" (in other words, your business model is in someone else's hands) is also true?
tychoish
· 4 months ago
While original comment is 6 months old, and I'm not sure if the events of the past six months are worth anything (and I don't have the time at this moment to consider these implications), here's a clarification:
- I don't mean to present a black and white moral system on this issue. It can be both "not a sin," and also "not a good thing." It's not, I think, a transgression to develop for a closed platform in an open way, and also I think it's true that putting users/developers at the mercy of another party is something that should be avoided.
- Perhaps I should have said "isn't neccessarily a bad thing."
We have examples of people developing tools aplenty. Emacs/Cgywin/Notepad++ and so forth all exist for windows and people put a lot of time into that, and that's not bad, but I think everyone's a bit wary of that kind of work becasue of the way it requires them to have their work exist at the mercy of other people.
Or hell, I think the whole mono debate is contentious *because* of this tension...
Cheers, sam
Kevin
· 11 months ago
i asked alex about removing the rate limit. he asnwered that that their new year resolution. I was under impression that there wont be any rlimit on APIs.. though 20K is much more cmpared to existing 100 limits... however as applications grows and traffic increases 20K will look like a small figure
william
· 11 months ago
The Elephant in the room
If you are an Open Source Developer or Content creator (If you are a member of either service you are a content creator) you should not use Facebook or Twitter.
By using Facebook or Twitter you are essentially raising the value of their companies and applications. Both Facebook and Twitter are closed source content silos that do not allow you to control the content that you create. Neither Facebook or Twitter put the content creator/members at the top of their pyramids when thinking about revenue models. Each of these companies puts their Companies first above the members and communities that have given them value and money.
If you are a developer you may be able to make some money by creating applications for Facebook or Twitter ; but I do not believe that Facebook or Twitter will ever allow your application to eat into their user base or their revenue. Because they are both closed source companies that have the ability to literally cut you off by changing the code/api or by using their proprietary knowledge to build an application that you can not possibly compete with. As a coder understand that when you build and extend Facebook or Twitters propitiatory platforms that you undermine the longevity of the Open Internet.
Content owners and Developers do not help these closed source companies (Twitter and Facebook) in their goal of creating another closed source content trap that will extract hundreds of Millions on dollars from their member and developer communities and give nothing back in return.
Bertil
· 11 months ago
I'm not sure the competition is between closed and open-source services. It is an important aspect — but adoption of the service is the first step. Compatibility with other service is the only way out of a monopoly, and then, you have to argue in favor of open code.
I say that because the source behind Twitter is not the hardest asset to copy: leveraging a community of practice is. The people behind the Open Stack and DiSo realized that their arch-enemy, Facebook Connect, not only offers a better experience (thanks to good UI and market-base) but also drive massive press and attention to their solutions. Just like them, you need to push Twitter (and its equivalents/competitors) to learn about how to do micro-blogging properly, and let people learn about micro-blogging.
Cesare
· 11 months ago
Out of curiosity, how do you split requests and "process them concurrently"? Is there a way to make a cumulative requests to get the list of followers? Don't you have to make one request for each page of followers?
Anyway I think that relationship data (e.g. who follows who) are crucial for the next generation of microblogging applications. Twiitter and friends will have to address this issue sooner or later.
Christopher_Ross
· 11 months ago
I can appreciate the concerns and thank you for going through the numbers for your readers. However, I was hoping that somewhere within your post you would have offered an olive branch in the form of a payment structure you would fine reasonable, given the value twitter and their API offer to you.
What would you find a reasonable starting point?
Jesse Stay
· 11 months ago
Christopher, I haven't really thought of that yet, to tell you the truth. If we did pay for API access, we would have to pass it back to our users in some form in the form of subscription or advertising. How much would you be willing to pay? I guess that would be the determining factor.
Bertil
· 11 months ago
Rather then considering the hard-to-estimate value-added, you might want to consider costs to Twitter; this would make tour service neutral to them, and you'd be in a safer position later to negotiate the (then better known) value created by being the largest information network.
Otherwise, advertising might be the better solution (like spam-filtering, your service is valuable, but hard to put a price on). Were would you be able to put some?
Christopher_Ross
· 11 months ago
Well, I don't know. But I think that this (throttling API calls) will cause a good disruption in the twitter application community. If a developer is not able to find an audience to support their product, they'll either have to dig deep and cover their costs until they do, or rethink the relevance of their product in the market.
The bar gets set higher, and I think we're all the better for it as both developers and end users.
Stiennon
· 11 months ago
This is crazy. Thanks for publicizing it. Twitter could make their life a lot easier by following your (and Andy's) advice and make the API's more like their own database queries.
I thought Twitter was already on AWS.
-Stiennon
F. Andy Seidl
· 11 months ago
20,000 / hr is not the crux of the problem. The real problem is that Twitter should provide more powerful API methods. For example, if there were an API method such as getFollowersSince(timestamp) you could retrieve Guy Kawasaki's new followers in a single calls rather than having to make 350 calls. Better yet, a method such as getFollowerChangesSince(timestamp) would let you get all new and followers AND discontinued followers in a single call.
By offering only very primitive API methods, Twitter forces developers to make large number of calls to get a job done and this, as you point out, does not scale. But the answer is not unlimited access because that does not scale either. As Twitter grows in importance, it must offer more powerful API methods like this to enable continued, large-scale adoption by application developers such as yourself.
Jesse Stay
· 11 months ago
Andy, exactly. I've discussed this with Alex and they have no intentions in improving that however.
jonconley
· 11 months ago
Just want to second the fact this would be a null issue if the API was improved. I agree with you both and the more the API lacks, the more things like this will continue to creep up.
abrudtkuhl
· 11 months ago
Great post Jesse.. I think API limits are BS and you are exactly right - this pretty much limits any growth for developers building on top of the Twitter's API.
Mihai Secasiu
· 11 months ago
isn't this limit per account. And if it is then why not use your user's credentials to do the api requests. If it's a per ip limit just get more ips, they are relatively cheap . Of course this is a short term solution, they are probably going to charge for extra api requests. But you should have seen this coming, it's impossible to handle any number of api requests from anyone. there's no such thing as scaling a system to infinite and for free too.
Jesse Stay
· 11 months ago
Mihai that won't work for a Scobleizer or Guy Kawasaki, or anyone with over 1,000 followers because we exceed their rate-limit just by going through their friends. It's per IP, not account.
I wish they would charge for API requests - imposing a limit is not the solution. Facebook has no request limits. FriendFeed has no request limits. If I want to grow, why should I build on top of Twitter now?
Mihai Secasiu
· 11 months ago
ok, I see you have a real problem now. my guess is this is either a temporary problem and it will be fixed once they manage to upgrade their system to scale better or it's the first step to paid API access
Jesse Stay
· 11 months ago
Mihai I suggest it's neither. I think it's just a dumb move that they think will fix too many requests on their servers. I've talked to them privately about this and they have made no hint at either being the case. If so I would welcome a paid model - I just wish they would implement it before imposing the rate limit. It makes no sense why they wouldn't.
sephillips
· 11 months ago
Is there a chance they will announce a premium fee based API service with rate tiers? Or perhaps your software could become a client side app?
Jesse Stay
· 11 months ago
Our software is based on server-based automation. Installing on the client isn't possible. And I don't see how this fixes the underlying issue Twitter has of too many requests. TweetDeck violates this, but they won't be punished because they're spread out across many IPs. I don't see why they should be treated any differently.
strangedesign
· 11 months ago
Agreed the API limits are just a band-aid on poor API development. But can you expand on who TweetDeck violates this because technically they don't. They aren't being punished because they don't tax the system (as bad as it is) anymore than a normal user can do on their own. Lucky, maybe, but violators I don't think so.
Jesse Stay
· 11 months ago
strangedesign, I'm referring to the fact that they are a single app, and while not a single IP, across multiple machines they are putting as much load, if not more, as any of us on the Twitter servers. I don't want TweetDeck to go away, but I'm stating that to show that this limit is ridiculous.
strangedesign
· 11 months ago
See where I differ slightly from your thinking is while yes they are a single app they are not using their own API account. They use MY API calls which limit me the individual user. If I choose to have a lot of groups in the client that is my choice. I have to decide how to use my 100/hr and I decide to give them to tweetdeck that is my choice. Where as Socialtoo (disclosure I am a user of SocialToo and LOVE IT) is is using a DEV account. Not saying it is right, but the spirit of the change from twitter IMHO is not about limiting applications, it is about resource preservation pure and simple.
Now I have worked on large implementations (talking millions of concurrent users) and Yes they need to fix their API but also they need to protect themselves from scriptkiddies and hacks who don't know how to code and therefore use up too many API calls. just my $0.02.
Jesse Stay
· 11 months ago
Chris, I agree with you that they need to protect their system, but this is not the solution. It kills SocialToo's auto-follow and stats services, along with many others you love. It completely cripples any app needing to base on a server from growing to its potential.
strangedesign
· 11 months ago
But what if that limit is what get's them through Inauguration and MacWorlds, etc? I would think keeping Jesse and his 10K users being able to auto-follow/unfollow/DM running is not their concern. Keeping their system running is what matters. At this point the only thing you can really be firm about is the communication and timing of this change is BS. Whether it is justified, the right thing to do, etc can't be answered without more information.
Jesse Stay
· 11 months ago
Chris, so, again, why would any developer in their right mind develop on Twitter if their apps can't grow to their potential? Twitter's own instability cripples developers based on your logic.
strangedesign
· 11 months ago
Absolutely is does. They are treating a symptom and not a cause but it is still probably what is driving the decision IMHO. A lot cheaper and quicker to limit the API, slow growth, and piss of what is a relatively small number of users than add unlimited capacity. Bad for business yes. Will it hurt them in the long run, who knows.
Scott C. Lemon
· 11 months ago
But Jesse ... of course installing on the client is possible! There are several solutions to work around the limits, and then carefully enroll the folks at twitter about the benefits of updating and enhancing their API.
Whining and complaining will probably not work ... :-(
abrudtkuhl
· 11 months ago
Good point - Maybe this is a push towards a premium API service
Jesse Stay
· 11 months ago
I don't see any hint towards that. If so, they should charge first rather than impose a hard limit. Let the market determine how many requests get made to their servers.
abrudtkuhl
· 11 months ago
Agreed completely... API limits are BS - especially for twitter where most of these apps are HUGE added value to the service and are a big reason for their continued growth.
There is no reason now that they have Bezos and AWS architecture behind them that they should impose rate limits.
Thom Allen
· 11 months ago
I agree with abrudtkuhl first post. A premium service is needed. You can't expect a business to provide a free service that lets YOU build a business for FREE, and get upset when they lookout for themselves. Jesse I know you would make some changes if you were in their shoes. Maybe not the same choice, but you would protect you business first.
I mean come on, Twitter was never meant to be an API for someone to build a business on. I like a lot of the services that use the Twitter API, but I certainly can't blame them. Maybe a discussion with the development community should have taken place first, just to gage the acceptance level, but you really don't know where they sit.
Mostly I think we should work with them instead of building a negative campaign against them.
Jesse Stay
· 11 months ago
Thom, as I mentioned in the article, this discussion has gone on far before I posted this article. Alex has made it fully clear they don't intend to budge on this rate limit. They're just finally making it official. He has also made it clear they don't intend to make it any more easy to find out the followers or friends a user has.
You bet I have plans around this, but it involves much less Twitter and much more of their competitors. I would be willing to pay (although the cost would have to be passed to our users) if Twitter charged for this. Charge for the API first, then let that throttle API requests. Don't just cut off your developers after giving them full access and telling them they are allowed to do what they want in the parameters you gave them.
Thom Allen
· 11 months ago
Yeah, it probably wasn't a good idea to pull the carpet that fast, and like I said, Twitter should have engaged the development community first, that would have made the transition easier. Could have been handled a little better.
But what do you think about external services causing issues with the overall service that affects the average person who just uses Twitter. Should they suffer because companies want complete and unfettered use of the API and it's resources? Doesn't sound too fair too me. I understand you position on timing, they have more to worry about than external services.
shinerweb
· 11 months ago
How long before they start charging for API access? Have to agree with you in that they do need to fix their "creaking" API. The cynic in me does wonder if this is a method of limiting others from building success "out of" the success of Twitter itself. Possibly very damaging to all Twitter based apps and not good news at all.
Chris
Jesse Stay
· 11 months ago
Shineweb, I don't see any hint towards them charging in any manner. I wish they would if too many requests are the issue so we aren't limited in growth. Make us figure out how to pay for the bandwidth, don't cut off our heads.
shinerweb
· 11 months ago
Totally agree with you again. That was the cynic within me again wondering (and only wondering since I haven't seen evidence) if in future they would offer a "paid for - unlimited/controlled!" access. As you say, it would make much more sense than simply slamming the door shut.
They will also be doing themselves no favours either. Were it not for apps like SocialToo, Twitter doesn't have a fraction of the appeal it does at the moment because of those apps. (Cynic head on again, perhaps developing their own 3rd party apps is on the agenda?)
Chris
Matik72
· 11 months ago
This does seem a bit backwards. Did they give any more explanation as to their rationale?
Jesse Stay
· 11 months ago
The rationale is they think we're making too many requests on their servers. My rationale is that limits our growth then.
theretrometro
· 11 months ago
Maybe Twitter is trying to "shake" the apps that help those of us who are trying to truly get folks who WANT to follow us and WANT serious followers. I'm noticing a surge in porn and dating tweets. If Twitter can't come up with a better way to get rid of this type of traffic outside of "block", (even though this is a free service), I may be better off with one of it's competitors.
I have noticed a difference in my "follows" since I joined ST. I no longer waste time doing my own deletions of folks who simply are looking for "temporary follow time" and "numbers" as they are auto deleted the minute they "unfollow". Since the porn/dating stuff is more recent, I'm hoping to notice a reduction over the next couple of days of this traffic.
Eventually, I would like to see some sort of statistical information on why others "unfollow" me. It will help me learn if I've posted something potentially offensive or if I've just lost their interest.
Jesse Stay
· 11 months ago
Theretrometro, I'd like to provide that for you, too. I think we're making Twitter a better place with the services we provide, but it won't last long if this rate limit remains in place.
DShan
· 11 months ago
this is an interesting development in light of the uncertainty as to twitter's business model. i can understand their intent to try and disincentivize automation of the twitter experience but i think you're right in suggesting they work on making applications like yours viable by tweaking their API. i wonder if they want more of these types of 'services' under their roof, eventually, which i might suggest if i were on the board looking for a way to make money on the whole venture.
Matik72
· 11 months ago
I am wondering if this is not a move towards a "pay to play" revenue model?
Jesse Stay
· 11 months ago
Matik72, you can ask Alex on the dev list, but he has made no hint of that. I would welcome it if they offered it. Imposing limits before imposing a pay-to-play model makes no sense though. They may as well impose the pay-to-play and let the market decide traffic to their servers.
Erik Giberti
· 11 months ago
Seems like there could be a layer of caching that you can do that would alleviate at least some of the pains.
Yes it limits your potential growth should every twitter user decide to use the service, but once you've scraped the 20,000 accounts associated with Scoble/Pirillo or others, you can cache that information for a few hours. Then you can query your internal caches as many times as needed. While the auto follow/un-follow won't be as real time, it will still work and Twitter's API is happier as a result.
I think it's time to get creative with the solutions. I don't expect complaining will solve much.
Erik
Jesse Stay
· 11 months ago
Erik, I believe we've exhausted about every resource we can. We do cache much of the data as-is. The fact is, there's a hard limit that stops any Twitter App from reaching their potential.
Chris Charabaruk
· 11 months ago
Limiting the growth of the services which are based on Twitter will ultimately limit Twitter itself, unless Twitter plans to provide those services itself. But I highly doubt it will come to that. This is a ridiculously stupid act of self-strangulation.
F. Andy Seidl
· 11 months ago
I do not classify it as ridiculously stupid to limit API calls to 20K/hour. In fact, speaking as a web service provider, I think that is a very generous limit. However, I would consider it foolish if Twitter did not introduce more powerful/efficient API methods. That would indeed limit their ability to grow.
Chris Charabaruk
· 11 months ago
Jesse's math in the article shows why I think 20K is stupid and unreasonable. All it takes is the likes of Scoble or another power user to max out a service for an entire hour. Would you like it if a service you made could only deal with one customer per hour because of changes by a third party?
F. Andy Seidl
· 11 months ago
>>Would you like it if a service you made could only deal with one customer per hour because of changes by a third party?<< Of course not, but that is not what I am suggesting.
What am suggesting is that the ideal approach is not to raise the limit on the number of requests per hour, but raise the limit on how much work each request can accomplish.
For many reasons, it would be far more efficient to perform many operation right at the service (i.e., at Twitter) rather than by requiring the client to do it in hundreds of bits and pieces. If you had to unload a pick-up truck full of sand, it would be very slow going if you were forced to make thousands of trips with a teaspoon. Far better would be to make a few dozen trips with a wheelbarrel.
From Twitter's point of view, they are allowing each service to make 5.5 requests per second on average, 24 hours per day. That is actually very generous, IMO, for a free service. By offering smarter API methods, they could increase by orders of magnitute the work that a client application could accomplish in the same number of request. And in many cases, this would also result in LESS CPU time AND LESS bandwidth. E.g., getting Guy K.'s new subscribers would result in transfering a relatively short list of results compared to transfering his entire subscriber list every time. It is also likely to take less CPU time (in aggregate) on Twitter's end by eliminating all the overhead of hundreds of API invocations, credential verification, etc.
Jesse Stay
· 11 months ago
Andy, I agree - that would make things magnitudes easier and more possible.
Mark Maximum
· 11 months ago
This might be step one of the 'making money by end of the year' plan. One way to do this could be to start building decent features for the power users (twitter lacks, well, everything, and they know it) - but before that they need to gently wind down the tens of services that proliferated around these missing features (doing it after would be a pr disaster). It's easy to see how socialtoo could solve the problem (at least for a good while) by disabling the services for who has more than, say, 500 followers. The problem with that, even if just as a very temporary measure, is that any possible revenue is likely to come from those very people - the twitter broadcasters. This is true for socialtoo as for any other twitter-based service - and twitter itself; so I think it's quite unlikely they will ever lift the usage cap or address those API issues. Not because it's difficult, not because of design decisions - but because of *business* decisions.
Jason Egan
· 11 months ago
This does seem like a rather short-sited thing to do on the part of Twitter. What if Apple had told people that they could not build accessories designed for the iPod? Not that they could do that, but you see where I'm going. Twitter is limiting the growth potential.
It's my guess that Twitter is doing this so that they can force users down a certain path that will correspond with their future plans for monetization.
Chris
· 11 months ago
I agree that the API limit is going to hurt larger developments, but then again we don't know the true extent Twitters servers are getting hammered for at present. Considering that this API limit has come after the Inauguration then perhaps they have made the change for a valid reason purely to protect their own servers and to keep Twitter running smoothly for it's users. Of course, paid API access would be a good solution however a paid API access solution is not something that can be implemented in a few days/weeks. If the servers are getting hammered hard right now to the extent that it's causing problems/affecting performance then I can understand the limit. I'd certainly rather see a limit imposed rather than have my Twitter experience ruined by slowdowns and downtime due to overloading.
Just because there are no hints towards a paid API model, or towards improving the API doesn't mean it's not going to happen surely? I would imagine that Twitter are aware of how important the development community is and in the long term I seriously doubt they would want to cut that off. At the same time, they also have a business to run and interests to protect and as such they are likely trying not to give away too much information about what they are doing. I'd imagine they have a lot of ideas and changes to implement which they are keeping tight lipped about until it's been finalised.
Just my 2c anyway. I guess only time will tell!
warzabidul
· 11 months ago
Whilst out on a walk today I was thinking about twitter and how it behaves. A year and a half ago they were struggling with the amounts of tweets people were both sending and receiving. They changed many things around and for a company that has been around for as long as they have (in startup terms) they have done very little to personaly improve the usability of their site. In fact since I started using twitter the biggest change is the replies arrow to avoid typing a name.
Where this will hurt them most is on mobile devices. If you're at a desk you can visit the page and although not as convenient it's not that much of a hinderance. Come to mobile devices though and you see how bad it gets. Several times recently I have hit the api limit on mobile 3rd party apps. As a result I simply didn't tweet as much as I would.
I am moving towards friendfeed, more intersted in more dynamic websites than twitter.
Scoble was speaking about the new twitter for friendfeed api to import friends and how, due to his vast number of followers he was running into problems.
As a side note I must admit to skimming through the post as I am not a developer.
David Meerman Scott
· 11 months ago
Your service is valuable. I, for one, would be willing to pay if it comes to that in order to weed out some of your users. Not ideal, but...
dccrowley
· 11 months ago
I'm blogging this post Jesse. It's bad news for you and for me (a socialtoo user). But even other services like Seesmic could be affected. It's going to be painful and widespread if this goes ahead. I understand twitter ... kind of. But imagine being able to make only 2 phone calls a day! In the end it just does not work. It will limit the service... and get Dave Winer back on his hobby horse, decentralized twitter which Bezos will hate.
It will do damage that's for sure. For instance I see the BBC following the inauguration yesterday and peoples reactions. The have loads of twitter quotes. So the media is waking up to the concept of 'the back channel'. Except when you are a BBC, NBC etc. and you want to do stuff! 20.000 requests should be peanuts I hope.
Jesse Stay
· 11 months ago
Thanks for the mention! This is important stuff, for sure.
bpm140
· 11 months ago
Jesse -- no idea if this will help your current bind, but if there are sections of your app that just rely on public twitter data, then Gnip can help tremendously by sending notifications when there is new data to be grabbed. This keeps you from wasting polls on endpoints that have not changed.
Eric, unfortunately the data I'm most interested in are the followers and friend data. I don't think Gnip provides that in any open feed does it? I'd love to talk if it does.
bpm140
· 11 months ago
Yeah, we're not a solution for you in the short term. We're rolling out distributed polling fairly soon, but it will be a while before we release *authenticated* distributed polling.
@JoeHobot
· 11 months ago
I guess my twitter mobile app is going to change too! that's bad news!
How many requests per hour do you make from a single IP?
@JoeHobot
· 11 months ago
I thought this is not "by IP" but over all 20.000 requests ( but to answer your question between 50-200) i would say. now multiply that by 1000 or 2000 users (not that I have that many) I would be screwed too!
I feel bad for SocialToo but I do feel bad for Twitter as well, they dont have any revenue income yet from "Ads" yet their servers suck big time. They could of at least hire nice CCNP to clear some data
jsinkeywest
· 11 months ago
Twitter to me is a bunch of snob losers that want you to check out their HOT thing and they never have any intention of reciprocating,if Google bought Twitter and closed it down I wouldn't miss a beat. 2009 is going to be rough people are to picky and don't appreciate JACK sadly like a old prostitute society on the internet will use you abuse you and walk away :) Despite all that :) It's great to see a QUALITY blog that is Do follow,mine is also and I'm on the LOOKOUT to network with cool people. If you know what I mean Bring IT :) Nice blog Stumbled
david
· 7 months ago
Using smart caching locally can help.
Jesse Stay
· 7 months ago
David, only to an extent. The only solution I've been able to come up with is scale to lots of servers, which I'm doing now.
sikis
· 6 months ago
platforms/sites end up moving things in a direction that helps their business, but hurts the crowd, are *very frustrating* but also not surprising. Alex is a great guy, twitter is a very inspiring platform (that secedes because of the number of users, and the tools that Alex has helped provide for), but it's still a business that really has to serve it's own interest, and if this doesn't illustrate the worth and utility
Jeff Donchez
· 5 months ago
yeah, now it's down to 150 requests per hour.. my client's twitter account has 11000+ friends and 11000+ followers.. I can't even do an api request to get their names (just their ids) because trying to get their names from the list of ips gives me the "over the limit" error
Jhonny
· 4 months ago
You guys should check out this twitter app www.sponsoraloser.com It's about posting twitter losers and finding them the answer to their problems, It's very funny! and It's all about helping others.
All Comments
Regarding the specific features that you want, I would make a distinction between:
* new followers (an information users can get by e-mail, and that they might agree to forward to you, with a simple IMAP filter); and
* people who stopped following a certain user.
As pointed out by the extensive literature, interrupted relations or more generally most negative news impair greatly a social service. It's the first question I get asked about Facebook or Twitter, and I can give you examples of services whose usage dropped days after implementing a look-back feature. By empowering Guy, you remove my ability (and some would say 'right') to stop being fed his every move without him knowing. Guy is a great guy, and he knows he twits too much, and probably wouldn't be offended. Others might like that with less detachment, and drama will not only happen, but get front-page treatment. That is threatening to Twitter because, with services such as yours and your competitor's, they cannot guarantee the same freedom to browse feeds.
Twitter, or Facebook who recently banned the Whooper Sacrifice application, do not want the balance of information tilt that way. Given Facebook massive lack of personnel, any of they move is most likely triggered by users. I do not think they decided to remove such a popular app without good reasons, namely users offended either by the fact that they were un-friended, or who forgot they had the application active and un-friended someone who took offense. It could be that non-important friends, aka 'weak-ties' are essential to their success (they are) but I doubt ten of those were worth the ban.
It's a matter of opinion, and would be hard to prove, but the success of a technology such as yours demands that you balance the right to know with the right to remain discreet; a majority of experts would agree with me.
Finally, I would love to have an alert saying that the recent user that friended me is apparently a spambot — provided you give the people behind those the ability to challenge your call. Twitter would love to have the ability to list those, and find out which one are useless, or do not respect the terms of services to ban them, and save some money; maybe they already do.
1. XMPP. if they provided XMPP pub/sub nodes for users feeds and data.... data could be pushed, if people were smart about things, it would scale better than repeated HTTP polling, and it would be awesome.
2. Open Microblogging, Laconica is compatible with the twitter api, provides XMPP *and* services could build implementations of open microblogging that let them subscribe to their clients via that network, and not have to deal with polling limits.
heart of this issue.
OMB doesn't directly solve the API problem, but it does so indirectly: no OMB server needs to "hold" the entire microblogging world (like twitter needs to), so the scaling problems that twitter has with its API limits aren't likely to be as much of a sore point. Secondly, you can set up a local/private OMB instance that would allow you to write your App/site against a "local" platform/site, rather than against a third party site...
---
Having said that, these kinds of problem where "crowd sourced" platforms/sites end up moving things in a direction that helps their business, but hurts the crowd, are *very frustrating* but also not surprising. Alex is a great guy, twitter is a very inspiring platform (that secedes because of the number of users, and the tools that Alex has helped provide for), but it's still a business that really has to serve it's own interest, and if this doesn't illustrate the worth and utility of open source, and the software freedom approach, then I don't know what would.
The heart of twitter's issue, is that when they thought they were providing a "status message service" they never really expected that they'd be creating what amounts to a public messaging and communications service; combined with the fact that non-niche web-services are incredibly difficult to monetize/collect subscriber fee's for...
Developing for closed platforms isn't a bad thing at all--often--but it does put developers completely at the mercy of another party, and that's pretty tough to stomach.
- I don't mean to present a black and white moral system on this issue. It can be both "not a sin," and also "not a good thing." It's not, I think, a transgression to develop for a closed platform in an open way, and also I think it's true that putting users/developers at the mercy of another party is something that should be avoided.
- Perhaps I should have said "isn't neccessarily a bad thing."
We have examples of people developing tools aplenty. Emacs/Cgywin/Notepad++ and so forth all exist for windows and people put a lot of time into that, and that's not bad, but I think everyone's a bit wary of that kind of work becasue of the way it requires them to have their work exist at the mercy of other people.
Or hell, I think the whole mono debate is contentious *because* of this tension...
Cheers,
sam
If you are an Open Source Developer or Content creator (If you are a member of either service you are a content creator) you should not use Facebook or Twitter.
By using Facebook or Twitter you are essentially raising the value of their companies and applications. Both Facebook and Twitter are closed source content silos that do not allow you to control the content that you create. Neither Facebook or Twitter put the content creator/members at the top of their pyramids when thinking about revenue models. Each of these companies puts their Companies first above the members and communities that have given them value and money.
If you are a developer you may be able to make some money by creating applications for Facebook or Twitter ; but I do not believe that Facebook or Twitter will ever allow your application to eat into their user base or their revenue. Because they are both closed source companies that have the ability to literally cut you off by changing the code/api or by using their proprietary knowledge to build an application that you can not possibly compete with. As a coder understand that when you build and extend Facebook or Twitters propitiatory platforms that you undermine the longevity of the Open Internet.
Content owners and Developers do not help these closed source companies (Twitter and Facebook) in their goal of creating another closed source content trap that will extract hundreds of Millions on dollars from their member and developer communities and give nothing back in return.
I say that because the source behind Twitter is not the hardest asset to copy: leveraging a community of practice is. The people behind the Open Stack and DiSo realized that their arch-enemy, Facebook Connect, not only offers a better experience (thanks to good UI and market-base) but also drive massive press and attention to their solutions. Just like them, you need to push Twitter (and its equivalents/competitors) to learn about how to do micro-blogging properly, and let people learn about micro-blogging.
Is there a way to make a cumulative requests to get the list of followers?
Don't you have to make one request for each page of followers?
Anyway I think that relationship data (e.g. who follows who) are crucial for the next generation of
microblogging applications. Twiitter and friends will have to address this issue sooner or later.
What would you find a reasonable starting point?
If we did pay for API access, we would have to pass it back to our users in
some form in the form of subscription or advertising. How much would you be
willing to pay? I guess that would be the determining factor.
Otherwise, advertising might be the better solution (like spam-filtering, your service is valuable, but hard to put a price on). Were would you be able to put some?
The bar gets set higher, and I think we're all the better for it as both developers and end users.
I thought Twitter was already on AWS.
-Stiennon
By offering only very primitive API methods, Twitter forces developers to make large number of calls to get a job done and this, as you point out, does not scale. But the answer is not unlimited access because that does not scale either. As Twitter grows in importance, it must offer more powerful API methods like this to enable continued, large-scale adoption by application developers such as yourself.
improving that however.
If it's a per ip limit just get more ips, they are relatively cheap .
Of course this is a short term solution, they are probably going to charge for extra api requests.
But you should have seen this coming, it's impossible to handle any number of api requests from anyone. there's no such thing as scaling a system to infinite and for free too.
1,000 followers because we exceed their rate-limit just by going through
their friends. It's per IP, not account.
I wish they would charge for API requests - imposing a limit is not the
solution. Facebook has no request limits. FriendFeed has no request limits.
If I want to grow, why should I build on top of Twitter now?
my guess is this is either a temporary problem and it will be fixed once they manage to upgrade their system to scale better
or it's the first step to paid API access
will fix too many requests on their servers. I've talked to them privately
about this and they have made no hint at either being the case. If so I
would welcome a paid model - I just wish they would implement it before
imposing the rate limit. It makes no sense why they wouldn't.
isn't possible. And I don't see how this fixes the underlying issue Twitter
has of too many requests. TweetDeck violates this, but they won't be
punished because they're spread out across many IPs. I don't see why they
should be treated any differently.
while not a single IP, across multiple machines they are putting as much
load, if not more, as any of us on the Twitter servers. I don't want
TweetDeck to go away, but I'm stating that to show that this limit is
ridiculous.
Now I have worked on large implementations (talking millions of concurrent users) and Yes they need to fix their API but also they need to protect themselves from scriptkiddies and hacks who don't know how to code and therefore use up too many API calls. just my $0.02.
not the solution. It kills SocialToo's auto-follow and stats services, along
with many others you love. It completely cripples any app needing to base on
a server from growing to its potential.
Twitter if their apps can't grow to their potential? Twitter's own
instability cripples developers based on your logic.
Whining and complaining will probably not work ... :-(
than impose a hard limit. Let the market determine how many requests get
made to their servers.
There is no reason now that they have Bezos and AWS architecture behind them that they should impose rate limits.
I mean come on, Twitter was never meant to be an API for someone to build a business on. I like a lot of the services that use the Twitter API, but I certainly can't blame them. Maybe a discussion with the development community should have taken place first, just to gage the acceptance level, but you really don't know where they sit.
Mostly I think we should work with them instead of building a negative campaign against them.
I posted this article. Alex has made it fully clear they don't intend to
budge on this rate limit. They're just finally making it official. He has
also made it clear they don't intend to make it any more easy to find out
the followers or friends a user has.
You bet I have plans around this, but it involves much less Twitter and much
more of their competitors. I would be willing to pay (although the cost
would have to be passed to our users) if Twitter charged for this. Charge
for the API first, then let that throttle API requests. Don't just cut off
your developers after giving them full access and telling them they are
allowed to do what they want in the parameters you gave them.
But what do you think about external services causing issues with the overall service that affects the average person who just uses Twitter. Should they suffer because companies want complete and unfettered use of the API and it's resources? Doesn't sound too fair too me. I understand you position on timing, they have more to worry about than external services.
Have to agree with you in that they do need to fix their "creaking" API.
The cynic in me does wonder if this is a method of limiting others from building success "out of" the success of Twitter itself.
Possibly very damaging to all Twitter based apps and not good news at all.
Chris
they would if too many requests are the issue so we aren't limited in
growth. Make us figure out how to pay for the bandwidth, don't cut off our
heads.
That was the cynic within me again wondering (and only wondering since I haven't seen evidence) if in future they would offer a "paid for - unlimited/controlled!" access.
As you say, it would make much more sense than simply slamming the door shut.
They will also be doing themselves no favours either. Were it not for apps like SocialToo, Twitter doesn't have a fraction of the appeal it does at the moment because of those apps.
(Cynic head on again, perhaps developing their own 3rd party apps is on the agenda?)
Chris
My rationale is that limits our growth then.
I have noticed a difference in my "follows" since I joined ST. I no longer waste time doing my own deletions of folks who simply are looking for "temporary follow time" and "numbers" as they are auto deleted the minute they "unfollow". Since the porn/dating stuff is more recent, I'm hoping to notice a reduction over the next couple of days of this traffic.
Eventually, I would like to see some sort of statistical information on why others "unfollow" me. It will help me learn if I've posted something potentially offensive or if I've just lost their interest.
Twitter a better place with the services we provide, but it won't last long
if this rate limit remains in place.
I would welcome it if they offered it. Imposing limits before imposing a
pay-to-play model makes no sense though. They may as well impose the
pay-to-play and let the market decide traffic to their servers.
Yes it limits your potential growth should every twitter user decide to use the service, but once you've scraped the 20,000 accounts associated with Scoble/Pirillo or others, you can cache that information for a few hours. Then you can query your internal caches as many times as needed. While the auto follow/un-follow won't be as real time, it will still work and Twitter's API is happier as a result.
I think it's time to get creative with the solutions. I don't expect complaining will solve much.
Erik
much of the data as-is. The fact is, there's a hard limit that stops any
Twitter App from reaching their potential.
Of course not, but that is not what I am suggesting.
What am suggesting is that the ideal approach is not to raise the limit on the number of requests per hour, but raise the limit on how much work each request can accomplish.
For many reasons, it would be far more efficient to perform many operation right at the service (i.e., at Twitter) rather than by requiring the client to do it in hundreds of bits and pieces. If you had to unload a pick-up truck full of sand, it would be very slow going if you were forced to make thousands of trips with a teaspoon. Far better would be to make a few dozen trips with a wheelbarrel.
From Twitter's point of view, they are allowing each service to make 5.5 requests per second on average, 24 hours per day. That is actually very generous, IMO, for a free service. By offering smarter API methods, they could increase by orders of magnitute the work that a client application could accomplish in the same number of request. And in many cases, this would also result in LESS CPU time AND LESS bandwidth. E.g., getting Guy K.'s new subscribers would result in transfering a relatively short list of results compared to transfering his entire subscriber list every time. It is also likely to take less CPU time (in aggregate) on Twitter's end by eliminating all the overhead of hundreds of API invocations, credential verification, etc.
possible.
It's easy to see how socialtoo could solve the problem (at least for a good while) by disabling the services for who has more than, say, 500 followers. The problem with that, even if just as a very temporary measure, is that any possible revenue is likely to come from those very people - the twitter broadcasters. This is true for socialtoo as for any other twitter-based service - and twitter itself; so I think it's quite unlikely they will ever lift the usage cap or address those API issues. Not because it's difficult, not because of design decisions - but because of *business* decisions.
It's my guess that Twitter is doing this so that they can force users down a certain path that will correspond with their future plans for monetization.
Just because there are no hints towards a paid API model, or towards improving the API doesn't mean it's not going to happen surely? I would imagine that Twitter are aware of how important the development community is and in the long term I seriously doubt they would want to cut that off. At the same time, they also have a business to run and interests to protect and as such they are likely trying not to give away too much information about what they are doing. I'd imagine they have a lot of ideas and changes to implement which they are keeping tight lipped about until it's been finalised.
Just my 2c anyway. I guess only time will tell!
Where this will hurt them most is on mobile devices. If you're at a desk you can visit the page and although not as convenient it's not that much of a hinderance. Come to mobile devices though and you see how bad it gets. Several times recently I have hit the api limit on mobile 3rd party apps. As a result I simply didn't tweet as much as I would.
I am moving towards friendfeed, more intersted in more dynamic websites than twitter.
Scoble was speaking about the new twitter for friendfeed api to import friends and how, due to his vast number of followers he was running into problems.
As a side note I must admit to skimming through the post as I am not a developer.
It will do damage that's for sure. For instance I see the BBC following the inauguration yesterday and peoples reactions. The have loads of twitter quotes. So the media is waking up to the concept of 'the back channel'. Except when you are a BBC, NBC etc. and you want to do stuff! 20.000 requests should be peanuts I hope.
Eric Marcoullier
CEO
Gnip (www.gnipcentral.com)
friend data. I don't think Gnip provides that in any open feed does it? I'd
love to talk if it does.
http://m.mwd.com
I feel bad for SocialToo but I do feel bad for Twitter as well, they dont have any revenue income yet from "Ads" yet their servers suck big time. They could of at least hire nice CCNP to clear some data
2009 is going to be rough people are to picky and don't appreciate JACK
sadly like a old prostitute society on the internet will use you abuse you and walk away :)
Despite all that :) It's great to see a QUALITY blog that is Do follow,mine is also and I'm
on the LOOKOUT to network with cool people. If you know what I mean Bring IT :)
Nice blog Stumbled
is scale to lots of servers, which I'm doing now.