Crowdsourcing Content with Mechanical Turk

Here's most of my BarCampNYC2 presentation. I had the video that was taken (which didn't start at the beginning) transcribed (via Mechanical Turk!) and done some very minor editing and formatting. I wanted to get it posted so other's could see it, and Identify themselves, and help fix up some of the discussion part at the bottom. If you were there, and you recall what was said, where there is currently (???) in the transcription, please leave the correction in a comment.

I Have a Content Site

(missing from video - I'll back fill here later)

I Do Keyword Research

(missing from video - I'll back fill here later)

I Discover Mechanical Turk

(missing from video - I'll back fill here later)

I Create a Request (HIT)

Pauli: . . .So I hopped on their interface and I said, you know, please describe your favorite icebreaker/game. And I asked for 9 assignments. And I offered a dollar reward.

And a dollar is pretty high for a mechanical turk environment. I hadn't researched what people were paying. I subsequently found that I get quality results back for paying about 20 cents a question. Because the people who are looking at these HITS and answering these questions, they're not making them as a calculation how much is my time worth. They've got time to kill, they're checking out the site. They think, “gee, can I make a little bit of change and is there some interesting task to do to kill some time. . . oh I know the answer to that question” and they hop in and tell you because it's something that comes from their experiences that they're sharing. Within half an hour I got nine ice breaker or game descriptions.

I Post an Article

And I wrote an article. I created a couple paragraphs of wrapper and then I filled in the games. I did a little bit of editing and then I laid it out nicely. I found I had more games than would reasonably fit on one page but not enough games to fit a full second page. So I went back to the well and I got some more games and I created two pages on the site.

Something Strange Happens

So then a funny thing happened. You guys have got to remember this is a little content site and I don't have a lot of traction with Google. All of a sudden I got a number one ranking on a keyword. Now it's not a big keyword, you know, it's not a key-word-of-all-keywords that search engine optimizers are going to actually want to get there for, but it was a pretty big deal for me.

Google Likes It!

Not only did I get a number one ranking on that key word but I have a top 10 ranking for 1, 2, 4 variations of ice breaker and I got a number 33 on a (???).

Man 1: This was after you posted the content from Amazon. It was just like magical content.

Pauli: Yeah, it was magical content. Obviously there was a selection process going on. I had to decide what content people were looking for, had to phrase the request in a way that people would give me the right content, then I had to package the content in a reasonable format: take advantage of the key words that my key word research had already identified as there being demand for. But I put it together and I put it up on the website and within three weeks, Google loves this content because people love this content. I'm giving people what they want, I'm giving them value. And I'm getting that content from anonymous people, I have no idea who they are. They're identified to me from this little 20 digit worker i.d. Through the Mechanical Turk service.

Man 1: So you have the copyright to the. . .

Pauli: I own the copyright because it's part of the terms of the service. If you're submitting work through mechanical turk, you agree that you are conveying full rights.

Man 2: To what extent are you just another layer between copy based (???) and adding someone else.

Pauli: Well in my request I specified that it had to be in the submitters own words. That they couldn't go out somewhere else on the web and copy and paste it in. And it was pretty clear to me that in fact was the case. There was compliance there. They were not copying and pasting. And that's something that you do probably want to check when you use mechanical turk to get content, you probably want to do a copy (???) search or some other search to see whether that content came from some other search on the web or whether it was original content that is supposed to be. Recall that ideas cannot be copyrighted, only the expression of these ideas. So in the contents of these games, I don't care where they learned the games. Did they read a book, did somebody teach it to them, did they learn it from somewhere else. It doesn't really matter, as long as they're telling me how to play the game in their own words. They've made the copyright to me and I'm able to publish it. And yes, through the terms of the service, I am copyrighting.

Man 1: Did you do more tests?

Pauli: I don't have any more results like this. I have more content that I've accumulated but I haven't put it together and posted it. Because I've gotten a little bit distracted by being involved in building a ruby library to wrap the mechanical turk AVI so that I can ultimately have a content management application that's integrated with mechanical turk through the back end. So I can submit requests, have them feed straight through to mechanical turk; get the results back, read them in my application, say yes, no, yes, no, yes, no, submit the acceptance, rejections back through the mechanical turk and then build pages interactively in this application. So i”Ive gotten somewhat distracted from the ultimate goal of producing this content and putting it out on the website because I'm building infrastructure. That said I do have a page of teaching songs that will probably go up in the next day or two.

Man 1: Who has used mechanical turk? Anybody? Anybody else doesn't know what it is?

Man 3: I didn't use it but I attended a java (???) in January that a guy from Amazon was talking about it. And he mentioned that getting started with rebuilding a (???) . . . . amazon.com (???)

Man 1: On behalf of mechanical turk and s3, I don't know if you know about that.

Man 3: s3, . . . largely.

Man 1: It's unlimited.

Man 2: the one key is that all the payment transactions go through the amazon, ecommerce, right? Your own account.

Man 1: Pretty much they give you amazon dollars and you spend it in the amazon store.

Pauli: so basically this is the mechanical turk interface. These are the requests that I've submitted. All seven of them. All of them are expired and entitled my results. Here is the first one that I submitted . . . pull this over a little bit. So that's the full request.

Man 1: Can you describe the request a little bit. Like what are you trying to have them do for you?

Pauli: Well basically in the presentation I basically, let me pop up this slide. Basically this is a tool I used, key word research. I had identified that ice breaker games was something that my audience was looking for. About 36 variations on ice breaker games. And it was coming back with various levels of searches in a given month. But combined all 36 variations you got some search traffic going there. I said, there's some content that I want to put on my site, to satisfy my audience. The problem being that I don't' know any ice breaker games. I don't' have a wealth of ice breaker games. I'm not going to spit that content out off the top of my head. But there are people out there who do know some ice breaker games, who have experience with these games. And that's when I went out to mechanical turk for in the first instance. And that's what this request represents, describe a game. So I got nine people telling me how to play one or another ice breaker game.

Man 3: So you said that this artificial intelligence. Some things computers can't do as well as humans, that's what we're told.

Pauli: A lot of the initial traffic was, look at this image, what color is it. Look at this image, does it have a person in it or something. Things that machines cannot do or cannot do well. But machines also don't compose original content well either. They don't have a wealth of experience that they can draw on.

Man 4: Ok, I'm looking here at one. Translation of a 36 minute podcast. Reward: $3.06. Who's pockets are paying $3.06?

Pauli: You as the requester.

Man 4: Out of my amazon account, it's going to go to someone else.

Pauli: Ok, you fund your account with Amazon. You have to prefund your account. So I threw $25.00 in. Then you have to wait for your account through amazon to clear. They don't use credit cards. In the data right now you have to provide an ECH transaction with your checking account. If you're not willing to do that you can't use mechanical turk right now. When it comes out of beta, maybe there will be some alternative funding provisions. But right now, they avoid having to pay the fees associated with having to pay credit cards.

Man 4: (???)

So now you have these funds sitting in your mechanical turk account with which to pay the workers. So you set up a request and you define how much you're willing to pay. As you specify a request, you specify the reward. So you can pay a penny for having somebody tell you what there's a person in the image. And you could pay $3 or $15 for transcribing a podcast depending on how long it is. How you decide how much you're going to pay for a task? Well there's an element, there's a market. People are not taking you up on your offer, you're not getting the work done, then you may have to raise your amount. I have found that I can get this kind of question answered for 20 cents each and I do not need to pay a dollar. I'm personally unwilling to go less than 20 cents because I already thing 20 cents is ridiculous.

Man 5: How quickly have you gotten results back and as you scale the pay does the quality actually increase?

Pauli: The quality doesn't increase. The results will change. I got all nine answers back in half an hour the first time. But the thing is when I went out later for 15 answers for 20 cents each it took 24 hours to get them all back. But in 24 hours I got 15 answers of acceptable quality, equivalent quality. You get an answer from somebody who's passionate about the question at hand, they're gonna go on for 3 paragraphs and say, email me if you want to know more. Or you've got somebody who has little knowledge and doesn't mind throwing it out for 20 cents or whatever you're paying and they'll give you the minimum that they feel will satisfy the question and they're done and they get their 20 cents. And you're done because you got the results. Can you use it? That's up to you. If you specify the requirements carefully, you're going to get acceptable results. And if they haven't met your requirements you can reject the results and not pay for it.

Man 5: Have you already seen it?

Pauli: You've already seen it. But if you go around rejecting all the results and using them anyway, people are going to complain. There's statistics when people look at your hit as a requester they can look at how frequently people reject your results. They can make a judgment. Personally I've only ever rejected one, maybe 2. It was actually in this set. I couldn't figure out how to play the game. And since I'm not using, at the time I wasn't using the api. Communicating back through the worker is only available through the api. The form interface doesn't give you the ability to say, “hey, I didn't understand this, could you expand.” So I had to reject it. I had to either decide to pay for even though I couldn't understand how to play the game or to reject it and I ultimately decided to reject it. As far as I'm concerned and the terms of service are if they've actually complied and done your request, you should pay them. It doesn't matter whether you like the answer, as long as it's a valid answer. So if I'd gotten two people who have given me the exact same game, I would have been honor bound to pay both of them. I didn't.

Man 6: Is there a rating system?

Pauli: There's not a rating system. There is statistics. Amazon tracks how many times you reject work. That's reportable. It also tracks how many times a particular worker's work is rejected. So if you're using the api, you have the ability to establish a qualification. You can say only people who have a certain qualification can do this work. So in the case of Casting Words, a podcast transcription company, their qualification is based on an acceptable sample. So if you want to get a qualification, you have to do the test and they've provided some kind of a test. I haven't taken it so I can't give you exact details on it. But as a requester you have the ability to say, only workers with an acceptance ratio above x are allowed to do this job. Or only workers with a certain number of accepted hits that they've previously worked on are allowed to do this job. So you have the ability to set those qualifications. But that is only accessible through the api, you have to be able to write code to be able to do that. That's why I'm busy writing my little ruby library so that I can write code.

Man 7: Are there existing packages that are in terms with the api . . .

Pauli: There is a company called hit builder. Hitbuilder.com They are in beta.

Man 8: isn't it hyphen builder?

Pauli: I'm not sure whether there's a hyphen or not. But they provide an interface and services around the use of the mechanical turk system. And then there is a developer resource website. Then there is sample code that they've provided is in java I think maybe php. That's on amazon. awsdeveloper.com. I can find it for you and read it out directly.

Man 2: I think this is one of the coolest things that's ever been invented. That's why I said people is (???) when I introduced myself. When I spoke to this lady. . .

Woman: So here at the bottom of the thing, you click over here and it says developers and, let's see.

Man ?: (???)

Man 9: Are there any numbers on the amazon, any traffic on the hits?

Pauli: That amazon turk itself is getting?

Man 9: Yeah, that if you got 9 hits in half an hour?

Pauli: yeah it gets a lot of traffic. A little bit less now than in its inception because there's not a lot of requesters right now. Basically at the inception, I wasn't there at the inception I was a little bit late to the party. At the inception, amazon was building its A9 product which is some kind of web search portal type thing. And they needed to do a lot of tagging and key word something. I don't even know what they were doing but they needed to do a lot of it. But they basically built this infrastructure for their own use and decided that they were sell it.

Man ?: pictures taking pictures of their blocks, pictures off streets. Is there (???) being used for that?

Pauli: Yeah, it's also being used for that. I'm trying to find the link to the developers resource center. There it is. So it's developer.amazonwebservices.com Then you get links to all these web services. Amazon's web services, products are really kind of amazing. I don't know if any of you have looked at the simple queue, the simple storage service, s3, and now the elastic (???) cloud. Pretty (???) stuff there.

Man 2: It seems that (???) there.

Pauli: Well I remember them saying that they built good muck. Infrastructure, the back end. The stuff that you need to be able to build products. They've really spent a lot of time building all this infrastructure at really tremendous scale for their own purposes. And now they're opening up the infrastructure to the world so that people can take advantage of the muck, this infrastructure that's kind of 80 percent of the job of building a good web application. You can take advantage of their good infrastructure and build the fun stuff and not have to worry about the back end so much. Of course then you have tremendous exposure to amazon and if they change their services, you may be out high and dry. So it's not as simple as that, but it is pretty exciting stuff. So anyway here's the mechanical turk.

Man 10: Do you think there are actually people out there making a living on mechanical turk?

Pauli: You cannot make a living doing mechanical turk hits. People are not paying the kind of money that you need to live on.

Man 10: Do you have a lot that are really offshore?

Pauli: No, there's nothing really offshore because unless amazon has a banking arrangements or shipping arrangements, you can't use the funds that end up in your amazon account. Amazon will write you a check but it will be too costly to cash that check. Because it will be US dollars thrown on a US bank. If it's too costly to cash that check, what are you going to do? And you can purchase merchandise off the amazon site but if you can't get it shipped to you at a reasonable cost. So really what it is is people that have time on their hands?

Man 11: Over there on your list that says quote, “I make $1.45 a week and I love it.”

(laughing)

Pauli: Exactly. That's an article. It's on the salon.com site. It's really kind of a fun little article. It basically gives you an idea of whose actually doing these jobs. It's people who have got time on their hands who otherwise might have gotten lost in wikipedia following link after link after link. They got on the mechanical turk to see if there were interesting little jobs they could do to see if they could make a penny here, a penny there, answering a question that I throw out there. There was a lot of traffic in the beginning when amazon was doing this A9 system. Right now there's not really that much. But this is where your developer tools are. In your developer.amazonwebservices.com site.

Man ?: I just (???). Compared to anything else.

Man 2: if they solve the problem of the payments, I think it could really improve the livelihood of a lot of people in the (???) world.

Man ?: So were you writing all the content for your website yourself?

Pauli: yeah, before that I was writing most of it myself. I did teach Sunday school. So I did write stuff for my own experience. The motivation or building a website was because I know what people are looking for. I also know how hard it is to find. People put stuff out on the web but it's not optimized in terms for search. It's not easy to find.

Man ?: So do you get most of your material from amazon turk now, do you still write most of it?

Pauli: Right now it's a little of both. I write what I can and where I don't have a . . . I wanted to show you what it looks like on the other side. OK, here we go. This is what people who can work on mechanical turk see. So, find hits to work on. Let's see what people are looking for right now. So for my content recently. . . actually right now I'm working on an article that's kind of a combination. . . So these are hits available to me. So this one is for 20 cents. Retrieve contact information from requester info. Let's see what it actually looks like. I'm writing an article about using songs in a lesson, the value of using songs to reinforce the lesson content in children. So it's basically an article about a teaching tactic with young children. That's an article that I can just write from research. Then I collected some songs, tell me a Bible teaching songs. And I got about 10 or so little songs that, to the tune of BINGO, to the tune of Farmer in the Dell, little songs. So it's a combination, real content that I wrote and content that I got from mechanical turkers. So this is somebody else's hit. Search via the web or call to retrieve contact information for real estate if the person responsible for real estate site selection for new stores for the company. So basically this is somebody who is selling real estate and wants leads. They identified some companies and they want to know who is the go to person. So for 20 cents each they'll pay you to find out who the go to person is. So are people taking these hits? I don't know. There's 40 of them available. We could come back tomorrow and see if any of them are left.

Man: Do you have a feel for what the sweet spot is for the complexity of the task?

Pauli: it has to be simple. If you have a really complex task, then you have to pay a little more. So here's a podcast interview with bonus for 11 minutes. They're offering a dollar. A dollar to transcribe a podcast for 11 minutes. I don't know who does the transcription for a dollar. I mean, 11 minutes so you can type 600 words for a dollar. So you can type 600 words for a dollar?They posted back, you have to read it and listen to it and then you have to say is it ok or not? You also have to take the qualification. Take a qualification to be allowed to even do it. It's not open to everybody.

Man: So they want 4 people to transcribe the same podcast?

Pauli: No, different ones. This one is 11 hours and 44 minutes! No no, it expires in 11 hours and 44 minutes. Alright, now this is 11 minutes, this one is 8 minutes, this one is 5 minutes.

Man: They type it in 3 times you compare them things . . .

Pauli: yeah, you can do that too. I haven't, Casting words has their own business model. I haven't really worried about how they do things.

Man: Do these people all have amazon ids? Do you ever find out who they are, who they really are, what culture they're from?

Woman: This wedding anniversary celebration stories. This is Renee Michaels and she's paying 5 cents. Let's see what she wants for 5 cents? Would you like your romantic anniversary celebration story published on my website, SweetLoveGifts.com? If so, please write one or two short paragraphs and I'll include it ASAP! (Note: it appears to be working well for her.)

Man: Does she ever find out who is sending it?

Pauli: You get an id. You never find out who the other person is who sends the . ..

Man: Or get a qualification.

Pauli: And if you provide a qualification that is a test, they pass that test.

Man: You pass something tangible.

Pauli: The qualification that you require can be based on the statistics of that site. You can write a test. If you write a test that can be scored. Because it's a multiple choice with known answers, you can provide the answer and the mechanical turk service will score the qualification and grant it immediately. If it's not something that they can assess in that fashion, you have to look at the results and decide whether you're going to grant the qualification or not

Man: There was a scandal. Somebody split up a long article and told people to do the test first. As a result got back all things translated said, no I don't think it's good enough.

Pauli: because the qualification isn't even a task. Oh, so they went in to do the qualification and it was the job?

Man: Well, that's not really a scandal.

Man: It could apply most of all to places where people don't make that much money. However because of the banking issue and payment issues it can only be used in the United states mostly. As a result people were complaining like, who's going to do this much work for not that much money? Then you have people on top of that rip you off.

Man: That's an emotional issue.

Man: When you click on a worker's i.d. Does that list everything they're good at?

Pauli: You can't click on a worker's i.d. You can submit a message to amazon for amazon to (???) to those workers so you can send them an email. If they reply to your email outside of amazon, should you provide that access, you can start communicating directly. But you do not know through the amazon service any identifying information for any workers. Now the workers can see your amazon account name. If you, like (???) here, signed up as an individual, it's going to be your first and last name. If you have a corporation and you provided your corporate credentials and you funded your account with your corporate bank account, like CastingWords, it's going to be your company name. So workers can see who you are but you can't see who they are. But you can communicate and you can provide means for workers to identify themselves to you should they choose to?

Man: Do you have anything you can show through your api?

Woman: I can show you that I have money on my account. Basically it's this tiny bit of ruby code and all I can do right now is pull back my list of hits that I've submitted. So it's extremely early early stages. So the very first day they came back here is $17.50. That's how much money I have in my amazon.com turk account. that's how much money I have to pay people. Oh I'm sorry. My machine is a wide screen. I just ran this little program. It's a ruby program.

Man: (???)

Woman: it's coded in the config files. It's open. My password is not visible on this screen. So your password, should you use this, you could use a ruby support project. It is as secure as you can make that file. When you make your api call, I'm using their rest service. When you make your api call you get back xml. So your document has the results. So for instance I have a list of hits that I have processed into an object. And later on down here when I actually call this, it's basically this is right now calling search hits. Search hits basically pulls back a list of all hits that you created in the mechanical turk service. Then inside the search hits object I have an array of hit objects. Basically it has the hit id, the type id, the creation time, title of description time, status, reward, my hits are all expired so it has no reward with them. So it comes back as null. Auto approval delay. that's a feature of a hit, if you never come back and approve any of your hits after a certain time, amazon will approve them for you and pay the worker. So that's a release of pay.

Man: (???)

Man: No, I was not related to this at all. Well, here's the deal. I thought about mechanical turk a long time ago, and I thought it wasn't feasible. I'm very happy that to see amazon built it but the biggest problem that I encountered was the payment part and it was outside. . .

Man: Outside?

Man: Outside of the US. I only thought that it was being used for very small things. Quite frankly, I'm very surprised and happy that people are using it and they are doing this, like what he said, that people actually generate content for a little bit of money, it opens up a whole new world to me.

Man: What do you do in your. . . are you??

Man: Yeah, I am, we went last year. I think this is one of the nicest things since sliced bread, to use one of the (???) expressions. You can include people in your code, put tasks in the queue, people do it, you pay them, that's it. You don't need to interact with them. This lady has used the web forming division, leading programming to the whole api's. It's just amazing what you can do. Once they solve the whole problem with the payments this can really really take off. There's 4 billion people, 3 billion of whom are not wealthy. I worked for a venture capital company, who had a company in Indonesia. 120 employees, 100 of those made less than 60 bucks a month. If I can change that lifestyle for them to make 200 dollars. For us it's nothing, for them it's a big big difference.

Man: Sorry, I came in late but what is the problem with the micro payments?

Man: Well, the problem for amazon is that they are not a bank, they pay like a dollar. For us a dollar is nothing, but a dollar somewhere else is more and they can't draw, because of tax treaties, IRS and I don't know what. They can't transfer this money from here to there to developing worlds, nations, cheaply.

Man: So how do you get paid with mechanical turk?

Pauli: Ok, you have an account with mechanical turk. To be a worker on mechanical turk, you have an amazon. They credit your account.

Man: So you can only have amazon good is what you're saying.

Pauli: If you build up a sufficient balance, they will cut you a check. But the thing is it's on US funds drawn on a US bank. So the thing is if it's a hundred dollar check and you're sitting in Indonesia, how much is it going to cost you to cash that check? You're going to lose it, half of it, in bank fees.

Man: You're gotta put it in pay pal, draw (???)

Man: (???)

Woman: See, there are many parts of the world that you cannot have a papal account.

Man: Of course you can, I've got an Australia paypal account. I transfer my mortgage, so money I make here in the US, I transfer into paypal and draw it out of paypal in Australia. It costs me nothing with papal. With Chase, it cost me $40 to send $6500 US back to Australia each month. And it cost me nothing. Now they screw you on the exchange rate, but I've figured it out. It works out to be about 20 bucks.

Man: What works out to be about 20 bucks.

Woman: That he loses on the exchange rate because amazon doesn't give him the market on the exchange rate. 2 percent or something. No it's not a flat rate.

Man: He said 20 bucks.

Man: So you send yourself a bill, I find it very interesting, you send yourself a bill and then pay it. With your credit card or something like that?

Man: Out of my US account into paypal, us, into paypal, into my Australia account.

Man: Is this for gambling purposes?

Man: If you were in Hungary or Romania, I'm just saying these countries. If you were in some countries you cannot get a paypal account.

Woman: It's like Western Union.

Man: It's going to be incredibly difficult and not practical.

Man: So they can buy books.

Woman: Some countries it's not even practical. Amazon won't even ship at that costs.

Man: What I don't get is if it's not even reputable. I want to know who these people are, I want to know beyond an amazon id, what's going on here. That's not giving me these answers.

Woman: No, it's not. So,. . .

Man: If it's a population of goodwill and people that are logged on somewhere and that's all I know.

Woman: And if that's sufficient for your task then you can use the service. And if it's not, then you can't.

Man: So this could get to the point where it requires higher levels of audience involved with . . .

Woman: The other thing that you are perfectly able to do this is that you can solicit workers independent of mechanical turk and then use the platform. So you solicit workers however you're soliciting them. You invite them to take your qualification. You provide them a special code that only invitees will have. They submit the code, they get the qualification. They are now known to you because, if you do it right, you can associate the code with the person you've invited. So now you can associate the worker id with the person you've invited. And you have that link.

Man: You have your .(???) . . within this anonymous...Good.

Man: Wouldn't it be nice if you had a task that directly could not be done by a human and then somebody finds a machine to do it anyway. That would be pretty interesting and if it's just a penny then it doesn't matter. You have a machine that can do it right nine hundred and ninety nine times.

Woman: This Ruby project that I'm working right now. It's off RubyForge. The project is called rturk. I'm the junior developer. The senior developer has disappeared. I don't know. I've only ever emailed him twice and the second time he never responded. I emailed him the first time and said, “oh good you're working on this, I'll help.” And he said “sure!” And I was added to the project. And then I said, “Ok, so any design documents, any plans, anything?” And nothing, no answer, no response. So I'm on my own. I'm doing the best I can. I'm having fun learning Ruby. When it's working I'll throw it out there on the site. If you don't want a ruby interface, or you don't want to wait for me, you can go out to the amazon developer website, pull out the sample code they've got out there, you just go to town.

Man: i like the best part is that both you and the person solving it can both use the interface.

Woman: The workers do not have an api. The workers, no. These are my available hits as a requester.

Man: I'm pretty sure that the workers can retrieve the HITs.

Woman: Third party persons have written screen scraping scripts to pull out hits. Amazon does not provide anything beyond. .. .

Man: That's what hit builder does?

Woman: No, hit builder is a requester-side service. If you don't want to write the programs, you can work with them. They provide an interface and the ability to handle the mechanical turk service through their services and their website. Anybody else?

Man: it would be nice if you submitted content and requested them to edit it?

Woman: Yes, no people do that.

Man: (???). If you can take feedback from somebody and create a new task based on the feedback of somebody else. The whole process (???)

Man: This was a presentation of the (???). Someone can't (???). actually it's being videotaped. Thank you for getting us interested in this.

Man: Look at s3 too. It's unlimited storage. It's pretty cheap. The only problem is (???). You (??) open the backup service.

Man: What are you running there?

Man: Just the (???)

Woman: it's access to their raid arrays.