Google Developers Day US – Intro to Data APIs

(My Original Blog Post: -*

Attention! API Garbage (might be unreadable):

data Google actually

I have a town with the Google developer programs and I ll be talking today about the Google data APIs this is going to be an introductory sections on them to try and cover a lot of ground in the shallow depth of production want to talk about and I get pretty excited when I start talking about is something I work with everyday so you tried your brief and save some for the event session which is going on later today so yet with that so it started on and I think the word a little late so I m an attorney keep it short and I get her out for lunch so of dimension that I get pretty excited about the Google data APIs may not know of a lot about the APIs that I have a shorthand summary people had used the APIs or okay for him right end of so there s been a lot of buzz going on about these new services that Google is providing and the way they were making them programmatically accessible so that a quote here from one of our friends over at Yahoo Andover got another one that comes was from the blogosphere that s a little more compact so people are our I have excited about this and what is it exactly that it s going on that s getting people so excited old Google has started to release whole bunch of APIs for rheumatic interfaces that allow you to do interesting things using Google services we ve got the toolkits and libraries and Web services and saw to be centering in on the Google data APIs and we have there that a little title in full fidelity APIs or saying when we release a full fidelity as we want you to be able to do everything you can do through the front end application but through a programmatic interface so you can write software to automate any of the tasks that you normally have to do by hand chain of sitting in front of the browser so these are the goals for our dual data APIs and I wanted to contrast this with the some APIs that you may be more familiar with some of the more visual and functionally oriented presentation APIs or Ajax APIs to do things like maps or Ajax search things that you can embed in your webpage and you don t really have to do any preprocessing of the data you re here getting something that is very visual in just plug it in to contrast that the data APIs or really about the data that sitting in our servers on the backend and so we pulled understatedly to format it manipulated transform it and, with some kind of presentation so this is this is where the mashup comes and it s a great way to get data to combine different sources or to use these different services to simplify the process of creating a rich application so we set out to design these APIs we were trying to do several things I mention the full fidelity aspect are trying to give you for rheumatic access clothing you can use a browser but we wanted to do it in a way that was simple that was easy-to-use that was based on open standards of a new programmers hate to to read specs documents and design so we wanted to try and make it simple and straightforward as possible so it uses a rest-based approach to to create update retrieved delete functionality data manipulation we needed a way to query to get the information you re looking for because some of these data sources can be very large if you re for example looking at all the events in your calendar well how do you find the one you want and then once we designed this this API we wanted to be able to use it in as many different services as possible so once you ve learned how to use the Google data APIs should be able to use the same knowledge with Google base calendar spreadsheets Google apps watercooled notebook costs well above those Google code search and we also wanted to enable you to do that in any language so I m we ve released a client libraries open source libraries in several languages and they re actually sounded launchers in the community have donated or created their open source so that I ll look at some binoculars or goals or how did we designed this system how does it actually work on it we wanted something that was simple simple data format in a simple protocol and we wanted to use something that was existing in open standard not try to create something new that would be confusing and in something floor and so we looked at the atom syndication format which is used for a lot of different feeds already things like blog since it s very similar to the RSS if you re familiar with that and we also wanted a simple protocol so we went with the HUD fears the underlying system using that for each UDP verbs before that post put and delete to to accomplish the crud operations that you need and there is an existing protocol that s tied in with the with Adam called the atom publishing protocol so the services that were designed for G. data are designed to be fully atom publishing protocol compliant so if you know the atom publishing protocol you can work with G. data and a lot of the tools that the already exists for the atom publishing protocol with the with little to no modification you can use them with faulty services so that was kind of our starting point was the atom publishing protocol from there we had to add a few things that we needed a suit to really make this work for all of these various services so we added in some some extensions on top of this atom publishing protocol s and extensible open protocol so we were free to add new things to it so we we created new data models that extends the standard atom feed and entry and we added in the query concurrency controls what happens if two people are trying to change the same data at once and we added an authentication we re working with simple itchy GP but a lot of the solutions that are there for each UDP authentication are fully secure so we ve designed some new things we had to roll around for him so going to go through these different extensions that have a high level of not really digging into a lot of details and later today at the advanced session will be going into a few more so before we go any further of Canada talking about this data format and thought it would be important to show it so we ve got a whole lot of XML here this is one entry so the atom publishing protocol deals with two large XML constructs feed and entry sore feet is a collection of entries entries have all kinds of information on all represented as XML and this happens to be a recipe actually Google bases is for you can store all kinds of different information and metadata about things so what of the popular item types in Google bases recipes so here s an example we have a recipe and I ll try to highlight kind of the human readable parts right answers a lot of the XML parsing that needs to be done were actually going to be looking at a lot of the XML throughout this thing so oh you know it s details aren t all that importance of interest have to come to get a feel for how the data is laid out notices the data API there s not a lot of flashy front and type stuff it s mostly in a dealing with large amounts of XML so let s begin now with authentication so we created two different ways to authenticate with the Google data services there was a client base where maybe you have a desktop application something you ve installed where you trust a system to give it your username and password so you login and then there was a browser-based system we re using redirects to send you to sign in to page and then send you on to a application web application for both of these are token-based schemes so with the sign in a secure and then you get a token which is used in expires and you can revoke all those kinds of things so that only are token is exposed through that each UDP traffic all right so they re moving on Aquarius at Google we like queries and so we ve come up with lots of different ways to query your data we have fulltext search categories you can query on things like the updated time when an entry is changed then we have the custom ones a lot of the different services that have implemented Google data APIs have created specific services like that example there is from spreadsheets you can query for all of your off all the rows in your spreadsheet or you can just ask for between this room in this row on Google base actually does have gone all out and created their own prayer language which is a pretty complicated and do some really interesting things with the bell so you can specify different output formats protocol itself works using Adam but it when you re pulling your information you can get it in a variety of forms whether it s RSS or juice on you can also do it but she signed scripts like a Tucson P. type callback into a JavaScript application so there s lots of options and you ll have to dig through the documentation to look at female differences in how all of these things work but we ve tried to really make a rich query infrastructure where you can get just the data you want so now its talk about the data model is shown in example before this is a different entry this one comes from calendar and what I wanted to show us how we extended the data model so for example this is some this is the XML that comes from the atom standard so we have an entry it has an idea hasn t updated time there s a title and content and in addition we added several things so for calendar are specifically there s quite a few are things we have reminders we have a time we have status of the event so I was really nice in this regard and that you can just plug in XML you can just add whatever else you want and actually some of the services allow yourself to define new elements and the service will just take those and store them in index them and allow you to search through those as if it were part of the standard itself I mentioned the concurrency so we use a system of optimistic concurrency since this is a rest-based arrests I ll Web service we give each item a unique URL in addition to that we have an edit your own edit link which has a version number appended to the end of it so if you when your friend are both pulling down the no different versions of say your spreadsheet and she updates her cell and your data so that it works out so that you know you re both doing it at nearly the same time it because it s in their first and you would ve overwritten when did the version number will changed after your friend did did do is update so you ll actually get data for nine conflict air and saying you know this version number change in the pulldown menu version to be able to change it flows those with a pretty simple way of solving this problem away those examiners and easy-to-use noticing we added were cheated at times so we have were using the entry for example over and over in all of these different services they each have their own flavor of entry so you know if if you are aggregating feeds and you were looking at funerals these different entries how do you tell which of your entries came from calendar spreadsheets or blogger so we added in kinds of ways of categorization using links actually schema links so we also use this for some of the XML attributes or properties elements that are reused over different services over examples for numbers there at all notes can small for people in the back but sound virtue phone numbers and one is labeled as a work and the second is labeled as a pager using that scheme your so now here s an example if you wanted to see for example calendar and look at all the entries you would just go to that URL you get an atom feed and there are a namespace extensions used to identify semantic entities and it items not easily knock one out of the RSS so it is since this is an atom feed some of the browsers for example are both understand how to interpret the data like as if it was a blog so if I open this up for example so will secure his or feed the hand to holiday we ve got all these different entries for different holidays okay so here are some of the services that tie into Google data APIs on that highlighted Google spreadsheets down there at the bottom because I m going to have walking through an example using spreadsheets have show you how this actually plays out so our goal here for this simple example is to change the contents over road in a spreadsheet so the first thing we have to do it for one of the changing a row we re going to have to authenticate because of their permission controls on your spreadsheets you can say who can and can t edit the spreadsheet so you just log in and the system will actually be able to show you can actually query for a list of all the spreadsheets that you are authorized to edit for example so this is the spreadsheet were going to be changing only small and simple were looking at the hours that people have worked in the items they created and in those hours and then we ve calculated items per minute out there so if something breaks so you authenticate were going to need to center information I ve chosen to do a client log in example so client login is done over a HUD PS using a post so there s an example of the information you send my highlighted the service name down there when you authenticate you authenticate for one service at a time so they each have kind of code words that we ve chosen arbitrarily people ask me why the spreadsheet services named wise and I have no idea okay so once you authenticate you your token back and then you use that token you included in the HUD header of all of your following requests so the system will check to make sure that you re allowed to see the spreadsheet serves an example were looking at changing a row so were going to be pulling down the list feed you can see down there this is our get URL of the feeds list the SID is a spreadsheet ID which is this really long ridiculous the strength and worksheet ID identifies a particular page within worksheet number going to be looking at the private and full representation of the data others have a lot of information that URL there and I m not going to get into it right now but I m basic ideas of their lots of ways that you can ask for your data to be represented you can view a spreadsheet as a series of cells or as a series of rows were choosing to look at it as a series of roads in our different levels of detail you can ask for centuries so here s our results this is what our spreadsheet looks like it s a big mess of XML this is why we have XML parsers so if we cut out some of the stuff that were not interested in maybe some of the namespace declarations and things like that we get something a little more readable but here s the data that we really care about right when were were were dealing with changing a spreadsheet so if your member back to the column names we had our spreadsheet we the the first row in the spreadsheet had named ours items and the IPM items per minute so those of them translated in the list feed into custom XML elements so there actually names according to the first row spreadsheet and then you have the contents there of those two different routes to date are so we have via the edit URL and mentioned before that you know we were using optimistic concurrency so we need to pay attention to the edit link for going to be changing the status so you can see that the at the end says you know Pvt.: then there s a few random strings of identifying the person so now that we have changed our information we actually send back an entry we do output to that edit link and that s the data were sent to the funny thing is we didn t actually change anything but we did update the row just doesn t look any different so that was a short example of a wanted the kind of show a few of the projects that are using our APIs because of it I wanted to give you kind of feel of the basics on how how you do a bit of some of the common operations but so once you realize you know that you have these resources these services and you have a simple way to do you know custom often up operation scripts things or a no manipulate data there s a lot of possibilities so people come up with some interesting things on our oh website PIC data we actually have a several long highlighted projects of people of created and then if you look through all the different services most of those also have example projects things that developers in the community up and creating a so for example here s a list of the applications that are using Google base slot of these cover a wide variety of different functionality and are interested in different types of information on why also wanted to show a couple of screenshots some that are a kind of interesting to talk very briefly about envy clock is actually a system where you can plug in your calendar and it ll create a graphical representation of what you re busy free time and a scan of boys think they re there planning to make a clock or something to me it looks like right now you can get it as a Google gadget and others of different ways to to use it but said it would be really cool if some of these things translated into physical devices separate from your computer found there s also like an office kind of planning application called timesheets which attached the other spreadsheets and how it renders some of photo editors that use the top Picasso web service so who see the hillside were doing okay I m time I wanted to do simple demo I mentioned some of the client libraries that that we have and wanted to show you how easy it can be using some of these tools so hopefully you can see this actually might be kind of hard on the back but I m going to be using a python of the Google data library I have a personal interest in showing the swamp boats see this is what you call like alter extreme programming right because I don t have one person looking over my shoulder I have like 200 so that always makes you feel really at their review 70 import fields okay so I ve imported the library found for spreadsheets and also I wrote a little helper module that does things like types in my username and password so you guys don t have to seal that ends some of us start by creating a clients and then I m going to have my hope rewarded okay so I mentioned before of doing a doing a query just asking for all of the rows in my spreadsheet so I m asking for a list feed and I m going to give it the spreadsheet ID which I ve also hardcoded into my helper module okay so looks like feed works and different print out my feed of a lot of XML so that s good so that I can look and see you know how many entries are in my okay the virtue which if I look over at my spreadsheet here yes there are in fact two churros and then if I go wanted to for example change oneself I could look at the soles feet in giving it the spreadsheet ID okay for this feed it s even longer and if we look at the links here to see how many entries are in it say that they re at 12 so if you look at this week out of a 4 x 3 so it gives you the full cells that have data so that now when you change one of these and the I mentioned that I m a changed ours because the items per minute this is calculated so as I change that you should see the items from the net change so we re going to change cell to grow to hand would ve changed the hours to 15 okay looks like it worked and we should be seeing this change appear if it doesn t I m in trouble hopefully I m looking at the same spreadsheet that I m editing it might be the problem actually because it looks like it did change you lots of bumper at a of a trying find that I m worried that it would just drag on what I open up for questions and at the end all the all proof that this actually works a no actually it s a bit. Dates you the question was I don t need to refresh the spreadsheet and the I might try and see usually need to identify did need to okay a thank you excellent excellent question alright so we have some microphone set up for questions if you fool would mind stepping up to those and I m which she embodies the checksum is a visitor checks full Unicode and the pursuit of its most Unicode is a beautiful Unicode assumes that I could take him an English US hosted Google data service” Hindu you were Japanese orchestra or any Unicode character to those values and have come back intact You actually tried that a few weeks ago and so I know it works for Google base at it works for everything you so the question is so well you did deprecated video Google search souls API is used to us for the fact that Ajax API to see you out if you deprive the developers of ability to manipulate the display of the search results are now hearing about this Google data APIs so used to about developers to benefit this awesome results so could you please explain the difference one you choose to restrict it to spend one set of tools to open up for the other tools of I m actually not sure what why things at the minute we decided to do things a different way in this respect I think part of the reason that we allow people to manipulate the data here is that it s their own data you know this is information in my spreadsheets and I need to be able to get the data interested of cushion at the very beginning you mentioned the associated APIs as full fidelity it is so clearly the dirt on all their right to get everything you need to do to you is that is that sort of the stated goal for all users to use the for them to be feature complete you is for you yes so of that is the goal and I think a lot is left up to the product teams or you know developing the whole system as to what makes sense whether all this is definitely to allow you to do as much as possible through the API free from having to sit in front of a browser that didn t do this for some of the okay great oh it s so wickedly amusing something like this and do you know it s out of Hong Kong high-traffic website of the one body is always hard to make sure that there s enough of performance on the other and add it to the response and not finding my website or something on Jacob is it irresponsibly did I ll see if I got to go somewhere that it is a response to your stuff if I had to do with timeouts and think about sort work does the is the API good enough to regard as a response to the gas so I don t think there really guarantees on things like a know response times timeouts service-level type agreements I think you should probably worry about timeouts although of course were were always trying to make sure that doesn t happen to you right where a working to speed these things up as much as possible make sure the scale but the API itself is just a GDP so it s were trying to keep it simple and so there s not really anything built-in that would protect you from things like HTP timeouts was her okay okay widow a lot of work on data over with feet Google ad words if you are basic and literacy solution to a problem of up to version 8 and granularity of and stability… which is to change it pretty much constantly which is hard for developers to keep up with the goods of life in a wife as a theme for four google and/or the community in general DC for kind of the architecture of the remodel you expose how quickly version to know what is okay or can you slow down why don t think we probably want to slow down this as difficult as that may make life I m not sure on our broader grand strategy of the kind of how we re trying to address this problem we were riveted that is an issue that you know having people run to catch up is and isn t a good thing but you know at the same time we don t want to hold back functionality so it s it s a difficult problem to be sure and I m not really sure kind of our strategy or direction as a whole s success will follow over the social general is in terms of organization within the company is their elected in API specialist group and they could ve put up its practices adopted or to think of it as a crosscutting across all be different if Edwards versus base versus map some of the stuff for first leg with your role in writing the share of so it s a little of both kind of as far as you know dealing with some kind of crosscutting support organization for creating APIs in NL and then letting each team do their own thing so my role is actually two work with a lot of the developers who are using these APIs and help them understand how it works developed tools create examples in a dinosaur I don t know with the needs of few wanted to jump on that one regatta of the member who s going to be giving up the advance talk later today so will a seat at our office feature for the advanced data talks with Richard Cypher to come to understand if you go Jeff go to Google I had a question about how this is a Google and where the data specialists of the job of developing the do you protocol we see it that way and you ll see some different approaches taken with 13 saves and Edwards are typically different API and there s a lot of cross collaboration with tapes we also are very much aware of the issue of the future with great to see clients we will make sure that we preserve as much of its multiple plate right now I m looking forward to were very much of thisworked if we listened but is a figure which is then a follow-up story that notes are the resident judge on the second microphone that will submission about Yahoo pipes for a fee to you that compared to G. data with Joe Biden selected link style that uses bold about that in the debate using civil protocols with you, so I m not really a pipes expert so let me preface it that way but I know that ice is involved in accepting fees and allowing you to manipulate the data and present a new feet up on the output so they actually have used a lot of the Google data feeds it speaks atom format and can also resent an RSS feed which you can consume and pipes so pipes actually has a plug-in for or recalled the data source for Google base decided they been it out using a lot of G. data things and do that so I think that some do they work well together as Windows JG 18 first brainstorm about the set authentication mechanism why do you discard 90 right of God not really sure actually I ve looked a little bit open ID and a guy don t really know death of a not sure that it solves all the problems we need part of for well but when I go into it because I really don t know well enough to be able to speak intelligently about Sirius either am mostly seceding out the Indian clock up there because I m the guy who wrote it as XLS to say this one I actually have a totally all right and is following that went to the disaster tries to take it off the unit but interesting places like on your fridge is so infinitely today and since they don t know what to do with Genoa to put it on the what new stuff that s up from grimy sink to his good clock is actually written in Ruby and I noticed I was on list of the riots supported languages I m not sure just how actually these advance slides I ve seen this fuller list of all of the languages that either people are contributed or know people at Google s in the bar suit the skewed some image earlier found that the search soap API was deprecated some point which I was usually unpopular people who do using a what assurance do I have is developers using she did it if you don t specifically for calendar but does you the same thing won t happen so that it admitted more concrete question is what it will community you have looked at these APIs in the past year was the economic value to Google there are ads going through the right it seems like the right thing to do but you know that s not always a good reason for keeping supplies forever at first so we talk a little bit about the the motivation and why and I wanted to provide these services are we ve created the services that we think are useful a lot of fun and I know we ve created web applications around them like for example calendar right that there lots of interesting things that you can do with calendar that we haven t thought of or that we haven t had the time to support out to release so by creating these APIs then we can greatly improve the user experience as far as providing new features right because now you can download an application that will that will do something new that our product doesn t do or you can combine them in interesting ways like there is an example that was recently released that some takes a spreadsheet of the People s birthdays like renaming a birthday and a link to a picture and then creates entries in your calendar for those people with embedded photos of stuff soak notes kind of a neat way to allow people to create useful things that we have a lot of good answers and it looks like were out of the question so let me just say it out through one more solicitous asks let me follow up on him so great that APIs empower us developers to come up with cool new features that make the experience of calendar reveals that produces better so take the next step is there is strong economic value in that activity to Google does it like a lock in users into using calendar somehow wonders Google for this is because Google could see strong economic value from locking people into our cool widgets is is that we rely on the Google getting a graduate that s why don t you believe the Iowa was just fun as resulting from the detectives were all both sound good to a I m actually not sure about how to the broader nation of Indians yes please oh I ll try to answer questions about your upward where the engineers will make all the decisions for it asserts a fiasco for a return to this value to our APIs to make sure developers are the winners and a fundamentally speaking at the effort to talk about this for a pretty high-level it was accompanied by the talk law that that uses only the and open-ended data from our point of view that sees that they configure to go somewhere else and get absolute winners are the winners are big users if they have that data is all in the mouth they can take that video to the products they consider nice with another killer piece of paper for him we specifically the question is right so the question was a does Google use these these APIs is Web services internally will psych testing for calendar exactly a vote we have we do use the bed API answer yes I know there are groups are using the APIs as far as I m calendar does you you may have heard of Google apps right were providing him all these Google webpages and services through your own domain so a system ourselves a 30 others a lot of internal use for it for the data APIs is that she can t labor it is sometimes found some indications that I thought they are a little on second largest of the political will in the future API down the blink lately to add those are I don t like getting second one ends and with nothing but air the Madisonville closets but I cannot get enough of being okay you re talking about spreadsheet that she s exactly a so for a okay so you re saying column manipulation using it is those federal manipulations you want to get a little on a set values to the political one needs I ve had a actually think you can clear a row by sending a general update but not Lake right and shift the other is a sure okay so I guess that s one of the limitations will have to go to work that hard things okay so of me just say that that so will we have some Google data station some of the laptops out by the beanbag chairs and there should be a group of us in in the black T-shirts answer questions or if you want to come talk to us feel free to drop by and thank you very much


~ by bitahatini on December 14, 2008.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: