next 
Martin Görner, Alexis Moussine-Pouchkine, Didier Girard
To help you in this challenge, you can use a powerful tool: App Engine. Our cloud experts will guide you along the way.
One more thing, you have only 2 hours.
Ready to scale ?
You won the contract of your life. Your job is to build the web site announcing, on D-day, the results of the presidential election.
The site will have to withstand a peak of potentially 300 million users on election day. Traffic can peak at above 10 milion simultaneous users at 8PM when the results are announced. Traffic will then taper off to zero over the next few days. The site must have the social features users expect: comments, satisfaction votes, image publishing, ...
The site cannot fail,
especially not at peak traffic!
You should now have a "g" icon in the Eclipse toolbar.
Eclipse and Java 7 troubleshooting:
Where is my JDK Java 7 ? On MacOS, run this in a terminal: On Windows, usually: |
Eclipse starts on JRE 6 and the Google plugin does not like it:
force java 7 in eclipse.ini (next to the Eclipse executable, in the app bundle on mac)
|
||||||
Which JDK does Eclipse use to compile my projects ? Eclipse > Preferences > Java > Installed JREs |
What Java system libraries does Eclipse use for my project ? Project > Properties > Java Build Path (Librairies tab) |
How to compile in Java 7 ? Project > Properties > Java Compiler |
|||||
Google Plugin for Eclipse Google App Engine Java SDKhttp://dl.google.com/eclipse/plugin/4.3Maven is a project format and build system for Java. Maven projects are portable between IDEs or continuous integration tools (Eclipse, NetBeans, IntelliJ, Jenkins). They include dependencies, including our dependency on the App Engine SDK which will be downloaded automatically.
Your first "Hello App Engine" servlet:
com.google.appengine.archetypes and select skeleton-archetype (1.7.5)
PresidentServlet.java withresponse.getWriter().write("Hello App Engine");
m2e-wtp Maven plugin for Eclipse to install from Maven Centralhttp://repo1.maven.org/maven2
Structure of the Web project:
When you use the assistant to create the servlet, the standard file web.xml is updated to map the servlet.
The file appengine-web.xml contains default values and will be used later to specify the application id as well as other App Engine specific parameters.
"port occupied" when launching your app, use the red "Terminate" button in the "Console" (Window > Show View > Console).
(Next time, to write cleaner code, prefer GWT or a JavaScript framework like AngularJS)
file: war/comments.jsp
<! DOCTYPE html> <%@ page contentType="text/html;charset=UTF-8" language="java" %> <html> <head><title>The president is...</title></head> <body> <H1>The president is...</H1> <img src="obama.png"> <p><%=request.getProtocol() %></p> </body> </html>
This page can be set as the default landing page in WEB-INF/web.xml
You fill find pictures of the candidates behind the first slide.
The president is...
The name of the new president is not known yet so we display something else, for instance:
request.getProtocol()
file: war/comments.jsp
<! DOCTYPE html>
<%@ page contentType="text/html;charset=UTF-8" language="java" %>
<%@ page import="com.google.common.html.HtmlEscapers" %> <!-- Guava -->
<html>
<body>
<H1>The president is...</H1>
<img src="romney.png" />
<%
String comment = request.getParameter("user-comment");
if (comment != null)
comment = HtmlEscapers.htmlEscaper().escape(comment);
%>
<p>You: <%=comment %></p>
<form action="" method="post">
<textarea name="user-comment" ></textarea><br/>
<input type="submit" value="OK" />
</form>
</body>
</html>
XSS vulnerability WARNING! DO NOT insert user-generated text into an HTML page without escaping.
The president is...
You need an input box:
<form action="" method="post"> <textarea name="user-comment" ></textarea><input type="submit" value="OK" /> </form>
And a way to get request parameters:
request.getParameter("user-comment")
file: war/comments.jsp
<! DOCTYPE html> <%@ page contentType="text/html;charset=UTF-8" language="java" %> <%@ page import="com.google.appengine.api.users.*" %> <%@ page import="com.google.common.html.HtmlEscapers" %> <!-- Guava --> <html><body> <H1>The president is...</H1> <img src="obama.jpg" /> <% UserService userService = UserServiceFactory.getUserService(); User user = userService.getCurrentUser(); if (user == null) {%> <p>Hello, <a href="<%= userService.createLoginURL(request.getRequestURI()) %>"> please log in</a> to post comments.</p> <%} else { String comment = request.getParameter("user-comment"); if (comment != null) comment = HtmlEscapers.htmlEscaper().escape(comment); %> <p><b><%=user.getNickname()%>:</b> <%=comment%></p> <form action="" method="post"> <textarea name="user-comment"></textarea><br/> <input type="submit" value="OK" /> <!-- logout link --> <a href="<%= userService.createLogoutURL(request.getRequestURI()) %>">logout</a> </form> <% } %> </body></html>
The president is...
Use the App Engine UserService API:
UserService userService = UserServiceFactory.getUserService(); User user = userService.getCurrentUser(); String n = user.getNickname(); String e = user.getEmail();
You will also need the login and logout URLs:
userService.createLoginURL("myTargetURL")
userService.createLogoutURL("myTargetURL");
Data layout: think "XML" rather than "database table"
Indexed queries only.
Eventually consistent queries.
Entity = kind + id +
(multivalued) properties
file: src/com/me/president/Comments.java
package com.me.president;
import java.util.*;
import com.google.appengine.api.datastore.*;
import com.google.appengine.api.datastore.Entity;
public class Comments {
static public void store(String text, String user) {
Entity commentEntity = new Entity("Comment");
commentEntity.setProperty("user", user);
commentEntity.setProperty("date", new Date());
commentEntity.setProperty("text", text);
DatastoreService datastore = DatastoreServiceFactory.getDatastoreService();
datastore.put(commentEntity);
}
static public List<Entity> retrieveAll() {
DatastoreService datastore = DatastoreServiceFactory.getDatastoreService();
Query query = new Query("Comment");
query.addSort("date", Query.SortDirection.DESCENDING);
return datastore.prepare(query).asList(FetchOptions.Builder.withLimit(100));
}
}
Usage:
// store:
Comments.store("Hello", "Nicolas");
// retrieve:
List<Entity> comments = Comments.retrieveAll();
for (Entity commentEntity: comments) { ... commentEntity.getProperty("text") ... }
The president is...
Low level datastore access:
DatastoreService datastore = DatastoreServiceFactory.getDatastoreService();
Write:
Entity commentEntity = new Entity("Comment");
commentEntity.setProperty("date", new Date());
commentEntity.setProperty("text", text);
datastore.put(commentEntity);
Read: 100 results sorted by date
Query query = new Query("Comment");
query.addSort("date", Query.SortDirection.DESCENDING);
List<Entity> results =
datastore.prepare(query)
.asList(FetchOptions.Builder.withLimit(100));
file: src/com/me/president/ObjectifyDAO.java
package com.me.president;
import com.googlecode.objectify.Objectify; import com.googlecode.objectify.ObjectifyService;
public class ObjectifyDAO
{
static { ObjectifyService.register(Comment.class); } // register all @Entity classes
static Objectify ofy() { return ObjectifyService.ofy(); } // proxy for Objectify
}
file: src/com/me/president/Comment.java
package com.me.president;
import java.util.Date; import java.util.List;
import com.googlecode.objectify.annotation.Entity;
import com.googlecode.objectify.annotation.Id;
import com.googlecode.objectify.annotation.Index;
import static com.me.president.ObjectifyDAO.ofy; // now ofy() goes through the DAO
@Entity public class Comment
{
@Id Long id; @Index Date date; public String user, text;
public Comment() {}
public Comment(String text, String user) { this.user = user; this.text = text; this.date = new Date(); }
public static void store(String text, String user) { ofy().save().entity(new Comment(text, user)); }
public static List<Comment> retrieveAll()
{ return ofy().load().type(Comment.class).order("-date").limit(100).list(); }
}
@Entity
public class Comment
{
@Id Long id;
@Index Date date;
String user;
String text;
public Comment() {}
...
}
If the Id is of type Long, it will be generated automatically on write.
Registering data classes in a Data Access Object:
public class ObjectifyDAO
{// register all @Entity classes
static { ObjectifyService.register(Comment.class); }
// proxy for Objectify
static Objectify ofy() {return ObjectifyService.ofy();}
}
// then use ofy() thanks to the declaration:
import static com.me.president.ObjectifyDAO.ofy;
Write:
ofy().save().entity(new Comment(text, user));
Read: 100 results sorted by date
List<Comment> list = ofy().load().type(Comment.class)
.order("-date").limit(100).list();
for (Comment c: list) { ... }
file: /src/com/me/president/Comment.java
The Comment class must be Serializable
public class Comment implements Serializable
Create a new method for retrieving comments with caching:
public static List<Comment> retrieveAllCached() {
MemcacheService cache = MemcacheServiceFactory.getMemcacheService();
List<Comment> comments = (ArrayList<Comment>)cache.get("100Comments");
if (comments == null) {
// read from Datastore
comments = retrieveAll();
// write to Memcache
cache.put("100Comments", new ArrayList<Comment>(comments));
}
return comments;
}
Finally, invalidate the cache on each new comment:
import static com.me.president.ObjectifyDAO.ofy;
public static void store(String text, String user) {
ofy().save().entity(new Comment(text, user));
MemcacheServiceFactory.getMemcacheService().delete("100Comments"); // flush cache
}
The interface:
MemcacheService cache =
MemcacheServiceFactory
.getMemcacheService();
cache.put("myKey", Object);
cache.get("myKey");
cache.delete("myKey");
Memcache makes sure all your instances see the same value.
Just keep the result of the last request in Memcache (serialized) until a new comment is added.
Memcache offers fast access and less contention for data primarily fetched from the datastore.
file: war/WEB-INF/appengine-web.xml
<?xml version="1.0" encoding="utf-8"?>
<appengine-web-app xmlns="http://appengine.google.com/ns/1.0">
<!-- Your application id as defined on http://appengine.google.com -->
<application>thepresident-is</application>
<!-- The version of your application, it can be a string (alpha, beta, ...) -->
<version>1</version>
...
</appengine-web-app>
Reference documentation on appengine-web.xml.
This configuration is also possible via the Eclipse plugin:
http://myappid.appspot.comappengine-web.xml
|
More on App Engine: |
And beyond: |
The cron servlet writes the name of the elected president into the datastore.
file: /src/com/me/president/CronServlet.java
public class CronServlet extends HttpServlet
{ @Override
protected void doGet(HttpServletRequest req, HttpServletResponse resp) throws IOException
{
Entity Elected = new Entity("Elected");
Elected.setProperty("name", "Barack obama"); //or Mitt Romney
DatastoreServiceFactory.getDatastoreService().put(Elected);
}
}
Servlet mapping in WEB-INF/web.xml, with access restrictions:
<servlet> <servlet-name>CronServlet</servlet-name> <servlet-class>com.me.thepresident.CronServlet</servlet-class> </servlet> <servlet-mapping> <servlet-name>CronServlet</servlet-name> <url-pattern>/cronAllowPresidentDisplay</url-pattern> </servlet-mapping> <security-constraint> <web-resource-collection> <url-pattern>/cron*</url-pattern> </web-resource-collection> <auth-constraint> <role-name>admin</role-name> </auth-constraint> </security-constraint>
Conditional display (file war/comments.jsp):
Entity e = DatastoreServiceFactory.getDatastoreService().prepare(new Query("Elected")).asSingleEntity();
if (e == null) /*wait message*/ else /*display president*/ e.getProperty("name");
The president is:
Cron jobs are servlets. Their URL is defined in web.xml.
Their periodicity can be set in WEB-INF/cron.xml (full syntax):
<?xml version="1.0" ?>
<cronentries>
<cron>
<url>/cronAllowPresidentDisplay</url>
<schedule>6 of May 20:00</schedule>
<!-- every 2 mins, every sunday 9:00 -->
<timezone>Europe/Paris</timezone>
</cron>
</cronentries>
Writes to the same datastore entity are limited to a few
writes per second: so how can I implement a counter ?
⇒Atomic increment in Memcache and a sharded counter in Datastore.
import static com.me.president.ObjectifyDAO.ofy;
@Entity
public class Counter {
@Id String key;
@Index String name;
@Index Long value;
Counter() {} // Empty constructor
Counter(String name, Integer shard_N, Long val) {
this.name = name;
this.value = val;
key = name + "_shard" + shard_N.toString();
}
public static void increment(String name) {
MemcacheService cache =
MemcacheServiceFactory.getMemcacheService();
Long val = cache.increment(name, 1); // atomic
if (val == null) {
val = Counter.read(name);
Long init = (val == null) ? 0L : val;
// atomic increment with initial value
val = cache.increment(name, 1, init);
}
Counter.write(name, val);
}
// Counter.java continued
public static Long value(String name) {
MemcacheService cache =
MemcacheServiceFactory.getMemcacheService();
Long val = (Long)cache.get(name);
if (val == null) {
val = Counter.read(name);
if (val != null) {
cache.put(name, val);
}
}
return val;
}
static Long read(String name) {
Counter c = ofy().load().type(Counter.class)
.filter("name", name).order("-value")
.first().get();
return (c == null ? null : c.value);
}
static void write(String name, Long value) {
Random r = new Random();
Integer shard_N = r.nextInt(10);
ofy().save().entity(
new Counter(name, shard_N, value));
}
}
//key = name+shard_N
public class @Entity Counter {@Id String key; @Index String name; @Index Long value;}
The president is:
+1 (17K) | -1 (10K) |
Atomic increment in Memcache:
Long C = MemcacheServiceFactory
.getMemcacheService().increment("myCounter", 1);
Save the new value in one of 10 counters in the datastore:
Integer shard_N = new Random().nextInt(10); ofy().save().entity(new Counter(name,shard_N,C));
Read: if Memcache fails, MAX of the counters in the datastore:
ofy().load().type(Counter.class).filter("name",
name).order("-value").first().get();
file: war/WEB-INF/appengine/datastore-indexes.xml
<?xml version="1.0" encoding="utf-8"?>
<datastore-indexes autoGenerate="false">
<datastore-index kind="Counter" ancestor="false" source="manual">
<property name="name" direction="asc"/>
<property name="value" direction="desc"/>
</datastore-index>
</datastore-indexes>
The SDK generates this file automatically in WEB-INF/appengine-generated/datastore-indexes-auto.xml copy it to WEB-INF/appengine/datastore-indexes.xml
@Index.file: war/admin.jsp
<%@ page contentType="text/html;charset=UTF-8" language="java" %>
<%@page import="com.google.appengine.api.blobstore.BlobstoreServiceFactory"%>
<form action="<%= BlobstoreServiceFactory.getBlobstoreService().createUploadUrl("/upload") %>"
method="post" enctype="multipart/form-data">
<label>President's name : </label> <input type="text" name="president-name"/>
<label>President's picture : </label> <input type="file" name="president-pic"/>
<input type="submit"/>
</form>
file: UploadServlet.java (map it on /upload)
package com.me.president;
public class UploadServlet extends HttpServlet {
@Override protected void doPost(HttpServletRequest req, HttpServletResponse resp)
throws ServletException, IOException {
BlobstoreService bs = BlobstoreServiceFactory.getBlobstoreService();
Map<String, List<BlobKey>> blobs = bs.getUploads(req);
BlobKey key = blobs.get("president-pic").get(0);
String presUrl = ImagesServiceFactory.getImagesService().getServingUrl(key);
String presName = req.getParameter("president-name");
resp.sendRedirect(presUrl); // just for debug
// TODO: keep the name and the pic URL of the president in the datastore (using an
// Objectify class) and use these values to display the elected president.
}
}
Create a form for uploading a photo to a URL given by the Blobstore:
<% url = BlobstoreServiceFactory.getBlobstoreService().createUploadUrl("/upload"); %>
<form action="<%= url %>" method="post" enctype="multipart/form-data">
<input type="file" name="president-pic"/> ...
Get the Blobstore key of the picture in the servlet
/upload:
void doPost(HttpServletRequest req, HttpServletResponse resp) {
BlobstoreService bs = BlobstoreServiceFactory.getBlobstoreService();
BlobKey key = bs.getUploads(req).get("president-pic").get(0); ...
Serve it to the client using the ImageService:
ImagesServiceFactory.getImagesService().getServingUrl(key);
file: war/WEB-INF/web.xml
<security-constraint>
<web-resource-collection>
<url-pattern>/cron*</url-pattern>
<url-pattern>/admin.jsp</url-pattern>
<url-pattern>/upload</url-pattern>
</web-resource-collection>
<auth-constraint>
<role-name>admin</role-name>
</auth-constraint>
</security-constraint>
file: src/com/me/president/MailHandler.java
package com.me.president;
import java.util.Properties;
import javax.mail.*;
import javax.servlet.ServletException;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
public class MailHandlerServlet extends HttpServlet {
public void doPost(HttpServletRequest req, HttpServletResponse resp) throws Exception {
Properties props = new Properties();
Session session = Session.getDefaultInstance(props, null);
MimeMessage message = new MimeMessage(session, req.getInputStream());
Comment.store((String)((Multipart)message.getContent()).getBodyPart(0).getContent()
, ((InternetAddress)message.getFrom()[0]).getAddress());
Message reply = message.reply(false);
reply.setFrom(new InternetAddress("noreply@cloudpresident.appspotmail.com"));
reply.setText("Thank you for your opinion.");
Transport.send(reply);
}
}
Enable email reception on address name@id_application.appspotmail.com
<inbound-services>
<service>mail</service>
</inbound-services>
Create a servlet mapped on /_ah/mail/*
To read the contents of an email:
Properties props = new Properties(); Session session = Session.getDefaultInstance(props, null); MimeMessage message = new MimeMessage(session, req.getInputStream());
Contact :
+Alexis Moussine-Pouchkine, Google
+Martin Görner, Google,
gplus.to/martin.gorner
+Didier Girard, SFEIR
This presentation is available online:
http://cloudpresident.appspot.com
More info
http://developers.google.com/appengine