\documentclass{beamer}
\definecolor{dblackcolor}{rgb}{0.0,0.0,0.0}
\definecolor{dbluecolor}{rgb}{.01,.02,0.7}
\definecolor{dredcolor}{rgb}{0.6,0,0}
\definecolor{dgraycolor}{rgb}{0.30,0.3,0.30}
\newcommand{\dblue}{\color{dbluecolor}}
\newcommand{\dred}{\color{dredcolor}}
\newcommand{\dblack}{\color{dblackcolor}}

\usepackage{listings} 
\lstdefinelanguage{Sage}[]{Python}
{morekeywords={True,False,sage,singular},
sensitive=true}
\lstset{frame=none,
          showtabs=False,
          showspaces=False,
          showstringspaces=False,
          commentstyle={\ttfamily\color{dredcolor}},
          keywordstyle={\ttfamily\color{dbluecolor}\bfseries},
          stringstyle ={\ttfamily\color{dgraycolor}\bfseries},
          language = Sage,
	  basicstyle={\scriptsize \ttfamily},
	  aboveskip=.3em,
	  belowskip=.1em
          }
\usepackage{url}
\usepackage{hyperref}
\hypersetup{colorlinks=true, urlcolor=blue}
\usepackage{comment}
\usepackage{colortbl}
\usepackage{fancybox}
\usepackage{beamerarticle}
\usepackage[utf8x]{inputenc}
\mode<presentation>
{
%  \usetheme{Rochester}
% \usetheme{Berkeley}
 \usetheme{PaloAlto}
 %\usecolortheme{crane}
% \usecolortheme{orchid}
 \usecolortheme{whale}
% \usecolortheme{lily}
 \setbeamercovered{transparent}
  % or whatever (possibly just delete it)
}
\usepackage{ngerman}

\title[Database Architecture]{My Next Modular Forms Database}
\date{October 2010}
\author[W. Stein]{William Stein (joint work with Mike Hansen)}

%\newcommand{\todo}[1]{[[#1]]}
\newcommand{\todo}[1]{}

\begin{document}

\begin{frame}
\titlepage
\end{frame}

\begin{frame}{Abstract}

\begin{block}{I Have Data}
  I have oodles of data on web pages and in files only I know how to
  use, and with Sage I can generate much more.  I am putting all of
  this data into a web-accessible database server.  Thanks to the NSF
  I can allocate terabytes of disk space to this database, and have
  money to buy extra computers for redundancy.
\end{block}

\begin{block}{Today's Technology is Better}
 Last time
  I tried putting together a database like this was in 2003;
  technology has dramatically improved since then.  Most computers are
  64-bit, which removes numerous annoying barriers, and there are good
  documented-oriented databases.  This talk is about how I intend to
  to put together this database, and will be of interest to
  others with similar goals. 
\end{block}

\end{frame}

\section{The Overall Architecture}

\begin{frame}{Database}
\begin{block}{Servers}
  \begin{itemize}
   \item MongoDB master -- disk.math.washington.edu (in Seattle)
   \item MongoDB slave 1 -- in Seattle on William Stein's OS X desktop (?)
   \item MongoDB slave 2 -- in Waterloo on Mike Rubinstein OS X computer
   \end{itemize}
 \end{block}
 
\begin{block}{Disk Space?}
\begin{itemize}
\item Try to limit the database footprint for this project to 4
 terabytes, so that a single $\sim$ \$350 USB disk plugged into any
 computer (Linux, OS X, etc.)  can server as a redundant MongoDB
 slave.  
\item But, if things get too big,  I'll use ``sharding''.
\end{itemize}
\end{block}
\end{frame}

\begin{frame}{Security and Users}
\begin{itemize}
\item
  The {\em master MongoDB server} will run directly on our big Ubuntu Linux
  fileserver, listening only on localhost. 

\item  A user who needs direct {\em write} access to the database will have their ssh
  key added to a limited account on this machine, and via ssh port
  forwarding, they will be able to access the database, using a login
  and password that gives them access to a subset of the databases or
  collections served by MongoDB.  

\item A single MongoDB server can {\em simultaneously} serve numerous
  completely independent databases, and independent requests from
  different users.
\end{itemize}
\end{frame}

\begin{frame}{Web Interface}
  \begin{block}{Software}
    \begin{itemize}
     \item Flask microframework: \url{http://flask.pocoo.org/}
      \item Apache: via {\tt mod\_wsgi}
     \end{itemize}
   \end{block}

\begin{itemize}
\item Use the Flask Python library (Flask is
    from the same group that brought us Jinja, Sphinx, etc.) to
    develop a web front end for to the database.
  \item Webpage will enable anybody to easily make fast queries.
  \item Will create indexes in the MongoDB database that optimize
    queries available through the web interface.
\item Will deploy our Flask application using Apache's {\tt
      mod\_wsgi} module, which is scales well.
  \end{itemize}
\end{frame}

  
\begin{frame}{Use MongoDB from C, C++, Javascript, Python (Sage), Perl, etc.}

\begin{itemize} 
\item Will run several {\em read-only MongoDB slave servers}.  Good
  for arbitrary queries against the database.  Some queries involve
  server side javascript run on millions of documents, and can take a
  long time and put a heavy load on the database server.

\item MongoDB officially supports accessing a MongoDB server from any
  of C, C++, Java, Javascript, Perl, PHP, Python (hence Sage!), and
  Ruby.  There are numerous other languages that are not officially
  supported, but are here:
  \url{http://www.mongodb.org/display/DOCS/Drivers}\\No math software
  besides Sage, e.g., none of Magma, Mathematica, Maple, or Matlab, is
  in that list.
  \end{itemize}
\end{frame}    
  
\begin{frame}{Web Upload?}

\begin{itemize}
\item Data upload is done by connecting to the {\em master} database
  via a programming language.  
\item There is {\em no web page upload for data} as part of my planned
  architecture, due to security issues and time constraints.
  However, if somebody else makes a web upload system, they could act
  as an ``editor'' and submit the results of uploads to my MongoDB
  database.
\end{itemize}

\end{frame}

\section{The Database -- MongoDB}

\begin{frame}{MongoDB: a Documented Oriented Database}
  At the Paris meeting, David Farmer has put forth an idea that ``the
  basic building blocks in the project are the individual homepages of
  each object of interest.''  

MongoDB (\url{http://www.mongodb.org/})
\begin{enumerate}
\item is a new free open source {\em documented oriented} database
  management system, written in C++.
\item is much different than a SQL database such as SQLite, MySQL,
  or PostgreSQL. 
\item data model corresponds to Farmer's idea of
  homepages.  
\item easily builds indexes and does elaborate
  optimized queries.
\item automatically {\em replicates}
  to any number of backup servers.
\end{enumerate}
\end{frame}

\begin{frame}{MongoDB makes your data ``feel smaller''}

\begin{itemize}
\item This summer I tested using MongoDB to deal with masses of data I
  generated related to modular forms for a research project with
  Barry Mazur.
\item I also tested putting all of the Cremona and Stein-Watkins
  tables of elliptic curves in a single big MongoDB database.
\item It made a vast amount of data (hundreds of gigabytes) feel
  ``small''.
\end{itemize}

I have {\em never} had this feeling before with huge number theory
tables using any other database, including
PostgreSQL, MySQL, sqlite, ZODB, and custom filesystem based stores.

\end{frame}

\begin{frame}{How to Learn MongoDB}
\begin{block}{}
Go to \url{http://www.mongodb.org/} and start browsing.
\end{block}
\begin{itemize}
\item Tons of quickstarts, tutorials, articles, and videos of
talks, slides, etc.  
\item A Company is behind MongoDB; but don't worry, MongoDB is free and open source
\item Some not-quite-finished {\em books} about MongoDB; I read them
  by temporarily signing up for an O'Reilly Safari books membership
  (\url{http://my.safaribooksonline.com}), reading them, then
  unsubscribing.
\end{itemize}

\end{frame}

\begin{frame}{Setting up a MongoDB Server}
\begin{block}{Step by step (more details below)}
\begin{enumerate}
\item Binaries (Linux, OS X, Windows, Solaris) from \url{http://www.mongodb.org/downloads}.
\item Start a MongoDB server running by typing {\tt mongod}.
\item Connect by typing {\tt mongo} in another window.
\item Connect from Python (or Sage) using {\tt pymongo}.
\end{enumerate}
\end{block}
\end{frame}

\begin{frame}[fragile]{Starting a MongoDB server}
I run my mongod server by typing:

\begin{block}{}\begin{lstlisting}
mongod --dbpath /lvm/array/lmfdb/mongodb  \
          --bind_ip localhost  --port 29000
\end{lstlisting}\end{block}

The dbpath option specifies where the files for the database are
stored and the bind\_ip and port options makes it so mongod accepts
connections on localhost port 29000; otherwise, anybody in the world
could just connect to your mongodb and delete all your data!!  If you
want to run mongod on a remote server somewhere, but easily connect to
it from your laptop (say), setup an ssh tunnel by simply typing:
\begin{block}{}
\begin{lstlisting}
ssh -L 29000:localhost:29000 remote.computer.edu
\end{lstlisting}
\end{block}
%Then you can pretend that port 29000 on your laptop {\em is} port
%29000 on the remote server, and things will just work.  
It's also
possible to create accounts with various permissions from the mongo
console.
% (see the mongo documentation for details).  However, if you
%setup accounts make sure to use an ssh tunnel whenever using them,
%since mongod itself doesn't use a secure socket, so your password
%would get sent in the clear.
\end{frame}

\begin{frame}[fragile]{Connecting to MongoDB via the console}

\begin{block}{Connect to your new MongoDB server with the Mongo console}\begin{lstlisting}
wstein@disk$ mongo localhost:29000
MongoDB shell version: 1.6.1
connecting to: localhost:29000/test
> show dbs
admin
local
research
> help 
        db.help()     help on db methods
        ...
\end{lstlisting}\end{block}

\end{frame}

\begin{frame}[fragile]{Connecting to MongoDB from Sage (Python)}
\begin{block}{Install Pymongo (about 10 seconds)}\begin{lstlisting}
sage: !easy_install pymongo
...
sage: quit   # important! 
\end{lstlisting}\end{block}

\begin{block}{Use it}\begin{lstlisting}
sage: import pymongo
sage: C = pymongo.Connection('localhost:29000')
sage: C.database_names()
[u'research', u'admin', u'local']
sage: R = C.research; R
Database(Connection('localhost', 29000), u'research')
sage: R.[tab key] ...
sage: R.collection_names()
[u'mazur_irreg.done', ..., u'fs.chunks', u'fs.files']
\end{lstlisting}\end{block}
\end{frame}

\begin{frame}{MongoDB's structure: Databases, Collections and Documents}

\begin{itemize}
\item    A MongoDB server serves a collection of {\em independent}
    databases.
\item A {\em database} is a set of collections, and a {\em collection}
    is a set of documents.
\item A {\em document} is like a
    Python dictionary, but only a limited number of datatypes are
    allowed.
\item Technically, a document is a ``BSON'' document, where BSON
    is a format very similar to JSON.
\end{itemize}

\end{frame}

\begin{frame}[fragile]{MongoDB documents are limited}
A MongoDB document must be at most 4MB in size.  Let's push
the limits, to see what this means in practice:
\begin{block}{Use it}\begin{lstlisting}
sage: foo = R.foo
sage: foo.insert({'test':'a'*(4*10^6)})
ObjectId('4cae369075688b3eab000006')
sage: foo.insert({'test':'a'*(5*10^6)})
Traceback (most recent call last):
...
InvalidDocument: document too large - BSON documents are limited to 4 MB
\end{lstlisting}\end{block}

So you could store a string with 4 million characters, but not 5
million; for reference, 4 million characters is about {\em 1,000 typed
  pages of text}.
\end{frame}

\begin{frame}[fragile]{How to Store Huge Stuff: GridFS}
\begin{itemize}
\item Recall: MongoDB documents can be at most 4MB in size!
\item GridFS get arounds this; stores gigantic data in
MongoDB
\item No indexing and searching capabilities
\item GridFS is just a key:value store, built on top of MongoDB.
\end{itemize}

\begin{block}{Using GridFS}\begin{lstlisting}
sage: import gridfs
sage: G = gridfs.GridFS(R)  # we defined the db R above
sage: G.put('a'*(5*10^6), filename='test1')
ObjectId('4cae3ba075688b3eab000008')
sage: x = G.get_last_version('test1').read(); len(x)
5000000
sage: x[:30]
'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaa'
\end{lstlisting}\end{block}
\end{frame}

\begin{frame}[fragile]{Using GridFS to Store Pickles}

You can store arbitrary Sage objects using dumps and loads, which are
wrappers around Python's pickle module:

\begin{block}{Store a mathematical object}\begin{lstlisting}
sage: M = ModularSymbols(389, 2)
sage: G.put(dumps(M), filename='modsym389')
ObjectId('4cae3c1a75688b3eab00001d')
sage: loads(G.get_last_version('modsym389').read())
Modular Symbols space of dimension 65 for Gamma_0(389)
of weight 2 with sign 0 over Rational Field
\end{lstlisting}\end{block}

You get one GridFS per database, so if you have documents in all sorts
of collections that somehow point to GridFS ``files'', you'll need to
choose some systematic way of naming the files.

\end{frame}



\section{The Web Interface -- Flask, mod\_wsgi, Apache}

\begin{frame}[fragile]{Flask: a web development ``micro-framework''}

\begin{block}{FLASK}
\url{http://flask.pocoo.org/}
\end{block}
     
\begin{block}{``Hello world'' written using Flask}\begin{lstlisting}
from flask import Flask
app = Flask(__name__)

@app.route("/")
def hello():
    return "Hello World!"

if __name__ == "__main__": app.run()
\end{lstlisting}\end{block}

\begin{block}{Put the above in a file hello.py and...}\begin{lstlisting}
$ easy_install Flask    # 30 seconds?
$ python hello.py
\end{lstlisting}\end{block}
\end{frame}

\begin{frame}{Using Flask}
\begin{itemize}
\item I don't have time in this talk to go into detail about how to use Flask in
general.
\item The documentation at \url{http://flask.pocoo.org/docs/} is excellent.
\item You use decorators to construct the URL mapping, deal with GET and POST requests, etc.
\item You can also put static/ and templates/ subdirectories in your Python project, and
relevant files will get pulled.
\item You need to learn the Jinja2 templating engine: \url{http://jinja.pocoo.org/2/}.
\end{itemize}

\end{frame}

\section{Demo Site}
\begin{frame}[fragile]{Demo Site}
\begin{block}{The DEMO is here}
\url{http://db.modform.org/}
\end{block}      
\begin{itemize}
\item Mike Hansen and I built a demo site.
\item Illustrates the architecture sketched above by providing access to a large table of
over a hundred million elliptic curves (Cremona plus Stein-Watkins)
\item Will form the core for the new modular forms database.
\end{itemize}
{\em Try it out!} It's running on boxen.math.washington.edu, in Seattle. 
\end{frame}


\begin{frame}[fragile]{Apache Setup}
\begin{block}{/etc/apache2/sites-available/lmfdb}
\begin{lstlisting}
NameVirtualHost db.modform.org:80
<VirtualHost db.modform.org:80>
    ServerName db.modform.org
    WSGIDaemonProcess lmfdb threads=5
    WSGIScriptAlias / /home/mhansen/lmfdb/lmfdb.wsgi
    <Directory /home/mhansen/lmfdb>
        WSGIProcessGroup lmfdb
        WSGIApplicationGroup %{GLOBAL}
        Order deny,allow
        Allow from all
    </Directory>
</VirtualHost>
\end{lstlisting}
\end{block}
And a symbolic link:
\begin{block}{}
\begin{lstlisting}
/etc/apache2/sites-available/lmfdb --->
               /etc/apache2/sites-enabled/lmfdb 
\end{lstlisting}
\end{block}
\end{frame}

\begin{frame}[fragile]{WSGI Setup}
\begin{block}{The WSGI application is defined by this file:}
\url{http://sage.math.washington.edu/home/mhansen/lmfdb/lmfdb.wsgi}
\end{block}

The main thing that this file has to do is define some object called
``application'' which will obey the WSGI protocol.  There are a few
other things in there to let it know about the environment.  Here are
the contents:
\begin{block}{}\begin{lstlisting}
import os, sys
sys.path.append('/home/mhansen/lmfdb')
os.environ['PYTHON_EGG_CACHE'] = '/home/mhansen/lmfdb/.python-eggs'
activate_this = '/home/mhansen/lmfdb/env/bin/activate_this.py'
execfile(activate_this, dict(__file__=activate_this))
from lmfdb import app as application
\end{lstlisting}\end{block}
(It is important to look at the files mentioned above.)
\end{frame}


\begin{frame}[fragile]{Python/Flask Code}

Look at the files in
\begin{block}{}
\small
\url{http://wstein.org/talks/2010-10-lmfdb/demo.tar.bz2}
\end{block}

In addition to the templates, there's a file lmfdb.py:
\begin{block}{}\begin{lstlisting}
from flask import Flask, url_for, render_template, request
app = Flask(__name__)
from pymongo import Connection
db = Connection(port=int(29000)).research 
...
@app.route('/ellcurves/rank/<int:rank>/')
@ellcurves_list
def ellcurves_of_rank(rank):
    curves = db.ellcurves.find({'r':rank}).sort('level')
    return locals()
...
\end{lstlisting}\end{block}
This file defines what happens when a URL
is accessed. 
\end{frame}

\section{Summary}

\begin{frame}{Summary}
  This talk has laid out the architecture that I will be using for my
  new web-based databases.  It uses the following free open source
  tools together in a natural way:
\begin{itemize}
     \item {\bf Python}: a high quality programming language
     \item {\bf MongoDB}: a scalable documented-oriented database
     \item {\bf Flask}: a "micro-framework" for Python-based web apps
     \item {\bf Jinja2}: a general purpose templating language
     \item {\bf Apache + WSGI}: scalable web server
\end{itemize}
     
You can try out a demo that combines the above right now at:
     \url{http://db.modform.org}

\end{frame}
               

\end{document}











%%% Local Variables: 
%%% mode: latex
%%% TeX-master: t
%%% End: