+++ /dev/null
-% Databases and SQL
-% Greg Wilson
-% June 2013
-
-In the late 1920s and early 1930s,
-William Dyer,
-Frank Pabodie,
-and Valentina Roerich led expeditions to the
-[Pole of Inaccessibility](http://en.wikipedia.org/wiki/Pole_of_inaccessibility)
-in the South Pacific,
-and then onward to Antarctica.
-Two years ago,
-Gina Geographer discovered their expedition journals
-in a storage locker at Miskatonic University.
-She has scanned and OCR'd the data they contain,
-and wants to store that information
-in a way that will make search and analysis easy.
-
-Gina basically has three options:
-text files,
-a spreadsheet,
-or a database.
-Text files are easiest to create,
-and work well with version control,
-but she would then have to build all her search and analysis herself.
-Spreadsheets are good for doing simple analysis,
-but as she found in her last project,
-they don't handle large or complex data sets very well.
-She would therefore like to put her data in a database,
-and this chapter will show her how.
-
-As many scientists have found out the hard way,
-if collecting data is the first 90% of the work,
-managing it is the other 90%.
-In this chapter,
-we'll see how to use a database to store and analyze field observations.
-The techniques we will explore apply directly to other kinds of databases as well,
-and as we'll see,
-knowing how to get information *out* of a database is essential to
-figuring out how to put data *in*.
-
-## For Instructors {.guide}
-
-Relational databases are not as widely used in science as in business,
-but they are still a common way to store large data sets with complex structure.
-Even when the data itself isn't in a database,
-the metadata could be:
-for example,
-meteorological data might be stored in files on disk,
-but data about when and where observations were made,
-data ranges,
-and so on could be in a database
-to make it easier for scientists to find what they want to.
-
-The first few sections
-(up to [Ordering Results](#s:sort))
-usually go very quickly.
-The pace usually slows down a bit when null values and aggregation are discussed,
-mostly because learners have a lot of details to keep straight by this point.
-Things *really* slow down during the discussion of joins,
-but this is the key idea in the whole lesson:
-important ideas like primary keys and referential integrity only make sense
-once learners have seen how they're used in joins.
-It's worth going over things a couple of times if necessary
-(with lots of examples).
-
-The final three sections are independent of each other,
-and can be dropped if time is short.
-Of the three,
-people seem to care most about how to add data
-(which only takes a few minutes to demonstrate),
-and how to use databases from inside "real" programs.
-The material on transactions is more abstract than the rest,
-and should be omitted if [web programming](web.html)
-isn't being taught.
-Overall,
-this material takes three hours to present
-assuming that a short exercise is done with each topic.
-
-### Prerequisites {.prereq}
-
-Everything up to the [final section](#s:programming)
-only requires some understanding of Boolean operators,
-data types,
-and pipelines,
-and what's needed can actually be introduced on the fly.
-That [final section](#s:programming),
-which shows how to use databases from within programs,
-depends on most of the [basic Python material](python.html).
-
-### Teaching Notes {.notes}
-
-* It isn't necessary to cover [sets and dictionaries](setdict.html)
- before this material,
- but if that has been discussed,
- it's helpful to point out that a relational table is a generalized dictionary.
-* Simple calculations are actually easier to do in a spreadsheet,
- the advantages of using a database become clear
- as soon as filtering and joins are needed.
- Instructors may therefore want to show a spreadsheet
- with the information from the four database tables
- consolidated into a single sheet,
- and demonstrate what's needed in both systems to answer questions like,
- "What was the average radiation reading in 1931?"
-* Some learners may have heard that NoSQL databases
- (i.e., ones that don't use the relational model)
- are the next big thing,
- and ask why we're not teaching those.
- The answers are:
- * Relational databases are far more widely used than NoSQL databases.
- * We have far more experience with relational databases
- than with any other kind,
- so we have a better idea of what to teach
- and how to teach it.
- * NoSQL databases are as different from each other
- as they are from relational databases.
- Until a leader emerges,
- it isn't clear *which* NoSQL database we should teach.
-* This discussion is a useful companion to that of vectorization
- in the lesson on [numerical computing](numpy.html):
- in both cases,
- the key point is to describe *what* to do,
- and let the computer figure out *how* to do it.
-
-## Selecting {#s:select}
-
-### Learning Objectives {.objectives}
-
-* Explain the difference between a table, a database, and a database manager.
-* Explain the difference between a field and a record.
-* Select specific fields from specific tables, and display them in a specific order.
-
-Duration: 15 minutes (not including time required to download database file and connect to it)
-
-### Lesson
-
-A [relational database](../gloss.html#relational-database)
-is a way to store and manipulate information
-that is arranged as [tables](../gloss.html#table).
-Each table has columns (also known as [fields](../gloss.html#field-database)) which describe the data,
-and rows (also known as [records](../gloss.html#record-database)) which contain the data.
-
-<a id="a:dbms"></a>
-When we are using a spreadsheet,
-we put formulas into cells to calculate new values based on old ones.
-When we are using a database,
-we send commands
-(usually called [queries](../gloss.html#query))
-to a [database manager](../gloss.html#database-manager):
-a program that manipulates the database for us.
-The database manager does whatever lookups and calculations the query specifies,
-returning the results in a tabular form
-that we can then use as a starting point for further queries.
-
-> ### Under the Hood {.box}
->
-> Every database manager—Oracle,
-> IBM DB2, PostgreSQL, MySQL, Microsoft Access, and SQLite—stores
-> data in a different way,
-> so a database created with one cannot be used directly by another.
-> However,
-> every database manager can import and export data in a variety of formats,
-> so it *is* possible to move information from one to another.
-
-Queries are written in a language called [SQL](../gloss.html#sql),
-which stands for "Structured Query Language".
-SQL provides hundreds of different ways to analyze and recombine data;
-we will only look at a handful,
-but that handful accounts for most of what scientists do.
-
-[Figure 1](#f:survey_db) shows
-a simple database that stores some of the data
-Gina extracted from the logs of those long-ago expeditions.
-It contains four tables:
-
-<div class="db">
-
-Table Purpose
--------------------- --------------------
-`Person` People who took readings.
-`Site` Locations of observation sites.
-`Visited` When readings were taken at specific sites.
-`Survey` The actual measurement values.
-
-</div>
-
-**Person**
-
-<div class="db">
-
-ident personal family
--------------------- -------------------- --------------------
-dyer William Dyer
-pb Frank Pabodie
-lake Anderson Lake
-roe Valentina Roerich
-danforth Frank Danforth
-
-</div>
-
-**Survey**
-
-<div class="db">
-
-taken person quant reading
--------------------- -------------------- -------------------- --------------------
-619 dyer rad 9.82
-619 dyer sal 0.13
-622 dyer rad 7.8
-622 dyer sal 0.09
-734 pb rad 8.41
-734 lake sal 0.05
-734 pb temp -21.5
-735 pb rad 7.22
-735 NULL sal 0.06
-735 NULL temp -26.0
-751 pb rad 4.35
-751 pb temp -18.5
-751 lake sal 0.1
-752 lake rad 2.19
-752 lake sal 0.09
-752 lake temp -16.0
-752 roe sal 41.6
-837 lake rad 1.46
-837 lake sal 0.21
-837 roe sal 22.5
-844 roe rad 11.25
-
-</div>
-
-**Site**
-
-<div class="db">
-
-name lat long
--------------------- -------------------- --------------------
-DR-1 -49.85 -128.57
-DR-3 -47.15 -126.72
-MSK-4 -48.87 -123.4
-
-</div>
-
-**Visited**
-
-<div class="db">
-
-ident site dated
--------------------- -------------------- --------------------
-619 DR-1 1927-02-08
-622 DR-1 1927-02-10
-734 DR-3 1939-01-07
-735 DR-3 1930-01-12
-751 DR-3 1930-02-26
-752 DR-3 NULL
-837 MSK-4 1932-01-14
-844 DR-1 1932-03-22
-
-</div>
-
-<figcaption>Figure 1: Survey Database</figcaption>
-
-Notice that three entries—one in the `Visited` table,
-and two in the `Survey` table—are shown as `NULL`.
-We'll return to these values [later](#s:null).
-For now,
-let's write an SQL query that displays scientists' names.
-We do this using the SQL command `select`,
-giving it the names of the columns we want and the table we want them from.
-Our query and its output look like this:
-
- sqlite> select family, personal from Person;
-
-<div class="db">
-
--------------------- --------------------
-Dyer William
-Pabodie Frank
-Lake Anderson
-Roerich Valentina
-Danforth Frank
--------------------- --------------------
-
-</div>
-
-The semi-colon at the end of the query
-tells the database manager that the query is complete and ready to run.
-If we enter the query without the semi-colon,
-or press 'enter' part-way through the query,
-the SQLite interpreter will give us a different prompt
-to show us that it's waiting for more input:
-
- sqlite> select family, personal
- ...> from Person
- ...> ;
-
-<div class="db">
-
--------------------- --------------------
-Dyer William
-Pabodie Frank
-Lake Anderson
-Roerich Valentina
-Danforth Frank
--------------------- --------------------
-
-</div>
-
-From now on,
-we won't bother to display the prompt(s) with our commands.
-
-> ### Case and Consistency {.box}
->
-> We have written our command and the column names in lower case,
-> and the table name in title case,
-> but we could use any mix:
-> SQL is [case insensitive](../gloss.html#case-insensitive),
-> so we could write them all in upper case,
-> or even like this:
->
-> ``` {.sql}
-> SeLeCt famILY, PERSonal frOM PERson;
-> ```
->
-> But please don't:
-> large SQL queries are hard enough to read
-> without the extra cognitive load of random capitalization.
-
-> ### Displaying Results {.box}
->
-> Exactly *how* the database displays the query's results
-> depends on what kind of interface we are using.
-> If we are running SQLite directly from the shell,
-> its default output looks like this:
->
-> Dyer|William
-> Pabodie|Frank
-> Lake|Anderson
-> Roerich|Valentina
-> Danforth|Frank
->
-> If we are using a graphical interface,
-> such as the [SQLite Manager](https://addons.mozilla.org/en-US/firefox/addon/sqlite-manager/) plugin for Firefox
-> or the [database extension](https://github.com/catherinedevlin/ipython-sql) for the IPython Notebook,
-> our output will be displayed graphically
-> ([Figure 2](#f:firefox_output)
-> and [Figure 3](#f:notebook_output)).
-> We'll use a simple table-based display in these notes.
->
-> <figure id="f:firefox_output">
-> <img src="db/firefox_output.png" alt="Firefox SQLite Manager Output" />
-> <figcaption>Figure 2: Firefox SQLite Manager Output</figcaption>
-> </figure>
->
-> <figure id="f:notebook_output">
-> <img src="db/notebook_output.png" alt="IPython Notebook Database Extension Output" />
-> <figcaption>Figure 3: IPython Notebook Database Extension Output</figcaption>
-> </figure>
-
-Going back to our query,
-it's important to understand that
-the rows and columns in a database table aren't actually stored in any particular order.
-They will always be *displayed* in some order,
-but we can control that in various ways.
-For example,
-we could swap the columns in the output by writing our query as:
-
-``` {.sql}
-select personal, family from Person;
-```
-
-<div class="db">
-
--------------------- --------------------
-William Dyer
-Frank Pabodie
-Anderson Lake
-Valentina Roerich
-Frank Danforth
--------------------- --------------------
-
-</div>
-
-or even repeat columns:
-
-``` {.sql}
-select ident, ident, ident from Person;
-```
-
-<div class="db">
-
--------------------- -------------------- --------------------
-dyer dyer dyer
-pb pb pb
-lake lake lake
-roe roe roe
-danforth danforth danforth
--------------------- -------------------- --------------------
-
-</div>
-
-We will see ways to rearrange the rows [later](#s:sort).
-
-As a shortcut, we can select all of the columns in a table
-using the wildcard `*`:
-
-``` {.sql}
-select * from Person;
-```
-
-<div class="db">
-
--------------------- -------------------- --------------------
-dyer William Dyer
-pb Frank Pabodie
-lake Anderson Lake
-roe Valentina Roerich
-danforth Frank Danforth
--------------------- -------------------- --------------------
-
-</div>
-
-### Summary {.keypoints}
-
-* A relational database stores information in tables with fields and records.
-* A database manager is a program that manipulates a database.
-* The commands or queries given to a database manager are usually written in a specialized language called SQL.
-* SQL is case insensitive.
-* The rows and columns of a database table aren't stored in any particular order.
-* Use `select *fields* from *table*` to get all the values for specific fields from a single table.
-* Use `select * from *table*` to select everything from a table.
-
-### Challenges {.challenges}
-
-* Write a query that selects only site names from the `Site` table.
-
-* Many people format queries as:
-
- ```
- SELECT personal, family FROM person;
- ```
-
- or as:
-
- ```
- select Personal, Family from PERSON;
- ```
-
- What style do you find easiest to read, and why?
-
-## Removing Duplicates {#s:distinct}
-
-### Learning Objectives {.objectives}
-
-* Write queries that only display distinct results once.
-
-Duration: 5 minutes.
-
-### Lesson
-
-Data is often redundant,
-so queries often return redundant information.
-For example,
-if we select the quantitites that have been measured
-from the `survey` table,
-we get this:
-
-``` {.sql}
-select quant from Survey;
-```
-
-<div class="db">
-
---------------------
-rad
-sal
-rad
-sal
-rad
-sal
-temp
-rad
-sal
-temp
-rad
-temp
-sal
-rad
-sal
-temp
-sal
-rad
-sal
-sal
-rad
---------------------
-
-</div>
-
-We can eliminate the redundant output
-to make the result more readable
-by adding the `distinct` keyword
-to our query:
-
-``` {.sql}
-select distinct quant from Survey;
-```
-
-<div class="db">
-
---------------------
-rad
-sal
-temp
---------------------
-
-</div>
-
-If we select more than one column—for example,
-both the survey site ID and the quantity measured—then
-the distinct pairs of values are returned:
-
-``` {.sql}
-select distinct taken, quant from Survey;
-```
-
-<div class="db">
-
--------------------- --------------------
-619 rad
-619 sal
-622 rad
-622 sal
-734 rad
-734 sal
-734 temp
-735 rad
-735 sal
-735 temp
-751 rad
-751 temp
-751 sal
-752 rad
-752 sal
-752 temp
-837 rad
-837 sal
-844 rad
--------------------- --------------------
-
-</div>
-
-Notice in both cases that duplicates are removed
-even if they didn't appear to be adjacent in the database.
-Again,
-it's important to remember that rows aren't actually ordered:
-they're just displayed that way.
-
-### Summary {.keypoints}
-
-* Use `distinct` to eliminate duplicates from a query's output.
-
-### Challenges {.challenges}
-
-* Write a query that selects distinct dates from the `Site` table.
-
-* If you are using SQLite from the command line,
- you can run a single query by passing it to the interpreter
- right after the path to the database file:
-
- ```
- $ sqlite3 survey.db 'select * from Person;'
- ```
-
- <div class="db">
-
- -------------------- -------------------- --------------------
- dyer William Dyer
- pb Frank Pabodie
- lake Anderson Lake
- roe Valentina Roerich
- danforth Frank Danforth
- -------------------- -------------------- --------------------
-
- </div>
-
- Fill in the missing commands in the pipeline below
- so that the output contains no redundant values.
-
- ```
- $ sqlite3 survey.db 'select person, quant from Survey;' | ____ | ____
- ```
-
- Do you think this is less efficient, just as efficient, or more efficient
- that using `distinct` for large data?
-
-## Filtering {#s:filter}
-
-### Learning Objectives {.objectives}
-
-* Write queries that select records based on the values of their fields.
-* Write queries that select records using combinations of several tests on their fields' values.
-* Build up complex filtering criteria incrementally.
-* Explain the logical order in which filtering by field value and displaying fields takes place.
-
-Duration: 5-10 minutes.
-
-### Lesson
-
-One of the most powerful features of a database is
-the ability to [filter](../gloss.html#filter) data,
-i.e.,
-to select only those records that match certain criteria.
-For example,
-suppose we want to see when a particular site was visited.
-We can select these records from the `Visited` table
-by using a `where` clause in our query:
-
-``` {.sql}
-select * from Visited where site='DR-1';
-```
-
-<div class="db">
-
--------------------- -------------------- --------------------
-619 DR-1 1927-02-08
-622 DR-1 1927-02-10
-844 DR-1 1932-03-22
--------------------- -------------------- --------------------
-
-</div>
-
-The database manager executes this query in two stages
-([Figure 4](#f:pipeline_where)).
-First,
-it checks at each row in the `Visited` table
-to see which ones satisfy the `where`.
-It then uses the column names following the `select` keyword
-to determine what columns to display.
-
- <figure id="f:pipeline_where">
- <img src="db/pipeline_where.png" alt="Two-Stage Query Processing Pipeline" />
- <figcaption>Figure 4: Two-Stage Query Processing Pipeline</figcaption>
- </figure>
-
-This processing order means that
-we can filter records using `where`
-based on values in columns that aren't then displayed:
-
-``` {.sql}
-select ident from Visited where site='DR-1';
-```
-
-<div class="db">
-
---------------------
-619
-622
-844
---------------------
-
-</div>
-
-We can use many other Boolean operators to filter our data.
-For example,
-we can ask for all information from the DR-1 site collected since 1930:
-
-``` {.sql}
-select * from Visited where (site='DR-1') and (dated>='1930-00-00');
-```
-
-<div class="db">
-
--------------------- -------------------- --------------------
-844 DR-1 1932-03-22
--------------------- -------------------- --------------------
-
-</div>
-
-(The parentheses around the individual tests aren't strictly required,
-but they help make the query easier to read.)
-
-> ### Working With Dates {#a:dates .box}
->
-> Most database managers have a special data type for dates.
-> In fact, many have two:
-> one for dates,
-> such as "May 31, 1971",
-> and one for durations,
-> such as "31 days".
-> SQLite doesn't:
-> instead,
-> it stores dates as either text
-> (in the ISO-8601 standard format "YYYY-MM-DD HH:MM:SS.SSSS"),
-> real numbers
-> (the number of days since November 24, 4714 BCE),
-> or integers
-> (the number of seconds since midnight, January 1, 1970).
-> If this sounds complicated,
-> it is,
-> but not nearly as complicated as figuring out
-> [historical dates in Sweden](http://en.wikipedia.org/wiki/Swedish_calendar).
-
-If we want to find out what measurements were taken by either Lake or Roerich,
-we can combine the tests on their names using `or`:
-
-``` {.sql}
-select * from Survey where person='lake' or person='roe';
-```
-
-<div class="db">
-
--------------------- -------------------- -------------------- --------------------
-734 lake sal 0.05
-751 lake sal 0.1
-752 lake rad 2.19
-752 lake sal 0.09
-752 lake temp -16.0
-752 roe sal 41.6
-837 lake rad 1.46
-837 lake sal 0.21
-837 roe sal 22.5
-844 roe rad 11.25
--------------------- -------------------- -------------------- --------------------
-
-</div>
-
-Alternatively,
-we can use `in` to see if a value is in a specific set:
-
-``` {.sql}
-select * from Survey where person in ('lake', 'roe');
-```
-
-<div class="db">
-
--------------------- -------------------- -------------------- --------------------
-734 lake sal 0.05
-751 lake sal 0.1
-752 lake rad 2.19
-752 lake sal 0.09
-752 lake temp -16.0
-752 roe sal 41.6
-837 lake rad 1.46
-837 lake sal 0.21
-837 roe sal 22.5
-844 roe rad 11.25
--------------------- -------------------- -------------------- --------------------
-
-</div>
-
-We can combine `and` with `or`,
-but we need to be careful about which operator is executed first.
-If we *don't* use parentheses,
-we get this:
-
-``` {.sql}
-select * from Survey where quant='sal' and person='lake' or person='roe';
-```
-
-<div class="db">
-
--------------------- -------------------- -------------------- --------------------
-734 lake sal 0.05
-751 lake sal 0.1
-752 lake sal 0.09
-752 roe sal 41.6
-837 lake sal 0.21
-837 roe sal 22.5
-844 roe rad 11.25
--------------------- -------------------- -------------------- --------------------
-
-</div>
-
-which is salinity measurements by Lake,
-and *any* measurement by Roerich.
-We probably want this instead:
-
-``` {.sql}
-select * from Survey where quant='sal' and (person='lake' or person='roe');
-```
-
-<div class="db">
-
--------------------- -------------------- -------------------- --------------------
-734 lake sal 0.05
-751 lake sal 0.1
-752 lake sal 0.09
-752 roe sal 41.6
-837 lake sal 0.21
-837 roe sal 22.5
--------------------- -------------------- -------------------- --------------------
-
-</div>
-
-Finally,
-we can use `distinct` with `where`
-to give a second level of filtering:
-
-``` {.sql}
-select distinct person, quant from Survey where person='lake' or person='roe';
-```
-
-<div class="db">
-
--------------------- --------------------
-lake sal
-lake rad
-lake temp
-roe sal
-roe rad
--------------------- --------------------
-
-</div>
-
-But remember:
-`distinct` is applied to the values displayed in the chosen columns,
-not to the entire rows as they are being processed.
-
-> ### Growing Queries {.box}
->
-> What we have just done is how most people "grow" their SQL queries.
-> We started with something simple that did part of what we wanted,
-> then added more clauses one by one,
-> testing their effects as we went.
-> This is a good strategy—in fact,
-> for complex queries it's often the *only* strategy—but
-> it depends on quick turnaround,
-> and on us recognizing the right answer when we get it.
->
-> The best way to achieve quick turnaround is often
-> to put a subset of data in a temporary database
-> and run our queries against that,
-> or to fill a small database with synthesized records.
-> For example,
-> instead of trying our queries against an actual database of 20 million Australians,
-> we could run it against a sample of ten thousand,
-> or write a small program to generate ten thousand random (but plausible) records
-> and use that.
-
-### Summary {.keypoints}
-
-* Use `where *test*` in a query to filter records based on Boolean tests.
-* Use `and` and `or` to combine tests.
-* Use `in` to check if a value is in a set.
-* Build up queries a bit at a time, and test them against small data sets.
-
-### Challenges {.challenges}
-
-* Gina wants to select all sites that lie within 30° of the equator.
- Her query is:
-
- ``` {.sql}
- select * from Site where (lat > -30) or (lat < 30);
- ``` {.sql}
-
- Explain why this is wrong,
- and rewrite the query so that it is correct.
-
-* Normalized salinity readings are supposed to be between 0.0 and 1.0.
- Write a query that selects all records from `Survey`
- with salinity values outside this range.
-
-* The SQL test `*column-name* like *pattern*`
- is true if the value in the named column
- matches the pattern given;
- the character '%' can be used any number of times in the pattern
- to mean "match zero or more characters".
-
- Expression Value
- -------------------- --------------------
- `'a' like 'a'` `True`
- `'a' like '%a'` `True`
- `'b' like '%a'` `False`
- `'alpha' like 'a%'` `True`
- `'alpha' like 'a%p%'` `True`
- `'beta' like 'a%p%'` `False`
-
- The expression `*column-name* not like *pattern*`
- inverts the test.
- Using `like`,
- write a query that finds all the records in `Visited`
- that *aren't* from sites labelled 'DR-something'.
-
-## Calculating New Values {#s:calc}
-
-### Learning Objectives {.objectives}
-
-* Write queries that do arithmetic using the values in individual records.
-
-Duration: 5 minutes.
-
-### Lesson
-
-After carefully reading the expedition logs,
-Gina realizes that the radiation measurements they report
-may need to be corrected upward by 5%.
-Rather than modifying the stored data,
-she can do this calculation on the fly
-as part of her query:
-
-``` {.sql}
-select 1.05 * reading from Survey where quant='rad';
-```
-
-<div class="db">
-
---------------------
-10.311
-8.19
-8.8305
-7.581
-4.5675
-2.2995
-1.533
-11.8125
---------------------
-
-</div>
-
-When we run the query,
-the expression `1.05 * reading` is evaluated for each row.
-Expressions can use any of the fields,
-all of usual arithmetic operators,
-and a variety of common functions.
-(Exactly which ones depends on which database manager is being used.)
-For example,
-we can convert temperature readings from Fahrenheit to Celsius
-and round to two decimal places as follows:
-
-``` {.sql}
-select taken, round(5*(reading-32)/9, 2) from Survey where quant='temp';
-```
-
-<div class="db">
-
--------------------- --------------------
-734 -29.72
-735 -32.22
-751 -28.06
-752 -26.67
--------------------- --------------------
-
-</div>
-
-We can also combine values from different fields,
-for example by using the string concatenation operator `||`:
-
-``` {.sql}
-select personal || ' ' || family from Person;
-```
-
-<div class="db">
-
---------------------
-William Dyer
-Frank Pabodie
-Anderson Lake
-Valentina Roerich
-Frank Danforth
---------------------
-
-</div>
-
-> ### A Note on Names {.box}
->
-> It may seem strange to use `personal` and `family` as field names
-> instead of `first` and `last`,
-> but it's a necessary first step toward handling cultural differences.
-> For example,
-> consider the following rules:
->
-> Full Name Alphabetized Under Reason
-> -------------------- -------------------- --------------------
-> Liu Xiaobo Liu Chinese family names come first
-> Leonardo da Vinci Leonardo "da Vinci" just means "from Vinci"
-> Catherine de Medici Medici family name
-> Jean de La Fontaine La Fontaine family name is "La Fontaine"
-> Juan Ponce de Leon Ponce de Leon full family name is "Ponce de Leon"
-> Gabriel Garcia Marquez Garcia Marquez double-barrelled Spanish surnames
-> Wernher von Braun von *or* Braun depending on whether he was in Germany or the US
-> Elizabeth Alexandra May Windsor Elizabeth monarchs alphabetize by the name under which they reigned
-> Thomas a Beckett Thomas and saints according to the names by which they were canonized
->
-> Clearly,
-> even a two-part division into "personal" and "family"
-> isn't enough...
-
-### Summary {.keypoints}
-
-* Use expressions as fields to calculate per-record values.</li>
-
-### Challenges {.challenges}
-
-* After further reading,
- Gina realizes that Valentina Roerich
- was reporting salinity as percentages.
- Write a query that returns all of her salinity measurements
- from the `Survey` table
- with the values divided by 100.
-
-* The `union` operator combines the results of two queries:
-
- ``` {.sql}
- select * from Person where ident='dyer' union select * from Person where ident='roe';
- ```
-
- <div class="db">
-
- -------------------- -------------------- --------------------
- dyer William Dyer
- roe Valentina Roerich
- -------------------- -------------------- --------------------
-
- </div>
-
- Use `union` to create a consolidated list of salinity measurements
- in which Roerich's, and only Roerich's,
- have been corrected as described in the previous challenge.
- The output should be something like:
-
- <div class="db">
-
- -------------------- --------------------
- 619 0.13
- 622 0.09
- 734 0.05
- 751 0.1
- 752 0.09
- 752 0.416
- 837 0.21
- 837 0.225
- -------------------- --------------------
-
- </div>
-
-* The site identifiers in the `Visited` table have two parts
- separated by a '-':
-
- ``` {.sql}
- select distinct site from Visited;
- ```
-
- <div class="db">
-
- --------------------
- DR-1
- DR-3
- MSK-4
- --------------------
-
- </div>
-
- Some major site identifiers are two letters long and some are three.
- The "in string" function `instr(X, Y)`
- returns the 1-based index of the first occurrence of string Y in string X,
- or 0 if Y does not exist in X.
- The substring function `substr(X, I)`
- returns the substring of X starting at index I.
- Use these two functions to produce a list of unique major site identifiers.
- (For this data,
- the list should contain only "DR" and "MSK").
-
-* Pabodie's journal notes that all his temperature measurements
- are in °F,
- but Lake's journal does not report whether he used °F or °C.
- How should Gina treat his measurements,
- and why?
-
-## Ordering Results {#s:sort}
-
-### Learning Objectives {.objectives}
-
-* Write queries that order results according to fields' values.
-* Write queries that order results according to calculated values.
-* Explain why it is possible to sort records using the values of fields that are not displayed.
-
-Duration: 5 minutes.
-
-### Lesson
-
-As we mentioned earlier,
-database records are not stored in any particular order.
-This means that query results aren't necessarily sorted,
-and even if they are,
-we often want to sort them in a different way,
-e.g., by the name of the project instead of by the name of the scientist.
-We can do this in SQL by adding an `order by` clause to our query:
-
-``` {.sql}
-select reading from Survey where quant='rad' order by reading;
-```
-
-<div class="db">
-
---------------------
-1.46
-2.19
-4.35
-7.22
-7.8
-8.41
-9.82
-11.25
---------------------
-
-</div>
-
-By default,
-results are sorted in ascending order
-(i.e.,
-from least to greatest).
-We can sort in the opposite order using `desc` (for "descending"):
-
-``` {.sql}
-select reading from Survey where quant='rad' order by reading desc;
-```
-
-<div class="db">
-
---------------------
-11.25
-9.82
-8.41
-7.8
-7.22
-4.35
-2.19
-1.46
---------------------
-
-</div>
-
-(And if we want to make it clear that we're sorting in ascending order,
-we can use `asc` instead of `desc`.)
-
-We can also sort on several fields at once.
-For example,
-this query sorts results first in ascending order by `taken`,
-and then in descending order by `person`
-within each group of equal `taken` values:
-
-``` {.sql}
-select taken, person from Survey order by taken asc, person desc;
-```
-
-<div class="db">
-
--------------------- --------------------
-619 dyer
-619 dyer
-622 dyer
-622 dyer
-734 pb
-734 pb
-734 lake
-735 pb
-735
-735
-751 pb
-751 pb
-751 lake
-752 roe
-752 lake
-752 lake
-752 lake
-837 roe
-837 lake
-837 lake
-844 roe
--------------------- --------------------
-
-</div>
-
-This is easier to understand if we also remove duplicates:
-
-``` {.sql}
-select distinct taken, person from Survey order by taken asc, person desc;
-```
-
-<div class="db">
-
--------------------- --------------------
-619 dyer
-622 dyer
-734 pb
-734 lake
-735 pb
-735
-751 pb
-751 lake
-752 roe
-752 lake
-837 roe
-837 lake
-844 roe
--------------------- --------------------
-
-</div>
-
-Since sorting happens before columns are filtered,
-we can sort by a field that isn't actually displayed:
-
-``` {.sql}
-select reading from Survey where quant='rad' order by taken;
-```
-
-<div class="db">
-
---------------------
-9.82
-7.8
-8.41
-7.22
-4.35
-2.19
-1.46
-11.25
---------------------
-
-</div>
-
-We can also sort results by the value of an expression.
-In SQLite,
-for example,
-the `random` function returns a pseudo-random integer
-each time it is called
-(i.e.,
-once per record):
-
-``` {.sql}
-select random(), ident from Person;
-```
-
-<div class="db">
-
--------------------- --------------------
--6309766557809954936 dyer
--2098461436941487136 pb
--2248225962969032314 lake
-6062184424509295966 roe
--1268956870222271271 danforth
--------------------- --------------------
-
-</div>
-
-So to randomize the order of our query results,
-e.g., when doing clinical trials,
-we can sort them by the value of this function:
-
-``` {.sql}
-select ident from Person order by random();
-```
-
-<div class="db">
-
---------------------
-danforth
-pb
-dyer
-lake
-roe
---------------------
-
-</div>
-
-``` {.sql}
-select ident from Person order by random();
-```
-
-<div class="db">
-
---------------------
-roe
-dyer
-pb
-lake
-danforth
---------------------
-
-</div>
-
-Our query pipeline now has four stages
-([Figure 5](#f:pipeline_sort_distinct)):
-
-* Select the rows that pass the `where` criteria.
-* Sort them if required.
-* Filter the columns according to the `select` criteria.
-* Remove duplicates if required.
-
- <figure id="f:pipeline_sort_distinct">
- <img src="db/pipeline_sort_distinct.png" alt="Four-Stage Query Processing Pipeline" />
- <figcaption>Figure 5: Four-Stage Query Processing Pipeline</figcaption>
- </figure>
-
-### Summary {.keypoints}
-
-* Use `order by` (with `asc` or `desc`) to order a query's results.
-* Use `random` to generate pseudo-random numbers.
-
-### Challenges {.challenges}
-
-* Create a list of sites identifiers
- and their distance from the equator in kilometers,
- sorted from furthest to closest.
- (A degree of latitude corresponds to 111.12 km.)
-
-* Gina needs a list of radiation measurements from all sites
- sorted by when they were taken.
- The query:
-
- ``` {.sql}
- select * from Survey where quant='rad' order by taken;
- ```
-
- produces the correct answer for the data used in our examples.
- Explain when and why it might produce the wrong answer.
-
-## Missing Data {#s:null}
-
-### Learning Objectives {.objectives}
-
-* Explain what databases use the special value `NULL` to represent.
-* Explain why databases should *not* uses their own special values (like 9999 or "N/A") to represent missing or unknown data.
-* Explain what atomic and aggregate calculations involving `NULL` produce, and why.
-* Write queries that include or exclude records containing `NULL`.
-
-Duration: 10-20 minutes
-(depending on whether or not the instructor includes an anecdote about
-what happens when you *don't* take missing data into account).
-
-### Lesson
-
-Real-world data is never complete—there are always holes.
-Databases represent these holes using special value called `null`.
-`null` is not zero, `False`, or the empty string;
-it is a one-of-a-kind value that means "nothing here".
-Dealing with `null` requires a few special tricks
-and some careful thinking.
-
-To start,
-let's have a look at the `Visited` table.
-There are eight records,
-but #752 doesn't have a date—or rather,
-its date is null:
-
-``` {.sql}
-select * from Visited;
-```
-
-<div class="db">
-
--------------------- -------------------- --------------------
-619 DR-1 1927-02-08
-622 DR-1 1927-02-10
-734 DR-3 1939-01-07
-735 DR-3 1930-01-12
-751 DR-3 1930-02-26
-752 DR-3
-837 MS-4 1932-01-14
-844 DR-1 1932-03-22
--------------------- -------------------- --------------------
-
-</div>
-
-> ### Displaying Nulls {.box}
->
-> Different databases display nulls differently.
-> Unfortunately,
-> SQLite's default is to print nothing at all,
-> which makes nulls easy to overlook
-> (particularly if they're in the middle of a long row).
-
-Null doesn't behave like other values.
-If we select the records that come before 1930:
-
-``` {.sql}
-select * from Visited where dated<'1930-00-00';
-```
-
-<div class="db">
-
--------------------- -------------------- --------------------
-619 DR-1 1927-02-08
-622 DR-1 1927-02-10
--------------------- -------------------- --------------------
-
-</div>
-
-we get two results,
-and if we select the ones that come during or after 1930:
-
-``` {.sql}
-select * from Visited where dated>='1930-00-00';
-```
-
-<div class="db">
-
--------------------- -------------------- --------------------
-734 DR-3 1939-01-07
-735 DR-3 1930-01-12
-751 DR-3 1930-02-26
-837 MS-4 1932-01-14
-844 DR-1 1932-03-22
--------------------- -------------------- --------------------
-
-</div>
-
-we get five,
-but record #752 isn't in either set of results.
-The reason is that
-`null<'1930-00-00'`
-is neither true nor false:
-null means, "We don't know,"
-and if we don't know the value on the left side of a comparison,
-we don't know whether the comparison is true or false.
-Since databases represent "don't know" as null,
-the value of `null<'1930-00-00'`
-is actually `null`.
-`null>='1930-00-00'` is also null
-because we can't answer to that question either.
-And since the only records kept by a `where`
-are those for which the test is true,
-record #752 isn't included in either set of results.
-
-Comparisons aren't the only operations that behave this way with nulls.
-`1+null` is `null`,
-`5*null` is `null`,
-`log(null)` is `null`,
-and so on.
-In particular,
-comparing things to null with = and != produces null:
-
-``` {.sql}
-select * from Visited where dated=NULL;
-```
-
-``` {.sql}
-select * from Visited where dated!=NULL;
-```
-
-To check whether a value is `null` or not,
-we must use a special test `is null`:
-
-``` {.sql}
-select * from Visited where dated is NULL;
-```
-
-<div class="db">
-
--------------------- --------------------
-752 DR-3
--------------------- --------------------
-
-</div>
-
-or its inverse `is not null`:
-
-``` {.sql}
-select * from Visited where dated is not NULL;
-```
-
-<div class="db">
-
--------------------- -------------------- --------------------
-619 DR-1 1927-02-08
-622 DR-1 1927-02-10
-734 DR-3 1939-01-07
-735 DR-3 1930-01-12
-751 DR-3 1930-02-26
-837 MS-4 1932-01-14
-844 DR-1 1932-03-22
--------------------- -------------------- --------------------
-
-</div>
-
-Null values cause headaches wherever they appear.
-For example,
-suppose we want to find the all of salinity measurements
-that weren't taken by Dyer.
-It's natural to write the query like this:
-
-``` {.sql}
-select * from Survey where quant='sal' and person!='lake';
-```
-
-<div class="db">
-
--------------------- -------------------- -------------------- --------------------
-619 dyer sal 0.13
-622 dyer sal 0.09
-752 roe sal 41.6
-837 roe sal 22.5
--------------------- -------------------- -------------------- --------------------
-
-</div>
-
-but this query filters omits the records
-where we don't know who took the measurement.
-Once again,
-the reason is that when `person` is `null`,
-the `!=` comparison produces `null`,
-so the record isn't kept in our results.
-If we want to keep these records
-we need to add an explicit check:
-
-``` {.sql}
-select * from Survey where quant='sal' and (person!='lake' or person is null);
-```
-
-<div class="db">
-
--------------------- -------------------- -------------------- --------------------
-619 dyer sal 0.13
-622 dyer sal 0.09
-735 sal 0.06
-752 roe sal 41.6
-837 roe sal 22.5
--------------------- -------------------- -------------------- --------------------
-
-</div>
-
-
-We still have to decide whether this is the right thing to do or not.
-If we want to be absolutely sure that
-we aren't including any measurements by Lake in our results,
-we need to exclude all the records for which we don't know who did the work.
-
-> ### What Happens When You Forget {.box}
->
-> Several years ago,
-> I was helping a group who were looking at
-> the spread of drug-resistant tuberculosis (DRTB)
-> in industrialized countries.
-> In particular,
-> they wanted to know if it was spreading faster among less affluent people.
->
-> We tackled the problem by combining two data sets.
-> The first gave us skin and blood test results for DRTB
-> along with patients' postal codes
-> (the only identifying information we were allowed---we didn't even have gender).
-> The second was Canadian census data that gave us
-> median income per postal code.
-> Since a PC is about 300-800 people,
-> we felt justified in joining the first with the second
-> to estimate incomes for people with positive and negative test results.
->
-> To our surprise,
-> we didn't find a correlation between income and infection.
-> We were just about to publish when someone spotted the mistake I'd made.
->
-> Question: Who *doesn't* have a postal code?
->
-> Answer: Homeless people.
->
-> When I did the join,
-> I was throwing away homeless people,
-> which introduced a statistically significant error in my results.
-> But I couldn't just set the income of anyone without a postal code to zero,
-> because our sample included another set of people without postal codes:
-> 16-21 year olds whose addresses were suppressed
-> because they had tested positive for sexually-transmitted diseases.
->
-> At this point the problem is no longer a database issue,
-> but rather a question of statistics.
-> The takeaway is,
-> checking your queries when you're programming is as important as
-> checking your samples when you're doing chemistry.
-
-### Summary {.keypoints}
-
-* Use `null` in place of missing information.
-* Almost every operation involving `null` produces `null` as a result.
-* Test for nulls using `is null` and `is not null`.
-
-### Challenges {.challenges}
-
-* Write a query that sorts the records in `Visited` by date,
- omitting entries for which the date is not known
- (i.e., is null).
-
-* What do you expect the query:
-
- ``` {.sql}
- select * from Visited where dated in ('1927-02-08', null);
- ```
-
- to produce?
- What does it actually produce?
-
-* Some database designers prefer to use
- a [sentinel value](../gloss.html#sentinel-value)
- to mark missing data rather than `null`.
- For example,
- they will use the date "0000-00-00" to mark a missing date,
- or -1.0 to mark a missing salinity or radiation reading
- (since actual readings cannot be negative).
- What does this simplify?
- What burdens or risks does it introduce?
-
-## Aggregation {#s:aggregate}
-
-### Learning Objectives {.objectives}
-
-* Write queries that combine values from many records to create a single aggregate value.
-* Write queries that put records into groups based on their values.
-* Write queries that combine values group by group.
-* Explain what is displayed for *unaggregated* fields when some fields are aggregated.
-
-Duration: 10 minutes.
-
-### Lesson
-
-Gina now wants to calculate ranges and averages for her data.
-She knows how to select all of the dates from the `Visited` table:
-
-``` {.sql}
-select dated from Visited;
-```
-
-<div class="db">
-
---------------------
-1927-02-08
-1927-02-10
-1939-01-07
-1930-01-12
-1930-02-26
-
-1932-01-14
-1932-03-22
---------------------
-
-</div>
-
-but to combine them,
-she must use an [aggregation function](../gloss.html#aggregation-function)
-such as `min` or `max`.
-Each of these functions takes a set of records as input,
-and produces a single record as output:
-
-``` {.sql}
-select min(dated) from Visited;
-```
-
-<div class="db">
-
---------------------
-1927-02-08
---------------------
-
-</div>
-
-``` {.sql}
-select max(dated) from Visited;
-```
-
-<div class="db">
-
---------------------
-1939-01-07
---------------------
-
-</div>
-
-`min` and `max` are just two of
-the aggregation functions built into SQL.
-Three others are `avg`,
-`count`,
-and `sum`:
-
-``` {.sql}
-select avg(reading) from Survey where quant='sal';
-```
-
-<div class="db">
-
---------------------
-7.20333333333
---------------------
-
-</div>
-
-``` {.sql}
-select count(reading) from Survey where quant='sal';
-```
-
-<div class="db">
-
---------------------
-9
---------------------
-
-</div>
-
-``` {.sql}
-select sum(reading) from Survey where quant='sal';
-```
-
-<div class="db">
-
---------------------
-64.83
---------------------
-
-</div>
-
-We used `count(reading)` here,
-but we could just as easily have counted `quant`
-or any other field in the table,
-or even used `count(*)`,
-since the function doesn't care about the values themselves,
-just how many values there are.
-
-SQL lets us do several aggregations at once.
-We can,
-for example,
-find the range of sensible salinity measurements:
-
-``` {.sql}
-select min(reading), max(reading) from Survey where quant='sal' and reading<=1.0;
-```
-
-<div class="db">
-
--------------------- --------------------
-0.05 0.21
--------------------- --------------------
-
-</div>
-
-We can also combine aggregated results with raw results,
-although the output might surprise you:
-
-``` {.sql}
-select person, count(*) from Survey where quant='sal' and reading<=1.0;
-```
-
-<div class="db">
-
--------------------- --------------------
-lake 7
--------------------- --------------------
-
-</div>
-
-Why does Lake's name appear rather than Roerich's or Dyer's?
-The answer is that when it has to aggregate a field,
-but isn't told how to,
-the database manager chooses an actual value from the input set.
-It might use the first one processed,
-the last one,
-or something else entirely.
-
-Another important fact is that when there are no values to aggregate,
-aggregation's result is "don't know"
-rather than zero or some other arbitrary value:
-
-``` {.sql}
-select person, max(reading), sum(reading) from Survey where quant='missing';
-```
-
-One final important feature of aggregation functions is that
-they are inconsistent with the rest of SQL in a very useful way.
-If we add two values,
-and one of them is null,
-the result is null.
-By extension,
-if we use `sum` to add all the values in a set,
-and any of those values are null,
-the result should also be null.
-It's much more useful,
-though,
-for aggregation functions to ignore null values
-and only combine those that are non-null.
-This behavior lets us write our queries as:
-
-``` {.sql}
-select min(dated) from Visited;
-```
-
-<div class="db">
-
---------------------
-1927-02-08
---------------------
-
-</div>
-
-instead of always having to filter explicitly:
-
-``` {.sql}
-select min(dated) from Visited where dated is not null;
-```
-
-<div class="db">
-
---------------------
-1927-02-08
---------------------
-
-</div>
-
-### Summary {.keypoints}
-
-* Use aggregation functions like `sum` and `max` to combine query results.
-* Use `count` function to count the number of results.
-* If some fields are aggregated and others are not, the database manager chooses an arbitrary result for the unaggregated field.
-* Most aggregation functions skip nulls when combining values.
-
-### Challenges {.challenges}
-
-* How many temperature readings did Frank Pabodie record,
- and what was their average value?
-
-* The average of a set of values is the sum of the values
- divided by the number of values.
- Does this mean that the `avg` function returns 2.0 or 3.0
- when given the values 1.0, `null`, and 5.0?
-
-* Gina wants to calculate the difference between
- each individual radiation reading
- and the average of all the radiation readings.
- She writes the query:
-
- ``` {.sql}
- select reading-avg(reading) from Survey where quant='rad';
- ```
-
- What does this actually produce, and why?
-
-* The function `group_concat(field, separator)`
- concatenates all the values in a field
- using the specified separator character
- (or ',' if the separator isn't specified).
- Use this to produce a one-line list of scientists' names,
- such as:
-
- ``` {.sql}
- William Dyer, Frank Pabodie, Anderson Lake, Valentina Roerich, Frank Danforth
- ```
-
- Can you find a way to order the list by surname?
-
-## Grouping {#s:grouping}
-
-### Learning Objectives {.objectives}
-
-* Group results to be aggregated separately.
-* Explain when grouping occurs in the processing pipeline.
-
-Duration: 5 minutes.
-
-### Lesson
-
-Aggregating all records at once doesn't always make sense.
-For example,
-suppose Gina suspects that there is a systematic bias in her data,
-and that some scientists' radiation readings are higher than others.
-We know that this doesn't work:
-
-``` {.sql}
-select person, count(reading), round(avg(reading), 2)
-from Survey
-where quant='rad';
-```
-
-<div class="db">
-
--------------------- -------------------- --------------------
-roe 8 6.56
--------------------- -------------------- --------------------
-
-</div>
-
-because the database manager selects a single arbitrary scientist's name
-rather than aggregating separately for each scientist.
-Since there are only five scientists,
-she could write five queries of the form:
-
-``` {.sql}
-select person, count(reading), round(avg(reading), 2)
-from Survey
-where quant='rad'
-and person='dyer';
-```
-
-<div class="db">
-
--------------------- -------------------- --------------------
-dyer 2 8.81
--------------------- -------------------- --------------------
-
-</div>
-
-but this would be tedious,
-and if she ever had a data set with fifty or five hundred scientists,
-the chances of her getting all of those queries right is small.
-
-What we need to do is
-tell the database manager to aggregate the hours for each scientist separately
-using a `group by` clause:
-
-``` {.sql}
-select person, count(reading), round(avg(reading), 2)
-from Survey
-where quant='rad'
-group by person;
-```
-
-<div class="db">
-
--------------------- -------------------- --------------------
-dyer 2 8.81
-lake 2 1.82
-pb 3 6.66
-roe 1 11.25
--------------------- -------------------- --------------------
-
-</div>
-
-`group by` does exactly what its name implies:
-groups all the records with the same value for the specified field together
-so that aggregation can process each batch separately.
-Since all the records in each batch have the same value for `person`,
-it no longer matters that the database manager
-is picking an arbitrary one to display
-alongside the aggregated `reading` values
-([Figure 6](#f:grouped_aggregation)).
-
- <figure id="f:grouped_aggregation">
- <img src="db/grouped_aggregation.png" alt="Grouped Aggregation" />
- <figcaption>Figure 6: Grouped Aggregation</figcaption>
- </figure>
-
-Just as we can sort by multiple criteria at once,
-we can also group by multiple criteria.
-To get the average reading by scientist and quantity measured,
-for example,
-we just add another field to the `group by` clause:
-
-``` {.sql}
-select person, quant, count(reading), round(avg(reading), 2)
-from Survey
-group by person, quant;
-```
-
-<div class="db">
-
--------------------- -------------------- -------------------- --------------------
- sal 1 0.06
- temp 1 -26.0
-dyer rad 2 8.81
-dyer sal 2 0.11
-lake rad 2 1.82
-lake sal 4 0.11
-lake temp 1 -16.0
-pb rad 3 6.66
-pb temp 2 -20.0
-roe rad 1 11.25
-roe sal 2 32.05
--------------------- -------------------- -------------------- --------------------
-
-</div>
-
-Note that we have added `person` to the list of fields displayed,
-since the results wouldn't make much sense otherwise.
-
-Let's go one step further and remove all the entries
-where we don't know who took the measurement:
-
-``` {.sql}
-select person, quant, count(reading), round(avg(reading), 2)
-from Survey
-where person is not null
-group by person, quant
-order by person, quant;
-```
-
-<div class="db">
-
--------------------- -------------------- -------------------- --------------------
-dyer rad 2 8.81
-dyer sal 2 0.11
-lake rad 2 1.82
-lake sal 4 0.11
-lake temp 1 -16.0
-pb rad 3 6.66
-pb temp 2 -20.0
-roe rad 1 11.25
-roe sal 2 32.05
--------------------- -------------------- -------------------- --------------------
-
-</div>
-
-Looking more closely,
-this query:
-
-* selected records from the `Survey` table
- where the `person` field was not null;
-
-* grouped those records into subsets
- so that the `person` and `quant` values in each subset
- were the same;
-
-* ordered those subsets first by `person`,
- and then within each sub-group by `quant`;
- and
-
-* counted the number of records in each subset,
- calculated the average `reading` in each,
- and chose a `person` and `quant` value from each
- (it doesn't matter which ones,
- since they're all equal).
-
-Our query processing pipeline now looks like
-[Figure 7](#f:pipeline_grouping).
-
- <figure id="f:pipeline_grouping">
- <img src="db/pipeline_grouping.png" alt="Query Processing Pipeline With Grouping" />
- <figcaption>Figure 7: Query Processing Pipeline With Grouping</figcaption>
- </figure>
-
-### Summary {.keypoints}
-
-* Use `group by` to group values for separate aggregation.
-
-### Challenges {.challenges}
-
-* Write a single query that finds the earliest and latest date
- that each site was visited.
-
-* Show the records produced by each stage of
- [Figure 7](#f:pipeline_grouping)
- for the following query:
-
- ``` {.sql}
- select min(reading), max(reading) from Survey
- where taken in (734, 735)
- and quant='temp'
- group by taken, quant;
- ```
-
-* How can the query in the previous challenge be simplified
- without changing its result?
-
-## Combining Data {#s:join}
-
-### Learning Objectives {.objectives}
-
-* Explain what primary keys and foreign keys are.
-* Write queries that combine information from two or more tables by matching keys.
-* Write queries using aliases for table names.
-* Explain why the `tablename.fieldname` notation is needed when tables are joined.
-* Explain the logical sequence of operations that occurs when two or more tables are joined.
-
-Duration: 20 minutes (and expect to have to walk through an example step-by-step).
-
-### Lesson
-
-In order to submit her data to a web site
-that aggregates historical meteorological data,
-Gina needs to format it as
-latitude, longitude, date, quantity, and reading.
-However,
-her latitudes and longitudes are in the `Site` table,
-while the dates of measurements are in the `Visited` table
-and the readings themselves are in the `Survey` table.
-She needs to combine these tables somehow.
-
-The SQL command to do this is `join`.
-To see how it works,
-let's start by joining the `Site` and `Visited` tables:
-
-``` {.sql}
-select * from Site join Visited;
-```
-
-<div class="db">
-
--------------------- -------------------- -------------------- -------------------- -------------------- --------------------
-DR-1 -49.85 -128.57 619 DR-1 1927-02-08
-DR-1 -49.85 -128.57 622 DR-1 1927-02-10
-DR-1 -49.85 -128.57 734 DR-3 1939-01-07
-DR-1 -49.85 -128.57 735 DR-3 1930-01-12
-DR-1 -49.85 -128.57 751 DR-3 1930-02-26
-DR-1 -49.85 -128.57 752 DR-3
-DR-1 -49.85 -128.57 837 MS-4 1932-01-14
-DR-1 -49.85 -128.57 844 DR-1 1932-03-22
-DR-3 -47.15 -126.72 619 DR-1 1927-02-08
-DR-3 -47.15 -126.72 622 DR-1 1927-02-10
-DR-3 -47.15 -126.72 734 DR-3 1939-01-07
-DR-3 -47.15 -126.72 735 DR-3 1930-01-12
-DR-3 -47.15 -126.72 751 DR-3 1930-02-26
-DR-3 -47.15 -126.72 752 DR-3
-DR-3 -47.15 -126.72 837 MS-4 1932-01-14
-DR-3 -47.15 -126.72 844 DR-1 1932-03-22
-MS-4 -48.87 -123.4 619 DR-1 1927-02-08
-MS-4 -48.87 -123.4 622 DR-1 1927-02-10
-MS-4 -48.87 -123.4 734 DR-3 1939-01-07
-MS-4 -48.87 -123.4 735 DR-3 1930-01-12
-MS-4 -48.87 -123.4 751 DR-3 1930-02-26
-MS-4 -48.87 -123.4 752 DR-3
-MS-4 -48.87 -123.4 837 MS-4 1932-01-14
-MS-4 -48.87 -123.4 844 DR-1 1932-03-22
--------------------- -------------------- -------------------- -------------------- -------------------- --------------------
-
-</div>
-
-`join` creates
-the [cross product](../gloss.html#cross-product)
-of two tables,
-i.e.,
-it joins each record of one with each record of the other
-to give all possible combinations.
-Since there are three records in `Site`
-and eight in `Visited`,
-the join's output has 24 records.
-And since each table has three fields,
-the output has six fields.
-
-What the join *hasn't* done is
-figure out if the records being joined have anything to do with each other.
-It has no way of knowing whether they do or not until we tell it how.
-To do that,
-we add a clause specifying that
-we're only interested in combinations that have the same site name:
-
-``` {.sql}
-select * from Site join Visited on Site.name=Visited.site;
-```
-
-<div class="db">
-
--------------------- -------------------- -------------------- -------------------- -------------------- --------------------
-DR-1 -49.85 -128.57 619 DR-1 1927-02-08
-DR-1 -49.85 -128.57 622 DR-1 1927-02-10
-DR-1 -49.85 -128.57 844 DR-1 1932-03-22
-DR-3 -47.15 -126.72 734 DR-3 1939-01-07
-DR-3 -47.15 -126.72 735 DR-3 1930-01-12
-DR-3 -47.15 -126.72 751 DR-3 1930-02-26
-DR-3 -47.15 -126.72 752 DR-3
-MS-4 -48.87 -123.4 837 MS-4 1932-01-14
--------------------- -------------------- -------------------- -------------------- -------------------- --------------------
-
-</div>
-
-`on` does the same job as `where`:
-it only keeps records that pass some test.
-(The difference between the two is that `on` filters records
-as they're being created,
-while `where` waits until the join is done
-and then does the filtering.)
-Once we add this to our query,
-the database manager throws away records
-that combined information about two different sites,
-leaving us with just the ones we want.
-
-Notice that we used `table.field` to specify field names
-in the output of the join.
-We do this because tables can have fields with the same name,
-and we need to be specific which ones we're talking about.
-For example,
-if we joined the `person` and `visited` tables,
-the result would inherit a field called `ident`
-from each of the original tables.
-
-We can now use the same dotted notation
-to select the three columns we actually want
-out of our join:
-
-``` {.sql}
-select Site.lat, Site.long, Visited.dated
-from Site join Visited
-on Site.name=Visited.site;
-```
-
-<div class="db">
-
--------------------- -------------------- --------------------
--49.85 -128.57 1927-02-08
--49.85 -128.57 1927-02-10
--49.85 -128.57 1932-03-22
--47.15 -126.72
--47.15 -126.72 1930-01-12
--47.15 -126.72 1930-02-26
--47.15 -126.72 1939-01-07
--48.87 -123.4 1932-01-14
--------------------- -------------------- --------------------
-
-</div>
-
-If joining two tables is good,
-joining many tables must be better.
-In fact,
-we can join any number of tables
-simply by adding more `join` clauses to our query,
-and more `on` tests to filter out combinations of records
-that don't make sense:
-
-``` {.sql}
-select Site.lat, Site.long, Visited.dated, Survey.quant, Survey.reading
-from Site join Visited join Survey
-on Site.name=Visited.site
-and Visited.ident=Survey.taken
-and Visited.dated is not null;
-```
-
-<div class="db">
-
--------------------- -------------------- -------------------- -------------------- --------------------
--49.85 -128.57 1927-02-08 rad 9.82
--49.85 -128.57 1927-02-08 sal 0.13
--49.85 -128.57 1927-02-10 rad 7.8
--49.85 -128.57 1927-02-10 sal 0.09
--47.15 -126.72 1939-01-07 rad 8.41
--47.15 -126.72 1939-01-07 sal 0.05
--47.15 -126.72 1939-01-07 temp -21.5
--47.15 -126.72 1930-01-12 rad 7.22
--47.15 -126.72 1930-01-12 sal 0.06
--47.15 -126.72 1930-01-12 temp -26.0
--47.15 -126.72 1930-02-26 rad 4.35
--47.15 -126.72 1930-02-26 sal 0.1
--47.15 -126.72 1930-02-26 temp -18.5
--48.87 -123.4 1932-01-14 rad 1.46
--48.87 -123.4 1932-01-14 sal 0.21
--48.87 -123.4 1932-01-14 sal 22.5
--49.85 -128.57 1932-03-22 rad 11.25
--------------------- -------------------- -------------------- -------------------- --------------------
-
-</div>
-
-<a id="a:keys"></a>
-We can tell which records from `Site`, `Visited`, and `Survey`
-correspond with each other
-because those tables contain
-[primary keys](../gloss.html#primary-key)
-and [foreign keys](../gloss.html#foreign-key).
-A primary key is a value,
-or combination of values,
-that uniquely identifies each record in a table.
-A foreign key is a value (or combination of values) from one table
-that identifies a unique record in another table.
-Another way of saying this is that
-a foreign key is the primary key of one table
-that appears in some other table.
-In our database,
-`Person.ident` is the primary key in the `Person` table,
-while `Survey.person` is a foreign key
-relating the `Survey` table's entries
-to entries in `Person`.
-
-Most database designers believe that
-every table should have a well-defined primary key.
-They also believe that this key should be separate from the data itself,
-so that if we ever need to change the data,
-we only need to make one change in one place.
-One easy way to do this is
-to create an arbitrary, unique ID for each record
-as we add it to the database.
-This is actually very common:
-those IDs have names like "student numbers" and "patient numbers",
-and they almost always turn out to have originally been
-a unique record identifier in some database system or other.
-As the query below demonstrates,
-SQLite automatically numbers records as they're added to tables,
-and we can use those record numbers in queries:
-
-``` {.sql}
-select rowid, * from Person;
-```
-
-<div class="db">
-
--------------------- -------------------- -------------------- --------------------
-1 dyer William Dyer
-2 pb Frank Pabodie
-3 lake Anderson Lake
-4 roe Valentina Roerich
-5 danforth Frank Danforth
--------------------- -------------------- -------------------- --------------------
-
-</div>
-
-### Summary {.keypoints}
-
-* Use `join` to create all possible combinations of records from two or more tables.
-* Use `join *tables* on *test*` to keep only those combinations that pass some test.
-* Use `*table*.*field*` to specify a particular field of a particular table.
-* Every record in a table should be uniquely identified by the value of its primary key.
-
-### Challenges {.challenges}
-
-* Write a query that lists all radiation readings from the DR-1 site.
-
-* Write a query that lists all sites visited by people named "Frank".
-
-* Describe in your own words what the following query produces:
-
- ``` {.sql}
- select Site.name from Site join Visited
- on Site.lat<-49.0 and Site.name=Visited.site and Visited.dated>='1932-00-00';
- ```
-
-* Why does the `Person` table have an `ident` field?
- Why do we not just use scientists' names in the `Survey` table?
-
-* Why does the table `Site` exist?
- Why didn't Gina just record latitudes and longitudes
- directly in the `Visited` and `Survey` tables?
-
-## Creating and Modifying Tables {#s:create}
-
-### Learning Objectives {.box}
-
-* Write queries that create database tables with fields of common types.
-* Write queries that specify the primary and foreign key relationships of tables.
-* Write queries that specify whether field values must be unique and/or are allowed to be `null`.
-* Write queries that erase database tables.
-* Write queries that add records to database tables.
-* Write queries that delete specific records from tables.
-* Explain what referential integrity is, and how a database can become inconsistent as data is changed.
-
-Duration: 10 minutes.
-
-### Lesson
-
-So far we have only looked at how to get information out of a database,
-both because that is more frequent than adding information,
-and because most other operations only make sense
-once queries are understood.
-If we want to create and modify data,
-we need to know two other pairs of commands.
-
-The first pair are `create table` and `drop table`.
-While they are written as two words,
-they are actually single commands.
-The first one creates a new table;
-its arguments are the names and types of the table's columns.
-For example,
-the following statements create the four tables in our survey database:
-
- create table Person(ident text, personal text, family text);
- create table Site(name text, lat real, long real);
- create table Visited(ident integer, site text, dated text);
- create table Survey(taken integer, person text, quant real, reading real);
-
-We can get rid of one of our tables using:
-
- drop table Survey;
-
-Be very careful when doing this:
-most databases have some support for undoing changes,
-but it's better not to have to rely on it.
-
-Different database systems support different data types for table columns,
-but most provide the following:
-
-<div class="db">
-
--------------------- --------------------
-`integer` A signed integer.
-`real` A floating point value.
-`text` A string.
-`blob` Any "binary large object" such as an image or audio file.
--------------------- --------------------
-
-</div>
-
-Most databases also support Booleans and date/time values;
-SQLite uses the integers 0 and 1 for the former,
-and represents the latter as discussed [earlier](#a:dates).
-An increasing number of databases also support geographic data types,
-such as latitude and longitude.
-Keeping track of what particular systems do or do not offer,
-and what names they give different data types,
-is an unending portability headache.
-
-When we create a table,
-we can specify several kinds of constraints on its columns.
-For example,
-a better definition for the `Survey` table would be:
-
- create table Survey(
- taken integer not null, -- where reading taken
- person text, -- may not know who took it
- quant real not null, -- the quantity measured
- reading real not null, -- the actual reading
- primary key(taken, quant),
- foreign key(taken) references Visited(ident),
- foreign key(person) references Person(ident)
- );
-
-Once again,
-exactly what constraints are avialable
-and what they're called
-depends on which database manager we are using.
-
-Once tables have been created,
-we can add and remove records using our other pair of commands,
-`insert` and `delete`.
-The simplest form of `insert` statement lists values in order:
-
- insert into Site values('DR-1', -49.85, -128.57);
- insert into Site values('DR-3', -47.15, -126.72);
- insert into Site values('MSK-4', -48.87, -123.40);
-
-We can also insert values into one table directly from another:
-
- create table JustLatLong(lat text, long TEXT);
- insert into JustLatLong select lat, long from site;
-
-Deleting records can be a bit trickier,
-because we have to ensure that the database remains internally consistent.
-If all we care about is a single table,
-we can use the `DELETE` command with a `WHERE` clause
-that matches the records we want to discard.
-For example,
-once we realize that Frank Danforth didn't take any measurements,
-we can remove him from the `Person` table like this:
-
- delete from Person where ident = "danforth";
-
-But what if we removed Anderson Lake instead?
-Our `Survey` table would still contain seven records
-of measurements he'd taken:
-
-``` {.sql}
-select count(*) from Survey where person='lake';
-```
-
-<div class="db">
-
---------------------
-7
---------------------
-
-</div>
-
-That's never supposed to happen:
-`Survey.person` is a foreign key into the `Person` table,
-and all our queries assume there will be a row in the latter
-matching every value in the former.
-
-This problem is called [referential integrity](../gloss.html#referential-integrity):
-we need to ensure that all references between tables can always be resolved correctly.
-One way to do this is to delete all the records
-that use `'lake'` as a foreign key
-before deleting the record that uses it as a primary key.
-If our database manager supports it,
-we can automate this
-using [cascading delete](../gloss.html#cascading-delete).
-However,
-this technique is outside the scope of this chapter.
-
-> ### Other Ways to Do It {#a:hybrid .box}
->
-> Many applications use a hybrid storage model
-> instead of putting everything into a database:
-> the actual data (such as astronomical images) is stored in files,
-> while the database stores the files' names,
-> their modification dates,
-> the region of the sky they cover,
-> their spectral characteristics,
-> and so on.
-> This is also how most music player software is built:
-> the database inside the application keeps track of the MP3 files,
-> but the files themselves live on disk.
-
-### Summary {.keypoints}
-
-* Use `create table *name*(...)` to create a table.
-* Use `drop table *name*` to erase a table.
-* Specify field names and types when creating tables.
-* Specify `primary key`, `foreign key`, `not null`, and other constraints when creating tables.
-* Use `insert into *table* values(...)` to add records to a table.
-* Use `delete from *table* where *test*` to erase records from a table.
-* Maintain referential integrity when creating or deleting information.
-
-### Challenges {.challenges}
-
-* Write an SQL statement to replace all uses of `null`
- in `Survey.person`
- with the string `'unknown'`.
-
-* One of Gina's colleagues has sent her a [CSV](../gloss.html#csv) file
- containing temperature readings by Robert Olmstead,
- which is formatted like this:
-
- ```
- Taken,Temp
- 619,-21.5
- 622,-15.5
- ```
-
- Write a small Python program that reads this file in
- and prints out the SQL `insert` statements needed
- to add these records to the survey database.
- Note: you will need to add an entry for Olmstead
- to the `Person` table.
- If you are testing your program repeatedly,
- you may want to investigate SQL's `insert or replace` command.
-
-* SQLite has several administrative commands that aren't part of the SQL standard.
- One of them is `.dump`,
- which prints the SQL commands needed to re-create the database.
- Another is `.load`,
- which reads a file created by `.dump` and restores the database.
- A colleague of yours thinks that storing dump files (which are text) in version control
- is a good way to track and manage changes to the database.
- What are the pros and cons of this approach?
-
-<section id="s:transactions">
-
-## Transactions
-
-### Learning Objectives {.objectives}
-
-* Explain what a race condition is.
-* Explain why database operations sometimes have to be placed ina transaction to ensure correct behavior.
-* Explain what it means to commit a transaction.</li>
-
-Duration: 10 minutes.
-
-### Lesson
-
-Suppose we have another table in our database that shows
-which pieces of equipment have been borrowed by which scientists:
-
-``` {.sql}
-select * from Equipment;
-```
-
-<div class="db">
-
--------------------- --------------------
-dyer CX-211 oscilloscope
-pb Greenworth balance
-lake Cavorite damping plates
--------------------- --------------------
-
-</div>
-
-(We should actually give each piece of equipment a unique ID,
-and use that ID here instead of the full name,
-just as we created a separate table for scientists earlier in this chapter,
-but we will bend the rules for now.)
-If William Dyer gives the oscilloscope to Valentina Roerich,
-we need to execute two statements to update this table:
-
- delete from Equipment where person="dyer" and thing="CX-211 oscilloscope";
- insert into Equipment values("roe", "CX-211 oscilloscope");
-
-This is all fine—unless our program happens to crash
-between the first statement and the second.
-If that happens,
-the `Equipment` table won't have a record for the oscilloscope at all.
-Such a crash may seem unlikely,
-but remember:
-if a computer can do two billion operations per second,
-that means there are two billion opportunities every second for something to go wrong.
-And if our operations take a long time to complete—as they will
-when we are working with large datasets,
-or when the database is being heavily used—the odds of failure increase.
-
-What we really want is a way to ensure that every operation is [ACID](../gloss.html#acid):
-[atomic](../gloss.html#atomic-operation) (i.e. indivisible),
-consistent, isolated, and durable.
-The precise meanings of these terms doesn't matter;
-what does is the notion that
-every logical operation on the database should either run to completion
-as if nothing else was going on at the same time,
-or fail without having any effect at all.
-
-The tool we use to ensure that this happens is called
-a [transaction](../gloss.html#transaction).
-Here's how we should actually write the statements
-to move the oscilloscope from one person to another:
-
- begin transaction;
- delete from Equipment where person="dyer" and thing="CX-211 oscilloscope";
- insert into Equipment values("roe", "CX-211 oscilloscope");
- end transaction;
-
-The database manager treats everything in the transaction as one large statement.
-If anything goes wrong inside,
-then none of the changes made in the transaction will actually be written to the database—it
-will be as if the transaction had never happened.
-Changes are only stored permanently
-when we [commit](../gloss.html#commit) them at the end of the transaction.
-
-> ### Transactions and Commits {.box}
->
-> We first used the term "transaction" in
-> [our discussion of version control](svn.html#b:basics:transaction).
-> That's not a coincidence:
-> behind the scenes,
-> tools like Subversion are using many of the same algorithms as database managers
-> to ensure that either everything happens consistently
-> or nothing happens at all.
-> We [use the term "commit"](svn.html#a:commit) for the same reason:
-> just as our changes to local files aren't written back to the version control repository
-> until we commit them,
-> our (apparent) changes to a database aren't written to disk
-> until we say so.
-
-Transactions serve another purpose as well.
-Suppose there is another table in the database called `Exposure`
-that records the number of days each scientist was exposed to
-higher-than-normal levels of radiation:
-
-``` {.sql}
-select * from Exposure;
-```
-
-<div class="db">
-
--------------------- --------------------
-pb 4
-dyer 1
-lake 5
--------------------- --------------------
-
-</div>
-
-After going through the journal entries for 1932,
-Gina wants to add two days to Lake's count:
-
-``` {.sql}
-update Exposure set days = days + 2 where person='lake';
-```
-
-However,
-her labmate has been doing through the journal entries for 1933
-to help Gina meet a paper deadline.
-At the same moment as Gina runs her command,
-her labmate runs this
-to add one more day to Lake's exposure:
-
-``` {.sql}
-update Exposure set days = days + 1 where person='lake';
-```
-
-After both operations have completed,
-the database should show that Lake was exposed for eight days
-(the original five, plus two from Gina, plus one from her labmate).
-However,
-there is a small chance that it won't.
-To see why,
-let's break the two queries into their respective read and write steps
-and place them side by side:
-
--------------------- --------------------
-`X = read Exposure('lake', __)` `Y = read Exposure('lake', __)`
-`write Exposure('lake', X+2)` `write Exposure('lake', Y+1)`
--------------------- --------------------
-
-The database can only actually do one thing at once,
-so it must put these four operations into some sequential order.
-That order has to respect the original order within each column,
-but the database can interleave the two columns any way it wants.
-If it orders them like this:
-
--------------------- --------------------
-`X = read Exposure('lake', __)` `X` is 5
-`write Exposure('lake', X+2)` database contains 7
-`Y = read Exposure('lake', __)` `Y` is 7
-`write Exposure('lake', Y+1)` database contains 8
--------------------- --------------------
-
-then all is well.
-But what if it interleaves the operations like this:
-
--------------------- --------------------
-`X = read Exposure('lake', __)` `X` is 5
-`Y = read Exposure('lake', __)` `Y` is 5
-`write Exposure('lake', X+2)` database contains 7
-`write Exposure('lake', Y+1)` database contains 6
--------------------- --------------------
-
-This ordering puts the initial value, 5, into both `X` and `Y`.
-It then writes 7 back to the database (the third statement),
-and then overwrites that with 6,
-since `Y` holds 5.
-
-This is called a [race condition](../gloss.html#race-condition),
-since the final result depends on a race between the two operations.
-Race conditions are part of what makes programming large systems with many components a nightmare:
-they are difficult to spot in advance
-(since they are caused by the interactions between components,
-rather than by anything in any one of those components),
-and can be almost impossible to debug
-(since they usually occur intermittently and infrequently).
-
-Transactions come to our rescue once again.
-If Gina and her labmate put their statements in transactions,
-the database will act as if it executed all of one and then all of the other.
-Whether or not it *actually* does this is up to whoever wrote the database manager:
-modern databases use very sophisticated algorithms to determine
-which operations actually have to be run sequentially,
-and which can safely be run in parallel to improve performance.
-The key thing is that
-every transaction will appear to have had the entire database to itself.
-
-### Summary {.keypoints}
-
-* Place operations in a transaction to ensure that they appear to be atomic, consistent, isolated, and durable.
-
-### Challenges {.challenges}
-
-* A friend of yours manages a database of aerial photographs.
- New records are added all the time,
- but existing records are never modified or updated.
- Your friend claims that because of this,
- he doesn't need to put his queries in transactions.
- Is he right or wrong, and why?
-
-## Programming With Databases {#s:programming}
-
-### Learning Objectives {.objectives}
-
-* Write a Python program that queries a database and processes the results.
-* Explain what an SQL injection attack is.
-* Write a program that safely interpolates values into queries.
-
-Duration: 20 minutes.
-
-### Lesson
-
-To end this chapter,
-let's have a look at how to access a database from
-a general-purpose programming language like Python.
-Other languages use almost exactly the same model:
-library and function names may differ,
-but the concepts are the same.
-
-Here's a short Python program that selects latitudes and longitudes
-from an SQLite database stored in a file called `survey.db`:
-
-``` {.python}
-import sqlite3
-connection = sqlite3.connect("survey.db")
-cursor = connection.cursor()
-cursor.execute("select site.lat, site.long from site;")
-results = cursor.fetchall()
-for r in results:
- print r
-cursor.close()
-connection.close()
-```
-
-The program starts by importing the `sqlite3` library.
-If we were connecting to MySQL, DB2, or some other database,
-we would import a different library,
-but all of them provide the same functions,
-so that the rest of our program does not have to change
-(at least, not much)
-if we switch from one database to another.
-
-Line 2 establishes a connection to the database.
-Since we're using SQLite,
-all we need to specify is the name of the database file.
-Other systems may require us to provide a username and password as well.
-Line 3 then uses this connection to create
-a [cursor](../gloss.html#cursor);
-just like the cursor in an editor,
-its role is to keep track of where we are in the database.
-
-On line 4, we use that cursor to ask the database to execute a query for us.
-The query is written in SQL,
-and passed to `cursor.execute` as a string.
-It's our job to make sure that SQL is properly formatted;
-if it isn't,
-or if something goes wrong when it is being executed,
-the database will report an error.
-
-The database returns the results of the query to us
-in response to the `cursor.fetchall` call on line 5.
-This result is a list with one entry for each record in the result set;
-if we loop over that list (line 6) and print those list entries (line 7),
-we can see that each one is a tuple
-with one element for each field we asked for.
-
-Finally, lines 8 and 9 close our cursor and our connection,
-since the database can only keep a limited number of these open at one time.
-Since establishing a connection takes time,
-though,
-we shouldn't open a connection,
-do one operation,
-then close the connection,
-only to reopen it a few microseconds later to do another operation.
-Instead,
-it's normal to create one connection that stays open for the lifetime of the program.
-
-> ### What Are The u's For? {.box}
->
-> You may have noticed that
-> each of the strings in our output has a lower-case 'u' in front of it.
-> That is Python's way of telling us that the string is stored in
-> [Unicode](../gloss.html#unicode).
-
-Queries in real applications will often depend on values provided by users.
-For example,
-a program might take a user ID as a command-line parameter
-and display the user's full name:
-
-``` {.python}
-import sys
-import sqlite3
-
-query = "select personal, family from Person where ident='%s';"
-user_id = sys.argv[1]
-
-connection = sqlite3.connect("survey.db")
-cursor = connection.cursor()
-
-cursor.execute(query % user_id)
-results = cursor.fetchall()
-print results[0][0], results[0][1]
-
-cursor.close()
-connection.close()
-```
-
-The variable `query` holds the statement we want to execute
-with a `%s` format string where we want to insert
-the ID of the person we're looking up.
-It seems simple enough,
-but what happens if someone gives the program this input?
-
- dyer"; drop table Survey; select "
-
-It looks like there's garbage after the name of the project,
-but it is very carefully chosen garbage.
-If we insert this string into our query,
-the result is:
-
-``` {.sql}
-select personal, family from Person where ident='dyer'; drop table Survey; select '';
-```
-
-Whoops:
-if we execute this,
-it will erase one of the tables in our database.
-
-This technique is called [SQL injection](../gloss.html#sql-injection),
-and it has been used to attack thousands of programs over the years.
-In particular,
-many web sites that take data from users insert values directly into queries
-without checking them carefully first.
-
-Since a villain might try to smuggle commands into our queries in many different ways,
-the safest way to deal with this threat is
-to replace characters like quotes with their escaped equivalents,
-so that we can safely put whatever the user gives us inside a string.
-We can do this by using a [prepared statement](../gloss.html#prepared-statement)
-instead of formatting our statements as strings.
-Here's what our example program looks like if we do this:
-
-``` {.python}
-import sys
-import sqlite3
-
-query = "select personal, family from Person where ident=?;"
-user_id = sys.argv[1]
-
-connection = sqlite3.connect("survey.db")
-cursor = connection.cursor()
-
-cursor.execute(query, [user_id])
-results = cursor.fetchall()
-print results[0][0], results[0][1]
-
-cursor.close()
-connection.close()
-```
-
-The key changes are in the query string and the `execute` call.
-Instead of formatting the query ourselves,
-we put question marks in the query template where we want to insert values.
-When we call `execute`,
-we provide a list
-that contains as many values as there are question marks in the query.
-The library matches values to question marks in order,
-and translates any special characters in the values
-into their escaped equivalents
-so that they are safe to use.
-
-### Summary {.keypoints}
-
-* Most applications that use databases embed SQL in a general-purpose programming language.
-* Database libraries use connections and cursors to manage interactions.
-* Programs can fetch all results at once, or a few results at a time.
-* If queries are constructed dynamically using input from users, malicious users may be able to inject their own commands into the queries.
-* Dynamically-constructed queries can use SQL's native formatting to safeguard against such attacks.
-
-### Challenges {.challenges}
-
-* Write a Python program that creates a new database
- in a file called `original.db`
- containing a single table called `Pressure`,
- with a single field called `reading`,
- and inserts 100,000 random numbers between 10.0 and 25.0.
- How long does it take this program to run?
- How long does it take to run a program
- that simply writes those random numbers to a file?
-
-* Write a Python program that creates a new database
- called `backup.db`
- with the same structure as `original.db`
- and copies all the values greater than 20.0
- from `original.db` to `backup.db`.
- Which is faster:
- filtering values in the query,
- or reading everything into memory and filtering in Python?
-
-## Summing Up {#s:summary}
-
-There are many things databases can't do,
-or can't do well
-(which is why we have general-purpose programming languages like Python).
-However,
-they are still the best tool available
-for managing many kinds of complex, structured data.
-Thousands of programmer-years have gone into their design and implementation
-so that they can handle very large datasets—terabytes or more—quickly and reliably.
-Queries allow for great flexibility in how you are able to analyze your data,
-which makes databases a good choice when you are exploring data.
projects / swc-workshop.git / commitdiff