Skip to content

Commit 13ecf9b

Browse files
vindarelvindarel
committed
DB++
1 parent d8fa3a5 commit 13ecf9b

File tree

3 files changed

+372
-106
lines changed

3 files changed

+372
-106
lines changed

‎content/building-blocks/database.md‎

Lines changed: 236 additions & 105 deletions
Original file line numberDiff line numberDiff line change
@@ -1,140 +1,271 @@
1-
21
+++
32
title = "Connecting to a database"
43
weight = 120
54
+++
65

7-
Please see the [databases section](databases.html). The Mito ORM
8-
supports SQLite3, PostgreSQL, MySQL, it has migrations and db schema
9-
versioning, etc.
6+
<!-- https://www.reddit.com/r/Common_Lisp/comments/1f7bfql/simple_session_management_with_hunchentoot/ -->
107

11-
### Checking a user is logged-in
8+
Let's study different use cases:
129

13-
A framework will provide a way to work with sessions. We'll create a
14-
little macro to wrap our routes to check if the user is logged in.
10+
- do you have an existing database you want to read data from?
11+
- do you want to create a new database?
12+
- do you prefer CLOS orientation
13+
- or to write SQL queries?
1514

16-
In Caveman, `*session*` is a hash table that represents the session's
17-
data. Here are our login and logout functions:
15+
We have many libraries to work with databases, let's have a recap first.
1816

19-
~~~lisp
20-
(defun login (user)
21-
"Log the user into the session"
22-
(setf (gethash :user *session*) user))
17+
<!-- ## Database libraries -->
2318

24-
(defun logout ()
25-
"Log the user out of the session."
26-
(setf (gethash :user *session*) nil))
27-
~~~
19+
The [#database section on the awesome-cl list](https://github.com/CodyReichert/awesome-cl#database)
20+
is a resource listing popular libraries to work with different kind of
21+
databases. We can group them roughly in those categories:
2822

29-
We define a simple predicate:
23+
- wrappers to one database engine (cl-sqlite, postmodern, cl-redis, cl-duckdb…),
24+
- ORMs (Mito),
25+
- interfaces to several DB engines (cl-dbi, cl-yesql…),
26+
- lispy SQL syntax (sxql…)
27+
- in-memory persistent object databases (bknr.datastore, cl-prevalence,…),
28+
- graph databases in pure Lisp (AllegroGraph, vivace-graph) or wrappers (neo4cl),
29+
- object stores (cl-store, cl-naive-store…)
30+
- and other tools (pgloader, [which was re-written from Python to Common Lisp](https://tapoueh.org/blog/2014/05/why-is-pgloader-so-much-faster/)).
3031

31-
~~~lisp
32-
(defun logged-in-p ()
33-
(gethash :user cm:*session*))
34-
~~~
32+
<!-- markdown-toc start - Don't edit this section. Run M-x markdown-toc-refresh-toc -->
33+
**Table of Contents**
3534

36-
and we define our `with-logged-in` macro:
35+
- [Connect](#connect)
36+
- [Run queries](#run-queries)
37+
- [Insert rows](#insert-rows)
38+
- [User-level API](#user-level-api)
39+
- [Close connections](#close-connections)
40+
- [The Mito ORM](#the-mito-orm)
41+
- [How to integrate the databases into the web frameworks](#how-to-integrate-the-databases-into-the-web-frameworks)
42+
- [Bonus: pimp your SQLite](#bonus-pimp-your-sqlite)
43+
- [References](#references)
3744

38-
~~~lisp
39-
(defmacro with-logged-in (&body body)
40-
`(if (logged-in-p)
41-
(progn ,@body)
42-
(render #p"login.html"
43-
'(:message "Please log-in to access this page."))))
44-
~~~
45+
<!-- markdown-toc end -->
4546

46-
If the user isn't logged in, there will nothing in the session store,
47-
and we render the login page. When all is well, we execute the macro's
48-
body. We use it like this:
4947

50-
~~~lisp
51-
(defroute "/account/logout" ()
52-
"Show the log-out page, only if the user is logged in."
53-
(with-logged-in
54-
(logout)
55-
(render #p"logout.html")))
56-
57-
(defroute ("/account/review" :method :get) ()
58-
(with-logged-in
59-
(render #p"review.html"
60-
(list :review (get-review (gethash :user *session*))))))
61-
~~~
48+
## How to query an existing database
6249

63-
and so on.
50+
Let's say you have a database called `db.db` and you want to extract
51+
data from it.
6452

53+
For our example, quickload this library:
6554

66-
### Encrypting passwords
55+
```lisp
56+
(ql:quickload "cl-dbi")
57+
```
6758

68-
#### With cl-pass
59+
[cl-dbi](https://github.com/fukamachi/cl-dbi/) can connect to major
60+
database engines: PostGres, SQLite, MySQL.
6961

70-
[cl-pass](https://github.com/eudoxia0/cl-pass) is a password hashing and verification library. It is as simple to use as this:
62+
Once `cl-dbi` is loaded, you can access its functions with the `dbi`
63+
package prefix.
7164

72-
~~~lisp
73-
(cl-pass:hash "test")
74-
;; "PBKDF2$sha256:20000$5cf6ee792cdf05e1ba2b6325c41a5f10$19c7f2ccb3880716bf7cdf999b3ed99e07c7a8140bab37af2afdc28d8806e854"
75-
(cl-pass:check-password "test" *)
76-
;; t
77-
(cl-pass:check-password "nope" **)
78-
;; nil
79-
~~~
65+
### Connect
8066

81-
You might also want to look at
82-
[hermetic](https://github.com/eudoxia0/hermetic), a simple
83-
authentication system for Clack-based applications.
67+
To connect to a database, use `dbi:connect` with paramaters the DB type, and its name:
8468

85-
#### Manually (with Ironclad)
69+
```lisp
70+
(defparameter *db-name* "db.db")
8671
87-
In this recipe we do the encryption and verification ourselves. We use the de-facto standard
88-
[Ironclad](https://github.com/froydnj/ironclad) cryptographic toolkit
89-
and the [Babel](https://github.com/cl-babel/babel) charset
90-
encoding/decoding library.
72+
(defvar *connection* nil "the DB connection")
9173
92-
The following snippet creates the password hash that should be stored in your
93-
database. Note that Ironclad expects a byte-vector, not a string.
74+
(defun connect ()
75+
(if (uiop:file-exists-p *db-name*)
76+
(setf *connection* (dbi:connect :sqlite3 :database-name (get-db-name)))
77+
(format t "The DB file ~a does not exist." *db-name*)))
78+
```
9479

95-
~~~lisp
96-
(defun password-hash (password)
97-
(ironclad:pbkdf2-hash-password-to-combined-string
98-
(babel:string-to-octets password)))
99-
~~~
80+
The available DB drivers are:
81+
- `:mysql`
82+
- `:sqlite3`
83+
- `:postgres`
10084

101-
`pbkdf2` is defined in [RFC2898](https://tools.ietf.org/html/rfc2898).
102-
It uses a pseudorandom function to derive a secure encryption key
103-
based on the password.
85+
For the username and password, use the key arguments `:username` and `:password`.
10486

105-
The following function checks if a user is active and verifies the
106-
entered password. It returns the user-id if active and verified and
107-
nil in all other cases even if an error occurs. Adapt it to your
108-
application.
87+
When you connect for the first time, cl-dbi will automatically
88+
quickload another dependency, depending on the driver. We advise to
89+
add the relevant one to your list of dependencies in your .asd file
90+
(or your binary will chok on a machine without Quicklisp, we learned
91+
this the hard way).
10992

110-
~~~lisp
111-
(defun check-user-password (user password)
112-
(handler-case
113-
(let* ((data (my-get-user-data user))
114-
(hash (my-get-user-hash data))
115-
(active (my-get-user-active data)))
116-
(when (and active (ironclad:pbkdf2-check-password (babel:string-to-octets password)
117-
hash))
118-
(my-get-user-id data)))
119-
(condition () nil)))
120-
~~~
93+
:dbd-sqlite3
94+
:dbd-mysql
95+
:dbd-postgres
96+
97+
We can now run queries.
98+
99+
### Run queries
121100

122-
And the following is an example on how to set the password on the
123-
database. Note that we use `(password-hash password)` to save the
124-
password. The rest is specific to the web framework and to the DB
125-
library.
101+
Running a query is done is 3 steps:
102+
103+
- write the SQL query (in a string, with a lispy syntax…)
104+
- `dbi:prepare` the query on a DB connection
105+
- `dbi:execute` it
106+
- and `dbi:fetch-all` results.
107+
108+
109+
```lisp
110+
(defparameter *select-products* "SELECT * FROM products LIMIT 100")
111+
112+
(dbi:fetch-all (dbi:execute (dbi:prepare *connection* *select-products*)))
113+
```
114+
115+
This returns something like:
116+
117+
```
118+
((:|id| 1 :|title| "Lisp Cookbook" :|shelf_id| 1 :|tags_id| NIL :|cover_url|
119+
"https://lispcookbook.github.io/cl-cookbook/orly-cover.png"
120+
:|created_at| "2024-11-07 22:49:23.972522Z" :|updated_at|
121+
"2024-12-30 20:55:51.044704Z")
122+
(:|id| 2 :|title| "Common Lisp Recipes" :|shelf_id| 1 :|tags_id| NIL
123+
:|cover_url| ""
124+
:|created_at| "2024-12-09 19:37:30.057172Z" :|updated_at|
125+
"2024-12-09 19:37:30.057172Z"))
126+
```
127+
128+
We got a list of records where each record is a *property list*, a
129+
list alternating a key (as a keyword) and a value.
130+
131+
Note how the keywords respect the case of our database fields with the `:|id|` notation.
132+
133+
With arguments, use a `?` placeholder in your SQL query and give a
134+
list of arguments to `dbi:execute`:
126135

127136
~~~lisp
128-
(defun set-password (user password)
129-
(with-connection (db)
130-
(execute
131-
(make-statement :update :web_user
132-
(set= :hash (password-hash password))
133-
(make-clause :where
134-
(make-op := (if (integerp user)
135-
:id_user
136-
:email)
137-
user))))))
137+
(defparameter *select-products* "SELECT * FROM products WHERE flag = ? OR updated_at > ?")
138+
139+
(let* ((query (dbi:prepare *connection* *select-products*))
140+
(query (dbi:execute query (list 0 "1984-01-01")))) ;; <--- list of arguments
141+
(loop for row = (dbi:fetch query)
142+
while row
143+
;; process "row".
144+
))
138145
~~~
139146

140-
*Credit: `/u/arvid` on [/r/learnlisp](https://www.reddit.com/r/learnlisp/comments/begcf9/can_someone_give_me_an_eli5_on_hiw_to_encrypt_and/)*.
147+
### Insert rows
148+
149+
*(straight from cl-dbi's documentation)*
150+
151+
`dbi:do-sql` prepares and executes a single statement. It returns the
152+
number of rows affected. It's typically used for non-`SELECT`
153+
statements.
154+
155+
```lisp
156+
(dbi:do-sql *connection*
157+
"INSERT INTO somewhere (flag, updated_at) VALUES (?, NOW())"
158+
(list 0))
159+
```
160+
161+
162+
### User-level API
163+
164+
`dbi` offers more functions to fetch results than `fetch-all`.
165+
166+
You can use `fetch` to get one result at a time or again `do-sql` to run any
167+
SQL statement.
168+
169+
170+
* connect [driver-name &amp; params] =&gt; &lt;dbi-connection&gt;
171+
* connect-cached [driver-name &amp; params] =&gt; &lt;dbi-connection&gt;
172+
* disconnect [&lt;dbi-connection&gt;] =&gt; T or NIL
173+
* prepare [conn sql] =&gt; &lt;dbi-query&gt;
174+
* prepare-cached [conn sql] =&gt; &lt;dbi-query&gt;
175+
* execute [query &amp;optional params] =&gt; something
176+
* fetch [result] =&gt; a row data as plist
177+
* fetch-all [result] =&gt; a list of all row data
178+
* do-sql [conn sql &amp;optional params]
179+
* list-all-drivers [] =&gt; (&lt;dbi-driver&gt; ..)
180+
* find-driver [driver-name] =&gt; &lt;dbi-driver&gt;
181+
* with-transaction [conn]
182+
* begin-transaction [conn]
183+
* commit [conn]
184+
* rollback [conn]
185+
* ping [conn] =&gt; T or NIL
186+
* row-count [conn] =&gt; a number of rows modified by the last executed INSERT/UPDATE/DELETE
187+
* with-connection [connection-variable-name &body body]
188+
189+
### Close connections
190+
191+
You should take care of closing the DB connection.
192+
193+
`dbi` has a macro for that:
194+
195+
```lisp
196+
(dbi:with-connection (conn :sqlite3 :database-name "/home/fukamachi/test.db")
197+
(let* ((query (dbi:prepare conn "SELECT * FROM People"))
198+
(query (dbi:execute query)))
199+
(loop for row = (dbi:fetch query)
200+
while row
201+
do (format t "~A~%" row))))
202+
```
203+
204+
Inside this macro, `conn` binds to the current connection.
205+
206+
There is more but enough, please refer to cl-dbi's README.
207+
208+
209+
## The Mito ORM
210+
211+
The [Mito ORM](https://github.com/fukamachi/mito/) provides a nice
212+
object-oriented way to define schemas and query the database.
213+
214+
It supports SQLite3, PostgreSQL and MySQL, it has automatic
215+
migrations, db schema versioning, and more features.
216+
217+
For example, this is how one can define a `user` table with two columns:
218+
219+
```lisp
220+
(mito:deftable user ()
221+
((name :col-type (:varchar 64))
222+
(email :col-type (or (:varchar 128) :null))))
223+
```
224+
225+
Once we create the table, we can create and insert `user` rows with
226+
methods such as `create-dao`:
227+
228+
```lisp
229+
(mito:create-dao 'user :name "Eitaro Fukamachi" :email "e.arrows@gmail.com")
230+
```
231+
232+
Once we edit the table definition (aka the class definition), Mito
233+
will (by default) automatically migrate it.
234+
235+
There is much more to say, but we refer you to Mito's good
236+
documentation and to the Cookbook.
237+
238+
## How to integrate the databases into the web frameworks
239+
240+
The web frameworks / web servers we use in this guide do not need
241+
anything special. Just use a DB driver and fetch results in your
242+
routes.
243+
244+
Using a DB connection per request with `dbi:with-connection` is a good idea.
245+
246+
## Bonus: pimp your SQLite
247+
248+
SQLite is a great database. It loves backward compatibility. As such,
249+
its default settings may not be optimal for a web application seeing
250+
some load. You might want to set some [PRAGMA
251+
statements](https://www.sqlite.org/pragma.html) (SQLite settings).
252+
253+
To set them, look at your DB driver how to run a raw SQL query.
254+
255+
With `cl-dbi`, this would be `dbi:do-sql`:
256+
257+
```lisp
258+
(dbi:do-sql *connection*
259+
"PRAGMA auto_vacuum;")
260+
```
261+
262+
Here's a nice list of pragmas useful for web development:
263+
264+
- [https://dev.to/briandouglasie/sensible-sqlite-defaults-5ei7](https://dev.to/briandouglasie/sensible-sqlite-defaults-5ei7)
265+
266+
267+
## References
268+
269+
- [CL Cookbook#databases](https://lispcookbook.github.io/cl-cookbook/databases.html)
270+
- [Mito](https://github.com/fukamachi/mito/)
271+
- [cl-dbi](https://github.com/fukamachi/cl-dbi/)

‎content/building-blocks/errors-interactivity.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,7 @@ it do when an error happens?
1111
- not interactive but developper friendly: it can show the lisp error message on the HTML page,
1212
- as well as the full Lisp backtrace
1313
- it can let the developper have the interactive debugger: in that
14-
case the framework doesn't catch the error, it lets it through, and
14+
case the framework doesn't catch the error, it lets it pass through, and
1515
we the developper deal with it as in a normal Slime session.
1616

1717
We see this by default:

0 commit comments

Comments
 (0)