chore(tazjin): Move my presentations to my user directory

Change-Id: I72b25680e7167c3a55477111c28b1d4936c60e2c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/606
Reviewed-by: tazjin <mail@tazj.in>
This commit is contained in:
Vincent Ambo 2020-06-26 20:38:47 +01:00 committed by tazjin
parent 1d0e421cb8
commit 2e3b03b5ae
40 changed files with 0 additions and 0 deletions

View file

@ -0,0 +1,5 @@
These are the slides for a talk I gave at the Norwegian Unix User Group on
2018-03-13.
There is more information and a recording on the [event
page](https://www.nuug.no/aktiviteter/20180313-reproduible-compiler/).

View file

@ -0,0 +1,50 @@
# This derivation builds the LaTeX presentation.
{ pkgs, ... }:
with pkgs;
let tex = texlive.combine {
inherit (texlive)
beamer
beamertheme-metropolis
etoolbox
euenc
extsizes
fontspec
lualibs
luaotfload
luatex
minted
ms
pgfopts
scheme-basic
translator;
};
in stdenv.mkDerivation {
name = "nuug-bootstrapping-slides";
src = ./.;
FONTCONFIG_FILE = makeFontsConf {
fontDirectories = [ fira fira-code fira-mono ];
};
buildInputs = [ tex fira fira-code fira-mono ];
buildPhase = ''
# LaTeX needs a cache folder in /home/ ...
mkdir home
export HOME=$PWD/home
# ${tex}/bin/luaotfload-tool -ufv
# As usual, TeX needs to be run twice ...
function run() {
${tex}/bin/lualatex presentation.tex
}
run && run
'';
installPhase = ''
mkdir -p $out
cp presentation.pdf $out/
'';
}

Binary file not shown.

After

Width:  |  Height:  |  Size: 241 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 88 KiB

View file

@ -0,0 +1,89 @@
#+TITLE: Bootstrapping, reproducibility, etc.
#+AUTHOR: Vincent Ambo
#+DATE: <2018-03-10 Sat>
* Compiler bootstrapping
This section contains notes about compiler bootstrapping, the
history thereof, which compilers need it - and so on:
** C
** Haskell
- self-hosted compiler (GHC)
** Common Lisp
CL is fairly interesting in this space because it is a language
that is defined via an ANSI standard that compiler implementations
normally actually follow!
CL has several ecosystem components that focus on making
abstracting away implementation-specific calls and if a self-hosted
compiler is written in CL using those components it can be
cross-bootstrapped.
** Python
* A note on runtimes
Sometimes the compiler just isn't enough ...
** LLVM
** JVM
* References
https://github.com/mame/quine-relay
https://manishearth.github.io/blog/2016/12/02/reflections-on-rusting-trust/
https://tests.reproducible-builds.org/debian/reproducible.html
* Slide thoughts:
1. Hardware trust has been discussed here a bunch, most recently
during the puri.sm talk. Hardware trust is important, as we see
with IME, but it's striking that people often take a leap to "I'm
now on my trusted Debian with free software".
Unless you built it yourself from scratch (Spoiler: you haven't)
you're placing trust in what is basically foreign binary blobs.
Agenda: Implications/attack vectors of this, state of the chicken
& egg, the topic of reproducibility, what can you do? (Nix!)
2. Chicken-and-egg issue
It's an important milestone for a language to become self-hosted:
You begin doing a kind of dogfeeding, you begin to enforce
reliability & consistency guarantees to avoid having to redo your
own codebase constantly and so on.
However, the implication is now that you need your own compiler
to compile itself.
Common examples:
- C/C++ compilers needed to build C/C++ compilers:
GCC 4.7 was the last version of GCC that could be built with a
standard C-compiler, nowadays it is mostly written in C++.
Certain versions of GCC can be built with LLVM/Clang.
Clang/LLVM can be compiled by itself and also GCC.
- Rust was originally written in OCAML but moved to being
self-hosted in 2011. Currently rustc-releases are always built
with a copy of the previous release.
It's relatively new so we can build the chain all the way.
Notable exceptions: Some popular languages are not self-hosted,
for example Clojure. Languages also have runtimes, which may be
written in something else (e.g. Haskell -> C runtime)
* How to help:
Most of this advice is about reproducible builds, not bootstrapping,
as that is a much harder project.
- fix reproducibility issues listed in Debian's issue tracker (focus
on non-Debian specific ones though)
- experiment with NixOS / GuixSD to get a better grasp on the
problem space of reproducibility
If you want to contribute to bootstrapping, look at
bootstrappable.org and their wiki. Several initiatives such as MES
could need help!

View file

@ -0,0 +1,251 @@
\documentclass[12pt]{beamer}
\usetheme{metropolis}
\newenvironment{code}{\ttfamily}{\par}
\title{Where does \textit{your} compiler come from?}
\date{2018-03-13}
\author{Vincent Ambo}
\institute{Norwegian Unix User Group}
\begin{document}
\maketitle
%% Slide 1:
\section{Introduction}
%% Slide 2:
\begin{frame}{Chicken and egg}
Self-hosted compilers are often built using themselves, for example:
\begin{itemize}
\item C-family compilers bootstrap themselves \& each other
\item (Some!) Common Lisp compilers can bootstrap each other
\item \texttt{rustc} bootstraps itself with a previous version
\item ... same for many other languages!
\end{itemize}
\end{frame}
\begin{frame}{Chicken, egg and ... lizard?}
It's not just compilers: Languages have runtimes, too.
\begin{itemize}
\item JVM is implemented in C++
\item Erlang-VM is C
\item Haskell runtime is C
\end{itemize}
... we can't ever get away from C, can we?
\end{frame}
%% Slide 3:
\begin{frame}{Trusting Trust}
\begin{center}
\huge{Could this be exploited?}
\end{center}
\end{frame}
%% Slide 4:
\begin{frame}{Short interlude: A quine}
\begin{center}
\begin{code}
((lambda (x) (list x (list 'quote x)))
\newline\vspace*{6mm} '(lambda (x) (list x (list 'quote x))))
\end{code}
\end{center}
\end{frame}
%% Slide 5:
\begin{frame}{Short interlude: Quine Relay}
\begin{center}
\includegraphics[
keepaspectratio=true,
height=\textheight
]{quine-relay.png}
\end{center}
\end{frame}
%% Slide 6:
\begin{frame}{Trusting Trust}
An attack described by Ken Thompson in 1983:
\begin{enumerate}
\item Modify a compiler to detect when it's compiling itself.
\item Let the modification insert \textit{itself} into the new compiler.
\item Add arbitrary attack code to the modification.
\item \textit{Optional!} Remove the attack from the source after compilation.
\end{enumerate}
\end{frame}
%% Slide 7:
\begin{frame}{Damage potential?}
\begin{center}
\large{Let your imagination run wild!}
\end{center}
\end{frame}
%% Slide 8:
\section{Countermeasures}
%% Slide 9:
\begin{frame}{Diverse Double-Compiling}
Assume we have:
\begin{itemize}
\item Target language compilers $A$ and $T$
\item The source code of $A$: $ S_{A} $
\end{itemize}
\end{frame}
%% Slide 10:
\begin{frame}{Diverse Double-Compiling}
Apply the first stage (functional equivalence):
\begin{itemize}
\item $ X = A(S_{A})$
\item $ Y = T(S_{A})$
\end{itemize}
Apply the second stage (bit-for-bit equivalence):
\begin{itemize}
\item $ V = X(S_{A})$
\item $ W = Y(S_{A})$
\end{itemize}
Now we have a new problem: Reproducibility!
\end{frame}
%% Slide 11:
\begin{frame}{Reproducibility}
Bit-for-bit equivalent output is hard, for example:
\begin{itemize}
\item Timestamps in output artifacts
\item Non-deterministic linking order in concurrent builds
\item Non-deterministic VM \& memory states in outputs
\item Randomness in builds (sic!)
\end{itemize}
\end{frame}
\begin{frame}{Reproducibility}
\begin{center}
Without reproducibility, we can never trust that any shipped
binary matches the source code!
\end{center}
\end{frame}
%% Slide 12:
\section{(Partial) State of the Union}
\begin{frame}{The Desired State}
\begin{center}
\begin{enumerate}
\item Full-source bootstrap!
\item All packages reproducible!
\end{enumerate}
\end{center}
\end{frame}
%% Slide 13:
\begin{frame}{Bootstrapping Debian}
\begin{itemize}
\item Sparse information on the Debian-wiki
\item Bootstrapping discussions mostly resolve around new architectures
\item GCC is compiled by depending on previous versions of GCC
\end{itemize}
\end{frame}
\begin{frame}{Reproducing Debian}
Debian has a very active effort for reproducible builds:
\begin{itemize}
\item Organised information about reproducibility status
\item Over 90\% reproducibility in Debian package base!
\end{itemize}
\end{frame}
\begin{frame}{Short interlude: Nix}
\begin{center}
\includegraphics[
keepaspectratio=true,
height=0.7\textheight
]{nixos-logo.png}
\end{center}
\end{frame}
\begin{frame}{Short interlude: Nix}
\begin{center}
\includegraphics[
keepaspectratio=true,
height=0.90\textheight
]{drake-meme.png}
\end{center}
\end{frame}
\begin{frame}{Short interlude: Nix}
\begin{center}
\includegraphics[
keepaspectratio=true,
height=0.7\textheight
]{nixos-logo.png}
\end{center}
\end{frame}
\begin{frame}{Bootstrapping NixOS}
Nix evaluation can not recurse forever: The bootstrap can not
simply depend on a previous GCC.
Workaround: \texttt{bootstrap-tools} tarball from a previous
binary cache is fetched and used.
An unfortunate magic binary blob ...
\end{frame}
\begin{frame}{Reproducing NixOS}
Not all reproducibility patches have been ported from Debian.
However: Builds are fully repeatable via the Nix fundamentals!
\end{frame}
\section{Future Developments}
\begin{frame}{Bootstrappable: stage0}
Hand-rolled ``Cthulhu's Path to Madness'' hex-programs:
\begin{itemize}
\item No non-auditable binary blobs
\item Aims for understandability by 70\% of programmers
\item End goal is a full-source bootstrap of GCC
\end{itemize}
\end{frame}
\begin{frame}{Bootstrappable: MES}
Bootstrapping the ``Maxwell Equations of Software'':
\begin{itemize}
\item Minimal C-compiler written in Scheme
\item Minimal Scheme-interpreter (currently in C, but intended to
be rewritten in stage0 macros)
\item End goal is full-source bootstrap of the entire GuixSD
\end{itemize}
\end{frame}
\begin{frame}{Other platforms}
\begin{itemize}
\item Nix for Darwin is actively maintained
\item F-Droid Android repository works towards fully reproducible
builds of (open) Android software
\item Mobile devices (phones, tablets, etc.) are a lost cause at
the moment
\end{itemize}
\end{frame}
\begin{frame}{Thanks!}
Resources:
\begin{itemize}
\item bootstrappable.org
\item reproducible-builds.org
\end{itemize}
@tazjin | mail@tazj.in
\end{frame}
\end{document}

Binary file not shown.

After

Width:  |  Height:  |  Size: 51 KiB

View file

@ -0,0 +1,142 @@
[file]
result
[last_saved_slide]
10
[font_size]
20000
[notes]
### 1
- previous discussions of hardware trust (e.g. purism presentation)
- people leap to "now I'm on my trusted Debian!"
- unless you built it from scratch (spoiler: you haven't) you're *trusting* someone
Agenda: Implications of trust with focus on bootstrap paths and reproducibility, plus how you can help.### 2
self-hosting:
- C-family: GCC pre/post 4.7, Clang
- Common Lisp: Sunshine land! (with SBCL)
- rustc: Bootstrap based on previous versions (C++ transpiler underway!)
- many other languages also work this way!
(Noteable counterexample: Clojure is written in Java!)### 3
- compilers are just one bit, the various runtimes exist, too!### 4
Could this be exploited?
People don't think about where their compiler comes from.
Even if they do, they may only go so far as to say "I'll just recompile it using <other compiler>".
Unfortunately, spoiler alert, life isn't that easy in the computer world and yes, exploitation is possible.### 5
- describe what a quine is
- classic Lisp quine
- explain demo quine
- demo demo quine
- this is interesting, but not useful - can quines do more than that?### 6
- quine-relay: "art project" with 128-language circular quine
- show source of quine-relay
- (demo quine relay?)
- side-note: this program is very, very trustworthy!### 7
Ken Thompson (designer of UNIX and a couple other things!) received Turing award in 1983, and described attack in speech.
- figure out how to detect self-compilation
- make that modification a quine
- insert modification into new compiler
- add attack code to modification
- remove attack from source, distributed binary will still be compromised! it's like evolution :)### 8
damage potential is basically infinite:
- classic "login" attack
=> also applicable to other credentials
- attack (weaken) crypto algorithms
- you can probably think of more!### 10
idea being: potential vulnerability would have to work across compilers:
the more compilers we can introduce (e.g. more architectures, different versions, different compilers), the harder it gets for a vulnerability to survive all of those
The more compilers, the merrier! Lisps are pretty good at this.### 11
if we get a bit-mismatch after DDC, not all hope is lost: Maybe the thing just isn't reproducible!
- many reasons for failures
- timestamps are a classic! artifacts can be build logs, metadata in ZIP-files or whatever
- non-determinism is the devil
- sometimes people actively introduce build-randomness (NaCl)### 12
- Does that binary download on the project's website really match the source?
- Your Linux packages are signed by someone - cool - but what does that mean?### 13
Two things should be achieved - gross oversimplification - to get to the ideal "desired state of the union":
1. full-source bootstrap: without ever introducing any binaries, go from nothing to a full Linux distribution
2. when packages are distributed, we should be able to know the expected output of a source package beforehand
=> suddenly binary distributions become a cache! But more on Nix later.### 14
- Debian project does not seem as concerned with bootstrapping as with reproducibility
- Debian mostly bootstraps on new architectures (using cross-compilation and similar techniques, from an existing binary base)
- core bootstrap (GCC & friends) is performed with previous Debian version and depending on GCC### 15
... however! Debian cares about reproducibility.
- automated testing of reproducibility
- information about the status of all packages is made available in repos
- Over 90% packages of packages are reproducible!
< show reproducible builds website >
Debian is still fundamentally a binary distribution though, but it doesn't have to be that way.### 16
Nix - a purely functional package manager
It's not a new project (10+ years), been discussed here before, has multiple components: package manager, language, NixOS.
Instead of describing *how* to build a thing, Nix describes *what* to build:### 17
### 19
In Nix, it's impossible to say "GCC is the result of applying GCC to the GCC source", because that happens to be infinite recursion.
Bootstrapping in Nix works by introducing a binary pinned by its full-hash, which was built on some previous Nix version.
Unfortunately also just a magic binary blob ... ### 20
NixOS is not actively porting all of Debian's reproducibility patches, but builds are fully repeatable:
- introducing a malicious compiler would produce a different input hash -> different package
Future slide: hope is not lost! Things are underway.### 21
- bootstrappable.org (demo?) is an umbrella page for several projects working on bootstrappability
- stage0 is an important piece: manually, small, auditable Hex programs to get to a Hex macro expander
- end goal is a full-source bootrap, but pieces are missing### 22
MES is out of the GuixSD circles (explain Guix, GNU Hurd joke)
- idea being that once you have a Lisp, you have all of computing (as Alan Key said)
- includes MesCC in Scheme -> can *almost* make a working tinyCC -> can *almost* make a working gcc 4.7
- minimal Scheme interpreter, currently built in C to get the higher-level stuff to work, goal is rewrite in hex
- bootstrapping Guix is the end goal### 23
- userspace in Darwin has a Nix project
- unsure about other BSDs, but if anyone knows - input welcome!
- F-Droid has reproducible Android packages, but that's also userspace only
- All other mobile platforms are a lost cause
Generally, all closed-source software is impossible to trust.

View file

@ -0,0 +1,6 @@
These are the slides for a presentation I gave for the Oslo javaBin meetup in
2016.
Unfortunately there is no recording of the presentation due to a technical error
(video was recorded, but no audio). This is a bit of a shame because I think
these are some of the best slides I've ever made.

View file

@ -0,0 +1,222 @@
slidenumbers: true
Erlang.
======
### Fault-tolerant, concurrent programming.
---
## A brief history of Erlang
---
![](https://www.ericsson.com/thinkingahead/the-networked-society-blog/wp-content/uploads/2014/09/bfW5FSr.jpg)
^ Telefontornet in Stockholm, around 1890. Used until 1913.
---
![](https://3.bp.blogspot.com/-UF7W9yTUO2g/VBqw-1HNTzI/AAAAAAAAPeg/KvsMbNSAcII/s1600/6835942484_1531372d8f_b.jpg)
^ Telephones were operated manually at Switchboards. Anyone old enough to remember? I'm certainly not.
---
![fit](https://russcam.github.io/fsharp-akka-talk/images/ericsson-301-AXD.png)
^ Eventually we did that in software, and we got better at it over time. Ericsson AXD 301, first commercial Erlang switch. But lets take a step back.
---
## Phone switches must be ...
Highly concurrent
Fault-tolerant
Distributed
(Fast!)
![right 150%](http://learnyousomeerlang.com/static/img/erlang-the-movie.png)
---
## ... and so is Erlang!
---
## Erlang as a whole:
- Unique process model (actors!)
- Built-in fault-tolerance & error handling
- Distributed processes
- Three parts!
---
## Part 1: Erlang, the language
- Functional
- Prolog-inspired syntax
- Everything is immutable
- *Extreme* pattern-matching
---
### Hello Joe
```erlang
hello_joe.
```
---
### Hello Joe
```erlang
-module(hello1).
-export([hello_joe/0]).
hello_joe() ->
hello_joe.
```
---
### Hello Joe
```erlang
-module(hello1).
-export([hello_joe/0]).
hello_joe() ->
hello_joe.
% 1> c(hello1).
% {ok,hello1}
% 2> hello1:hello_joe().
% hello_joe
```
---
### Hello Joe
```erlang
-module(hello2).
-export([hello/1]).
hello(Name) ->
io:format("Hello ~s!~n", [Name]).
% 3> c(hello2).
% {ok,hello2}
% 4> hello2:hello("Joe").
% Hello Joe!
% ok
```
---
## [fit] Hello ~~world~~ Joe is boring!
## [fit] Lets do it with processes.
---
### Hello Server
```erlang
-module(hello_server).
-export([start_server/0]).
start_server() ->
spawn(fun() -> server() end).
server() ->
receive
{greet, Name} ->
io:format("Hello ~s!~n", [Name]),
server()
end.
```
---
## [fit] Some issues with that ...
- What about unused messages?
- What if the server crashes?
---
## [fit] Part 2: Open Telecom Platform
### **It's called Erlang/OTP for a reason.**
---
# OTP: An Application Framework
- Supervision - keep processes alive!
- OTP Behaviours - common process patterns
- Extensive standard library
- Error handling, debuggers, testing, ...
- Lots more!
^ Standard library includes lots of things from simple network libraries over testing frameworks to cryptography, complete LDAP clients etc.
---
# Supervision
![inline](http://erlang.org/doc/design_principles/sup6.gif)
^ Supervision keeps processes alive, different restart behaviours, everything should be supervised to avoid "process" (and therefore memory) leaks
---
# OTP Behaviours
* `gen_server`
* `gen_statem`
* `gen_event`
* `supervisor`
^ gen = generic. explain server, explain statem, event = event handling with registered handlers, supervisor ...
---
`gen_server`
---
## [fit] Part 3: BEAM
### Bogdan/Bjørn Erlang Abstract machine
---
## A VM for Erlang
* Many were written, BEAM survived
* Concurrent garbage-collection
* Lower-level bytecode than JVM
* Very open to new languages
(Elixir, LFE, Joxa, ...)
---
## What next?
* Ole's talk, obviously!
* Learn You Some Erlang!
www.learnyousomeerlang.com
* Watch *Erlang the Movie*
* (soon!) Join the Oslo BEAM meetup group
---
# [fit] Questions?
`@tazjin`

View file

@ -0,0 +1,5 @@
-module(hello).
-export([hello_joe/0]).
hello_joe() ->
hello_joe.

View file

@ -0,0 +1,5 @@
-module(hello1).
-export([hello_joe/0]).
hello_joe() ->
hello_joe.

View file

@ -0,0 +1,11 @@
-module(hello2).
-export([hello/1]).
hello(Name) ->
io:format("Hey ~s!~n", [Name]).
% 3> c(hello2).
% {ok,hello2}
% 4> hello2:hello("Joe").
% Hello Joe!
% ok

View file

@ -0,0 +1,12 @@
-module(hello_server).
-export([start_server/0, server/0]).
start_server() ->
spawn(fun() -> server() end).
server() ->
receive
{greet, Name} ->
io:format("Hello ~s!~n", [Name]),
hello_server:server()
end.

View file

@ -0,0 +1,36 @@
-module(hello_server2).
-behaviour(gen_server).
-compile(export_all).
%%% Start callback for supervisor
start_link() ->
gen_server:start_link({local, ?MODULE}, ?MODULE, [], []).
%%% gen_server callbacks
init([]) ->
{ok, sets:new()}.
handle_call({greet, Name}, _From, State) ->
io:format("Hello ~s!~n", [Name]),
NewState = sets:add_element(Name, State),
{reply, ok, NewState};
handle_call({bye, Name}, _From, State) ->
io:format("Goodbye ~s!~n", [Name]),
NewState = sets:del_element(Name, State),
{reply, ok, NewState}.
terminate(normal, State) ->
[io:format("Goodbye ~s!~n", [Name]) || Name <- State],
ok.
%%% Unused gen_server callbacks
code_change(_OldVsn, State, _Extra) ->
{ok, State}.
handle_info(_Info, State) ->
{noreply, State}.
handle_cast(_Request, State) ->
{noreply, State}.

View file

@ -0,0 +1,24 @@
-module(hello_sup).
-behaviour(supervisor).
-export([start_link/0, init/1]).
%%% Module API
start_link() ->
supervisor:start_link({local, ?MODULE}, ?MODULE, []).
%%% Supervisor callbacks
init([]) ->
Children = [hello_spec()],
{ok, { {one_for_one, 5, 10}, Children}}.
%%% Private
hello_spec() ->
#{id => hello_server2,
start => {hello_server2, start_link, []},
restart => permanent,
shutdown => 5000,
type => worker,
module => [hello_server2]}.

View file

@ -0,0 +1,8 @@
all: slides
slides:
lualatex --shell-escape slides.tex
clean:
rm -f slides.aux slides.log slides.nav \
slides.out slides.toc slides.snm

View file

@ -0,0 +1,7 @@
These are the slides for my presentation about [servant][] at [Oslo Haskell][].
A full video recording of the presentation is available [on Vimeo][].
[servant]: https://haskell-servant.github.io/
[Oslo Haskell]: http://www.meetup.com/Oslo-Haskell/events/227107530/
[on Vimeo]: https://vimeo.com/153901805

Binary file not shown.

View file

@ -0,0 +1,75 @@
[file]
slides.pdf
[font_size]
10897
[notes]
### 1
13### 2
Let's talk about servant, which is several things:
API description DSL, we'll speak about how this DSL works
and why it's at the type level
Interpretations of the types resulting from that DSL, for example in
web servers or API clients
Servant is commonly used or implementing services with APIs, or for accessing
other APIs with a simple, typed client
### 3
Why type-level DSLs?
Type-level DSL: express *something*, e.g. endpoints of API, on type level by combining types. Types can be uninhabited
Phil Wadler's: expression problem: things should be extensible both in the cases of a type, and in the functions operating on the type
Normal data types: can't add new constructors easily
Servant lifts thisup to simply allow the declaration of new types that can be included in the DSL, and new interpretations that can be attached to the types through typeclasses
APIs become first-class citizens, can pass them around, combine them etc, they are separate from interpretations such as server implementations. In contrast, in most webframeworks, API declaration is implicit
(Mention previous attemps at type-safe web, Yesod / web-routes + boomerang etc)
### 4
Three extensions are necessary:
TypeOperators lets us use infix operators on the type level as constructors
DataKinds promotes new type declarations to the kind level, makes type-level literals (strings and natural numbers) available, lets us use type-level lists and pairs in combination with typeoperators
TypeFamilies: Type-level functions, map one set of types to another, come in two forms (type families, non-injective; data families, injective), more powerful than associated types
### 5
Here you can see servant's general syntax, we define an API type as a simple alias of some other type combinations
strings are type-level strings, not actually values, represent path elements
endpoints are separated by :<|>, all endpoints end in a method with content types and return types
Capture captures path segments, but there are other combinators, for example for headers
Everything that is used from the request is expressed in types, enforcing checkability, no "escape hatch" inside handlers to get request
Every combinator has associated interpretations through typeclasses
### 6
Explain type alias, point out Capture
Server is a type level function (type family), as mentioned earlier
### 7
If we expand server (in ghci with kind!) we can see the actual type of the
function
### 8
Lets speak about some interpretations of these things
### 9
Servant server is the main interpretation that people are interested in, it's used
for taking a type specification and creating a server from it
Based on WAI, the web application interface, common abstraction for web servers which came out of the Yesod project. Implemented by the web server warp, which Yesod runs on
### 10
Explain snippet, path gets removed from server type (irrelevant for handler),
route extracts string to value level
### 11
Explain echo server quickly
### 12
servant client allows generation of Haskell functions that query the API with the same types
this makes for easy to use RPC for example
### 13
A lot of other interpretations exist for all kinds of things, mock servers for testing, foreign functions in various languages, documentation ...
### 14
Demo!
1. Go quickly through code
2. Run server, query with curl
3. Open javascript function
4. Show JS code in the thing
5. Open the map itself
6. Open GHCi, use client
7. Generate docs
### 15
Conclusion
Servant is pretty good, it's very easy to get started and it's great to raise the level of things that the compiler can tell you about when you do them wrong.
### 16
Drawbacks.

View file

@ -0,0 +1,137 @@
\documentclass[12pt]{beamer}
\usetheme{metropolis}
\usepackage{minted}
\newenvironment{code}{\ttfamily}{\par}
\title{servant}
\subtitle{Defining web APIs at the type-level}
\begin{document}
\metroset{titleformat frame=smallcaps}
\setminted{fontsize=\scriptsize}
\maketitle
\section{Introduction}
\begin{frame}{Type-level DSLs?}
\begin{itemize}
\item (Uninhabited) types with attached ``meaning''
\item The Expression Problem (Wadler 1998)
\item API representation and interpretation are separated
\item APIs become first-class citizens
\end{itemize}
\end{frame}
\begin{frame}{Haskell extensions}
\begin{itemize}
\item TypeOperators
\item DataKinds
\item TypeFamilies
\end{itemize}
\end{frame}
\begin{frame}[fragile]{A servant example}
\begin{minted}{haskell}
type PubAPI = "pubs" :> Get [JSON] [Pub]
:<|> "pubs" :> "tagged"
:> Capture "tag" Text
:> Get [JSON] [Pub]
\end{minted}
\end{frame}
\begin{frame}[fragile]{Computed types}
\begin{minted}{haskell}
type TaggedPubs = "tagged" :> Capture "tag" Text :> ...
taggedPubsHandler :: Server TaggedPubs
taggedPubsHandler tag = ...
\end{minted}
\end{frame}
\begin{frame}[fragile]{Computed types}
\begin{minted}{haskell}
type TaggedPubs = "tagged" :> Capture "tag" Text :> ...
taggedPubsHandler :: Server TaggedPubs
taggedPubsHandler tag = ...
Server TaggedPubs ~
Text -> EitherT ServantErr IO [Pub]
\end{minted}
\end{frame}
\section{Interpretations}
\begin{frame}{servant-server}
The one everyone is interested in!
\begin{itemize}
\item Based on WAI, can run on warp
\item Interprets combinators with a simple \texttt{HasServer c} class
\item Easy to use!
\end{itemize}
\end{frame}
\begin{frame}[fragile]{HasServer ...}
\begin{minted}{haskell}
instance (KnownSymbol path, HasServer sublayout)
=> HasServer (path :> sublayout) where
type ServerT (path :> sublayout) m = ServerT sublayout m
route ...
where
pathString = symbolVal (Proxy :: Proxy path)
\end{minted}
\end{frame}
\begin{frame}[fragile]{Server example}
\begin{minted}{haskell}
type Echo = Capture "echo" Text :> Get [PlainText] Text
echoAPI :: Proxy Echo
echoAPI = Proxy
echoServer :: Server Echo
echoServer = return
\end{minted}
\end{frame}
\begin{frame}{servant-client}
\begin{itemize}
\item Generates Haskell client functions for API
\item Same types as API specification: For RPC the whole ``web layer'' is abstracted away
\item Also easy to use!
\end{itemize}
\end{frame}
\begin{frame}{servant-docs, servant-js ...}
Many other interpretations exist already, for example:
\begin{itemize}
\item Documentation generation
\item Foreign function export (e.g. Elm, JavaScript)
\item Mock-server generation
\end{itemize}
\end{frame}
\section{Demo}
\section{Conclusion}
\begin{frame}{Drawbacks}
\begin{itemize}
\item Haskell has no custom open kinds (yet)
\item Proxies are ugly
\item Errors can be a bit daunting
\end{itemize}
\end{frame}
\begin{frame}{Questions?}
Ølkartet: github.com/tazjin/pubkartet \\
Slides: github.com/tazjin/servant-presentation
@tazjin
\end{frame}
\end{document}

View file

@ -0,0 +1,6 @@
slides.aux
slides.log
slides.nav
slides.out
slides.snm
slides.toc

View file

@ -0,0 +1 @@
No Nix files will ever be under this tree ...

View file

@ -0,0 +1,11 @@
all: slides.pdf
slides.toc:
lualatex slides.tex
slides.pdf: slides.toc
lualatex slides.tex
clean:
rm -f slides.aux slides.log slides.nav \
slides.out slides.toc slides.snm

View file

@ -0,0 +1,6 @@
This repository contains the slides for my systemd presentation at Hackeriet.
Requires LaTeX, [beamer][] and the [metropolis][] theme.
[beamer]: http://mirror.hmc.edu/ctan/macros/latex/contrib/beamer/
[metropolis]: https://github.com/matze/mtheme

View file

@ -0,0 +1,7 @@
[Unit]
Description=Demonstrate failing units
OnFailure=demo-notify@%n.service
[Service]
Type=oneshot
ExecStart=/usr/bin/false

View file

@ -0,0 +1,7 @@
[Unit]
Description=Limited resources demo
DefaultDependencies=no
Before=slices.target
[Slice]
CPUQuota=10%

View file

@ -0,0 +1,6 @@
[Unit]
Description=Demonstrate systemd templating by sending a notification
[Service]
Type=oneshot
ExecStart=/usr/bin/notify-send 'Systemd notification' '%i'

View file

@ -0,0 +1,6 @@
[Unit]
Description=Demonstrate systemd path units
[Path]
DirectoryNotEmpty=/tmp/hackeriet
Unit=demo.service

View file

@ -0,0 +1,6 @@
[Unit]
Description=Stress test CPU
[Service]
Slice=demo.slice
ExecStart=/usr/bin/stress -c 5

View file

@ -0,0 +1,12 @@
[Unit]
Description=Demonstrate systemd timers
[Timer]
OnActiveSec=2
OnUnitActiveSec=5
AccuracySec=5
Unit=demo.service
# OnCalendar=Thu,Fri 2016-*-1,5 11:12:13
[Install]
WantedBy=multi-user.target

View file

@ -0,0 +1,6 @@
[Unit]
Description=Demo unit for systemd
[Service]
Type=oneshot
ExecStart=/usr/bin/echo "Systemd unit activated. Hello Hackeriet."

View file

@ -0,0 +1,27 @@
# simple oneshot
Run `demo-notify@hello.service`
# simple timer
Run `demo-timer.timer`, show both
# enabling
Enable `demo-timer.timer`, go to symlink folder, disable
# OnError
Show & run `demo-error.service`
# cgroups demo
Start `demo-stress.service` without, show in htop, stop
Show slice unit, start slice unit
Add Slice=demo-limits.slice
daemon-reload
Start stress again
# Proper service
Look at nginx unit

Binary file not shown.

View file

@ -0,0 +1,85 @@
[file]
slides.pdf
[notes]
### 1
### 2
Let's start off by looking at what an init system is, how they used to work and what systemd does different before we go into more systemd-specific details.
### 3
system processes that are started include for example FS mounts, network settings, powertop...
system services are long-running processes such as daemons, e.g. SSH, database or web servers, session managers, udev ...
orphans: Process whose parent has finished somehow, gets adopted by init system
-> when a process terminates its parent must call wait() to get its exit() code, if there is no init system adopting orphans the process would become a zombie
### 4
Before systemd there were simple init systems that just did the tasks listed on the previous slide.
Init scripts -> increased greatly in complexity over time, look at incomprehensible skeleton for Debian service init scripts
Runlevels -> things such as single-user mode, full multiuser mode, reboot, halt
Init will run all the scripts, but it will not do much more than print information on success/failure of started scripts
Init scripts run strictly sequential
Init is unaware of inter-service dependencies, expressed through prefixing scripts with numbers etc.
Init will not watch processes after system is booted -> crashing daemons will not automatically restart
### 5
### 6
How systemd came to be
Considering the lack of process monitoring, problematic things about init scripts -> legacy init systems have drawbacks
Apple had already built launchd, a more featured init system that monitored running processes, could automatically restart them and allowed for certain advanced features -> however it is awful to use and wrap your head around
Lennart Poettering of Pulseaudio fame and Kay Sievers decided to implement a new init system to address these problems, while taking certain clues from Apple's design
### 7
Systemd's design goals
### 8
No more init scripts with opaque effects -> services are clearly defined units
Unit dependencies -> systemd can figure out what can be started in parallel
Process supervision: Unit can be configured in many ways, e.g. always restart, only restart on success etc
Service logs: We'll talk more about this later
### 9
Units are the core component of systemd that users deal with. They define services and everything else that systemd needs to start and manage.
Note that all these are the names of the respective man page on a system with systemd installed
Types:
systemd.service - processes controlled by systemd
systemd.target - equivalent to "runlevels", grouping of units for synchronisation
systemd.timer - more powerful replacement of cron that starts other units
systemd.path - systemd equvialent of inotify, watches files/folders -> launches units
systemd.socket - expose local IPC or network sockets, launch units on connections
systemd.device - trigger units when certain devices are connected
systemd.mount - systemd equivalent of fstab entries
systemd.swap - like mount
systemd.slice - unit groups for resource management purposes
... and a few more specialised ones
### 10
Linux cgroups are a new resource management feature added quite a long time ago, but not used much.
Cgroups can be created manually and processes can be moved into them in order to control resource utilisation
Few people used them before systemd, limits.conf was often much easier but not as fine-grained
Systemd changed this
### 11
Systemd collects standard output and stderr from all processes into its journal system
they provide a tool for querying the log, for example grouping service logs together with correct timestamps, querying,
### 12
Systemd tooling, most important one is systemctl for general service management
journalctl is the query and management tool for journald
systemd-analyze is used for figuring out performance issues, for example by analysing the boot process, can make cool graphs of dependencies
systemd-cgtop is like top, but not on a process level - it's on a cgroup/slice level, shows combined usage of cgroups
systemd-cgls lists contents of systemd's cgroups to see which services are in what group
there also exist a bunch of others that we'll skip for now
### 13
### 14
### 15
Systemd criticism comes from many directions and usually focuses on a few points
feature-creep: systemd is absorbing a lot of different services
### 16
explain diagram a bit
### 17
opaque: as a result, systemd has a lot more internal complexity that people can't easily wrap your mind around. However I argue that unless you're using something like suckless' sinit with your own scripts, you probably have no idea what your init does today anyways
unstable: this was definitely true even in the first stable release, with the binary log format getting corrupted for example. I haven't personally experienced any trouble with it recently though.
Another thing is that services start depending on systemd when they shouldn't, a problem for the BSD world (who cares (hey christoph!))
### 18
Despite criticism, systemd was adopted rapidly by large portions of the Linux
Initially in RedHat, because Poettering and co work there and it was clear from the beginning that it would be there
ArchLinux (which I'm using) and a few others followed suit quite quickly
Eventually, the big Debian init system discussion - after a lot of flaming - led to Debian adopting it as well, which had a ripple effect for related distros such as Ubuntu which abandoned upstart for it.

View file

@ -0,0 +1,160 @@
\documentclass[12pt]{beamer}
\usetheme{metropolis}
\newenvironment{code}{\ttfamily}{\par}
\title{systemd}
\subtitle{The standard Linux init system}
\begin{document}
\metroset{titleformat frame=smallcaps}
\maketitle
\section{Introduction}
\begin{frame}{What is an init system?}
An init system is the first userspace process (PID 1) started in a UNIX-like system. It handles:
\begin{itemize}
\item Starting system processes and services to prepare the environment
\item Adopting and ``reaping'' orphaned processes
\end{itemize}
\end{frame}
\begin{frame}{Classical init systems}
Init systems before systemd - such as SysVinit - were very simple.
\begin{itemize}
\item Services and processes to run are organised into ``init scripts''
\item Scripts are linked to specific runlevels
\item Init system is configured to boot into a runlevel
\end{itemize}
\end{frame}
\section{systemd}
\begin{frame}{Can we do better?}
\begin{itemize}
\item ``legacy'' init systems have a lot of drawbacks
\item Apple is taking a different approach on OS X
\item Systemd project was founded to address these issues
\end{itemize}
\end{frame}
\begin{frame}{Systemd design goals}
\begin{itemize}
\item Expressing service dependencies
\item Monitoring service status
\item Enable parallel service startups
\item Ease of use
\end{itemize}
\end{frame}
\begin{frame}{Systemd - the basics}
\begin{itemize}
\item No scripts are executed, only declarative units
\item Units have explicit dependencies
\item Processes are supervised
\item cgroups are utilised to apply resource limits
\item Service logs are managed and centrally queryable
\item Much more!
\end{itemize}
\end{frame}
\begin{frame}{Systemd units}
Units specify how and what to start. Several types exist:
\begin{code}
\small
\begin{columns}[T,onlytextwidth]
\column{0.5\textwidth}
\begin{itemize}
\item systemd.service
\item systemd.target
\item systemd.timer
\item systemd.path
\item systemd.socket
\end{itemize}
\column{0.5\textwidth}
\begin{itemize}
\item systemd.device
\item systemd.mount
\item systemd.swap
\item systemd.slice
\end{itemize}
\end{columns}
\end{code}
\end{frame}
\begin{frame}{Resource management}
Systemd utilises Linux \texttt{cgroups} for resource management, specifically CPU, disk I/O and memory usage.
\begin{itemize}
\item Hierarchical setup of groups makes it easy to limit resources for a set of services
\item Units can be attached to a \texttt{systemd.slice} for controlling resources for a group of services
\item Resource limits can also be specified directly in the unit
\end{itemize}
\end{frame}
\begin{frame}{journald}
Systemd comes with an integrated log management solution, replacing software such as \texttt{syslog-ng}.
\begin{itemize}
\item All process output is collected in the journal
\item \texttt{journalctl} tool provides many options for querying and tailing logs
\item Children of processes automatically log to the journal as well
\item \textbf{Caveat:} Hard to learn initially
\end{itemize}
\end{frame}
\begin{frame}{Systemd tooling}
A variety of CLI-tools exist for managing systemd systems.
\begin{code}
\begin{itemize}
\item systemctl
\item journalctl
\item systemd-analyze
\item systemd-cgtop
\item systemd-cgls
\end{itemize}
\end{code}
Let's look at some of them.
\end{frame}
\section{Demo}
\section{Controversies}
\begin{frame}{Systemd criticism}
Systemd has been heavily criticised, usually focusing around a few points:
\begin{itemize}
\item Feature-creep: Systemd absorbs more and more other services
\end{itemize}
\end{frame}
\begin{frame}{Systemd criticism}
\includegraphics[keepaspectratio=true,width=\textwidth]{systemdcomponents.png}
\end{frame}
\begin{frame}{Systemd criticism}
Systemd has been heavily criticised, usually focusing around a few points:
\begin{itemize}
\item Feature-creep: Systemd absorbs more and more other services
\item Opaque: systemd's inner workings are harder to understand than old \texttt{init}
\item Unstable: development is quick and breakage happens
\end{itemize}
\end{frame}
\begin{frame}{Systemd adoption}
Systemd was initially adopted by RedHat (and related distributions).
It spread quickly to others, for example ArchLinux.
Debian and Ubuntu were the last major players who decided to adopt it, but not without drama.
\end{frame}
\section{Questions?}
\end{document}

Binary file not shown.

After

Width:  |  Height:  |  Size: 228 KiB