site-josuah

/usr/josuah
Log | Files | Refs

commit f2f9cf9cff2b2c236f3488bdf68c1c4108ad83fc
parent 133d7f3d19aea51075621464e766ca62d3a0a11e
Author: Josuah Demangeon <me@josuah.net>
Date:   Fri, 17 Apr 2020 22:50:36 +0200

convert blog posts into wiki entries

Diffstat:
A.githooks/.post-update.swp | 0
Dblog/2019-06-29-dns-setup/index.md | 168-------------------------------------------------------------------------------
Dblog/2019-07-15-daemon-supervisors/index.md | 89-------------------------------------------------------------------------------
Mindex.md | 13++++++-------
Awiki/git-hooks/index.md | 98+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Awiki/supervisor/index.md | 81+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Awiki/tinydns/index.md | 98+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
7 files changed, 283 insertions(+), 264 deletions(-)

diff --git a/.githooks/.post-update.swp b/.githooks/.post-update.swp diff --git a/blog/2019-06-29-dns-setup/index.md b/blog/2019-06-29-dns-setup/index.md @@ -1,168 +0,0 @@ -Simple, easy, trivial DNS setup that works -========================================== - -To run nameservers, you need to maintain a bunch of interdependent -DNS information: - -* You first need NS entries that tells who resolve the domain. - -* You then need A and AAAA entries for those NS entries so that one - can join the nameservers. - -* It is good practice to keep PTR entries for these A and AAAA entries - by politeness. - -* You will then have MX entries so you can receive e-mail. - -* But MX entries needs to have matching PTR records, that itself - needs to ping back to the same IP as the MX record have (mail hosting - providers makes a difference between lazy spammers and someone - who wants to maintain its mail stack). - -* Finally you have a basic setup, you can add your A entries. But wait, - do I setup matching PTR entries for these A entries? There are already - PTR entries on some of these IPs. - -There need for this much configuration! - -By maintaining just a few lists of information that make sense, all -of this mess becomes crystal clear. - -I have these files with one entry per line: - -/etc/dns/rr.domain - A list of top and second level domain names. The first one - listed is the "technical" domain name. - - i.e: z0.is - josuah.net - -/etc/dns/rr.host - A list of "hostname without domain part", "IPv4", "IPv6". - This is the only file where IP are written. This is the - book keeping of the servers you address. - - Each line of this file leads to a hostname.technical.dom - record of type A, AAAA, and PTR. This makes debugging - simple: you know the hostname associated with each IP. - - i.e: kuntur 199.247.28.162 2001:19f0:5001:7ac::12 - harpyja 80.67.190.196 2a00:5884:8214::16 - -/etc/dns/rr.mx - A list of hostnames that run a mail server. Each domain - in "rr.domain" gets one MX record per mail server listed - here, with the form: hostname.technical.dom as generated - by the "rr.host" list. - - We now have matching MX, A, AAAA and PTR entries! - - i.e: kuntur - -/etc/dns/rr.ns - A list of hostnames that run a DNS name server. Like for - "rr.mx", each domain in "rr.domain" gets one NS record per - name server listed here. - - i.e: kuntur - harpyja - -/etc/dns/rr.alias - A list of regular domain records for your various servers - and services. You may - - i.e: harpyja www.josuah.net - harpyja git.josuah.net - kuntur josuah.z0.is - -In the end you have a clear model that you build for yourself that -fully leverage the DNS zone challenges. - -New computer, mail servers, name servers, top level domain... are -added by a single line on one of these files (same goes for removal). - -You never have to copy-paste IPs, nor maintain consistency between -different records. Each information is held at only at one place. - -I use the tinydns authoritative nameserver to publish these records. - -This below is the awk script that convert the list "i.e:" above to a -zone readable by tinydns. I run it in a makefile like this: - -/etc/dns/Makefile: - - all: data.cdb - - data = rr.domain rr.host rr.alias rr.mx rr.ns - data: data.awk ${data} - awk -f data.awk ${data} >$@ - - data.cdb: data - tinydns-data - - clean: - rm -f data data.cdb - -/etc/dns/data.awk: - - BEGIN { - FS = "[\t ]+" - } - - function ip6_hex(ip6) { - x = "" - sub("::", substr("::::::::", split(ip6, a, ":") - 1), ip6) - split(ip6, a, ":") - for (i = 1; i <= 8; i++) x = x substr("0000" a[i], length(a[i]) + 1) - return x - } - - function ip6_fmt(ip6) { - gsub("....", "&:", ip6) - sub(":$", "", ip6) - return ip6 - } - - FNR == 1 && FILENAME != "rr.domain" { - print "\n# " FILENAME "\n" - } - - FILENAME != "rr.domain" && FNR != 1 { - print "" - } - - FILENAME == "rr.domain" { - domain[++i] = $1 - } - - FILENAME == "rr.host" { - host[$1"4"] = $2 - host[$1"6"] = ip6_hex($3) - print "=" $1 "." domain[1] ":" host[$1"4"] - print "6" $1 "." domain[1] ":" host[$1"6"] - } - - FILENAME == "rr.alias" { - for (f = 2; f <= NF; f++) { - print "+" $f ":" host[$1"4"] - print "3" $f ":" host[$1"6"] - } - } - - FILENAME == "rr.ns" { - print "+" $1 "." domain[1] ":" host[$1"4"] - print "3" $1 "." domain[1] ":" host[$1"6"] - for (i in domain) { - print "." domain[i] "::" $1 "." domain[1] - } - } - - FILENAME == "rr.mx" { - for (i in domain) { - print "@" domain[i] "::" $1 "." domain[1] - } - } - - END { - print "" - } diff --git a/blog/2019-07-15-daemon-supervisors/index.md b/blog/2019-07-15-daemon-supervisors/index.md @@ -1,89 +0,0 @@ -Daemons, supervisors, and simplicity -==================================== - -When it comes to servers, daemon is a central concept: - - daemon - program that runs even after the user quit the shell - -That is basically a "daemon" right? - -A simple problem, a simple solution: make the process double-fork -so the user sitting at the terminal is free to keep going. -pgrep and ps are your friends now. - -This is one simple approach, but one single problem arises: - - -- How to run the same daemon twice? -- - -Knee-jerk reaction: "Why would one want to do that?!" - -- Different DHCP pool of addresses on different interfaces, -- Different mail filtering rules for different inside/outside networks, -- Different vHosts for a mail / http / ... daemon, -- Different identical daemon running as different users, -- Different agetty with one per TTY, -- ... - -The solution often encountered is handling the variety of roles -from the inside of the daemon instead of starting one dedicated -daemon per role: with configuration blocks that lets you handle -each different roles in a different way (per vhost, per network -interface, per tty, per tcp port...). - -That makes each and every daemon much more complex. They all need -a complex configuration file parser and have a much more complex -internal design. - -Why not one daemon per configuration block? Because when a daemon -puts itself to the background (through forking), we loose its PID -(Process ID) and then we cannot distinguish two identical daemons. -How do we know which one to restart then? - -Solutions? - -- Writing the PID into a file is not good, as if a daemon gets killed - while getting out of memory, the PID file remains, and another daemon - could get the PID of its parent (former Debian style). - -- Having a ${daemon}ctl command that talks to the daemon through a socket - works, but then each daemon needs to support it while a simple signal - handling would solve all use cases (*BSD style). - -- Matching the command line the daemon was started with with pgrep works - but it requires to adapt it to each daemon (OpenBSD style). - -- Keep the daemon in the foreground (s6 / runit / daemontools style). - -Wait what? - -Yes! In the end, not causing the problem might be a decent solution. -Though we still need to get the control back to the terminal after -launching the daemon... - -Then use one process to launch all the others: A "supervisor" that -starts the daemons, and keeps each daemon as a child proces. The -supervisor knows their PID without PID file or socket. That is how -fork() works: it returns the PID. - -How to organize daemons then? A trivial approach is to have one -run script per daemon to launch, that exec() into the daemon at the -end which stays at the foreground. - -Once the supervisor is is triggered, it can start each of these -./${daemon}/run scripts, keep them as child, and watch for more -to come, with some ./${daemon}/socket for listenning for commands -such as "restart", "stop", "alarm" to send signals to the daemon. - -The supervision system was complex to implement right and half baked -inside of each daemon with ${daemon}ctl, it is now done reliably once -for all daemons in a dedicated program: the supervisor. - -Running this/these extra processes does not consume much more memory -(one megabyte? two?) and makes each daemon smaller (which compensates -for the few megabytes lost). - -Yes, systemd also comes with its own set of solutions. But systemd -have an army of engineers (backed by one of the biggest tech company) -to go through the army of problems it may face. It is a complex -solution to a complex problem, while supervision trees make the -problem simple first, and then solve it simply. diff --git a/index.md b/index.md @@ -1,18 +1,17 @@ Welcome to my publication tool, [[josuah.net]]. -This is the home of: +You can find documentation about my software projects: - * The [[NotWiki]] project, a documentation tool that you currently see at work. + * The [[NotWiki]] project, a website/gophersite generation tool. - * Some [[qmail]]'s internals documentation (WIP) gathered through the - [[notqmail]] project + * My [[githooks]] scripts, tiny alternative to [[githooks.com]]'s setups. - * Some [[ASCII]] Art +As well as documentation on other people's software: - * A few fancy articles on my [[blog]] +And finally, [[ASCII]] Art and a soon-to-disappear [[blog]]. [notwiki]: //code.z0.is/notwiki/ -[qmail]: https://cr.yp.to/qmail.html +[githooks]: //josuah.net/wiki/git-hooks/ [ascii]: /ascii/ [blog]: /blog/ diff --git a/wiki/git-hooks/index.md b/wiki/git-hooks/index.md @@ -0,0 +1,98 @@ +Minimal Git Hooks configuration to get started +============================================== + +Git [[hooks]] permit to run commands on a range of git events, mainly: a git +commit. + +I use 3 shell scripts of roughly 10 lines each to configure git hooks, calling +scripts from within the repository itself: .githooks/<hookname> + +[hooks]: https://githooks.com/ + +git-hooks-run +------------- +This is what runs on every event, to put on /bare-repo.git/hook/<hookname>. +There is no point in running it by hand. + + #!/bin/sh -e + + hookname=$1 ref=$2 + + [ "$(git ls-tree "$ref" ".git$hookname")" ] || exit 0 + + echo "${0##*/}: running '$1' on '$ref'" + git cat-file blob "$ref:.git$hookname" | { + IFS='! ' read -r _ cmd args + exec "$cmd" "$args" "/dev/stdin" "$ref" + } + +It checks if there is a file called .githooks/<hookname> (git ls-tree "$ref" +...), and if so, extract this file from git (git cat-file blob ...), read the +shebang, and execute the rest with the command of the shebang ("| { ... }"). + + +git-hooks-install +----------------- +This setups the command above for a bare git repository: + + #!/bin/sh -e + # setup git hooks to run .githook from the inside of the repo + + for x; do + for hook in pre-commit pre-receive post-commit post-receive; do + echo "#!/usr/bin/env git-hooks-run" >"$x/hooks/$hook" + chmod +x "$x/hooks/$hook" + done + done + +It replace selected hooks at repo.git/hooks/<hookname> with only this shebang: + + #!/usr/bin/env git-hooks-run + +This has the effect of calling the git-hooks-run from above with +hook/<hookname> as argument, along with the extra arguments providedd by git, +which is all we need for our hooks. + + +git-hooks-workdir +----------------- +With only git-hooks-run, we lack a way to use the content of the repository +to interact with the hooks (it is not always needed to use the content). + +In case this is needed, this command extract the workdir of the commit pushed +into a new directory in /var/cache/git (that it delete in case of failure), +and print it out so that the hook script can use it: + + #!/bin/sh -e + # utility to run from hooks to create a temporary working directory + + ref=$1 + commit=$(git rev-parse "$ref") + workdir="/var/cache/git/$commit" + + mkdir -p "$workdir" + trap 'rm -rf "$workdir"' INT EXIT TERM HUP + git archive --prefix="$workdir/" --format="tar" "$ref" | (cd / && tar -xf -) + exec echo "$workdir" + +To use it from within the hook, to catch the workdir and make sure there is +no remaining file even in case of failure, thanks to the trap internal shell +command: + + #!/bin/sh -ex + tmp=$(git-hooks-workdir "$@") + trap 'rm -rf "$tmp"' INT TERM EXIT HUP + cd "$tmp" + +This might be the top of your hook script. + +The (optional) -x flags in the shebang, which will print every command as it is +executed. It will be printed on commiter side as "remote: $line". + +The (optional) -e flag is there to die if any command fails, such as the initial +cd to the "$tmp" directory. + +At that point, we will be in a workind directory containing the content of the +state at the commit pushed, and we can run commands, such as running unit test +(make test), send an email or anything we need. + diff --git a/wiki/supervisor/index.md b/wiki/supervisor/index.md @@ -0,0 +1,81 @@ +Daemons, supervisors, and simplicity +==================================== + +When it comes to servers, daemon is a central concept, here is a proposed +definition: + +daemon - program that runs even after the user quit the shell + +A simple problem, a simple solution: make the process double-fork so the user +sitting at the terminal is free to keep going. pgrep and ps are your friends +now. + +This is one simple approach, but lead to that problem how to run the same +daemon twice? + +Knee-jerk reaction: "Why would one want to do that?" + + * Different DHCP pool of addresses on different interfaces, + * Different mail filtering rules for different inside/outside networks, + * Different vHosts for a mail / http / ... daemon, + * Different users running the same daemon + * Different agetty with one per TTY, + * ... + +The solution often encountered is handling the variety of roles +from the inside of the daemon instead of starting one dedicated +daemon per role: with configuration blocks that lets you handle +each different roles in a different way (per vhost, per network +interface, per tty, per tcp port...). + +That makes each and every daemon much more complex. They all need +a complex configuration file parser and have a much more complex +internal design. + +Why not one daemon per configuration block? Because when a daemon +puts itself to the background (through forking), we loose its PID +(Process ID) and then we cannot distinguish two identical daemons. +How do we know which one to restart then? + +Solutions? + + * Writing the PID into a file is not good, as if a daemon gets killed while + getting out of memory, the PID file remains, and another daemon could get + the PID of its parent (SysV-style). + + * Having a ${daemon}ctl command that talks to the daemon through a socket + works, but then each daemon needs to support it while a simple signal handling + would solve all use cases (*BSD-style). + + * Matching the command line the daemon was started with with pgrep works but + it requires to adapt it to each daemon (also *BSD-style). + + * Use Linux-specific APIs to work around the issues lead by the diversity of + cases above (systemd). + + * Keep the daemon in the foreground (s6/runit/daemontools style). + +In the end, not causing the problem might be a decent solution: getting the +control back to the terminal was convenient for running programs from a shell, +but causes problems on other sides. +need to get the control back to the terminal after starting the program... + +Then use one process to launch all the others: A "supervisor" that starts the +daemons, and keeps each daemon as a child proces. The supervisor knows their +PID without pidfile or socket. That is how fork(2) works: it returns the PID +to the parent so that it can wait(2) for it, send it signals... + +How to organize daemons then? A trivial approach is to have one run script per +daemon to launch, that exec(2) into the daemon which stays at the foreground. + +Once the supervisor is is triggered, it can start each of these ./$daemon/run +scripts, keep them as child, and watch for more to come, with some +./$daemon/sock for listenning for commands such as "restart", "stop", "alarm" +to send signals to the daemon. + +The supervision system was complex to implement right and half baked inside of +each daemon with ${daemon}ctl, it is now done reliably once for all daemons in +a dedicated program: the supervisor. + +Running this/these extra processes does not consume much more memory (one +megabyte?) and makes each daemon smaller, compensating for the lost bytes. diff --git a/wiki/tinydns/index.md b/wiki/tinydns/index.md @@ -0,0 +1,98 @@ +Configuration of [[tinydns]] +============================ + +[tinydns]: https://cr.yp.to/djbdns.html + +To run nameservers, you need to maintain a bunch of interdependent DNS +information: + + * You first need NS entries that tells who resolve the domain. + + * You then need A and AAAA entries for those NS entries so that one can join + the nameservers. + + * It is good practice to keep PTR entries. + + * You will then have MX entries so you can receive e-mail. + + * Finally you have a basic setup, you can add your A entries. But wait, do I + setup matching PTR entries for these A entries? There are already PTR + entries on some of these IPs. + +To achieve this, I use these *input files*, that get converted into ./data by +an awk [[script]]: + +[script]: /wiki/tinydns/data.awk + + +/etc/dns/rr.domain +------------------ +A list of top and second level domain names. The first one listed is the +"technical" domain name. + + z0.is + josuah.net + + +/etc/tinydns/rr.host +-------------------- +A list of "hostname without domain part", "IPv4", "IPv6". This is the only file +where IP are written. This is the book keeping of the servers you address. + +Each line of this file leads to a hostname.technical.dom record of type A, +AAAA, and PTR: + + kuntur 199.247.28.162 2001:19f0:5001:7ac::12 + harpyja 80.67.190.196 2a00:5884:8214::16 + + +/etc/tinydns/rr.mx +------------------ +A list of hostnames that run a mail server. Each domain in "rr.domain" gets one +MX record per mail server listed here, with the form: hostname.technical.dom as +generated by the "rr.host" list. + +We now have matching MX, A, AAAA and PTR entries. + + kuntur + + +/etc/dns/rr.ns +-------------- +A list of hostnames that run a DNS name server. Like for "rr.mx", each domain +in "rr.domain" gets one NS record per name server listed here. + + kuntur ns1 + harpyja ns2 + + +/etc/dns/rr.alias +----------------- +A list of regular domain records for your various servers and services. You may + + harpyja www.josuah.net + harpyja git.josuah.net + kuntur josuah.z0.is + +In the end you have a clear model that you build for yourself that fully +leverage the DNS zone challenges. + +New computer, mail servers, name servers, top level domain... are added by a +single line on one of these files (same goes for removal). + +You never have to copy-paste IPs, nor maintain consistency between different +records. Each information is held at only at one place. + +I run it in a makefile like this: + + all: data.cdb + + data = rr.domain rr.host rr.alias rr.mx rr.ns + data: data.awk ${data} + awk -f data.awk ${data} >$@ + + data.cdb: data + tinydns-data + + clean: + rm -f data data.cdb