site-josuah

/usr/josuah
Log | Files | Refs

commit ade969426963039c898af4630c15476e8932fc0d
parent 01230001725676e4ec12e3495e93fd90644459ce
Author: Josuah Demangeon <me@josuah.net>
Date:   Sat, 30 May 2020 00:47:51 +0200

import the wiki back

Diffstat:
Mindex.md | 28+++++++++-------------------
Awiki/awk/index.md | 350+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Awiki/git-hooks/index.md | 88+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Awiki/jj/index.md | 244+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Awiki/jj/jj-in | 20++++++++++++++++++++
Awiki/jj/jj-join | 14++++++++++++++
Awiki/jj/jj-log | 156+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Awiki/jj/jj-tail | 14++++++++++++++
Awiki/jj/retail | 38++++++++++++++++++++++++++++++++++++++
Awiki/qmail/destination-mx/index.md | 156+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Awiki/qmail/index.md | 23+++++++++++++++++++++++
Awiki/supervisor/index.md | 74++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Awiki/tinydns/Makefile | 14++++++++++++++
Awiki/tinydns/data.awk | 59+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Awiki/tinydns/index.md | 86+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Awiki/tinydns/push.awk | 9+++++++++
16 files changed, 1354 insertions(+), 19 deletions(-)

diff --git a/index.md b/index.md @@ -2,29 +2,19 @@ Welcome to my publication tool. You can find documentation about my software projects: - * The [[NotWiki]] project, a website/gophersite generation tool. - * My [[githooks]] scripts, that regenerate my NotWiki sites. - -[notwiki]: //code.z0.is/notwiki/ -[githooks]: //code.z0.is/wiki/git-hooks/ + * The [NotWiki](//code.z0.is/notwiki/) project, a website/gophersite + generation tool. + * My [githooks](/wiki/git-hooks/) scripts, that regenerate my NotWiki sites. As well as documentation on other people's software: - * Use of [[awk]], the command/language. - * Use of [[jj]] with UCSPI and s6. - * Use of [[tinydns]] with an awk scripts for generating dns ./data. - * Notes on [[qmail]] implementation. - -[awk]: //code.z0.is/wiki/awk/ -[jj]: //code.z0.is/wiki/jj/ -[tinydns]: //code.z0.is/wiki/tinydns/ -[qmail]: //code.z0.is/wiki/qmail/ - -And finally, [ASCII][aa] Art and [quotes][qu] and [links][lk]. + * Use of [awk](/wiki/awk/), the command/language. + * Use of [jj](/wiki/jj/) with UCSPI and s6. + * Use of [tinydns](/wiki/tinydns/) with an awk scripts for generating dns + ./data. + * Notes on [qmail](/wiki/qmail/) implementation. -[aa]: /ascii/ -[qu]: /quotes/ -[lk]: /links/ +And finally, [ASCII](/ascii/) Art and [quotes](/quotes/) and [links](/links/). You can access to this server through tor: diff --git a/wiki/awk/index.md b/wiki/awk/index.md @@ -0,0 +1,350 @@ +AWK +=== + +AWK is a surprising efficient language, for both [performance][perf] and code +efficiency. This comes with the ubiquitous array structure, and splitting the +input in fields by default. + +Not everything is parsed efficiently with AWK, Type-Length-Value for instance, +but many things are. I use it for multiple projects: + + * [[NotWiki]], featuring a (not)markdown parser that does two passes on + to easen-up the parsing, + + * [[ics2txt]], a basic iCal to TSV or plain text converter (two directions), + + * [[jj]] by aaronNGi, a daemon with an awk engine to project that turns raw + IRC protocol into easily readable split log files + +[perf]: https://adamdrake.com/command-line-tools-can-be-235x-faster-than-your-hadoop-cluster.html +[notwiki]: //code.z0.is/notwiki/ +[ics2txt]: gopher://bitreich.org/1/scm/ics2txt +[parser]: //code.z0.is/git/notwiki/files/ +[jj]: /wiki/jj/ + +Below are multiple ways of using awk for getting the best out of it. These are +partly by myself, partly collected from what I saw in the wild. + + +CSV fields with header +---------------------- +Instead of trying to remember the number of the column, using the name of the +column is much easier, and permit to have new columns inserted in the .csv file +without breaking the script. + + $ cat input.txt + domain_name,expiry_date,creation_date,owner,account_id + nowhere.com,2020-03,2019-05,me,23535 + perdu.com,2020-04,2018-03,you,23535 + pa.st,2020-09,2014-05,them,23535 + + $ awk ' + BEGIN { FS = "," } + NR == 1 { for (i = 1; i <= NF; i++) F[$i] = i; next } + $F["domain_name"] ~ /\.com$/ { + print $F["expiry_date"], $F["owner"], $F["domain_name"] + } + ' input.txt + 2020-03 me nowhere.com + 2020-04 you perdu.com + + +UCL-style configuration +----------------------- +Parsing data that is not organised with line-column is also convenient and +efficient with awk, convenient for selecting one kind of value out of a +configuration file: + + $ cat input.txt + connections { + conn-faraway { + children { + localnet = fe80:123d:35d3::%vio1/64 + localnet = fe80:2e46:1d23::%vio2/64 + } + children { + localnet = fe80:546:23e4::%vio3/64 + } + } + conn-veryclose { + children { + localnet = fe80:b536:243f::%vio3/64 + localnet = fe80:34f3:23c3::%vio3/64 + localnet = fe80:546a:343d::%vio3/64 + } + } + } + + $ awk ' + $2 == "{" { F[lv++] = $1 } + $1 == "}" { delete F[--lv] } + F[0] == "connections" && F[2] == "children" && $1 == "localnet" { + print F[1], $3 + } + ' input.txt + conn-faraway fe80:123d:35d3::%vio1/64 + conn-faraway fe80:2e46:1d23::%vio2/64 + conn-faraway fe80:546:23e4::%vio3/64 + conn-veryclose fe80:b536:243f::%vio3/64 + conn-veryclose fe80:34f3:23c3::%vio3/64 + conn-veryclose fe80:546a:343d::%vio3/64 + + +Key-Value splitter +------------------ +Parsing key-value pairs can be mapped rather directly to an awk array, +for instance, to extract an abstract out of a basic iCal file: + + $ cat input.txt + BEGIN:VEVENT + METHOD:PUBLISH + UID:9189@FOSDEM20@fosdem.org + TZID:Europe-Brussels + DTSTART:20200201T170000 + DTEND:20200201T175000 + SUMMARY:State of the Onion + DESCRIPTION:Building usable free software to fight surveillance and censorship. + CLASS:PUBLIC + STATUS:CONFIRMED + CATEGORIES:Internet + LOCATION:Janson + END:VEVENT + $ awk ' + BEGIN { FS = ":" } + { F[$1] = $2 } + $1 == "END" { + print F["SUMMARY"] " - " F["DESCRIPTION"] + print F["DTSTART"], "(" F["TZID"] ")" + } + ' input.txt + State of the Onion - Building usable free software to fight surveillance and censorship. + 20200201T170000 (Europe-Brussels) + + +Edit variables passed to functions +---------------------------------- +For languages that support references, pointers, or objects, it is possible to +edit the variable passed to a function, so that the variable also gets edited +in the function that called it. + + void increment(int *i) { (*i)++; } + +Awk does not support changing integers or strings, but supports editing the +fields of an array: + + function increment_first(arr) { arr[1]++ } + + +Local variables in functions +---------------------------- +By default, all awk variables are global, which is inconvenient for writing +functions. The solution is to add an extra function argument at the end for +each local variable we need. + +Functions can be called with fewer arguments than they have. + + $ awk ' + function concat3(arg1, arg2, arg3, + local1) + { + local1 = arg1 arg2 arg3 + return local1 + } + + BEGIN { + local1 = 1 + print(concat3("a", "w", "k")) + print(local1) + } + ' + awk + 1 + +I learned this with the [jj] project. + +[jj]: https://github.com/aaronNGi/jj/ + + +A sort() function +----------------- +A very convenient feature lacking to awk is support for sorting members of an +array. Is possible to implement sort() in awk (this is a quicksort): + + function swap(array, a, b, + tmp) + { + tmp = array[a] + array[a] = array[b] + array[b] = tmp + } + + function sort(array, beg, end) + { + if (beg >= end) # end recursion + return + + a = beg + 1 # 1st is the pivot, so +1 + b = end + while (a < b) { + while (a < b && array[a] <= array[beg]) # beg: skip lesser + a++ + while (a < b && array[b] > array[beg]) # end: skip greater + b-- + swap(array, a, b) # found 2 misplaced + } + + if (array[beg] > array[a]) # put the pivot back + swap(array, beg, a) + + sort(array, beg, a - 1) # sort lower half + sort(array, a, end) # sort higher half + } + +This sorts the array values using integers keys: `array[1]`, `array[2]`, ... +It sorts from `array[beg]` to `array[end]` included, so you can choose your +array indices starting at 0 or 1, or sort just a part of the array. + +Example usage: with the both function above: + + { + LINES[NR] = $0 + } + + END { + sort(LINES, 1, NR) + for (i = 1; i <= NR; i++) + print(LINES[i]) + } + +Performance is far from terrible! + + $ od -An /dev/urandom | head -n 1000000 | time ./test.awk >/dev/null + real 0m 19.23s + user 0m 17.90s + sys 0m 0.12s + + $ od -An /dev/urandom | head -n 1000000 | time sort >/dev/null + real 0m 4.39s + user 0m 3.00s + sys 0m 0.10s + + +A gmtime() function +------------------- +POSIX awk as well as many implementations lack the [time functions][tf] present in +GNU awk. This gmtime() function split an epoch integer value (1587302158) into the +fields year, mon, mday, hour, min, sec (2020-04-19T15:15:58Z): + +[tf]: https://www.gnu.org/software/gawk/manual/html_node/Time-Functions.html + + function isleap(year) + { + return (year % 4 == 0) && (year % 100 != 0) || (year % 400 == 0) + } + + function mdays(mon, year) + { + return (mon == 2) ? (28 + isleap(year)) : (30 + (mon + (mon > 7)) % 2) + } + + function gmtime(sec, tm) + { + tm["year"] = 1970 + while (sec >= (s = 86400 * (365 + isleap(tm["year"])))) { + tm["year"]++ + sec -= s + } + + tm["mon"] = 1 + while (sec >= (s = 86400 * mdays(tm["mon"], tm["year"]))) { + tm["mon"]++ + sec -= s + } + + tm["mday"] = 1 + while (sec >= (s = 86400)) { + tm["mday"]++ + sec -= s + } + + tm["hour"] = 0 + while (sec >= 3600) { + tm["hour"]++ + sec -= 3600 + } + + tm["min"] = 0 + while (sec >= 60) { + tm["min"]++ + sec -= 60 + } + + tm["sec"] = sec + } + +The tm array will be filled with field names following the [[gmtime]] +function as you can see above. + +[gmtime]: https://pubs.opengroup.org/onlinepubs/9699919799/functions/gmtime.html + + +A localtime() function +---------------------- +For printing functions in the user's favorite timezone, gmtime's time needs +to be shifted. This can also be done in standard awk by calling the date(1) +command: + + function localtime(sec, tm, + tz, h, m) + { + if (!TZOFFSET) { + "date +%z" | getline tz + close("date +%z") + h = substr(tz, 2, 2) + m = substr(tz, 4, 2) + TZOFFSET = substr(date, 1, 1) (h * 3600 + m * 60) + } + return gmtime(sec + TZOFFSET, tm) + } + +Note that date(1) will only be called the first time localtime() is called, and +the TZOFFSET global variable will be used for the next calls. + + +A mktime() function +------------------- +Complementary function to gmtime is mktime for converting a `tm[]` array back +to an integer representation. This is useful for parsing time values back to +an unix timestamp: + + function isleap(year) + { + return (year % 4 == 0) && (year % 100 != 0) || (year % 400 == 0) + } + + function mdays(mon, year) + { + return (mon == 2) ? (28 + isleap(year)) : (30 + (mon + (mon > 7)) % 2) + } + + function mktime(tm, + sec, mon, day) + { + sec = tm["sec"] + tm["min"] * 60 + tm["hour"] * 3600 + + day = tm["mday"] - 1 + + for (mon = tm["mon"] - 1; mon > 0; mon--) + day = day + mdays(mon, tm["year"]) + + # constants: x * 365 + x / 400 - x / 100 + x / 4 + day = day + int(tm["year"] / 400) * 146097 + day = day + int(tm["year"] % 400 / 100) * 36524 + day = day + int(tm["year"] % 100 / 4) * 1461 + day = day + int(tm["year"] % 4 / 1) * 365 + + return sec + (day - 719527) * 86400 + } + +All the following fields of `tm[]` must be defined: "year", "mon", "mday", +"hour", "min", "sec". diff --git a/wiki/git-hooks/index.md b/wiki/git-hooks/index.md @@ -0,0 +1,88 @@ +Minimal Git Hooks configuration +=============================== + +Git [[hooks]] permit to run commands on a range of git events, mainly: a git +commit. + +I use 3 shell scripts of roughly 10 lines each to configure git hooks, calling +scripts from within the repository itself: `.githooks/<hookname>` + +[hooks]: https://githooks.com/ + +git-hooks-run +------------- +This is what runs on every event, to put on `/bare-repo.git/hook/<hookname>`. +There is no point in running it by hand. + + #!/bin/sh -e + hookname=$1 ref=${2:-master} + + echo "${0##*/}: running '$1' on '$ref'" + git cat-file blob "$ref:.git$hookname" | { + IFS='! ' read -r _ cmd args + exec "$cmd" "$args" "/dev/stdin" "$ref" + } + +It checks if there is a file called `.githooks/<hookname>` (git ls-tree "$ref" +...), and if so, extract this file from git (git cat-file blob ...), read the +shebang, and execute the rest with the command of the shebang ("| { ... }"). + + +git-hooks-install +----------------- +This setups the command above for a bare git repository: + + #!/bin/sh -e + for x; do + echo "#!/usr/bin/env git-hooks-run" >"$x/hooks/post-upate" + chmod +x "$x/hooks/post-update" + done + +It replace selected hooks at repo.git/hooks/post-update with only this shebang: + + #!/usr/bin/env git-hooks-run + +This has the effect of calling the git-hooks-run from above with +hook/post-update as argument, along with the extra arguments providedd by git, +which is all we need for our hook. + + +git-hooks-workdir +----------------- +With only git-hooks-run, we lack a way to use the content of the repository +to interact with the hooks (it is not always needed to use the content). + +In case this is needed, this command extract the workdir of the commit pushed +into a new directory in /var/cache/git (that it delete in case of failure), +and print it out so that the hook script can use it: + + #!/bin/sh -e + ref=$1 + commit=$(git rev-parse "$ref") + workdir="/var/cache/git/$commit" + + mkdir -p "$workdir" + trap 'rm -rf "$workdir"' INT EXIT TERM HUP + git archive --prefix="$workdir/" --format="tar" "$ref" | (cd / && tar -xf -) + exec echo "$workdir" + +To use it from within the hook, to catch the workdir and make sure there is +no remaining file even in case of failure, thanks to the trap internal shell +command: + + #!/bin/sh -ex + tmp=$(git-hooks-workdir "$@") + trap 'rm -rf "$tmp"' INT TERM EXIT HUP + cd "$tmp" + +This might be the top of your hook script. + +The (optional) -x flags in the shebang, which will print every command as it is +executed. It will be printed on commiter side as "remote: $line". + +The (optional) -e flag is there to die if any command fails, such as the initial +cd to the "$tmp" directory. + +At that point, we will be in a workind directory containing the content of the +state at the commit pushed, and we can run commands, such as running unit test +(make test), send an email or anything we need. diff --git a/wiki/jj/index.md b/wiki/jj/index.md @@ -0,0 +1,244 @@ + The jj IRC client +=================== + +[jj](https://github.com/aaronNGi/jj/) is an [Internet Relay Chat +(IRC)](https://invidio.us/watch?v=R8FOGlnYkgg) client based on the principles +of [ii](https://tools.suckless.org/ii/), using FIFO (named pipe) files to read +commands from the user, and simply logs the output of each channels to plain +files. + +Instead of being an ncurse program that runs into tmux(1), jj works as an +applicative router. A router for ISO layer 7. You may already know "applicative +routers" for different protocols already: + + * SMTP: [OpenSMTPD][m1], [qmail][m2], [Postfix][m3]... + * HTTP: [nginx][h1], [lighttpd][h2], [relayd][h3], [haproxy][h4]... + * SIP: [OpenSIPS][s1], [Kamailio][s2]... + * IRC: [ngircd][i1], [hybridircd][i2]... + +[m1]: https://www.opensmtpd.org/ +[m2]: https://cr.yp.to/qmail.html +[m3]: http://www.postfix.org/ +[h1]: https://nginx.org/ +[h2]: https://www.lighttpd.net/ +[h3]: https://bsd.plumbing/ +[h4]: https://haproxy.org/ +[s1]: https://opensips.org/ +[s2]: https://kamailio.org/w/ +[i1]: https://ngircd.barton.de/ +[i2]: http://ircd-hybrid.org/ + +IRC itself is much of a routing protocol server-side: routing messages to +the right client. Another aspect of applicative routing is client-side. + +Much like ii or sic, and all the modular bots in the wild, the jj acts as +a is a client-side IRC router through hooks and log files. + + +How jj works +------------ +As opposed to ii, which is plain C, jj uses an awk script to do the heavy +lifting of IRC parsing, called from a compiled binary coded in C, that only +does the fifo creation, and multiplexing of user and network input, and +write both to the awk script. + +It reads its entire configuration from environment variables, described on +the the project's README.md, such as $IRC_DIR, in which it create a direcTory +with the channel as a name. + +I set IRC_DIR to /var/irc, which gives us (full list on the README.md): + + * /var/irc/irc.freenode.net/channels/*.log - messages from users and channels. + * /var/irc/irc.freenode.net/in - the FIFO pipe to which write messages. + +There is one instance of jj per server conexion, which greatly simplifies +the software, makes debugging much easier, and permit to adapt and configure +it specifically for the requirements of each host to connect to. + +Recently, jj acquired the [UCSPI][u0] connexion interface, which is a way to +write programs without handling the network protocols into the program itself, +but instead expect an running connexion (TCP, TLS, SSH, SOCKS5...) from standard +input and output (for servers) or file descripTor 6 and 7 (for clients). + +This permits to run it through [s6-networking][s6], [ucspi][u1], [ucspi-ssl][u2], +[curvecp][cp], or any protocol that has an UCSPI adapter for it. + +[u0]: https://cr.yp.to/proto/ucspi.txt +[s6]: https://skarnet.org/software/s6-networking/ +[u1]: https://github.com/younix/ucspi/ +[u2]: https://www.fehcom.de/ipnet/ucspi-ssl.html +[cp]: https://curvecp.org/ + + +State of IRC client<->server connexions +--------------------------------------- +Because it is run by different people and projects, the connexion to IRC +servers varies greatly through the different cases: + + * Some servers only accept TCP connexions. + * Some servers only accept TLS connexions. + * Some servers permit to use a client TLS certificate to authenticate. + * Some servers support connexion coming form [[Tor]], providing the extra + privacy that the IRC protocol lacks + * Some servers refuse connexions coming from Tor. + * Some are published as Tor hidden services directly: so no need for TLS. + * Some servers still propose TLS over Tor, with certificate authentication. + * Some servers use a self-signed certificate, and publish a fingerprint + of their certificate. + * Some servers used a private certificate *authority* and publish their + root certificate. + +From this plethora of security fine tuning, it is necessary to have an irc +client with a good TLS implementation (lots of lines of code), and a socks +proxy (more lines of code), with a configuration interface (many many lines of +code). + + +How UCSPI helps +--------------- +I use jj under the [[s6]] and [[s6-rc]] supervision tree on a VPS (until I get +real hardware home). + +jj having one instance per host to connect to, and jj supporting UCSPI, all of +these are of a great fit: + +[s6]: https://skarnet.org/software/s6/ +[s6-rc]: https://skarnet.org/software/s6-rc/ +[Tor]: https://www.Torproject.org/ + +By using ucspi, we are entirely avoiding the problemm, as we can compose a +socks client to talk to the Tor daemon (one line of scripts), and once the +connexion has started, it is possible to start a tlsclient program that uses +the active connexion and start a TLS session within that Tor socket, which can +be configured as well to use a client certificate. + +All of the UCSPI tools work by performing their startup work (opening a TCP +connexion, initiating a TLS/SOCKS/... session), and starting a child program, +which was pased as argument to it, with a pipe, or executing them directly. + +This gives more or less long chain, aka [[chainloading]], that [[djb]] used +and popularized through its set of programs. + +[chainloading]: https://en.wikipedia.org/wiki/Chain_loading +[djb]: https://cr.yp.to/djb.html + + +How I use jj and s6/s6-rc and UCSPI together +-------------------------------------------- +The s6 and s6-rc package come with an execline shell-like language that makes +this style of pipling as natural + +This is how it looks like in an [[execline]] ./run script in practice, as povided by s6 for +configuration of daemon (jj) startup scripts: + +Boilerplate I have everywhere (sorry sk., #!/usr/bin/env...): + + #!/usr/bin/env execlineb + fdmove -c 2 1 + +This command reads the content of each file in ./env/ and export environment +variables to their content: env/IRC_HOST, env/IRC_USER, env/IRC_PASSWORD, ... + + s6-envdir "env" + +I use the "irc" user and the /var/irc/ direcTory: + + s6-setuidgid irc + +This starts a TCP connexion to the local Tor daemon running: + + s6-tcpclient 127.0.0.1 9050 + +This starts a SOCK5 proxy session for communicating with the [[hackint]] + + sockc "5ogdsfyoqk47ompu.onion" "6667" + +At that point, the program called by sockc will be able to communicate +directly with IRC commands, as the TCP connexion is ready, the socks +session is set, and the raw IRC protocol follows. + +It is now possible to call jjd directly, which will recognize the UCSPI +environment through the $PROTO environment variable. + + jjd + +In case there is another context, situation, other UCSPI commands can be +inserted or moved, including the setup of client certifiates, that often goes +through environment variables. + +This is the same script while run as #!/bin/sh. Yes, this is a single command! + + #!/bin/sh + fdmove -c 2 1 \ + s6-envdir "env" \ + s6-setuidgid irc \ + s6-tcpclient 127.0.0.1 9050 \ + sockc "5ogdsfyoqk47ompu.onion" "6667" \ + jjd + + +Making jj useable interactively +------------------------------- +Being log-based, wrappers tools are necessary for making jj useable as a main +IRC client. + + +### [jj-in](/wiki/jj/jj-in) + +*similar to jji from the jj package* + +Take lines from the user stdin with a prompt set to the name of the channel. +A better line-editing tool could be used, such as rlwrap around this. + + +### [jj-log](/wiki/jj/jj-log) + +*similar to jjp in the jj package* + +Get a human-readable feed out of the jj log: +Filter the log to add colors. One way to use it is: tail -f chan.log | jj-log + + +### [retail](/wiki/jj/retail) + +*not in original jj package* + +Print the multiple files sorted as if tail -f was printing them since the +beginning, with the same "==> filename <==" header. + + +### [jj-tail](/wiki/jj/jj-tail) + +*not in original jj package* + +Instead of going with one window per chatroom and switching between chatrooms, +jj-tail gives a combined feed with the output of all channels in a same stream +showing a headers for each channel. + + == irc.freenode.net/##networking == + 00:12 ikityik: it's like having a bad fever dream with this ISP + == irc.freenode.net/#cat-v == + 00:12 cicl: when usb is enabled sounds like a vm? + 00:12 cicl: whats the context? + == irc.freenode.net/#openbsd == + 00:12 thrig: Logan Runners? + 00:12 knot: with blackjack and hookers + +To avoid being bothered with too many channels at once, it takes a list of +find(1) -path $filters as argument that will match the name of the +server/channels: + + $ jj-tail + <log of all channels of all servers ...> + ^C + $ jj-tail freenode + <log of all channels of freenode/* ...> + ^C + $ jj-tail qmail smtp + <log of freenode/#qmail freenode/#opensmtpd ...> + ^C + $ jj-tail server + <server logs of all channels follows (wich hilights) ...> + +This way, you can reduce the scope of the conversation to one server, similar +channels on multiple servers, just the selected channels... diff --git a/wiki/jj/jj-in b/wiki/jj/jj-in @@ -0,0 +1,20 @@ +#!/bin/sh -e +# input wrapper for jj + +path=${1:?} +chan=${path#*/} +host=${path%%/*} + +cd "/var/irc/$host" +[ -p "in" -a -w "in" ] + +prompt() { [ -t 0 ] && printf '\033[37m%s\033[0m> ' "$1" >&2 || :; } + +while prompt "$chan" && IFS= read -r line || [ -n "$line" ]; do + case "$line" in + /[!/]*) line=${line#?} ;; + '') continue ;; + *) line="msg $chan $line" ;; + esac + printf '%s\n' "$line" +done >"in" diff --git a/wiki/jj/jj-join b/wiki/jj/jj-join @@ -0,0 +1,14 @@ +#!/bin/sh -e +# re-join all channels for which there is a log + +IRC_HOST=${IRC_HOST:-$1} +IRC_HOST=${IRC_HOST:?} +IRC_DIR=${IRC_DIR:-/var/irc} + +cd "$IRC_DIR/$IRC_HOST" +[ -p "in" ] || exit 1 +for IRC_CHANNEL in "channels/#"*; do + IRC_CHANNEL=${IRC_CHANNEL##*/} + IRC_CHANNEL=${IRC_CHANNEL%.log} + echo "JOIN $IRC_CHANNEL" >"in" +done diff --git a/wiki/jj/jj-log b/wiki/jj/jj-log @@ -0,0 +1,156 @@ +#!/usr/bin/awk -f + +function isleap(year) +{ + return (year % 4 == 0) && (year % 100 != 0) || (year % 400 == 0) +} + +function mdays(mon, year) +{ + return (mon == 2) ? (28 + isleap(year)) : (30 + (mon + (mon > 7)) % 2) +} + +# Split the time in seconds since epoch into a table, with fields named +# as with gmtime(3): tm["year"], tm["mon"], tm["mday"], tm["hour"], +# tm["min"], tm["sec"]. + +function gmtime(sec, tm) +{ + tm["year"] = 1970 + while (sec >= (s = 86400 * (365 + isleap(tm["year"])))) { + tm["year"]++ + sec -= s + } + + tm["mon"] = 1 + while (sec >= (s = 86400 * mdays(tm["mon"], tm["year"]))) { + tm["mon"]++ + sec -= s + } + + tm["mday"] = 1 + while (sec >= (s = 86400)) { + tm["mday"]++ + sec -= s + } + + tm["hour"] = 0 + while (sec >= 3600) { + tm["hour"]++ + sec -= 3600 + } + + tm["min"] = 0 + while (sec >= 60) { + tm["min"]++ + sec -= 60 + } + + tm["sec"] = sec +} + +function localtime(sec, tm, + tz, h, m) +{ + if (!TZOFFSET) { + "date +%z" | getline tz + close("date +%z") + h = substr(tz, 2, 2) + m = substr(tz, 4, 2) + TZOFFSET = substr(date, 1, 1) (h * 3600 + m * 60) + } + return gmtime(sec + TZOFFSET, tm) +} + +function parse_msg(s, msg, + nickend, text) +{ + nickend = index($2, ">") + localtime($1, TM) + msg["type"] = substr($2, nickend + 1) + msg["text"] = substr($0, index($0, ">") + 2 + length(msg["type"])) + msg["nick"] = substr($2, 2, nickend - 2) + msg["date"] = sprintf("%04d/%02d/%02d", TM["year"], TM["mon"], TM["mday"]) + msg["time"] = sprintf("%02d:%02d", TM["hour"], TM["min"]) +} + +function parse_bold(s, + tail) +{ + tail = s + s = "" + while (match(tail, "\002[^\002]*\002")) { + s = s substr(tail, 1, RSTART - 1) "\033[1m" + s = s substr(tail, RSTART + 1, RLENGTH - 2) "\033[m" + tail = substr(tail, RSTART + RLENGTH) + } + return s tail +} + +BEGIN { + FS = " " +} + +!NF { + next +} + +# Handle tail's "==> filename <==" headers. + +$1 == "==>" { + sub(".log <==", "") + sub("/channels/", " ") + sub("/server$", " server") + sub(".*/", "") + sub(" ", "/") + printf("\n== %s ==", $0) + fflush() + next +} + +{ + if (NF < 2) { + printf("\n%s", $0) + fflush() + next + } + + parse_msg(s, msg) + msg["text"] = parse_bold(msg["text"]) + + # Display date only when it actually changed. + if (LASTDATE != msg["date"]) { + LASTDATE = msg["date"] + printf("\n\n-- " msg["date"] " --") + } + + # Set the date color for highlights. + if (index(msg["type"], "!")) { + printf("\a") + dcolor = "33" + } else { + dcolor = "38;5;8" + } + + # Status message. + if (msg["nick"] == "-") { + printf("\n\033[%sm%s \033[%sm%s\033[0m", + dcolor, msg["time"], "38;5;8", msg["text"]) + fflush() + next + } + + ncolor = (index(msg["type"], "*") ? "37" : "31") + + # Handle ACTION. + if (index(msg["type"], "a")) { + msg["text"] = msg["nick"] " " msg["text"] + msg["nick"] = "*" + } + + printf("\n\033[%sm%s\033[0m \033[%sm%10s\033[0m: %s", + dcolor, msg["time"], ncolor, msg["nick"], msg["text"]) + fflush() + + delete msg +} diff --git a/wiki/jj/jj-tail b/wiki/jj/jj-tail @@ -0,0 +1,14 @@ +#!/bin/sh -e +# tail all jj channels in /var/irc + +# build a list of (-path "") to give to find(1) +[ "$#" = 0 ] && set -- "" +for x in "$@"; do + shift + set -- "$@" -o -path "*$x*.log" +done +shift # remove the leading -o + +# First print the history then print the live log +exec find "/var/irc" \( "$@" \) -exec retail {} + -exec tail -f -n 0 {} + \ + | exec jj-log diff --git a/wiki/jj/retail b/wiki/jj/retail @@ -0,0 +1,38 @@ +#!/usr/bin/awk -f +# print files as if tail -f was running since the beginning +# +# Non-POSIX tail(1) supports multiple files and printing a "==> filename <==" +# header. This gives a different result if you are running tail since 2 minutes +# with lines comming from the various files repeating the headers frequently, +# or if you re-run tail again on the files. +# +# This script sorts all lines from the different files, and print the header +# whenever the file being printed changes, which give the same output of a +# tail -f ... running since the beginning. +# +# Note that this requires the files'line to already be sorted, such as +# containing a timestamp. +# +# If you use a log with Apr Aug Dec Feb Jan Jun Jul Mar May Nov Sep instead of +# a real timestamp, you may convert the dates and pipe it to that script. + +BEGIN { + for (i = 1; i < ARGC; i++) + files[ARGV[i]] = "" + + for (f in files) + if ((getline files[f] <f) <= 0) + delete files[f] + + while (length(files) > 0) { + for (f in files) + if (!(min in files) || files[f] < files[min]) + min = f + if (last != min) + printf("\n==> %s <==\n", min) + last = min + print(files[min]) + if ((getline files[min] <min) <= 0) + delete files[min] + } +} diff --git a/wiki/qmail/destination-mx/index.md b/wiki/qmail/destination-mx/index.md @@ -0,0 +1,156 @@ +Choosing the destination MX +=========================== +*how does qmail choose its destination mail server?* + +In qmail-remote.c, a variable prefme is compared with the .pref field from each +item of an ipalloc struct (an array of struct { ip; pref; }). + + qmail-remote.c + ... + 333 static ipalloc ip = {0}; + ... + 396 for (i = 0;i < ip.len;++i) + 397 if (ipme_is(&ip.ix[i].ip)) + 398 if (ip.ix[i].pref < prefme) + 399 prefme = ip.ix[i].pref; + ... + +### What is .pref and where is it taking its data from? + +In dns.c, there is a function dns_mxip() (still with K&R declaration): + + dns.c + ... + 312 int dns_mxip(ia,sa,random) + 313 ipalloc *ia; + 314 stralloc *sa; + 315 unsigned long random; + 316 { + ... + +Inside, we have a call to findmx(), iterating on all the MX records found +from DNS. + + dns.c + ... + 350 while ((r = findmx(T_MX)) != 2) + 351 { + 352 if (r == DNS_SOFT) { alloc_free(mx); return DNS_SOFT; } + 353 if (r == 1) + ... + +The position, preference and IP of the MX record are passed through static +global variables to dns_mxip() (why not... no threads here). + + dns.c + ... + 25 static union { HEADER hdr; unsigned char buf[PACKETSZ]; } response; + 26 static int responselen; + 27 static unsigned char *responseend; + 28 static unsigned char *responsepos; + 29 + 30 static int numanswers; + 31 static char name[MAXDNAME]; + 32 static struct ip_address ip; + 33 unsigned short pref; + ... + +For each result, ia is filled with the IPs one by one through static int +dns_ipplus(ipalloc *ia, stralloc *sa, int pref): + + dns.c + ... + 369 while (nummx > 0) + 370 { + ... + 389 switch(dns_ipplus(ia,&mx[i].sa,mx[i].p)) + 390 { + 391 case DNS_MEM: case DNS_SOFT: + 392 flagsoft = 1; break; + 393 } + ... + 397 } + +And that is where qmail-remote gets its list of preferences: from DNS only, no +configuration impacting ip[].ix.pref at all. + +### What is the effect of comparing prefme with each ip[].ix.pref from DNS? + +Just below if (flagallaliases) prefme = 500000;, it chooses the first +ip.[].ix.pref ip that is lower than the prefme threshold. + + qmail-remote.c + ... + 404 for (i = 0;i < ip.len;++i) + 405 if (ip.ix[i].pref < prefme) + 406 break; + ... + +Just above if (flagallaliases) prefme = 500000;, the loop finds the lowest pref +value for IPs matching those of the server (SIOCGIFCONF) and store it into pref. + + qmail-remote.c + ... + 395 prefme = 100000; + 396 for (i = 0;i < ip.len;++i) + 397 if (ipme_is(&ip.ix[i].ip)) + 398 if (ip.ix[i].pref < prefme) + 399 prefme = ip.ix[i].pref; + ... + + ipme.c + ... + 42 struct ip_mx ix; + ... + 97 byte_copy(&ix.ip,4,&sin->sin_addr); + 98 if (ioctl(s,SIOCGIFFLAGS,x) == 0) + 99 if (ifr->ifr_flags & IFF_UP) + 100 if (!ipalloc_append(&ipme,&ix)) { close(s); return 0; } + ... + +That filters out all the MX entries that have a lower preference than qmail's +own IPs. That looks like having the effect of capturing the mail in case +qmail-remote has to send it out, but the server it send it to has the same IP +(in other words: send a mail to myself through IP). + +It is still possible to send a mail to another mail server among the MX +entries, even if there is an IP of the local interface in the pool of MX +responses : with a lower MX than the one that match our IPs. + +This looks like a safeguard against misconfiguration: a mail for a same IP as +one of which qmail listen on needs to be send through qmail-local, not through +qmail-remote! + + +### Why a high value for prefme in some cases? + + qmail-remote.c + ... + 401 if (relayhost) prefme = 300000; + 402 if (flagallaliases) prefme = 500000; + ... + +When the mail server needs to relay everything to somewhere, or for these +addrmangle cases, (special case) qmail-remote bypasses this mechanism: allow +all ips regardless of the context, and gonna do what it gotta do what it gotta +do: forward to the first of the IPs found. + + +### Why 500000, why not 5725 or 42? + + qmail-remote.c + ... + 395 prefme = 100000; + ... + 401 if (relayhost) prefme = 300000; + 402 if (flagallaliases) prefme = 500000; + ... + +Debugging purposes? [[rfc1035]] says DNS rr for MX preferences are unsigned 16 +bits, so it maxes out to 65536, and we cannot have a value so hight coming from +the DNS. + +A printf() after the two loop would show whether the mechanism was triggered or +not. + +[rfc1035]: https://tools.ietf.org/html/rfc1035#section-3.3.9 diff --git a/wiki/qmail/index.md b/wiki/qmail/index.md @@ -0,0 +1,23 @@ +(not)qmail implementation +========================= +*notes on [[qmail]] and [[notqmail]] implementation for self reference* + +[qmail]: https://cr.yp.to/qmail.html +[notqmail]: https://notqmail.org/ + +* [Choosing the destination MX][mx] + +[mx]: /wiki/qmail/destination-mx/ + +For documentation on how to use qmail, [life with qmail][lwq] is the practical +you are probably looking for. + +[lwq]: http://www.lifewithqmail.org/lwq.html + +Patching qmail: + +* Qmail Big DNS - [mailing list arcihve][p1] - the ANY lookup for a domain may + give a lot of different results, or not be supported at all by nameservers + (walking toward deprecation). Disabling the feature may help some deliveries. + +[p1]: https://lists.gt.net/qmail/users/138190 diff --git a/wiki/supervisor/index.md b/wiki/supervisor/index.md @@ -0,0 +1,74 @@ +Daemons, supervisors, and simplicity +==================================== + +When it comes to servers, daemon is a central concept, here is a proposed +definition: + +daemon - program that runs even after the user quit the shell + +A simple problem, a simple solution: make the process double-fork so the user +sitting at the terminal is free to keep going. pgrep and ps are your friends +now. + +This is one simple approach, but lead to that problem how to run the same +daemon twice? + +Knee-jerk reaction: "Why would one want to do that?" + + * Different DHCP pool of addresses on different interfaces, + * Different mail filtering rules for different inside/outside networks, + * Different vHosts for a mail / http / ... daemon, + * Different users running the same daemon + * Different agetty with one per TTY, + * ... + +The solution often encountered is handling the variety of roles from the inside +of the daemon instead of starting one dedicated daemon per role: with +configuration blocks that lets you handle each different roles in a different +way (per vhost, per network interface, per tty, per tcp port...). + +That makes each and every daemon much more complex. They all need a complex +configuration file parser and have a much more complex internal design. + +Solutions? + + * Writing the PID into a file is not good, as if a daemon gets killed while + getting out of memory, the PID file remains, and another daemon could get + the PID of its parent (SysV-style). + + * Having a ${daemon}ctl command that talks to the daemon through a socket + works, but then each daemon needs to support it while a simple signal handling + would solve all use cases (*BSD-style). + + * Matching the command line the daemon was started with with pgrep works but + it requires to adapt it to each daemon (also *BSD-style). + + * Use Linux-specific APIs to work around the issues lead by the diversity of + cases above (systemd). + + * Keep the daemon in the foreground (s6/runit/daemontools style). + +In the end, not causing the problem might be a decent solution: getting the +control back to the terminal was convenient for running programs from a shell, +but causes problems on other sides. +need to get the control back to the terminal after starting the program... + +Then use one process to launch all the others: A "supervisor" that starts the +daemons, and keeps each daemon as a child proces. The supervisor knows their +PID without pidfile or socket. That is how fork(2) works: it returns the PID +to the parent so that it can wait(2) for it, send it signals... + +How to organize daemons then? A trivial approach is to have one run script per +daemon to launch, that exec(2) into the daemon which stays at the foreground. + +Once the supervisor is is triggered, it can start each of these ./$daemon/run +scripts, keep them as child, and watch for more to come, with some +./$daemon/sock for listenning for commands such as "restart", "stop", "alarm" +to send signals to the daemon. + +The supervision system was complex to implement right and half baked inside of +each daemon with ${daemon}ctl, it is now done reliably once for all daemons in +a dedicated program: the supervisor. + +Running this/these extra processes does not consume much more memory (one +megabyte?) and makes each daemon smaller, compensating for the lost bytes. diff --git a/wiki/tinydns/Makefile b/wiki/tinydns/Makefile @@ -0,0 +1,14 @@ +all: data.cdb + +data = rr.soa rr.host rr.alias rr.mx rr.ns +data: data.awk ${data} + awk -f data.awk ${data} >$@ + +data.cdb: data + tinydns-data + +clean: + rm -f data data.cdb + +push: data.cdb rr.host rr.ns + awk -f push.awk rr.host rr.ns diff --git a/wiki/tinydns/data.awk b/wiki/tinydns/data.awk @@ -0,0 +1,59 @@ +#!/usr/bin/awk -f +# compose a tinydns-fromatted data file from multiple input files + +function ip6_hex(ip6, + hex, i, arr) +{ + sub("::", substr("::::::::", split(ip6, arr, ":") - 1), ip6) + split(ip6, arr, ":") + for (i = 1; i <= 8; i++) + hex = hex substr("0000" arr[i], length(arr[i]) + 1) + return hex +} + +/^$/ { next } + +FILENAME != "rr.soa" { + print(FNR > 1 ? "" : "\n# " FILENAME "\n") +} + +FILENAME == "rr.soa" { + domain[++i] = $1 +} + +FILENAME == "rr.host" { + for (i = 2; i <= NF; i++) { + if (index($i, ":") == 0) { + host4[$1] = $i + print("=" $1 "." domain[1] ":" host4[$1]) + } else { + host6[$1] = ip6_hex($i) + print("6" $1 "." domain[1] ":" host6[$1]) + } + } +} + +FILENAME == "rr.alias" { + for (i = 2; i <= NF; i++) { + if (host4[$1]) print("+" $i ":" host4[$1]) + if (host6[$1]) print("3" $i ":" host6[$1]) + } +} + +FILENAME == "rr.ns" { + if (host4[$1]) print("+" $2 "." domain[1] ":" host4[$1]) + if (host6[$1]) print("3" $2 "." domain[1] ":" host6[$1]) + for (i in domain) { + print("." domain[i] "::" $2 "." domain[1]) + } +} + +FILENAME == "rr.mx" { + for (i in domain) { + print("@" domain[i] "::" $1 "." domain[1]) + } +} + +END { + print("") +} diff --git a/wiki/tinydns/index.md b/wiki/tinydns/index.md @@ -0,0 +1,86 @@ +Configuration of [[tinydns]] +============================ + +[tinydns]: https://cr.yp.to/djbdns.html + +To run nameservers, you need to maintain a bunch of interdependent DNS +information. + +To achieve this, I write small input files that end up in data.cdb, read +by the tinydns program: + + (text) + rr.domain ┐ (text) (binary) + rr.host │ ┌────────┐ ┌────────────┐ + rr.mx ├─┤data.awk├─> data >─┤tinydns-edit├─> data.cdb + rr.ns │ └────────┘ └────────────┘ + rr.alias ┘ + +The workhorse here is [[data.awk]], and the whole chain is controlled by a +small [[Makefile]]. + +[data.awk]: /wiki/tinydns/data.awk +[makefile]: /wiki/tinydns/Makefile + + +/etc/tinydns/rr.domain +---------------------- +A list of top and second level domain names. The first one listed is the +"technical" domain name. + + z0.is + josuah.net + + +/etc/tinydns/rr.host +-------------------- +A list of "hostname without domain part", "IPv4", "IPv6". This is the only file +where IP are written. This is the book keeping of the servers you address. + +Each line of this file leads to a hostname.technical.dom record of type A, +AAAA, and PTR: + + kuntur 199.247.28.162 2001:19f0:5001:7ac::12 + harpyja 80.67.190.196 2a00:5884:8214::16 + + +/etc/tinydns/rr.mx +------------------ +A list of hostnames that run a mail server. Each domain in "rr.domain" gets one +MX record per mail server listed here, with the form: hostname.technical.dom as +generated by the "rr.host" list. + +We now have matching MX, A, AAAA and PTR entries. + + kuntur + + +/etc/tinydns/rr.ns +------------------ +A list of hostnames that run a DNS name server. Like for "rr.mx", each domain +in "rr.domain" gets one NS record per name server listed here. + + kuntur ns1 + harpyja ns2 + + +/etc/tinydns/rr.alias +--------------------- +A list of regular domain records for your various servers and services. You may + + harpyja www.josuah.net + harpyja git.josuah.net + kuntur josuah.z0.is + + +Uploading data.cdb to nameservers +--------------------------------- +"make push" update all necessary parts to build data.cdb as seen above, then +calls [[push.awk]]. + +[push.awk]: /wiki/tinydns/push.awk + +push.awk copies data.cdb to each nameserver in rr.ns, using the IP found in +rr.host (so even if your DNS is down, you can still update it from remote). +This example sends about 10KB over SSH in total, about 1.50 second over a DSL +line for 2 nameservers. diff --git a/wiki/tinydns/push.awk b/wiki/tinydns/push.awk @@ -0,0 +1,9 @@ +#!/usr/bin/awk -f + +FILENAME == "rr.host" { + ip[$1] = $2 +} + +FILENAME == "rr.ns" { + system("scp 'data.cdb' 'root@"ip[$1]":/etc/tinydns/data.cdb'") +}