site-josuah

/usr/josuah
Log | Files | Refs

commit 67e24190273cd722b6f366525c601e0adf5c8572
parent ab7428e704f044145365197aae3ffb3ab6f6ee15
Author: Josuah Demangeon <me@josuah.net>
Date:   Wed, 27 May 2020 19:28:10 +0200

move the wiki to a separate repo

Diffstat:
M.githooks/post-update | 2--
Mindex.md | 12++++++------
Rwiki/quotes/index.md -> quotes/index.md | 0
Dwiki/awk/index.md | 350-------------------------------------------------------------------------------
Dwiki/git-hooks/index.md | 89-------------------------------------------------------------------------------
Dwiki/jj/index.md | 254-------------------------------------------------------------------------------
Dwiki/jj/jj-in | 20--------------------
Dwiki/jj/jj-join | 14--------------
Dwiki/jj/jj-log | 156-------------------------------------------------------------------------------
Dwiki/jj/jj-tail | 14--------------
Dwiki/jj/retail | 38--------------------------------------
Dwiki/qmail/destination-mx/index.md | 156-------------------------------------------------------------------------------
Dwiki/qmail/index.md | 23-----------------------
Dwiki/supervisor/index.md | 74--------------------------------------------------------------------------
Dwiki/tinydns/Makefile | 14--------------
Dwiki/tinydns/data.awk | 59-----------------------------------------------------------
Dwiki/tinydns/index.md | 86-------------------------------------------------------------------------------
Dwiki/tinydns/push.awk | 9---------
18 files changed, 6 insertions(+), 1364 deletions(-)

diff --git a/.githooks/post-update b/.githooks/post-update @@ -9,8 +9,6 @@ notwiki-doc gph "/srv/gopher" * cp -r "ascii" "/srv/gopher/" cp -r "ascii" "/srv/www/htdocs/josuah" -cp -r "wiki" "/srv/gopher/" -cp -r "wiki" "/srv/www/htdocs/josuah" cp "style.css" "/srv/www/htdocs/josuah" notwiki-index-gph "ascii" >"/srv/gopher/ascii/index.gph" diff --git a/index.md b/index.md @@ -6,7 +6,7 @@ You can find documentation about my software projects: * My [[githooks]] scripts, that regenerate my NotWiki sites. [notwiki]: //code.z0.is/notwiki/ -[githooks]: /wiki/git-hooks/ +[githooks]: //code.z0.is/wiki/git-hooks/ As well as documentation on other people's software: @@ -15,15 +15,15 @@ As well as documentation on other people's software: * Use of [[tinydns]] with an awk scripts for generating dns ./data. * Notes on [[qmail]] implementation. -[awk]: /wiki/awk/ -[jj]: /wiki/jj/ -[tinydns]: /wiki/tinydns/ -[qmail]: /wiki/qmail/ +[awk]: //code.z0.is/wiki/awk/ +[jj]: //code.z0.is/wiki/jj/ +[tinydns]: //code.z0.is/wiki/tinydns/ +[qmail]: //code.z0.is/wiki/qmail/ And finally, [ASCII][aa] Art and [quotes][qu]. [aa]: /ascii/ -[qu]: /wiki/quotes/ +[qu]: /quotes/ You can access to this server through tor: diff --git a/wiki/quotes/index.md b/quotes/index.md diff --git a/wiki/awk/index.md b/wiki/awk/index.md @@ -1,350 +0,0 @@ - AWK -===== - -AWK is a surprising efficient language, for both [performance][perf] and code -efficiency. This comes with the ubiquitous array structure, and splitting the -input in fields by default. - -Not everything is parsed efficiently with AWK, Type-Length-Value for instance, -but many things are. I use it for multiple projects: - - * [[NotWiki]], featuring a (not)markdown parser that does two passes on - to easen-up the parsing, - - * [[ics2txt]], a basic iCal to TSV or plain text converter (two directions), - - * [[jj]] by aaronNGi, a daemon with an awk engine to project that turns raw - IRC protocol into easily readable split log files - -[perf]: https://adamdrake.com/command-line-tools-can-be-235x-faster-than-your-hadoop-cluster.html -[notwiki]: //code.z0.is/notwiki/ -[ics2txt]: gopher://bitreich.org/1/scm/ics2txt -[parser]: //code.z0.is/git/notwiki/files/ -[jj]: /wiki/jj/ - -Below are multiple ways of using awk for getting the best out of it. These are -partly by myself, partly collected from what I saw in the wild. - - -CSV fields with header ----------------------- -Instead of trying to remember the number of the column, using the name of the -column is much easier, and permit to have new columns inserted in the .csv file -without breaking the script. - - $ cat input.txt - domain_name,expiry_date,creation_date,owner,account_id - nowhere.com,2020-03,2019-05,me,23535 - perdu.com,2020-04,2018-03,you,23535 - pa.st,2020-09,2014-05,them,23535 - - $ awk ' - BEGIN { FS = "," } - NR == 1 { for (i = 1; i <= NF; i++) F[$i] = i; next } - $F["domain_name"] ~ /\.com$/ { - print $F["expiry_date"], $F["owner"], $F["domain_name"] - } - ' input.txt - 2020-03 me nowhere.com - 2020-04 you perdu.com - - -UCL-style configuration ------------------------ -Parsing data that is not organised with line-column is also convenient and -efficient with awk, convenient for selecting one kind of value out of a -configuration file: - - $ cat input.txt - connections { - conn-faraway { - children { - localnet = fe80:123d:35d3::%vio1/64 - localnet = fe80:2e46:1d23::%vio2/64 - } - children { - localnet = fe80:546:23e4::%vio3/64 - } - } - conn-veryclose { - children { - localnet = fe80:b536:243f::%vio3/64 - localnet = fe80:34f3:23c3::%vio3/64 - localnet = fe80:546a:343d::%vio3/64 - } - } - } - - $ awk ' - $2 == "{" { F[lv++] = $1 } - $1 == "}" { delete F[--lv] } - F[0] == "connections" && F[2] == "children" && $1 == "localnet" { - print F[1], $3 - } - ' input.txt - conn-faraway fe80:123d:35d3::%vio1/64 - conn-faraway fe80:2e46:1d23::%vio2/64 - conn-faraway fe80:546:23e4::%vio3/64 - conn-veryclose fe80:b536:243f::%vio3/64 - conn-veryclose fe80:34f3:23c3::%vio3/64 - conn-veryclose fe80:546a:343d::%vio3/64 - - -Key-Value splitter ------------------- -Parsing key-value pairs can be mapped rather directly to an awk array, -for instance, to extract an abstract out of a basic iCal file: - - $ cat input.txt - BEGIN:VEVENT - METHOD:PUBLISH - UID:9189@FOSDEM20@fosdem.org - TZID:Europe-Brussels - DTSTART:20200201T170000 - DTEND:20200201T175000 - SUMMARY:State of the Onion - DESCRIPTION:Building usable free software to fight surveillance and censorship. - CLASS:PUBLIC - STATUS:CONFIRMED - CATEGORIES:Internet - LOCATION:Janson - END:VEVENT - $ awk ' - BEGIN { FS = ":" } - { F[$1] = $2 } - $1 == "END" { - print F["SUMMARY"] " - " F["DESCRIPTION"] - print F["DTSTART"], "(" F["TZID"] ")" - } - ' input.txt - State of the Onion - Building usable free software to fight surveillance and censorship. - 20200201T170000 (Europe-Brussels) - - -Edit variables passed to functions ----------------------------------- -For languages that support references, pointers, or objects, it is possible to -edit the variable passed to a function, so that the variable also gets edited -in the function that called it. - - void increment(int *i) { (*i)++; } - -Awk does not support changing integers or strings, but supports editing the -fields of an array: - - function increment_first(arr) { arr[i]++ } - - -Local variables in functions ----------------------------- -By default, all awk variables are global, which is inconvenient for writing -functions. The solution is to add an extra function argument at the end for -each local variable we need. - -Functions can be called with fewer arguments than they have. - - $ awk ' - function concat3(arg1, arg2, arg3, - local1) - { - local1 = arg1 arg2 arg3 - return local1 - } - - BEGIN { - local1 = 1 - print(concat3("a", "w", "k")) - print(local1) - } - ' - awk - 1 - -I learned this with the [jj] project. - -[jj]: https://github.com/aaronNGi/jj/ - - -A sort() function ------------------ -A very convenient feature lacking to awk is support for sorting members of an -array. Is possible to implement sort() in awk (this is a quicksort): - - function swap(array, a, b, - tmp) - { - tmp = array[a] - array[a] = array[b] - array[b] = tmp - } - - function sort(array, beg, end) - { - if (beg >= end) # end recursion - return - - a = beg + 1 # 1st is the pivot, so +1 - b = end - while (a < b) { - while (a < b && array[a] <= array[beg]) # beg: skip lesser - a++ - while (a < b && array[b] > array[beg]) # end: skip greater - b-- - swap(array, a, b) # found 2 misplaced - } - - if (array[beg] > array[a]) # put the pivot back - swap(array, beg, a) - - sort(array, beg, a - 1) # sort lower half - sort(array, a, end) # sort higher half - } - -This sorts the values of the array using integers keys: array[1], array[2], ... -It sorts element between array[beg] and array[end] included, so you can choose -array starting at 0, at 1, or sort just a part of the array. - -Example usage: with the both function above: - - { - LINES[NR] = $0 - } - - END { - sort(LINES, 1, NR) - for (i = 1; i <= NR; i++) - print(LINES[i]) - } - -Performance is far from terrible! - - $ od -An /dev/urandom | head -n 1000000 | time ./test.awk >/dev/null - real 0m 19.23s - user 0m 17.90s - sys 0m 0.12s - - $ od -An /dev/urandom | head -n 1000000 | time sort >/dev/null - real 0m 4.39s - user 0m 3.00s - sys 0m 0.10s - - -A gmtime() function -------------------- -POSIX awk as well as many implementations lack the [time functions][tf] present in -GNU awk. This gmtime() function split an epoch integer value (1587302158) into the -fields year, mon, mday, hour, min, sec (2020-04-19T15:15:58Z): - -[tf]: https://www.gnu.org/software/gawk/manual/html_node/Time-Functions.html - - function isleap(year) - { - return (year % 4 == 0) && (year % 100 != 0) || (year % 400 == 0) - } - - function mdays(mon, year) - { - return (mon == 2) ? (28 + isleap(year)) : (30 + (mon + (mon > 7)) % 2) - } - - function gmtime(sec, tm) - { - tm["year"] = 1970 - while (sec >= (s = 86400 * (365 + isleap(tm["year"])))) { - tm["year"]++ - sec -= s - } - - tm["mon"] = 1 - while (sec >= (s = 86400 * mdays(tm["mon"], tm["year"]))) { - tm["mon"]++ - sec -= s - } - - tm["mday"] = 1 - while (sec >= (s = 86400)) { - tm["mday"]++ - sec -= s - } - - tm["hour"] = 0 - while (sec >= 3600) { - tm["hour"]++ - sec -= 3600 - } - - tm["min"] = 0 - while (sec >= 60) { - tm["min"]++ - sec -= 60 - } - - tm["sec"] = sec - } - -The tm array will be filled with field names following the [[gmtime]] -function as you can see above. - -[gmtime]: https://pubs.opengroup.org/onlinepubs/9699919799/functions/gmtime.html - - -A localtime() function ----------------------- -For printing functions in the user's favorite timezone, gmtime's time needs -to be shifted. This can also be done in standard awk by calling the date(1) -command: - - function localtime(sec, tm, - tz, h, m) - { - if (!TZOFFSET) { - "date +%z" | getline tz - close("date +%z") - h = substr(tz, 2, 2) - m = substr(tz, 4, 2) - TZOFFSET = substr(date, 1, 1) (h * 3600 + m * 60) - } - return gmtime(sec + TZOFFSET, tm) - } - -Note that date(1) will only be called the first time localtime() is called, and -the TZOFFSET global variable will be used for the next calls. - - -A mktime() function -------------------- -Complementary function to gmtime is mktime for converting a tm[] array back to -an integer representation. This is useful for parsing time values back to an -unix timestamp: - - function isleap(year) - { - return (year % 4 == 0) && (year % 100 != 0) || (year % 400 == 0) - } - - function mdays(mon, year) - { - return (mon == 2) ? (28 + isleap(year)) : (30 + (mon + (mon > 7)) % 2) - } - - function mktime(tm, - sec, mon, day) - { - sec = tm["sec"] + tm["min"] * 60 + tm["hour"] * 3600 - - day = tm["mday"] - 1 - - for (mon = tm["mon"] - 1; mon > 0; mon--) - day = day + mdays(mon, tm["year"]) - - # constants: x * 365 + x / 400 - x / 100 + x / 4 - day = day + int(tm["year"] / 400) * 146097 - day = day + int(tm["year"] % 400 / 100) * 36524 - day = day + int(tm["year"] % 100 / 4) * 1461 - day = day + int(tm["year"] % 4 / 1) * 365 - - return sec + (day - 719527) * 86400 - } - -All the following fields of tm[] must be defined: "year", "mon", "mday", -"hour", "min", "sec". diff --git a/wiki/git-hooks/index.md b/wiki/git-hooks/index.md @@ -1,89 +0,0 @@ -Minimal Git Hooks configuration -=============================== - -Git [[hooks]] permit to run commands on a range of git events, mainly: a git -commit. - -I use 3 shell scripts of roughly 10 lines each to configure git hooks, calling -scripts from within the repository itself: .githooks/<hookname> - -[hooks]: https://githooks.com/ - -git-hooks-run -------------- -This is what runs on every event, to put on /bare-repo.git/hook/<hookname>. -There is no point in running it by hand. - - #!/bin/sh -e - hookname=$1 ref=${2:-master} - - echo "${0##*/}: running '$1' on '$ref'" - git cat-file blob "$ref:.git$hookname" | { - IFS='! ' read -r _ cmd args - exec "$cmd" "$args" "/dev/stdin" "$ref" - } - -It checks if there is a file called .githooks/<hookname> (git ls-tree "$ref" -...), and if so, extract this file from git (git cat-file blob ...), read the -shebang, and execute the rest with the command of the shebang ("| { ... }"). - - -git-hooks-install ------------------ -This setups the command above for a bare git repository: - - #!/bin/sh -e - for x; do - echo "#!/usr/bin/env git-hooks-run" >"$x/hooks/post-upate" - chmod +x "$x/hooks/post-update" - done - -It replace selected hooks at repo.git/hooks/post-update with only this shebang: - - #!/usr/bin/env git-hooks-run - -This has the effect of calling the git-hooks-run from above with -hook/post-update as argument, along with the extra arguments providedd by git, -which is all we need for our hook. - - -git-hooks-workdir ------------------ -With only git-hooks-run, we lack a way to use the content of the repository -to interact with the hooks (it is not always needed to use the content). - -In case this is needed, this command extract the workdir of the commit pushed -into a new directory in /var/cache/git (that it delete in case of failure), -and print it out so that the hook script can use it: - - #!/bin/sh -e - ref=$1 - commit=$(git rev-parse "$ref") - workdir="/var/cache/git/$commit" - - mkdir -p "$workdir" - trap 'rm -rf "$workdir"' INT EXIT TERM HUP - git archive --prefix="$workdir/" --format="tar" "$ref" | (cd / && tar -xf -) - exec echo "$workdir" - -To use it from within the hook, to catch the workdir and make sure there is -no remaining file even in case of failure, thanks to the trap internal shell -command: - - #!/bin/sh -ex - tmp=$(git-hooks-workdir "$@") - trap 'rm -rf "$tmp"' INT TERM EXIT HUP - cd "$tmp" - -This might be the top of your hook script. - -The (optional) -x flags in the shebang, which will print every command as it is -executed. It will be printed on commiter side as "remote: $line". - -The (optional) -e flag is there to die if any command fails, such as the initial -cd to the "$tmp" directory. - -At that point, we will be in a workind directory containing the content of the -state at the commit pushed, and we can run commands, such as running unit test -(make test), send an email or anything we need. - diff --git a/wiki/jj/index.md b/wiki/jj/index.md @@ -1,254 +0,0 @@ - The jj irc client -=================== - -[[jj]] is an [Internet Relay Chat][irc] client based on the principles of -[[ii]], using FIFO (named pipe) files to read commands from the user, and -simply logs the output of each channels to plain files. - -[ii]: https://tools.suckless.org/ii/ -[jj]: https://github.com/aaronNGi/jj/ -[irc]: https://invidio.us/watch?v=R8FOGlnYkgg - -Instead of being an ncurse program that runs into tmux(1), jj works as an -applicative router. A router for ISO layer 7. You may already know "applicative -routers" for different protocols already: - - * SMTP: [OpenSMTPD][m1], [qmail]][m2], [Postfix][m3]... - * HTTP: [nginx][h1], [lighttpd][h2], [relayd][h3], [haproxy[h4]... - * SIP: [OpenSIPS][s1], [Kamailio][s2]... - * IRC: [ngircd][i1], [hybridircd][i2]... - -[m1]: https://www.opensmtpd.org/ -[m2]: https://cr.yp.to/qmail.html -[m3]: http://www.postfix.org/ -[h1]: https://nginx.org/ -[h2]: https://www.lighttpd.net/ -[h3]: https://bsd.plumbing/ -[h4]: https://haproxy.org/ -[s1]: https://opensips.org/ -[s2]: https://kamailio.org/w/ -[i1]: https://ngircd.barton.de/ -[i2]: http://ircd-hybrid.org/ - -IRC itself is much of a routing protocol server-side: routing messages to -the right client. Another aspect of applicative routing is client-side. - -Much like ii or sic, and all the modular bots in the wild, the jj acts as -a is a client-side IRC router through hooks and log files. - - -How jj works ------------- -As opposed to ii, which is plain C, jj uses an awk script to do the heavy -lifting of IRC parsing, called from a compiled binary coded in C, that only -does the fifo creation, and multiplexing of user and network input, and -write both to the awk script. - -It reads its entire configuration from environment variables, described on -the the project's README.md, such as $IRC_DIR, in which it create a direcTory -with the channel as a name. - -I set IRC_DIR to /var/irc, which gives us (full list on the README.md): - - * /var/irc/irc.freenode.net/channels/*.log - messages from users and channels. - * /var/irc/irc.freenode.net/in - the FIFO pipe to which write messages. - -There is one instance of jj per server conexion, which greatly simplifies -the software, makes debugging much easier, and permit to adapt and configure -it specifically for the requirements of each host to connect to. - -Recently, jj acquired the [[UCSPI]] connexion interface, which is a way to -write programs without handling the network protocols into the program itself, -but instead expect an running connexion (TCP, TLS, SSH, SOCKS5...) from standard -input and output (for servers) or file descripTor 6 and 7 (for clients). - -This permits to run it through [[s6-networking]], [[ucspi-y]], [[ucspi-ssl]], -[[curvecp]], or any protocol that has an UCSPI adapter for it. - -[s6-networking]: https://skarnet.org/software/s6-networking/ -[ucspi]: https://cr.yp.to/proto/ucspi.txt -[ucspi-y]: https://github.com/younix/ucspi/ -[ucspi-ssl]: https://www.fehcom.de/ipnet/ucspi-ssl.html -[curvecp]: https://curvecp.org/ - - -State of IRC client<->server connexions ---------------------------------------- -Because it is run by different people and projects, the connexion to IRC -servers varies greatly through the different cases: - - * Some servers only accept TCP connexions. - * Some servers only accept TLS connexions. - * Some servers permit to use a client TLS certificate to authenticate. - * Some servers support connexion coming form [[Tor]], providing the extra - privacy that the IRC protocol lacks - * Some servers refuse connexions coming from Tor. - * Some are published as Tor hidden services directly: so no need for TLS. - * Some servers still propose TLS over Tor, with certificate authentication. - * Some servers use a self-signed certificate, and publish a fingerprint - of their certificate. - * Some servers used a private certificate *authority* and publish their - root certificate. - -From this plethora of security fine tuning, it is necessary to have an irc -client with a good TLS implementation (lots of lines of code), and a socks -proxy (more lines of code), with a configuration interface (many many lines of -code). - - -How UCSPI helps ---------------- -I use jj under the [[s6]] and [[s6-rc]] supervision tree on a VPS (until I get -real hardware home). - -jj having one instance per host to connect to, and jj supporting UCSPI, all of -these are of a great fit: - -[s6]: https://skarnet.org/software/s6/ -[s6-rc]: https://skarnet.org/software/s6-rc/ -[Tor]: https://www.Torproject.org/ - -By using ucspi, we are entirely avoiding the problemm, as we can compose a -socks client to talk to the Tor daemon (one line of scripts), and once the -connexion has started, it is possible to start a tlsclient program that uses -the active connexion and start a TLS session within that Tor socket, which can -be configured as well to use a client certificate. - -All of the UCSPI tools work by performing their startup work (opening a TCP -connexion, initiating a TLS/SOCKS/... session), and starting a child program, -which was pased as argument to it, with a pipe, or executing them directly. - -This gives more or less long chain, aka [[chainloading]], that [[djb]] used -and popularized through its set of programs. - -[chainloading]: https://en.wikipedia.org/wiki/Chain_loading -[djb]: https://cr.yp.to/djb.html - - -How I use jj and s6/s6-rc and UCSPI together --------------------------------------------- -The s6 and s6-rc package come with an execline shell-like language that makes -this style of pipling as natural - -This is how it looks like in an [[execline]] ./run script in practice, as povided by s6 for -configuration of daemon (jj) startup scripts: - -Boilerplate I have everywhere (sorry sk., #!/usr/bin/env...): - - #!/usr/bin/env execlineb - fdmove -c 2 1 - -This command reads the content of each file in ./env/ and export environment -variables to their content: env/IRC_HOST, env/IRC_USER, env/IRC_PASSWORD, ... - - s6-envdir "env" - -I use the "irc" user and the /var/irc/ direcTory: - - s6-setuidgid irc - -This starts a TCP connexion to the local Tor daemon running: - - s6-tcpclient 127.0.0.1 9050 - -This starts a SOCK5 proxy session for communicating with the [[hackint]] - - sockc "5ogdsfyoqk47ompu.onion" "6667" - -At that point, the program called by sockc will be able to communicate -directly with IRC commands, as the TCP connexion is ready, the socks -session is set, and the raw IRC protocol follows. - -It is now possible to call jjd directly, which will recognize the UCSPI -environment through the $PROTO environment variable. - - jjd - -In case there is another context, situation, other UCSPI commands can be -inserted or moved, including the setup of client certifiates, that often goes -through environment variables. - -This is the same script while run as #!/bin/sh. Yes, this is a single command! - - #!/bin/sh - fdmove -c 2 1 \ - s6-envdir "env" \ - s6-setuidgid irc \ - s6-tcpclient 127.0.0.1 9050 \ - sockc "5ogdsfyoqk47ompu.onion" "6667" \ - jjd - - -Making jj useable interactively -------------------------------- -Being log-based, wrappers tools are necessary for making jj useable as a main -IRC client. - - -### [[jj-in]] - -*similar to jji from the jj package* - -Take lines from the user stdin with a prompt set to the name of the channel. -A better line-editing tool could be used, such as rlwrap around this. - -[jj-in]: /wiki/jj/jj-in - - -### [[jj-log]] - -*similar to jjp in the jj package* - -Get a human-readable feed out of the jj log: -Filter the log to add colors. One way to use it is: tail -f chan.log | jj-log - -[jj-log]: /wiki/jj/jj-log - - -### [[retail]] - -*not in original jj package* - -Print the multiple files sorted as if tail -f was printing them since the -beginning, with the same "==> filename <==" header. - -[retail]: /wiki/jj/retail - - -### [[jj-tail]] - -*not in original jj package* - -Instead of going with one window per chatroom and switching between chatrooms, -jj-tail gives a combined feed with the output of all channels in a same stream -showing a headers for each channel. - - == irc.freenode.net/##networking == - 00:12 ikityik: it's like having a bad fever dream with this ISP - == irc.freenode.net/#cat-v == - 00:12 cicl: when usb is enabled sounds like a vm? - 00:12 cicl: whats the context? - == irc.freenode.net/#openbsd == - 00:12 thrig: Logan Runners? - 00:12 knot: with blackjack and hookers - -To avoid being bothered with too many channels at once, it takes a list of -find(1) -path $filters as argument that will match the name of the -server/channels: - - $ jj-tail - <log of all channels of all servers ...> - ^C - $ jj-tail freenode - <log of all channels of freenode/* ...> - ^C - $ jj-tail qmail smtp - <log of freenode/#qmail freenode/#opensmtpd ...> - ^C - $ jj-tail server - <server logs of all channels follows (wich hilights) ...> - -This way, you can reduce the scope of the conversation to one server, similar -channels on multiple servers, just the selected channels... - -[jj-tail]: /wiki/jj/jj-tail diff --git a/wiki/jj/jj-in b/wiki/jj/jj-in @@ -1,20 +0,0 @@ -#!/bin/sh -e -# input wrapper for jj - -path=${1:?} -chan=${path#*/} -host=${path%%/*} - -cd "/var/irc/$host" -[ -p "in" -a -w "in" ] - -prompt() { [ -t 0 ] && printf '\033[37m%s\033[0m> ' "$1" >&2 || :; } - -while prompt "$chan" && IFS= read -r line || [ -n "$line" ]; do - case "$line" in - /[!/]*) line=${line#?} ;; - '') continue ;; - *) line="msg $chan $line" ;; - esac - printf '%s\n' "$line" -done >"in" diff --git a/wiki/jj/jj-join b/wiki/jj/jj-join @@ -1,14 +0,0 @@ -#!/bin/sh -e -# re-join all channels for which there is a log - -IRC_HOST=${IRC_HOST:-$1} -IRC_HOST=${IRC_HOST:?} -IRC_DIR=${IRC_DIR:-/var/irc} - -cd "$IRC_DIR/$IRC_HOST" -[ -p "in" ] || exit 1 -for IRC_CHANNEL in "channels/#"*; do - IRC_CHANNEL=${IRC_CHANNEL##*/} - IRC_CHANNEL=${IRC_CHANNEL%.log} - echo "JOIN $IRC_CHANNEL" >"in" -done diff --git a/wiki/jj/jj-log b/wiki/jj/jj-log @@ -1,156 +0,0 @@ -#!/usr/bin/awk -f - -function isleap(year) -{ - return (year % 4 == 0) && (year % 100 != 0) || (year % 400 == 0) -} - -function mdays(mon, year) -{ - return (mon == 2) ? (28 + isleap(year)) : (30 + (mon + (mon > 7)) % 2) -} - -# Split the time in seconds since epoch into a table, with fields named -# as with gmtime(3): tm["year"], tm["mon"], tm["mday"], tm["hour"], -# tm["min"], tm["sec"]. - -function gmtime(sec, tm) -{ - tm["year"] = 1970 - while (sec >= (s = 86400 * (365 + isleap(tm["year"])))) { - tm["year"]++ - sec -= s - } - - tm["mon"] = 1 - while (sec >= (s = 86400 * mdays(tm["mon"], tm["year"]))) { - tm["mon"]++ - sec -= s - } - - tm["mday"] = 1 - while (sec >= (s = 86400)) { - tm["mday"]++ - sec -= s - } - - tm["hour"] = 0 - while (sec >= 3600) { - tm["hour"]++ - sec -= 3600 - } - - tm["min"] = 0 - while (sec >= 60) { - tm["min"]++ - sec -= 60 - } - - tm["sec"] = sec -} - -function localtime(sec, tm, - tz, h, m) -{ - if (!TZOFFSET) { - "date +%z" | getline tz - close("date +%z") - h = substr(tz, 2, 2) - m = substr(tz, 4, 2) - TZOFFSET = substr(date, 1, 1) (h * 3600 + m * 60) - } - return gmtime(sec + TZOFFSET, tm) -} - -function parse_msg(s, msg, - nickend, text) -{ - nickend = index($2, ">") - localtime($1, TM) - msg["type"] = substr($2, nickend + 1) - msg["text"] = substr($0, index($0, ">") + 2 + length(msg["type"])) - msg["nick"] = substr($2, 2, nickend - 2) - msg["date"] = sprintf("%04d/%02d/%02d", TM["year"], TM["mon"], TM["mday"]) - msg["time"] = sprintf("%02d:%02d", TM["hour"], TM["min"]) -} - -function parse_bold(s, - tail) -{ - tail = s - s = "" - while (match(tail, "\002[^\002]*\002")) { - s = s substr(tail, 1, RSTART - 1) "\033[1m" - s = s substr(tail, RSTART + 1, RLENGTH - 2) "\033[m" - tail = substr(tail, RSTART + RLENGTH) - } - return s tail -} - -BEGIN { - FS = " " -} - -!NF { - next -} - -# Handle tail's "==> filename <==" headers. - -$1 == "==>" { - sub(".log <==", "") - sub("/channels/", " ") - sub("/server$", " server") - sub(".*/", "") - sub(" ", "/") - printf("\n== %s ==", $0) - fflush() - next -} - -{ - if (NF < 2) { - printf("\n%s", $0) - fflush() - next - } - - parse_msg(s, msg) - msg["text"] = parse_bold(msg["text"]) - - # Display date only when it actually changed. - if (LASTDATE != msg["date"]) { - LASTDATE = msg["date"] - printf("\n\n-- " msg["date"] " --") - } - - # Set the date color for highlights. - if (index(msg["type"], "!")) { - printf("\a") - dcolor = "33" - } else { - dcolor = "38;5;8" - } - - # Status message. - if (msg["nick"] == "-") { - printf("\n\033[%sm%s \033[%sm%s\033[0m", - dcolor, msg["time"], "38;5;8", msg["text"]) - fflush() - next - } - - ncolor = (index(msg["type"], "*") ? "37" : "31") - - # Handle ACTION. - if (index(msg["type"], "a")) { - msg["text"] = msg["nick"] " " msg["text"] - msg["nick"] = "*" - } - - printf("\n\033[%sm%s\033[0m \033[%sm%10s\033[0m: %s", - dcolor, msg["time"], ncolor, msg["nick"], msg["text"]) - fflush() - - delete msg -} diff --git a/wiki/jj/jj-tail b/wiki/jj/jj-tail @@ -1,14 +0,0 @@ -#!/bin/sh -e -# tail all jj channels in /var/irc - -# build a list of (-path "") to give to find(1) -[ "$#" = 0 ] && set -- "" -for x in "$@"; do - shift - set -- "$@" -o -path "*$x*.log" -done -shift # remove the leading -o - -# First print the history then print the live log -exec find "/var/irc" \( "$@" \) -exec retail {} + -exec tail -f -n 0 {} + \ - | exec jj-log diff --git a/wiki/jj/retail b/wiki/jj/retail @@ -1,38 +0,0 @@ -#!/usr/bin/awk -f -# print files as if tail -f was running since the beginning -# -# Non-POSIX tail(1) supports multiple files and printing a "==> filename <==" -# header. This gives a different result if you are running tail since 2 minutes -# with lines comming from the various files repeating the headers frequently, -# or if you re-run tail again on the files. -# -# This script sorts all lines from the different files, and print the header -# whenever the file being printed changes, which give the same output of a -# tail -f ... running since the beginning. -# -# Note that this requires the files'line to already be sorted, such as -# containing a timestamp. -# -# If you use a log with Apr Aug Dec Feb Jan Jun Jul Mar May Nov Sep instead of -# a real timestamp, you may convert the dates and pipe it to that script. - -BEGIN { - for (i = 1; i < ARGC; i++) - files[ARGV[i]] = "" - - for (f in files) - if ((getline files[f] <f) <= 0) - delete files[f] - - while (length(files) > 0) { - for (f in files) - if (!(min in files) || files[f] < files[min]) - min = f - if (last != min) - printf("\n==> %s <==\n", min) - last = min - print(files[min]) - if ((getline files[min] <min) <= 0) - delete files[min] - } -} diff --git a/wiki/qmail/destination-mx/index.md b/wiki/qmail/destination-mx/index.md @@ -1,156 +0,0 @@ -Choosing the destination MX -=========================== -*how does qmail choose its destination mail server?* - -In qmail-remote.c, a variable prefme is compared with the .pref field from each -item of an ipalloc struct (an array of struct { ip; pref; }). - - qmail-remote.c - ... - 333 static ipalloc ip = {0}; - ... - 396 for (i = 0;i < ip.len;++i) - 397 if (ipme_is(&ip.ix[i].ip)) - 398 if (ip.ix[i].pref < prefme) - 399 prefme = ip.ix[i].pref; - ... - -### What is .pref and where is it taking its data from? - -In dns.c, there is a function dns_mxip() (still with K&R declaration): - - dns.c - ... - 312 int dns_mxip(ia,sa,random) - 313 ipalloc *ia; - 314 stralloc *sa; - 315 unsigned long random; - 316 { - ... - -Inside, we have a call to findmx(), iterating on all the MX records found -from DNS. - - dns.c - ... - 350 while ((r = findmx(T_MX)) != 2) - 351 { - 352 if (r == DNS_SOFT) { alloc_free(mx); return DNS_SOFT; } - 353 if (r == 1) - ... - -The position, preference and IP of the MX record are passed through static -global variables to dns_mxip() (why not... no threads here). - - dns.c - ... - 25 static union { HEADER hdr; unsigned char buf[PACKETSZ]; } response; - 26 static int responselen; - 27 static unsigned char *responseend; - 28 static unsigned char *responsepos; - 29 - 30 static int numanswers; - 31 static char name[MAXDNAME]; - 32 static struct ip_address ip; - 33 unsigned short pref; - ... - -For each result, ia is filled with the IPs one by one through static int -dns_ipplus(ipalloc *ia, stralloc *sa, int pref): - - dns.c - ... - 369 while (nummx > 0) - 370 { - ... - 389 switch(dns_ipplus(ia,&mx[i].sa,mx[i].p)) - 390 { - 391 case DNS_MEM: case DNS_SOFT: - 392 flagsoft = 1; break; - 393 } - ... - 397 } - -And that is where qmail-remote gets its list of preferences: from DNS only, no -configuration impacting ip[].ix.pref at all. - -### What is the effect of comparing prefme with each ip[].ix.pref from DNS? - -Just below if (flagallaliases) prefme = 500000;, it chooses the first -ip.[].ix.pref ip that is lower than the prefme threshold. - - qmail-remote.c - ... - 404 for (i = 0;i < ip.len;++i) - 405 if (ip.ix[i].pref < prefme) - 406 break; - ... - -Just above if (flagallaliases) prefme = 500000;, the loop finds the lowest pref -value for IPs matching those of the server (SIOCGIFCONF) and store it into pref. - - qmail-remote.c - ... - 395 prefme = 100000; - 396 for (i = 0;i < ip.len;++i) - 397 if (ipme_is(&ip.ix[i].ip)) - 398 if (ip.ix[i].pref < prefme) - 399 prefme = ip.ix[i].pref; - ... - - ipme.c - ... - 42 struct ip_mx ix; - ... - 97 byte_copy(&ix.ip,4,&sin->sin_addr); - 98 if (ioctl(s,SIOCGIFFLAGS,x) == 0) - 99 if (ifr->ifr_flags & IFF_UP) - 100 if (!ipalloc_append(&ipme,&ix)) { close(s); return 0; } - ... - -That filters out all the MX entries that have a lower preference than qmail's -own IPs. That looks like having the effect of capturing the mail in case -qmail-remote has to send it out, but the server it send it to has the same IP -(in other words: send a mail to myself through IP). - -It is still possible to send a mail to another mail server among the MX -entries, even if there is an IP of the local interface in the pool of MX -responses : with a lower MX than the one that match our IPs. - -This looks like a safeguard against misconfiguration: a mail for a same IP as -one of which qmail listen on needs to be send through qmail-local, not through -qmail-remote! - - -### Why a high value for prefme in some cases? - - qmail-remote.c - ... - 401 if (relayhost) prefme = 300000; - 402 if (flagallaliases) prefme = 500000; - ... - -When the mail server needs to relay everything to somewhere, or for these -addrmangle cases, (special case) qmail-remote bypasses this mechanism: allow -all ips regardless of the context, and gonna do what it gotta do what it gotta -do: forward to the first of the IPs found. - - -### Why 500000, why not 5725 or 42? - - qmail-remote.c - ... - 395 prefme = 100000; - ... - 401 if (relayhost) prefme = 300000; - 402 if (flagallaliases) prefme = 500000; - ... - -Debugging purposes? [[rfc1035]] says DNS rr for MX preferences are unsigned 16 -bits, so it maxes out to 65536, and we cannot have a value so hight coming from -the DNS. - -A printf() after the two loop would show whether the mechanism was triggered or -not. - -[rfc1035]: https://tools.ietf.org/html/rfc1035#section-3.3.9 diff --git a/wiki/qmail/index.md b/wiki/qmail/index.md @@ -1,23 +0,0 @@ -(not)qmail implementation -========================= -*notes on [[qmail]] and [[notqmail]] implementation for self reference* - -[qmail]: https://cr.yp.to/qmail.html -[notqmail]: https://notqmail.org/ - -* [Choosing the destination MX][mx] - -[mx]: /wiki/qmail/destination-mx/ - -For documentation on how to use qmail, [life with qmail][lwq] is the practical -you are probably looking for. - -[lwq]: http://www.lifewithqmail.org/lwq.html - -Patching qmail: - -* Qmail Big DNS - [mailing list arcihve][p1] - the ANY lookup for a domain may - give a lot of different results, or not be supported at all by nameservers - (walking toward deprecation). Disabling the feature may help some deliveries. - -[p1]: https://lists.gt.net/qmail/users/138190 diff --git a/wiki/supervisor/index.md b/wiki/supervisor/index.md @@ -1,74 +0,0 @@ -Daemons, supervisors, and simplicity -==================================== - -When it comes to servers, daemon is a central concept, here is a proposed -definition: - -daemon - program that runs even after the user quit the shell - -A simple problem, a simple solution: make the process double-fork so the user -sitting at the terminal is free to keep going. pgrep and ps are your friends -now. - -This is one simple approach, but lead to that problem how to run the same -daemon twice? - -Knee-jerk reaction: "Why would one want to do that?" - - * Different DHCP pool of addresses on different interfaces, - * Different mail filtering rules for different inside/outside networks, - * Different vHosts for a mail / http / ... daemon, - * Different users running the same daemon - * Different agetty with one per TTY, - * ... - -The solution often encountered is handling the variety of roles from the inside -of the daemon instead of starting one dedicated daemon per role: with -configuration blocks that lets you handle each different roles in a different -way (per vhost, per network interface, per tty, per tcp port...). - -That makes each and every daemon much more complex. They all need a complex -configuration file parser and have a much more complex internal design. - -Solutions? - - * Writing the PID into a file is not good, as if a daemon gets killed while - getting out of memory, the PID file remains, and another daemon could get - the PID of its parent (SysV-style). - - * Having a ${daemon}ctl command that talks to the daemon through a socket - works, but then each daemon needs to support it while a simple signal handling - would solve all use cases (*BSD-style). - - * Matching the command line the daemon was started with with pgrep works but - it requires to adapt it to each daemon (also *BSD-style). - - * Use Linux-specific APIs to work around the issues lead by the diversity of - cases above (systemd). - - * Keep the daemon in the foreground (s6/runit/daemontools style). - -In the end, not causing the problem might be a decent solution: getting the -control back to the terminal was convenient for running programs from a shell, -but causes problems on other sides. -need to get the control back to the terminal after starting the program... - -Then use one process to launch all the others: A "supervisor" that starts the -daemons, and keeps each daemon as a child proces. The supervisor knows their -PID without pidfile or socket. That is how fork(2) works: it returns the PID -to the parent so that it can wait(2) for it, send it signals... - -How to organize daemons then? A trivial approach is to have one run script per -daemon to launch, that exec(2) into the daemon which stays at the foreground. - -Once the supervisor is is triggered, it can start each of these ./$daemon/run -scripts, keep them as child, and watch for more to come, with some -./$daemon/sock for listenning for commands such as "restart", "stop", "alarm" -to send signals to the daemon. - -The supervision system was complex to implement right and half baked inside of -each daemon with ${daemon}ctl, it is now done reliably once for all daemons in -a dedicated program: the supervisor. - -Running this/these extra processes does not consume much more memory (one -megabyte?) and makes each daemon smaller, compensating for the lost bytes. diff --git a/wiki/tinydns/Makefile b/wiki/tinydns/Makefile @@ -1,14 +0,0 @@ -all: data.cdb - -data = rr.soa rr.host rr.alias rr.mx rr.ns -data: data.awk ${data} - awk -f data.awk ${data} >$@ - -data.cdb: data - tinydns-data - -clean: - rm -f data data.cdb - -push: data.cdb rr.host rr.ns - awk -f push.awk rr.host rr.ns diff --git a/wiki/tinydns/data.awk b/wiki/tinydns/data.awk @@ -1,59 +0,0 @@ -#!/usr/bin/awk -f -# compose a tinydns-fromatted data file from multiple input files - -function ip6_hex(ip6, - hex, i, arr) -{ - sub("::", substr("::::::::", split(ip6, arr, ":") - 1), ip6) - split(ip6, arr, ":") - for (i = 1; i <= 8; i++) - hex = hex substr("0000" arr[i], length(arr[i]) + 1) - return hex -} - -/^$/ { next } - -FILENAME != "rr.soa" { - print(FNR > 1 ? "" : "\n# " FILENAME "\n") -} - -FILENAME == "rr.soa" { - domain[++i] = $1 -} - -FILENAME == "rr.host" { - for (i = 2; i <= NF; i++) { - if (index($i, ":") == 0) { - host4[$1] = $i - print("=" $1 "." domain[1] ":" host4[$1]) - } else { - host6[$1] = ip6_hex($i) - print("6" $1 "." domain[1] ":" host6[$1]) - } - } -} - -FILENAME == "rr.alias" { - for (i = 2; i <= NF; i++) { - if (host4[$1]) print("+" $i ":" host4[$1]) - if (host6[$1]) print("3" $i ":" host6[$1]) - } -} - -FILENAME == "rr.ns" { - if (host4[$1]) print("+" $2 "." domain[1] ":" host4[$1]) - if (host6[$1]) print("3" $2 "." domain[1] ":" host6[$1]) - for (i in domain) { - print("." domain[i] "::" $2 "." domain[1]) - } -} - -FILENAME == "rr.mx" { - for (i in domain) { - print("@" domain[i] "::" $1 "." domain[1]) - } -} - -END { - print("") -} diff --git a/wiki/tinydns/index.md b/wiki/tinydns/index.md @@ -1,86 +0,0 @@ -Configuration of [[tinydns]] -============================ - -[tinydns]: https://cr.yp.to/djbdns.html - -To run nameservers, you need to maintain a bunch of interdependent DNS -information. - -To achieve this, I write small input files that end up in data.cdb, read -by the tinydns program: - - (text) - rr.domain ┐ (text) (binary) - rr.host │ ┌────────┐ ┌────────────┐ - rr.mx ├─┤data.awk├─> data >─┤tinydns-edit├─> data.cdb - rr.ns │ └────────┘ └────────────┘ - rr.alias ┘ - -The workhorse here is [[data.awk]], and the whole chain is controlled by a -small [[Makefile]]. - -[data.awk]: /wiki/tinydns/data.awk -[makefile]: /wiki/tinydns/Makefile - - -/etc/tinydns/rr.domain ----------------------- -A list of top and second level domain names. The first one listed is the -"technical" domain name. - - z0.is - josuah.net - - -/etc/tinydns/rr.host --------------------- -A list of "hostname without domain part", "IPv4", "IPv6". This is the only file -where IP are written. This is the book keeping of the servers you address. - -Each line of this file leads to a hostname.technical.dom record of type A, -AAAA, and PTR: - - kuntur 199.247.28.162 2001:19f0:5001:7ac::12 - harpyja 80.67.190.196 2a00:5884:8214::16 - - -/etc/tinydns/rr.mx ------------------- -A list of hostnames that run a mail server. Each domain in "rr.domain" gets one -MX record per mail server listed here, with the form: hostname.technical.dom as -generated by the "rr.host" list. - -We now have matching MX, A, AAAA and PTR entries. - - kuntur - - -/etc/tinydns/rr.ns ------------------- -A list of hostnames that run a DNS name server. Like for "rr.mx", each domain -in "rr.domain" gets one NS record per name server listed here. - - kuntur ns1 - harpyja ns2 - - -/etc/tinydns/rr.alias ---------------------- -A list of regular domain records for your various servers and services. You may - - harpyja www.josuah.net - harpyja git.josuah.net - kuntur josuah.z0.is - - -Uploading data.cdb to nameservers ---------------------------------- -"make push" update all necessary parts to build data.cdb as seen above, then -calls [[push.awk]]. - -[push.awk]: /wiki/tinydns/push.awk - -push.awk copies data.cdb to each nameserver in rr.ns, using the IP found in -rr.host (so even if your DNS is down, you can still update it from remote). -This example sends about 10KB over SSH in total, about 1.50 second over a DSL -line for 2 nameservers. diff --git a/wiki/tinydns/push.awk b/wiki/tinydns/push.awk @@ -1,9 +0,0 @@ -#!/usr/bin/awk -f - -FILENAME == "rr.host" { - ip[$1] = $2 -} - -FILENAME == "rr.ns" { - system("scp 'data.cdb' 'root@"ip[$1]":/etc/tinydns/data.cdb'") -}