site-josuah

/usr/josuah
Log | Files | Refs

commit 1a5d510bc5330ef175987c5049a8756110e5b4c7
parent 829c20b8d7a6a2601c7d4745a14cabe8c91cebd7
Author: Josuah Demangeon <me@josuah.net>
Date:   Tue,  6 Oct 2020 13:35:13 +0200

flatten out

Diffstat:
Aawk/index.md | 435+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Rwiki/c-programming/index.md -> c-programming/index.md | 0
Rwiki/git-hooks/index.md -> git-hooks/index.md | 0
Mindex.md | 8++++----
Ajj/index.md | 258+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Rwiki/jj/jj-in -> jj/jj-in | 0
Rwiki/jj/jj-join -> jj/jj-join | 0
Rwiki/jj/jj-log -> jj/jj-log | 0
Rwiki/jj/jj-tail -> jj/jj-tail | 0
Rwiki/jj/retail -> jj/retail | 0
Mlinks/index.md | 2+-
Rwiki/qmail/destination-mx/index.md -> qmail/destination-mx/index.md | 0
Aqmail/index.md | 23+++++++++++++++++++++++
Mquotes/index.md | 4++--
Rwiki/supervisor/index.md -> supervisor/index.md | 0
Rwiki/tinydns/Makefile -> tinydns/Makefile | 0
Rwiki/tinydns/data.awk -> tinydns/data.awk | 0
Atinydns/index.md | 89+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Rwiki/tinydns/push.awk -> tinydns/push.awk | 0
Dwiki/awk/index.md | 434-------------------------------------------------------------------------------
Dwiki/jj/index.md | 258-------------------------------------------------------------------------------
Dwiki/qmail/index.md | 23-----------------------
Dwiki/tinydns/index.md | 89-------------------------------------------------------------------------------
23 files changed, 812 insertions(+), 811 deletions(-)

diff --git a/awk/index.md b/awk/index.md @@ -0,0 +1,435 @@ +AWK +=== +AWK is a surprising efficient language, for both [performance][perf] and code +efficiency. This comes with the ubiquitous array structure, and splitting the +input in fields by default. + +Not everything is parsed efficiently with AWK, Type-Length-Value for instance, +but many things are. I use it for multiple projects: + +* [[NotWiki]], featuring a (not)markdown parser that does two passes on + to easen-up the parsing, +* [ics2txt](//code.z0.is/ics2txt/), a iCalendar to other format converter. +* [jj](/~josuah/jj/) by aaronNGi, a daemon with an awk engine to project + that turns raw IRC protocol into *very* easy to parse split log files. + +[perf]: https://adamdrake.com/command-line-tools-can-be-235x-faster-than-your-hadoop-cluster.html +[notwiki]: /~josuah/notwiki/ +[ics2txt]: gopher://bitreich.org/1/scm/ics2txt +[parser]: /notwiki/notwiki-index-html + +Below are multiple ways of using awk for getting the best out of it. These are +partly by myself, partly collected from what I saw in the wild. + +CSV fields with header +---------------------- +Instead of trying to remember the number of the column, using the name of the +column is much easier, and permit to have new columns inserted in the .csv file +without breaking the script. + +``` +$ cat input.txt +domain_name,expiry_date,creation_date,owner,account_id +nowhere.com,2020-03,2019-05,me,23535 +perdu.com,2020-04,2018-03,you,23535 +pa.st,2020-09,2014-05,them,23535 + +$ awk ' + BEGIN { FS = "," } + NR == 1 { for (i = 1; i <= NF; i++) F[$i] = i; next } + $F["domain_name"] ~ /\.com$/ { + print $F["expiry_date"], $F["owner"], $F["domain_name"] + } +' input.txt +2020-03 me nowhere.com +2020-04 you perdu.com +``` + +UCL-style configuration +----------------------- +Parsing data that is not organised with line-column is also convenient and +efficient with awk, convenient for selecting one kind of value out of a +configuration file: + +``` +$ cat input.txt +connections { + conn-faraway { + children { + localnet = fe80:123d:35d3::%vio1/64 + localnet = fe80:2e46:1d23::%vio2/64 + } + children { + localnet = fe80:546:23e4::%vio3/64 + } + } + conn-veryclose { + children { + localnet = fe80:b536:243f::%vio3/64 + localnet = fe80:34f3:23c3::%vio3/64 + localnet = fe80:546a:343d::%vio3/64 + } + } +} +``` + +``` +$ awk ' + $2 == "{" { F[lv++] = $1 } + $1 == "}" { delete F[--lv] } + F[0] == "connections" && F[2] == "children" && $1 == "localnet" { + print F[1], $3 + } +' input.txt +conn-faraway fe80:123d:35d3::%vio1/64 +conn-faraway fe80:2e46:1d23::%vio2/64 +conn-faraway fe80:546:23e4::%vio3/64 +conn-veryclose fe80:b536:243f::%vio3/64 +conn-veryclose fe80:34f3:23c3::%vio3/64 +conn-veryclose fe80:546a:343d::%vio3/64 +``` + +Key-Value splitter +------------------ +Parsing key-value pairs can be mapped rather directly to an awk array, +for instance, to extract an abstract out of a basic iCal file: + +``` +$ cat input.txt +BEGIN:VEVENT +METHOD:PUBLISH +UID:9189@FOSDEM20@fosdem.org +TZID:Europe-Brussels +DTSTART:20200201T170000 +DTEND:20200201T175000 +SUMMARY:State of the Onion +DESCRIPTION:Building usable free software to fight surveillance and censorship. +CLASS:PUBLIC +STATUS:CONFIRMED +CATEGORIES:Internet +LOCATION:Janson +END:VEVENT +$ awk ' + BEGIN { FS = ":" } + { F[$1] = $2 } + $1 == "END" { + print F["SUMMARY"] " - " F["DESCRIPTION"] + print F["DTSTART"], "(" F["TZID"] ")" + } +' input.txt +State of the Onion - Building usable free software to fight surveillance and censorship. +20200201T170000 (Europe-Brussels) +``` + +Edit variables passed to functions +---------------------------------- +For languages that support references, pointers, or objects, it is possible to +edit the variable passed to a function, so that the variable also gets edited +in the function that called it. + +``` +void increment(int *i) { (*i)++; } +``` + +Awk does not support changing integers or strings, but supports editing the +fields of an array: + +``` +function increment_first(arr) { arr[1]++ } +``` + +Local variables in functions +---------------------------- +By default, all awk variables are global, which is inconvenient for writing +functions. The solution is to add an extra function argument at the end for +each local variable we need. + +Functions can be called with fewer arguments than they have. + +``` +$ awk ' + function concat3(arg1, arg2, arg3, + local1) + { + local1 = arg1 arg2 arg3 + return local1 + } + + BEGIN { + local1 = 1 + print(concat3("a", "w", "k")) + print(local1) + } +' +awk +1 +``` + +I learned this with the [jj](https://github.com/aaronNGi/jj/) project. + +A sort() function +----------------- +A very convenient feature lacking to awk is support for sorting members of an +array. Is possible to implement sort() in awk (this is a quicksort): + +``` +function swap(array, a, b, + tmp) +{ + tmp = array[a] + array[a] = array[b] + array[b] = tmp +} + +function sort(array, beg, end) +{ + if (beg >= end) # end recursion + return + a = beg + 1 # 1st is the pivot, so +1 + b = end + while (a < b) { + while (a < b && array[a] <= array[beg]) # beg: skip lesser + a++ + while (a < b && array[b] > array[beg]) # end: skip greater + b-- + swap(array, a, b) # found 2 misplaced + } + if (array[beg] > array[a]) # put the pivot back + swap(array, beg, a) + sort(array, beg, a - 1) # sort lower half + sort(array, a, end) # sort higher half +} +``` + +This sorts the array values using integers keys: `array[1]`, `array[2]`, ... +It sorts from `array[beg]` to `array[end]` included, so you can choose your +array indices starting at 0 or 1, or sort just a part of the array. + +Example usage: with the both function above: + +``` +{ + LINES[NR] = $0 +} + +END { + sort(LINES, 1, NR) + for (i = 1; i <= NR; i++) + print(LINES[i]) +} +``` + +Performance is far from terrible! + +``` +$ od -An /dev/urandom | head -n 1000000 | time ./test.awk >/dev/null +real 0m 19.23s +user 0m 17.90s +sys 0m 0.12s + +$ od -An /dev/urandom | head -n 1000000 | time sort >/dev/null +real 0m 4.39s +user 0m 3.00s +sys 0m 0.10s +``` + +Fill a static array +------------------- +With C and many other languages, there are convenient concise syntax such as +`{ "a", "b", "c", ...}`to fill an array with values. An well-known way is: + +``` +split("a b c ...", array, " ") +``` + +Note that this does not saves the length out of split, but in practice I +realized I rarely need it: + +``` +for (i = 1; i in split; i++) + print(i, array[i]) +``` + +A fold_line() function +----------------------- +Convenient to work with text documents or emails. This version does not +truncate very long words, such as some https:// links. + +``` +function fold_line(str, len, + head, tail, i) +{ + head = substr(str, 1, len + 1) + sub(" *$", "", head) + if (length(head) == len + 1) + sub(" *[^ ]*$", "", head) + if (length(head) == 0) { + tail = substr(str, len + 1) + head = substr(str, 1, len) + if ((i = index(tail, " ")) == 0) + return str + return head substr(tail, 1, i) + } + return head +} +``` + +To use it, call it in a while loop like this: + +``` +{ + while (line = fold_line($0, 72)) { + print line + $0 = substr($0, length(line) + 2) + } +} +``` + +A gmtime() function +------------------- +POSIX awk as well as many implementations lack the [time functions][tf] present in +GNU awk. This gmtime() function split an epoch integer value (1587302158) into the +fields year, mon, mday, hour, min, sec (2020-04-19T15:15:58Z): + +[tf]: https://www.gnu.org/software/gawk/manual/html_node/Time-Functions.html + +``` +function isleap(year) +{ + return (year % 4 == 0) && (year % 100 != 0) || (year % 400 == 0) +} + +function mdays(mon, year) +{ + return (mon == 2) ? (28 + isleap(year)) : (30 + (mon + (mon > 7)) % 2) +} + +function gmtime(sec, tm) +{ + tm["year"] = 1970 + while (sec >= (s = 86400 * (365 + isleap(tm["year"])))) { + tm["year"]++ + sec -= s + } + tm["mon"] = 1 + while (sec >= (s = 86400 * mdays(tm["mon"], tm["year"]))) { + tm["mon"]++ + sec -= s + } + tm["mday"] = 1 + while (sec >= (s = 86400)) { + tm["mday"]++ + sec -= s + } + tm["hour"] = 0 + while (sec >= 3600) { + tm["hour"]++ + sec -= 3600 + } + tm["min"] = 0 + while (sec >= 60) { + tm["min"]++ + sec -= 60 + } + tm["sec"] = sec +} +``` + +The tm array will be filled with field names following the [[gmtime]] +function as you can see above. + +[gmtime]: https://pubs.opengroup.org/onlinepubs/9699919799/functions/gmtime.html + +A localtime() function +---------------------- +For printing functions in the user's favorite timezone, gmtime's time needs +to be shifted. This can also be done in standard awk by calling the date(1) +command: + +``` +function localtime(sec, tm, + tz, h, m) +{ + if (!TZ) { + "date +%z" | getline tz + close("date +%z") + h = substr(tz, 2, 2) + m = substr(tz, 4, 2) + TZ = substr(tz, 1, 1) (h * 3600 + m * 60) + } + return gmtime(sec + TZ, tm) +} +``` + +Note that date(1) will only be called the first time localtime() is called, and +the TZ global variable will be used for the next calls. + +A timegm() function +------------------- +Complementary function to gmtime is timegm for converting a `tm[]` array back +to an integer representation. This is useful for parsing time values back to +an unix timestamp: + +``` +function isleap(year) +{ + return (year % 4 == 0) && (year % 100 != 0) || (year % 400 == 0) +} + +function mdays(mon, year) +{ + return (mon == 2) ? (28 + isleap(year)) : (30 + (mon + (mon > 7)) % 2) +} + +function timegm(tm, + sec, mon, day) +{ + sec = tm["sec"] + tm["min"] * 60 + tm["hour"] * 3600 + day = tm["mday"] - 1 + for (mon = tm["mon"] - 1; mon > 0; mon--) + day = day + mdays(mon, tm["year"]) + day = day + int(tm["year"] / 400) * 146097 + day = day + int(tm["year"] % 400 / 100) * 36524 + day = day + int(tm["year"] % 100 / 4) * 1461 + day = day + int(tm["year"] % 4 / 1) * 365 + return sec + (day - 719527) * 86400 +} +``` + +All the following fields of `tm[]` must be defined: "year", "mon", "mday", +"hour", "min", "sec". + +Convert MAC address to brand name +--------------------------------- +[MAC addresses](https://en.wikipedia.org/MAC_address) are composed by a +leading Organization Unique Identifier (OUI) of 3 bytes and a trailing 3-byte +number, unique for that OUI. + +Each vendor has its own OUI, so each OUI maps to a vendor. With the reference +list the [IEEE](https://ieee.org/) publishes, it is possible to convert MAC +address OUI digits to a human-readable name: + +``` +function oui_table(path, + url) +{ + url = "http://standards-oui.ieee.org/oui/oui.txt" + if (system("test -f '" path "'") > 0) + if (system("curl -L -o '" path "' " url) != 0) + return -1 + while (getline <path) + if ($2 " " $3 == "(base 16)") + OUI[$1] = substr($0, 22) + return 0 +} +``` + +Then a global `OUI` array does the MAC addresss to vendor name mapping: + +``` +BEGIN { + if (oui_table("/var/tmp/oui.txt") < 0) + exit(1) + print(OUI[toupper("84a991")]) +} +``` diff --git a/wiki/c-programming/index.md b/c-programming/index.md diff --git a/wiki/git-hooks/index.md b/git-hooks/index.md diff --git a/index.md b/index.md @@ -9,13 +9,13 @@ You can find documentation about my software projects: * The [passlock](/passlock/) tool, an authentication backend. * The [netini](/netini/) toolbox, to plot networks. * The [ics2txt](/ics2txt/) tool, to deal with the iCalendar format. -* My [githooks](/~josuah/wiki/git-hooks/) scripts, that regenerate my NotWiki sites. +* My [githooks](/~josuah/git-hooks/) scripts, that regenerate my NotWiki sites. As well as documentation on other people's software: -* Use of [awk](/~josuah/wiki/awk/), the command/language. -* Use of [jj](/~josuah/wiki/jj/) IRC client with UCSPI and s6. -* Use of [tinydns](/~josuah/wiki/tinydns/) with an awk scripts for generating dns +* Use of [awk](/~josuah/awk/), the command/language. +* Use of [jj](/~josuah/jj/) IRC client with UCSPI and s6. +* Use of [tinydns](/~josuah/tinydns/) with an awk scripts for generating dns ./data. * Notes on [notqmail](https://notqmail.z0.is/) implementation. diff --git a/jj/index.md b/jj/index.md @@ -0,0 +1,258 @@ +The jj IRC client +================== +[jj](https://github.com/aaronNGi/jj/) is an [Internet Relay Chat +(IRC)](https://invidio.us/watch?v=R8FOGlnYkgg) client based on the principles +of [ii](https://tools.suckless.org/ii/), using FIFO (named pipe) files to read +commands from the user, and simply logs the output of each channels to plain +files. + +Instead of being an ncurse program that runs into tmux(1), jj works as an +applicative router. A router for ISO layer 7. You may already know "applicative +routers" for different protocols already: + +* SMTP: [OpenSMTPD][m1], [qmail][m2], [Postfix][m3]... +* HTTP: [nginx][h1], [lighttpd][h2], [relayd][h3], [haproxy][h4]... +* SIP: [OpenSIPS][s1], [Kamailio][s2]... +* IRC: [ngircd][i1], [hybridircd][i2]... + +[m1]: https://www.opensmtpd.org/ +[m2]: https://cr.yp.to/qmail.html +[m3]: http://www.postfix.org/ +[h1]: https://nginx.org/ +[h2]: https://www.lighttpd.net/ +[h3]: https://bsd.plumbing/ +[h4]: https://haproxy.org/ +[s1]: https://opensips.org/ +[s2]: https://kamailio.org/w/ +[i1]: https://ngircd.barton.de/ +[i2]: http://ircd-hybrid.org/ + +IRC itself is much of a routing protocol server-side: routing messages to +the right client. Another aspect of applicative routing is client-side. + +Much like ii or sic, and all the modular bots in the wild, the jj acts as +a is a client-side IRC router through hooks and log files. + +How jj works +------------ +As opposed to ii, which is plain C, jj uses an awk script to do the heavy +lifting of IRC parsing, called from a compiled binary coded in C, that only +does the fifo creation, and multiplexing of user and network input, and +write both to the awk script. + +It reads its entire configuration from environment variables, described on +the the project's README.md, such as $IRC_DIR, in which it create a direcTory +with the channel as a name. + +I set IRC_DIR to /var/irc, which gives us (full list on the README.md): + +* /var/irc/irc.freenode.net/channels/*.log - messages from users and channels. +* /var/irc/irc.freenode.net/in - the FIFO pipe to which write messages. + +There is one instance of jj per server conexion, which greatly simplifies +the software, makes debugging much easier, and permit to adapt and configure +it specifically for the requirements of each host to connect to. + +Recently, jj acquired the [UCSPI][u0] connexion interface, which is a way to +write programs without handling the network protocols into the program itself, +but instead expect an running connexion (TCP, TLS, SSH, SOCKS5...) from standard +input and output (for servers) or file descripTor 6 and 7 (for clients). + +This permits to run it through [s6-networking][s6], [ucspi][u1], [ucspi-ssl][u2], +[curvecp][cp], or any protocol that has an UCSPI adapter for it. + +[u0]: https://cr.yp.to/proto/ucspi.txt +[s6]: https://skarnet.org/software/s6-networking/ +[u1]: https://github.com/younix/ucspi/ +[u2]: https://www.fehcom.de/ipnet/ucspi-ssl.html +[cp]: https://curvecp.org/ + +State of IRC client \<=\> server connexions +------------------------------------------- +Because it is run by different people and projects, the connexion to IRC +servers varies greatly through the different cases: + +* Some servers only accept TCP connexions. +* Some servers only accept TLS connexions. +* Some servers permit to use a client TLS certificate to authenticate. +* Some servers support connexion coming form [[Tor]], providing the extra + privacy that the IRC protocol lacks +* Some servers refuse connexions coming from Tor. +* Some are published as Tor hidden services directly: so no need for TLS. +* Some servers still propose TLS over Tor, with certificate authentication. +* Some servers use a self-signed certificate, and publish a fingerprint + of their certificate. +* Some servers used a private certificate *authority* and publish their + root certificate. + +From this plethora of security fine tuning, it is necessary to have an irc +client with a good TLS implementation (lots of lines of code), and a socks +proxy (more lines of code), with a configuration interface (many many lines of +code). + +How UCSPI helps +--------------- +I use jj under the [[s6]] and [[s6-rc]] supervision tree on a VPS (until I get +real hardware home). + +jj having one instance per host to connect to, and jj supporting UCSPI, all of +these are of a great fit: + +[s6]: https://skarnet.org/software/s6/ +[s6-rc]: https://skarnet.org/software/s6-rc/ +[Tor]: https://www.Torproject.org/ + +By using ucspi, we are entirely avoiding the problemm, as we can compose a +socks client to talk to the Tor daemon (one line of scripts), and once the +connexion has started, it is possible to start a tlsclient program that uses +the active connexion and start a TLS session within that Tor socket, which can +be configured as well to use a client certificate. + +All of the UCSPI tools work by performing their startup work (opening a TCP +connexion, initiating a TLS/SOCKS/... session), and starting a child program, +which was pased as argument to it, with a pipe, or executing them directly. + +This gives more or less long chain, aka [[chainloading]], that [[djb]] used +and popularized through its set of programs. + +[chainloading]: https://en.wikipedia.org/Chain_loading +[djb]: https://cr.yp.to/djb.html + +How I use jj and s6/s6-rc and UCSPI together +-------------------------------------------- +The s6 and s6-rc package come with an execline shell-like language that makes +this style of pipling as natural + +This is how it looks in an [execline](http://skarnet.org/software/execline/) +`./run` script in practice, as povided by s6 for configuration of daemon (jj) +startup scripts: + +Boilerplate I have everywhere (sorry sk., #!/usr/bin/env...): + +``` +#!/usr/bin/env execlineb +fdmove -c 2 1 +``` + +This command reads the content of each file in ./env/ and export environment +variables to their content: env/IRC_HOST, env/IRC_USER, env/IRC_PASSWORD, ... + +``` +s6-envdir "env" +``` + +I use the "irc" user and the /var/irc/ direcTory: + +``` +s6-setuidgid irc +``` + +This starts a TCP connexion to the local Tor daemon running: + +``` +s6-tcpclient 127.0.0.1 9050 +``` + +This starts a SOCK5 proxy session for communicating with the +[hackint](https://hackint.org/) + +``` +sockc "5ogdsfyoqk47ompu.onion" "6667" +``` + +At that point, the program called by sockc will be able to communicate +directly with IRC commands, as the TCP connexion is ready, the socks +session is set, and the raw IRC protocol follows. + +It is now possible to call jjd directly, which will recognize the UCSPI +environment through the $PROTO environment variable. + +``` +jjd +``` + +In case there is another context, situation, other UCSPI commands can be +inserted or moved, including the setup of client certifiates, that often goes +through environment variables. + +This is the same script while run as #!/bin/sh. Yes, this is a single command! + +``` +#!/bin/sh +fdmove -c 2 1 \ + s6-envdir "env" \ + s6-setuidgid irc \ + s6-tcpclient 127.0.0.1 9050 \ + sockc "5ogdsfyoqk47ompu.onion" "6667" \ + jjd +``` + +Making jj useable interactively +------------------------------- +Being log-based, wrappers tools are necessary for making jj useable as a main +IRC client. + + +### [jj-in](/~josuah/jj/jj-in) + +*similar to jji from the jj package* + +Take lines from the user stdin with a prompt set to the name of the channel. +A better line-editing tool could be used, such as rlwrap around this. + + +### [jj-log](/~josuah/jj/jj-log) + +*similar to jjp in the jj package* + +Get a human-readable feed out of the jj log: +Filter the log to add colors. One way to use it is: tail -f chan.log | jj-log + + +### [retail](/~josuah/jj/retail) + +*not in original jj package* + +Print the multiple files sorted as if tail -f was printing them since the +beginning, with the same "==> filename <==" header. + + +### [jj-tail](/~josuah/jj/jj-tail) + +*not in original jj package* + +Instead of going with one window per chatroom and switching between chatrooms, +jj-tail gives a combined feed with the output of all channels in a same stream +showing a headers for each channel. + +``` +== irc.freenode.net/##networking == +00:12 ikityik: it's like having a bad fever dream with this ISP +== irc.freenode.net/#cat-v == +00:12 cicl: when usb is enabled sounds like a vm? +00:12 cicl: whats the context? +== irc.freenode.net/#openbsd == +00:12 thrig: Logan Runners? +00:12 knot: with blackjack and hookers +``` + +To avoid being bothered with too many channels at once, it takes a list of +find(1) -path $filters as argument that will match the name of the +server/channels: + +``` +$ jj-tail +<log of all channels of all servers ...> +^C +$ jj-tail freenode +<log of all channels of freenode/* ...> +^C +$ jj-tail qmail smtp +<log of freenode/#qmail freenode/#opensmtpd ...> +^C +$ jj-tail server +<server logs of all channels follows (wich hilights) ...> +``` + +This way, you can reduce the scope of the conversation to one server, similar +channels on multiple servers, just the selected channels... diff --git a/wiki/jj/jj-in b/jj/jj-in diff --git a/wiki/jj/jj-join b/jj/jj-join diff --git a/wiki/jj/jj-log b/jj/jj-log diff --git a/wiki/jj/jj-tail b/jj/jj-tail diff --git a/wiki/jj/retail b/jj/retail diff --git a/links/index.md b/links/index.md @@ -194,7 +194,7 @@ System programming * <https://skarnet.org/software/> - Rewrite the layer between the kernel and the applications with minimalism. -* <https://en.wikipedia.org/wiki/Advanced_Programming_in_the_Unix_Environment> - +* <https://en.wikipedia.org/Advanced_Programming_in_the_Unix_Environment> - Programming book for feeling fine with all these syscalls. Crypto diff --git a/wiki/qmail/destination-mx/index.md b/qmail/destination-mx/index.md diff --git a/qmail/index.md b/qmail/index.md @@ -0,0 +1,23 @@ +(not)qmail implementation +========================= +*notes on [[qmail]] and [[notqmail]] implementation for self reference* + +[qmail]: https://cr.yp.to/qmail.html +[notqmail]: https://notqmail.org/ + +* [Choosing the destination MX][mx] + +[mx]: /~josuah/qmail/destination-mx/ + +For documentation on how to use qmail, [life with qmail][lwq] is the practical +you are probably looking for. + +[lwq]: http://www.lifewithqmail.org/lwq.html + +Patching qmail: + +* Qmail Big DNS - [mailing list arcihve][p1] - the ANY lookup for a domain may + give a lot of different results, or not be supported at all by nameservers + (walking toward deprecation). Disabling the feature may help some deliveries. + +[p1]: https://lists.gt.net/qmail/users/138190 diff --git a/quotes/index.md b/quotes/index.md @@ -29,13 +29,13 @@ awesome natural language ingenuity. > Usually the advice for computer problems is to turn it off and > then on again, so this might through them off a bit. -**[Roy Fielding](https://en.wikipedia.org/wiki/Roy_Fielding)**: +**[Roy Fielding](https://en.wikipedia.org/Roy_Fielding)**: > Some architectural styles are often portrayed as “silver bullet” solutions > for all forms of software. However, a good designer should select a style > that matches the needs of a particular problem being solved. -**[Roy Fielding](https://en.wikipedia.org/wiki/Roy_Fielding)**: +**[Roy Fielding](https://en.wikipedia.org/Roy_Fielding)**: > REST is designed to be efficient for large-grain hypermedia data transfer, > optimizing for the common case of the Web, but resulting in an interface that diff --git a/wiki/supervisor/index.md b/supervisor/index.md diff --git a/wiki/tinydns/Makefile b/tinydns/Makefile diff --git a/wiki/tinydns/data.awk b/tinydns/data.awk diff --git a/tinydns/index.md b/tinydns/index.md @@ -0,0 +1,89 @@ +Configuration of [[tinydns]] +============================ +[tinydns]: https://cr.yp.to/djbdns.html + +To run nameservers, you need to maintain a bunch of interdependent DNS +information. + +To achieve this, I write small input files that end up in data.cdb, read +by the tinydns program: + +``` +(text) +rr.domain ┐ (text) (binary) +rr.host │ ┌────────┐ ┌────────────┐ +rr.mx ├─┤data.awk├─> data >─┤tinydns-edit├─> data.cdb +rr.ns │ └────────┘ └────────────┘ +rr.alias ┘ +``` + +The workhorse here is [[data.awk]], and the whole chain is controlled by a +small [[Makefile]]. + +[data.awk]: /~josuah/tinydns/data.awk +[makefile]: /~josuah/tinydns/Makefile + +/etc/tinydns/rr.domain +---------------------- +A list of top and second level domain names. The first one listed is the +"technical" domain name. + +``` +z0.is +josuah.net +``` + +/etc/tinydns/rr.host +-------------------- +A list of "hostname without domain part", "IPv4", "IPv6". This is the only file +where IP are written. This is the book keeping of the servers you address. + +Each line of this file leads to a hostname.technical.dom record of type A, +AAAA, and PTR: + +``` +kuntur 199.247.28.162 2001:19f0:5001:7ac::12 +harpyja 80.67.190.196 2a00:5884:8214::16 +``` + +/etc/tinydns/rr.mx +------------------ +A list of hostnames that run a mail server. Each domain in "rr.domain" gets one +MX record per mail server listed here, with the form: hostname.technical.dom as +generated by the "rr.host" list. + +We now have matching MX, A, AAAA and PTR entries. + +``` +kuntur +``` + +/etc/tinydns/rr.ns +------------------ +A list of hostnames that run a DNS name server. Like for "rr.mx", each domain +in "rr.domain" gets one NS record per name server listed here. + +``` +kuntur ns1 +harpyja ns2 +``` + +/etc/tinydns/rr.alias +--------------------- +A list of regular domain records for your various servers and services. You may + +``` +harpyja www.josuah.net +harpyja git.josuah.net +kuntur josuah.z0.is +``` + +Uploading data.cdb to nameservers +--------------------------------- +"make push" update all necessary parts to build data.cdb as seen above, then +calls [push.awk](/~josuah/tinydns/push.awk). + +push.awk copies data.cdb to each nameserver in rr.ns, using the IP found in +rr.host (so even if your DNS is down, you can still update it from remote). +This example sends about 10KB over SSH in total, about 1.50 second over a DSL +line for 2 nameservers. diff --git a/wiki/tinydns/push.awk b/tinydns/push.awk diff --git a/wiki/awk/index.md b/wiki/awk/index.md @@ -1,434 +0,0 @@ -AWK -=== -AWK is a surprising efficient language, for both [performance][perf] and code -efficiency. This comes with the ubiquitous array structure, and splitting the -input in fields by default. - -Not everything is parsed efficiently with AWK, Type-Length-Value for instance, -but many things are. I use it for multiple projects: - -* [[NotWiki]], featuring a (not)markdown parser that does two passes on - to easen-up the parsing, -* [ics2txt](//code.z0.is/ics2txt/), a iCalendar to other format converter. -* [jj](/~josuah/wiki/jj/) by aaronNGi, a daemon with an awk engine to project - that turns raw IRC protocol into *very* easy to parse split log files. - -[perf]: https://adamdrake.com/command-line-tools-can-be-235x-faster-than-your-hadoop-cluster.html -[notwiki]: /~josuah/notwiki/ -[ics2txt]: gopher://bitreich.org/1/scm/ics2txt -[parser]: /notwiki/notwiki-index-html - -Below are multiple ways of using awk for getting the best out of it. These are -partly by myself, partly collected from what I saw in the wild. - -CSV fields with header ----------------------- -Instead of trying to remember the number of the column, using the name of the -column is much easier, and permit to have new columns inserted in the .csv file -without breaking the script. - -$ cat input.txt -domain_name,expiry_date,creation_date,owner,account_id -nowhere.com,2020-03,2019-05,me,23535 -perdu.com,2020-04,2018-03,you,23535 -pa.st,2020-09,2014-05,them,23535 - -$ awk ' - BEGIN { FS = "," } - NR == 1 { for (i = 1; i <= NF; i++) F[$i] = i; next } - $F["domain_name"] ~ /\.com$/ { - print $F["expiry_date"], $F["owner"], $F["domain_name"] - } -' input.txt -2020-03 me nowhere.com -2020-04 you perdu.com -``` - -UCL-style configuration ------------------------ -Parsing data that is not organised with line-column is also convenient and -efficient with awk, convenient for selecting one kind of value out of a -configuration file: - -``` -$ cat input.txt -connections { - conn-faraway { - children { - localnet = fe80:123d:35d3::%vio1/64 - localnet = fe80:2e46:1d23::%vio2/64 - } - children { - localnet = fe80:546:23e4::%vio3/64 - } - } - conn-veryclose { - children { - localnet = fe80:b536:243f::%vio3/64 - localnet = fe80:34f3:23c3::%vio3/64 - localnet = fe80:546a:343d::%vio3/64 - } - } -} -``` - -``` -$ awk ' - $2 == "{" { F[lv++] = $1 } - $1 == "}" { delete F[--lv] } - F[0] == "connections" && F[2] == "children" && $1 == "localnet" { - print F[1], $3 - } -' input.txt -conn-faraway fe80:123d:35d3::%vio1/64 -conn-faraway fe80:2e46:1d23::%vio2/64 -conn-faraway fe80:546:23e4::%vio3/64 -conn-veryclose fe80:b536:243f::%vio3/64 -conn-veryclose fe80:34f3:23c3::%vio3/64 -conn-veryclose fe80:546a:343d::%vio3/64 -``` - -Key-Value splitter ------------------- -Parsing key-value pairs can be mapped rather directly to an awk array, -for instance, to extract an abstract out of a basic iCal file: - -``` -$ cat input.txt -BEGIN:VEVENT -METHOD:PUBLISH -UID:9189@FOSDEM20@fosdem.org -TZID:Europe-Brussels -DTSTART:20200201T170000 -DTEND:20200201T175000 -SUMMARY:State of the Onion -DESCRIPTION:Building usable free software to fight surveillance and censorship. -CLASS:PUBLIC -STATUS:CONFIRMED -CATEGORIES:Internet -LOCATION:Janson -END:VEVENT -$ awk ' - BEGIN { FS = ":" } - { F[$1] = $2 } - $1 == "END" { - print F["SUMMARY"] " - " F["DESCRIPTION"] - print F["DTSTART"], "(" F["TZID"] ")" - } -' input.txt -State of the Onion - Building usable free software to fight surveillance and censorship. -20200201T170000 (Europe-Brussels) -``` - -Edit variables passed to functions ----------------------------------- -For languages that support references, pointers, or objects, it is possible to -edit the variable passed to a function, so that the variable also gets edited -in the function that called it. - -``` -void increment(int *i) { (*i)++; } -``` - -Awk does not support changing integers or strings, but supports editing the -fields of an array: - -``` -function increment_first(arr) { arr[1]++ } -``` - -Local variables in functions ----------------------------- -By default, all awk variables are global, which is inconvenient for writing -functions. The solution is to add an extra function argument at the end for -each local variable we need. - -Functions can be called with fewer arguments than they have. - -``` -$ awk ' - function concat3(arg1, arg2, arg3, - local1) - { - local1 = arg1 arg2 arg3 - return local1 - } - - BEGIN { - local1 = 1 - print(concat3("a", "w", "k")) - print(local1) - } -' -awk -1 -``` - -I learned this with the [jj](https://github.com/aaronNGi/jj/) project. - -A sort() function ------------------ -A very convenient feature lacking to awk is support for sorting members of an -array. Is possible to implement sort() in awk (this is a quicksort): - -``` -function swap(array, a, b, - tmp) -{ - tmp = array[a] - array[a] = array[b] - array[b] = tmp -} - -function sort(array, beg, end) -{ - if (beg >= end) # end recursion - return - a = beg + 1 # 1st is the pivot, so +1 - b = end - while (a < b) { - while (a < b && array[a] <= array[beg]) # beg: skip lesser - a++ - while (a < b && array[b] > array[beg]) # end: skip greater - b-- - swap(array, a, b) # found 2 misplaced - } - if (array[beg] > array[a]) # put the pivot back - swap(array, beg, a) - sort(array, beg, a - 1) # sort lower half - sort(array, a, end) # sort higher half -} -``` - -This sorts the array values using integers keys: `array[1]`, `array[2]`, ... -It sorts from `array[beg]` to `array[end]` included, so you can choose your -array indices starting at 0 or 1, or sort just a part of the array. - -Example usage: with the both function above: - -``` -{ - LINES[NR] = $0 -} - -END { - sort(LINES, 1, NR) - for (i = 1; i <= NR; i++) - print(LINES[i]) -} -``` - -Performance is far from terrible! - -``` -$ od -An /dev/urandom | head -n 1000000 | time ./test.awk >/dev/null -real 0m 19.23s -user 0m 17.90s -sys 0m 0.12s - -$ od -An /dev/urandom | head -n 1000000 | time sort >/dev/null -real 0m 4.39s -user 0m 3.00s -sys 0m 0.10s -``` - -Fill a static array -------------------- -With C and many other languages, there are convenient concise syntax such as -`{ "a", "b", "c", ...}`to fill an array with values. An well-known way is: - -``` -split("a b c ...", array, " ") -``` - -Note that this does not saves the length out of split, but in practice I -realized I rarely need it: - -``` -for (i = 1; i in split; i++) - print(i, array[i]) -``` - -A fold_line() function ------------------------ -Convenient to work with text documents or emails. This version does not -truncate very long words, such as some https:// links. - -``` -function fold_line(str, len, - head, tail, i) -{ - head = substr(str, 1, len + 1) - sub(" *$", "", head) - if (length(head) == len + 1) - sub(" *[^ ]*$", "", head) - if (length(head) == 0) { - tail = substr(str, len + 1) - head = substr(str, 1, len) - if ((i = index(tail, " ")) == 0) - return str - return head substr(tail, 1, i) - } - return head -} -``` - -To use it, call it in a while loop like this: - -``` -{ - while (line = fold_line($0, 72)) { - print line - $0 = substr($0, length(line) + 2) - } -} -``` - -A gmtime() function -------------------- -POSIX awk as well as many implementations lack the [time functions][tf] present in -GNU awk. This gmtime() function split an epoch integer value (1587302158) into the -fields year, mon, mday, hour, min, sec (2020-04-19T15:15:58Z): - -[tf]: https://www.gnu.org/software/gawk/manual/html_node/Time-Functions.html - -``` -function isleap(year) -{ - return (year % 4 == 0) && (year % 100 != 0) || (year % 400 == 0) -} - -function mdays(mon, year) -{ - return (mon == 2) ? (28 + isleap(year)) : (30 + (mon + (mon > 7)) % 2) -} - -function gmtime(sec, tm) -{ - tm["year"] = 1970 - while (sec >= (s = 86400 * (365 + isleap(tm["year"])))) { - tm["year"]++ - sec -= s - } - tm["mon"] = 1 - while (sec >= (s = 86400 * mdays(tm["mon"], tm["year"]))) { - tm["mon"]++ - sec -= s - } - tm["mday"] = 1 - while (sec >= (s = 86400)) { - tm["mday"]++ - sec -= s - } - tm["hour"] = 0 - while (sec >= 3600) { - tm["hour"]++ - sec -= 3600 - } - tm["min"] = 0 - while (sec >= 60) { - tm["min"]++ - sec -= 60 - } - tm["sec"] = sec -} -``` - -The tm array will be filled with field names following the [[gmtime]] -function as you can see above. - -[gmtime]: https://pubs.opengroup.org/onlinepubs/9699919799/functions/gmtime.html - -A localtime() function ----------------------- -For printing functions in the user's favorite timezone, gmtime's time needs -to be shifted. This can also be done in standard awk by calling the date(1) -command: - -``` -function localtime(sec, tm, - tz, h, m) -{ - if (!TZ) { - "date +%z" | getline tz - close("date +%z") - h = substr(tz, 2, 2) - m = substr(tz, 4, 2) - TZ = substr(tz, 1, 1) (h * 3600 + m * 60) - } - return gmtime(sec + TZ, tm) -} -``` - -Note that date(1) will only be called the first time localtime() is called, and -the TZ global variable will be used for the next calls. - -A timegm() function -------------------- -Complementary function to gmtime is timegm for converting a `tm[]` array back -to an integer representation. This is useful for parsing time values back to -an unix timestamp: - -``` -function isleap(year) -{ - return (year % 4 == 0) && (year % 100 != 0) || (year % 400 == 0) -} - -function mdays(mon, year) -{ - return (mon == 2) ? (28 + isleap(year)) : (30 + (mon + (mon > 7)) % 2) -} - -function timegm(tm, - sec, mon, day) -{ - sec = tm["sec"] + tm["min"] * 60 + tm["hour"] * 3600 - day = tm["mday"] - 1 - for (mon = tm["mon"] - 1; mon > 0; mon--) - day = day + mdays(mon, tm["year"]) - day = day + int(tm["year"] / 400) * 146097 - day = day + int(tm["year"] % 400 / 100) * 36524 - day = day + int(tm["year"] % 100 / 4) * 1461 - day = day + int(tm["year"] % 4 / 1) * 365 - return sec + (day - 719527) * 86400 -} -``` - -All the following fields of `tm[]` must be defined: "year", "mon", "mday", -"hour", "min", "sec". - -Convert MAC address to brand name ---------------------------------- -[MAC addresses](https://en.wikipedia.org/wiki/MAC_address) are composed by a -leading Organization Unique Identifier (OUI) of 3 bytes and a trailing 3-byte -number, unique for that OUI. - -Each vendor has its own OUI, so each OUI maps to a vendor. With the reference -list the [IEEE](https://ieee.org/) publishes, it is possible to convert MAC -address OUI digits to a human-readable name: - -``` -function oui_table(path, - url) -{ - url = "http://standards-oui.ieee.org/oui/oui.txt" - if (system("test -f '" path "'") > 0) - if (system("curl -L -o '" path "' " url) != 0) - return -1 - while (getline <path) - if ($2 " " $3 == "(base 16)") - OUI[$1] = substr($0, 22) - return 0 -} -``` - -Then a global `OUI` array does the MAC addresss to vendor name mapping: - -``` -BEGIN { - if (oui_table("/var/tmp/oui.txt") < 0) - exit(1) - print(OUI[toupper("84a991")]) -} -``` diff --git a/wiki/jj/index.md b/wiki/jj/index.md @@ -1,258 +0,0 @@ -The jj IRC client -================== -[jj](https://github.com/aaronNGi/jj/) is an [Internet Relay Chat -(IRC)](https://invidio.us/watch?v=R8FOGlnYkgg) client based on the principles -of [ii](https://tools.suckless.org/ii/), using FIFO (named pipe) files to read -commands from the user, and simply logs the output of each channels to plain -files. - -Instead of being an ncurse program that runs into tmux(1), jj works as an -applicative router. A router for ISO layer 7. You may already know "applicative -routers" for different protocols already: - -* SMTP: [OpenSMTPD][m1], [qmail][m2], [Postfix][m3]... -* HTTP: [nginx][h1], [lighttpd][h2], [relayd][h3], [haproxy][h4]... -* SIP: [OpenSIPS][s1], [Kamailio][s2]... -* IRC: [ngircd][i1], [hybridircd][i2]... - -[m1]: https://www.opensmtpd.org/ -[m2]: https://cr.yp.to/qmail.html -[m3]: http://www.postfix.org/ -[h1]: https://nginx.org/ -[h2]: https://www.lighttpd.net/ -[h3]: https://bsd.plumbing/ -[h4]: https://haproxy.org/ -[s1]: https://opensips.org/ -[s2]: https://kamailio.org/w/ -[i1]: https://ngircd.barton.de/ -[i2]: http://ircd-hybrid.org/ - -IRC itself is much of a routing protocol server-side: routing messages to -the right client. Another aspect of applicative routing is client-side. - -Much like ii or sic, and all the modular bots in the wild, the jj acts as -a is a client-side IRC router through hooks and log files. - -How jj works ------------- -As opposed to ii, which is plain C, jj uses an awk script to do the heavy -lifting of IRC parsing, called from a compiled binary coded in C, that only -does the fifo creation, and multiplexing of user and network input, and -write both to the awk script. - -It reads its entire configuration from environment variables, described on -the the project's README.md, such as $IRC_DIR, in which it create a direcTory -with the channel as a name. - -I set IRC_DIR to /var/irc, which gives us (full list on the README.md): - -* /var/irc/irc.freenode.net/channels/*.log - messages from users and channels. -* /var/irc/irc.freenode.net/in - the FIFO pipe to which write messages. - -There is one instance of jj per server conexion, which greatly simplifies -the software, makes debugging much easier, and permit to adapt and configure -it specifically for the requirements of each host to connect to. - -Recently, jj acquired the [UCSPI][u0] connexion interface, which is a way to -write programs without handling the network protocols into the program itself, -but instead expect an running connexion (TCP, TLS, SSH, SOCKS5...) from standard -input and output (for servers) or file descripTor 6 and 7 (for clients). - -This permits to run it through [s6-networking][s6], [ucspi][u1], [ucspi-ssl][u2], -[curvecp][cp], or any protocol that has an UCSPI adapter for it. - -[u0]: https://cr.yp.to/proto/ucspi.txt -[s6]: https://skarnet.org/software/s6-networking/ -[u1]: https://github.com/younix/ucspi/ -[u2]: https://www.fehcom.de/ipnet/ucspi-ssl.html -[cp]: https://curvecp.org/ - -State of IRC client \<=\> server connexions -------------------------------------------- -Because it is run by different people and projects, the connexion to IRC -servers varies greatly through the different cases: - -* Some servers only accept TCP connexions. -* Some servers only accept TLS connexions. -* Some servers permit to use a client TLS certificate to authenticate. -* Some servers support connexion coming form [[Tor]], providing the extra - privacy that the IRC protocol lacks -* Some servers refuse connexions coming from Tor. -* Some are published as Tor hidden services directly: so no need for TLS. -* Some servers still propose TLS over Tor, with certificate authentication. -* Some servers use a self-signed certificate, and publish a fingerprint - of their certificate. -* Some servers used a private certificate *authority* and publish their - root certificate. - -From this plethora of security fine tuning, it is necessary to have an irc -client with a good TLS implementation (lots of lines of code), and a socks -proxy (more lines of code), with a configuration interface (many many lines of -code). - -How UCSPI helps ---------------- -I use jj under the [[s6]] and [[s6-rc]] supervision tree on a VPS (until I get -real hardware home). - -jj having one instance per host to connect to, and jj supporting UCSPI, all of -these are of a great fit: - -[s6]: https://skarnet.org/software/s6/ -[s6-rc]: https://skarnet.org/software/s6-rc/ -[Tor]: https://www.Torproject.org/ - -By using ucspi, we are entirely avoiding the problemm, as we can compose a -socks client to talk to the Tor daemon (one line of scripts), and once the -connexion has started, it is possible to start a tlsclient program that uses -the active connexion and start a TLS session within that Tor socket, which can -be configured as well to use a client certificate. - -All of the UCSPI tools work by performing their startup work (opening a TCP -connexion, initiating a TLS/SOCKS/... session), and starting a child program, -which was pased as argument to it, with a pipe, or executing them directly. - -This gives more or less long chain, aka [[chainloading]], that [[djb]] used -and popularized through its set of programs. - -[chainloading]: https://en.wikipedia.org/wiki/Chain_loading -[djb]: https://cr.yp.to/djb.html - -How I use jj and s6/s6-rc and UCSPI together --------------------------------------------- -The s6 and s6-rc package come with an execline shell-like language that makes -this style of pipling as natural - -This is how it looks in an [execline](http://skarnet.org/software/execline/) -`./run` script in practice, as povided by s6 for configuration of daemon (jj) -startup scripts: - -Boilerplate I have everywhere (sorry sk., #!/usr/bin/env...): - -``` -#!/usr/bin/env execlineb -fdmove -c 2 1 -``` - -This command reads the content of each file in ./env/ and export environment -variables to their content: env/IRC_HOST, env/IRC_USER, env/IRC_PASSWORD, ... - -``` -s6-envdir "env" -``` - -I use the "irc" user and the /var/irc/ direcTory: - -``` -s6-setuidgid irc -``` - -This starts a TCP connexion to the local Tor daemon running: - -``` -s6-tcpclient 127.0.0.1 9050 -``` - -This starts a SOCK5 proxy session for communicating with the -[hackint](https://hackint.org/) - -``` -sockc "5ogdsfyoqk47ompu.onion" "6667" -``` - -At that point, the program called by sockc will be able to communicate -directly with IRC commands, as the TCP connexion is ready, the socks -session is set, and the raw IRC protocol follows. - -It is now possible to call jjd directly, which will recognize the UCSPI -environment through the $PROTO environment variable. - -``` -jjd -``` - -In case there is another context, situation, other UCSPI commands can be -inserted or moved, including the setup of client certifiates, that often goes -through environment variables. - -This is the same script while run as #!/bin/sh. Yes, this is a single command! - -``` -#!/bin/sh -fdmove -c 2 1 \ - s6-envdir "env" \ - s6-setuidgid irc \ - s6-tcpclient 127.0.0.1 9050 \ - sockc "5ogdsfyoqk47ompu.onion" "6667" \ - jjd -``` - -Making jj useable interactively -------------------------------- -Being log-based, wrappers tools are necessary for making jj useable as a main -IRC client. - - -### [jj-in](/~josuah/wiki/jj/jj-in) - -*similar to jji from the jj package* - -Take lines from the user stdin with a prompt set to the name of the channel. -A better line-editing tool could be used, such as rlwrap around this. - - -### [jj-log](/~josuah/wiki/jj/jj-log) - -*similar to jjp in the jj package* - -Get a human-readable feed out of the jj log: -Filter the log to add colors. One way to use it is: tail -f chan.log | jj-log - - -### [retail](/~josuah/wiki/jj/retail) - -*not in original jj package* - -Print the multiple files sorted as if tail -f was printing them since the -beginning, with the same "==> filename <==" header. - - -### [jj-tail](/~josuah/wiki/jj/jj-tail) - -*not in original jj package* - -Instead of going with one window per chatroom and switching between chatrooms, -jj-tail gives a combined feed with the output of all channels in a same stream -showing a headers for each channel. - -``` -== irc.freenode.net/##networking == -00:12 ikityik: it's like having a bad fever dream with this ISP -== irc.freenode.net/#cat-v == -00:12 cicl: when usb is enabled sounds like a vm? -00:12 cicl: whats the context? -== irc.freenode.net/#openbsd == -00:12 thrig: Logan Runners? -00:12 knot: with blackjack and hookers -``` - -To avoid being bothered with too many channels at once, it takes a list of -find(1) -path $filters as argument that will match the name of the -server/channels: - -``` -$ jj-tail -<log of all channels of all servers ...> -^C -$ jj-tail freenode -<log of all channels of freenode/* ...> -^C -$ jj-tail qmail smtp -<log of freenode/#qmail freenode/#opensmtpd ...> -^C -$ jj-tail server -<server logs of all channels follows (wich hilights) ...> -``` - -This way, you can reduce the scope of the conversation to one server, similar -channels on multiple servers, just the selected channels... diff --git a/wiki/qmail/index.md b/wiki/qmail/index.md @@ -1,23 +0,0 @@ -(not)qmail implementation -========================= -*notes on [[qmail]] and [[notqmail]] implementation for self reference* - -[qmail]: https://cr.yp.to/qmail.html -[notqmail]: https://notqmail.org/ - -* [Choosing the destination MX][mx] - -[mx]: /~josuah/wiki/qmail/destination-mx/ - -For documentation on how to use qmail, [life with qmail][lwq] is the practical -you are probably looking for. - -[lwq]: http://www.lifewithqmail.org/lwq.html - -Patching qmail: - -* Qmail Big DNS - [mailing list arcihve][p1] - the ANY lookup for a domain may - give a lot of different results, or not be supported at all by nameservers - (walking toward deprecation). Disabling the feature may help some deliveries. - -[p1]: https://lists.gt.net/qmail/users/138190 diff --git a/wiki/tinydns/index.md b/wiki/tinydns/index.md @@ -1,89 +0,0 @@ -Configuration of [[tinydns]] -============================ -[tinydns]: https://cr.yp.to/djbdns.html - -To run nameservers, you need to maintain a bunch of interdependent DNS -information. - -To achieve this, I write small input files that end up in data.cdb, read -by the tinydns program: - -``` -(text) -rr.domain ┐ (text) (binary) -rr.host │ ┌────────┐ ┌────────────┐ -rr.mx ├─┤data.awk├─> data >─┤tinydns-edit├─> data.cdb -rr.ns │ └────────┘ └────────────┘ -rr.alias ┘ -``` - -The workhorse here is [[data.awk]], and the whole chain is controlled by a -small [[Makefile]]. - -[data.awk]: /~josuah/wiki/tinydns/data.awk -[makefile]: /~josuah/wiki/tinydns/Makefile - -/etc/tinydns/rr.domain ----------------------- -A list of top and second level domain names. The first one listed is the -"technical" domain name. - -``` -z0.is -josuah.net -``` - -/etc/tinydns/rr.host --------------------- -A list of "hostname without domain part", "IPv4", "IPv6". This is the only file -where IP are written. This is the book keeping of the servers you address. - -Each line of this file leads to a hostname.technical.dom record of type A, -AAAA, and PTR: - -``` -kuntur 199.247.28.162 2001:19f0:5001:7ac::12 -harpyja 80.67.190.196 2a00:5884:8214::16 -``` - -/etc/tinydns/rr.mx ------------------- -A list of hostnames that run a mail server. Each domain in "rr.domain" gets one -MX record per mail server listed here, with the form: hostname.technical.dom as -generated by the "rr.host" list. - -We now have matching MX, A, AAAA and PTR entries. - -``` -kuntur -``` - -/etc/tinydns/rr.ns ------------------- -A list of hostnames that run a DNS name server. Like for "rr.mx", each domain -in "rr.domain" gets one NS record per name server listed here. - -``` -kuntur ns1 -harpyja ns2 -``` - -/etc/tinydns/rr.alias ---------------------- -A list of regular domain records for your various servers and services. You may - -``` -harpyja www.josuah.net -harpyja git.josuah.net -kuntur josuah.z0.is -``` - -Uploading data.cdb to nameservers ---------------------------------- -"make push" update all necessary parts to build data.cdb as seen above, then -calls [push.awk](/~josuah/wiki/tinydns/push.awk). - -push.awk copies data.cdb to each nameserver in rr.ns, using the IP found in -rr.host (so even if your DNS is down, you can still update it from remote). -This example sends about 10KB over SSH in total, about 1.50 second over a DSL -line for 2 nameservers.