Jekyll2022-07-08T06:31:56+02:00https://kitabi.eu/feed.xmlConstantly OutdatedFawzi's constantly outdated website and blog on physics, science, ethics, mathematics and programming...Fawzi MohamedUpdated Nomad Pages2022-07-08T00:00:00+02:002022-07-08T00:00:00+02:00https://kitabi.eu/blog/update-nomad-pages<p>I noticed that several links in the <a href="/nomad/">NOMAD pages</a> were outdated (as I changed the way the site is generated), they should be fixed now.
Also I created a <a href="https://github.com/fawzi/container-manager">clone of the container manager git</a> in github.</p>fawziI noticed that several links in the NOMAD pages were outdated (as I changed the way the site is generated), they should be fixed now. Also I created a clone of the container manager git in github.Back to Django IT2022-06-12T00:00:00+02:002022-06-12T00:00:00+02:00https://kitabi.eu/blog/django-it<p>Un gruppo di ricerca usava <a href="https://en.wikipedia.org/wiki/Microsoft_Access">Microsoft Access</a> per creare dei Database, e stava pensando alle proprie opzioni.</p>
<p>Queste sono alcune note di quello che gli ho detto, in quanto può essere rilevante ache per altri.</p>
<p>Sono tornato a <a href="https://djangoproject.com">Django</a>, che avevo già usato per <a href="http://mupris.net">mupris.net</a>, e sono giunto alla conclusione che probabilmente è la soluzione migliore per loro.</p>
<h1 id="note-generali">Note generali</h1>
<p>Database relazionale: la cosa più importante è definire lo schema. Quello si può fare discutendo ad alto livello su cosa si vuole registrare, e poi definendo le tabelle e i campi.</p>
<h2 id="perchè-django">Perchè Django</h2>
<p>Django non è il tool migliore per esperti, e in effetti non è quello che ho usato in <a href="/nomad">NOMAD</a>, ma ci sono atri aspetti che sono importanti, in particolare la possibilità di includere facilmente altre persone, in particolare esperti della materia, ma non di informatica.
Django normalmente incanala verso delle pratiche che non sono necessariamente le migliori, ma che hanno dimostrato la loro utilità molte volte.
Questo lo rende una buona soluzione in tanti casi.</p>
<ul>
<li>definizione del Db facile da capire per i non addetti ai lavori (molto importante)</li>
<li>supporto per migrazioni</li>
<li>supporto per più databasi:
<ul>
<li><a href="https://www.postgresql.org">Postgres</a> è una buona scelta (open source, gratuito, e buone performance & capacità)</li>
<li>mysql o mariadb (il fork più aperto) è ok in particolare se chi lo deve amministrare lo conosce di già</li>
<li>sqlite: il db è in un singlo file, molto comodo per uno sviluppatore locale</li>
</ul>
</li>
<li>possibiltà per nuovi developer di avere facilmente un installazione locale in cui sperimentare, e poi integrare le cose nella versione principale (via migrazioni)</li>
<li>codice e definizione del databse in git (<em>molto utile</em>, <a href="https://git-scm.com/doc">info su git</a>)</li>
<li>spinge verso un buon design del DB (la soluzione più ovvia in Django normalmente è una soluzione ben pensata e usata da molti). Esistono optioni più flessibili (SqlAlchemy per esempio) che possono essere più potenti per gli esperti, ma per i non esperti è più facile scegliere opzioni che creano problemi più tardi</li>
<li>interfaccia di admin: Uno vuole poter inserire i dati, e questo é fatto da utenti “speciali”.
Con django si può immediatamente aver un interfaccia per inserire i dati molto funzionale, e che può poi essere miglorata.</li>
</ul>
<p>l’interfaccia di admin secondo me è un punto fondamentale, e la ragione principale per scegliere django.
Migrazioni e interfaccia admin permettono di non rimanere ostaggi del database, e ruscire a cambiarlo in modo ragionevole.</p>
<ul>
<li><a href="https://www.django-rest-framework.org">Django REST Framework</a> è una buona opzione per API</li>
<li><a href="https://docs.djangoproject.com/en/4.0/ref/contrib/gis/tutorial/">GeoDjango</a> aiuta a supportate dati spaziali georiferiti (rilevante er il loro progetto)
<h2 id="alternative">Alternative</h2>
<p>Se a pagamento <a href="https://www.claris.com">Filemaker</a> ha un interfaccia abbastanza semplice da gestire per i non programmatori.
Il suo svantaggio principale è che necessitando una licenza è più difficile includere gente esterna.</p>
</li>
</ul>
<p>Se no <a href="https://www.sqlalchemy.org">Sqlalchemy</a> è un ottimo tool in python per gesture lo schema, migrazioni (nuove versioni del DB) e queries al database (più potente di Django, ma senza il supporto per creare le interfacce, si può combinare con altri tools).
Ci sono naturalmente altre possibilità per esempio (<a href="https://guides.rubyonrails.org/active_record_basics.html">Active Records</a> se si usa ruby on rails, o <a href="https://www.jooq.org">Jooq</a> in java), e a seconda di quello che si usa possono essere migliori, ma io comincerei da queste.</p>
<h1 id="genrazione-dello-schema">Genrazione dello schema</h1>
<p><a href="https://github.com/schemacrawler/SchemaCrawler">SchemaCrawler</a></p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>./bin/schemacrawler.sh <span class="nt">--server</span> sqlite <span class="nt">--database</span> ../rsite/db.sqlite3 <span class="nt">--info-level</span><span class="o">=</span>maximum <span class="nt">--password</span><span class="o">=</span> <span class="nt">--command</span><span class="o">=</span>schema <span class="nt">--output-format</span><span class="o">=</span>png <span class="nt">--output-file</span><span class="o">=</span>../rsite/schema.png
</code></pre></div></div>
<p>preso da <a href="http://www.tobias-weis.de/short-schema-diagram-from-an-existing-sqlite-database/">Short</a></p>
<p>potrebbe essere utile guardare anche
<a href="https://github.com/inukshuk/sqleton">Sqleton</a></p>
<h1 id="vedere-db-di-access-su-mac">vedere DB di Access su mac</h1>
<p><a href="https://eggerapps.at/mdbviewer/">Mdb Viewer</a></p>fawziUn gruppo di ricerca usava Microsoft Access per creare dei Database, e stava pensando alle proprie opzioni.Back to Django EN2022-06-12T00:00:00+02:002022-06-12T00:00:00+02:00https://kitabi.eu/blog/django<p>A research group was using <a href="https://en.wikipedia.org/wiki/Microsoft_Access">Microsoft Access</a> for
their databases and was thinking about their options.</p>
<p>I gave them some suggestions in italian, this is a rough translation of
<a href="/blog/2022-06-12-django-it/">them</a>, as they might be relevant also for others.</p>
<p>I came back to <a href="https://djangoproject.com">Django</a>, which I had already used in
<a href="http://mupris.net">mupris.net</a>, and I concluded that it was the best solution
for them.</p>
<p>Django is not necessarily the best tool for experts, and indeed it is not what I
choose for <a href="/nomad">NOMAD</a>.
There are several options, for example <a href="https://www.sqlalchemy.org">Sqlalchemy</a> is an
excellent python tool to handle relational databases, <a href="https://guides.rubyonrails.org/active_record_basics.html">Active Records</a>
is what would be used with ruby on rails (which I did in another project, and is a bit the
inspiration for Django approach), <a href="https://www.jooq.org">Jooq</a> is an excellet optio with
java or JVM languages like scala. I used all of these options, and there are even more.</p>
<p>Still there are other aspects that are relevant for a choice:</p>
<ul>
<li>possibility of involving field experts which are not computer science experts,</li>
<li>how much those persons can rely on good programmers and how much they should be able
to work independently.</li>
<li>how much the tool pushes you toward well tested solutions (maybe not the best, but
things that worked well in practice)</li>
</ul>
<p>Here I think that django, with its support for migrations and the admin interface, helping
one not to be kept hostage of the Database, beats other options.</p>
<p>See the <a href="/blog/2022-06-12-django-it/">full discussion in italian</a> for more details.</p>fawziA research group was using Microsoft Access for their databases and was thinking about their options.How do Algorithms become biased?2021-04-03T00:00:00+02:002021-04-03T00:00:00+02:00https://kitabi.eu/blog/algorithms-biased<p>Recently, via <a href="https://kottke.org/21/03/how-do-algorithms-become-biased">kottke</a> I came across a interesting Youtube video: <a href="https://www.youtube.com/watch?v=Ok5sKLXqynQ&t=2s">Are We Automating Racism</a>.
It takes a look to racism in algorithms, a <a href="https://kitabi.eu/thoughts/social-challenges-ai/#racism">topic dear to me</a>, and a nice introduction to some of the issues, especially for people outside the field.</p>Fawzi MohamedRecently, via kottke I came across a interesting Youtube video: Are We Automating Racism. It takes a look to racism in algorithms, a topic dear to me, and a nice introduction to some of the issues, especially for people outside the field.Hey 1Password2020-11-01T00:00:00+01:002020-11-01T00:00:00+01:00https://kitabi.eu/blog/hey-1password<p>After quite some delay I started to use <a href="https://hey.com">hey.com</a>, as my email.
It immediately looked like <a href="https://hey.com/how-it-works/">an approach</a> I would like:</p>
<ul>
<li>the imbox for important emails,</li>
<li>the feed for info you want to see at your own pace,</li>
<li>the paper trail for the receipts</li>
</ul>
<p>just sounds natural to me, and indeed I do like it.</p>
<p>What I don’t like is that I cannot use my kitabi.eu address. I understand why:</p>
<ul>
<li>It is simpler to handle</li>
<li>It helps with marketing</li>
</ul>
<p>The second point is quite clever: instead of having an intrusive “sent with hey.com” signature (also ugly when trying to sell being a privacy-conscious entity), it just allows @hey.com addresses, making recipients aware of hey.com (as much as the email client shows it).
This spreads awareness of hey.com in an understated but effective way.</p>
<p>Still, I will surely switch to using my own @kitabi.eu address as soon as it becomes available, having your own domain is the best way to maintain control of your email address, and being able to change who administers it (or administer it yourself), without a very disruptive change of address.</p>
<p>Anyway Hey asked me to set up 2-factor authentication (2fa), and that was the trigger for me to move to 1Password.
Apple does a good job with keychains, and keeping your passwords secure, while sharing them between your machines <em>if</em> you only use Apple devices.
When using Linux this quickly becomes an issue, also sharing passwords securely with others can be very useful.
I was using <a href="https://lastpass.com">LastPass</a>, which works and has a usable free tier, but I knew that from a security and reliability point of view <a href="https://1password.com">1Password</a> would be better.</p>
<p>Other <a href="https://twofactorauth.org">2fa</a> solutions exist (I did use Microsoft Authenticator), but I decided to use this as an impetus to switch to 1Password.
It went relatively smoothly, the only annoyance was importing apple keychain passwords.</p>
<p>Not being able to easily export all passwords can be a security feature, but is definitely also a lock-in “feature” that keeps you in the Apple ecosystem.
As I had only a few entries, so I just Ctrl/Right-Clicked on every entry to copy it in a .csv file, that I then imported in 1Password.
Exporting from LastPass on the other hand was easy, a point in its favor.</p>
<p>2fa with 1Password stores the key in the vault along with everything else.
This is convenient but means that whoever has access to the vault has access to everything, so it is less independent (but is still effective against someone guessing your password).
Indeed this is the drawback of password managers in general: the password manager itself becomes a high-value target and losing it is more damaging.
This is a well-known trade-off: the best security is the one you use, excellent security that you don’t use doesn’t really help.
And here 1Passwort shines: they really care for security, and also give options to change a bit the tradeoff between convenience and security: one can have local vaults that are not synchronized across devices.</p>
<p>Anyway, I went with the family option and so far I am satisfied, I prefer to pay for such a service, incentivize healthy relationship with the customers, time will tell if things will stay so.</p>Fawzi MohamedAfter quite some delay I started to use hey.com, as my email. It immediately looked like an approach I would like: the imbox for important emails, the feed for info you want to see at your own pace, the paper trail for the receiptsMy Backup script2020-10-25T00:00:00+02:002020-10-25T00:00:00+02:00https://kitabi.eu/blog/backup-script<p>Nothing like vacation to do and go back to <a href="../2020-02-05-backup/">my backup routine</a> and improve it a bit.</p>
<p>I mainly put everything in <a href="../../thoughts/do-bk.sh">a script</a>, added the the status and scrub of the <a href="https://www.ixsystems.com/community/attachments/backup_freenas_rsync-txt.21348/">backup_freenas_rsync</a> script by K. Raquel Sanborn.</p>
<p>In the script I do not try to handle all contingencies, simply to be careful, and bail out with a helpful error message if things are strange, assuming that then manual intervention will be required.
Normally I use the <a href="../../thoughts/bk-and-log.sh">following script</a> to run it and keep a log, after having mounted the backup media and started a screen as describe in <a href="../2020-02-05-backup/">my backup routine</a>.</p>
<p>For the record currently a scrub of the backup takes just shy of 12 hours.</p>Fawzi MohamedNothing like vacation to do and go back to my backup routine and improve it a bit.My Backup settings2020-02-05T00:00:00+01:002020-02-05T00:00:00+01:00https://kitabi.eu/blog/backup<h1 id="my-freenas-setup">My Freenas setup</h1>
<p>Having a good network storage to share the files one has at home is a very useful thing.
I have used for several years <a href="https://www.freenas.org">freenas</a>, it is free, open source, built on the top of <a href="http://open-zfs.org/">open zfs</a> arguably one of the first new generation files systems with lots of nice enterprise features: snapshots, checksums for everything, growable volumes, configurable redundancy…
It might be a bit more work, configuration than ready made solutions like synology, but it works well.</p>
<p>I use a very old system, that I assembled 10 years ago: it is an AMD Phenom(tm) II X2 555 (Back edition, 2 CPU 3.2GHz) with 20GB ECC memory on an Asus M4A89GTD PRO USB3.
I considered upgrading it, mainly because a newer system would use much less electricity, power efficiency did increase quite a bit, but every time I looked at it it was more expensive than I expected/was ready to expend to have a safe system with enough ECC RAM.
AMD Zen2 (Ryzen/EPIC2) might have changed that (Intel always had a large premium for ECC RAM).
Anyway, throwing away HW also has an environmental cost, so I am staying with it for now.</p>
<p>Freenas works mostly without issues, I have a pool with raid 1 disks (2x3T & 2x8T), an ssd for the OS.</p>
<h1 id="snapshots">Snapshots</h1>
<p>Against mistakes: deletions incorrect modifications…
I keep ~100 periodic snapshots via the <b>Tasks > Periodic Snapshot Tasks</b>:</p>
<ul>
<li>Every 1 Week for 6 months (~24).</li>
<li>Every 1 day for 2 weeks (~14).</li>
<li>Every 2 hours for 2 days (~24).</li>
<li>Every 15 mins for 10 hours (~40).</li>
</ul>
<p>This way I am able to recover older files.</p>
<h1 id="backup-procedure">Backup Procedure</h1>
<p>Still, in case of disaster an extra backup is needed.
I use a hot swappable SATA Enclosure built into my server tower to perform backups.
This way I can use naked 3.5’’ SATA Hard Disks that I store in storage boxes (I use renkforce storage box for 3.5’’ HDs) to protect them a bit.
Luckily I am still at a level where all my data fit in a single 8T HD.</p>
<p>What I changed recently is the filesystem I use on it, I switched from UFS to ZFS.
This means that my backup can contain snapshots, and is checksummed.
Clearly, if a disk fails the data is lost, but I always keep at least two backups (and I have begun to keep a backup off-site, now that I keep it encrypted.</p>
<p>It is possible to create a pool (<code class="highlighter-rouge">zpool create ...</code> for example <code class="highlighter-rouge">zpool create -m /mnt/Backup -O compression=lz4 Backup</code>), import a pool from previously created pool (<code class="highlighter-rouge">zpool import ...</code>), sync the disk (<code class="highlighter-rouge">sync</code>) from the command line, check the pool (<code class="highlighter-rouge">zpool status [pool]</code>), export a pool (<code class="highlighter-rouge">zpool export [pool]</code>) and remove the disk, check the available pools (<code class="highlighter-rouge">zpool list</code> and initially I did it, but it is cumbersome, and the GUI does not like it.</p>
<p>Finally, I decided to create and import single disk pools from the GUI.
This also makes it easier to create and handle encrypted volumes.</p>
<ul>
<li>
<p>To create a new volume (first time using an HD):
<b>Storage > Pools > Add > Create new pool</b> Choose a name (I use <code class="highlighter-rouge">bk_*</code>) activate Encryption, choose the disk, move it in the VDev and finally <b>Create</b> the pool.
Save the encryption key in a safe place.
I have a directory with a subdirectory for each pool name where I save the corresponding geli.key .
I also write the pool name on the disk.</p>
</li>
<li>
<p>To import an existing volume (all the following times to update a backup):
<b>Storage > Pools > Add > Import existing pool</b> Choose yes decrypt disks, upload the key</p>
</li>
</ul>
<p>I perform the actual backup from the command line, using the shell provided from the gui (I did not open the ssh port).
Thus being able to recover if the connection is lost is crucial.
For this I use the <code class="highlighter-rouge">screen</code> command (<code class="highlighter-rouge">screen -ls</code> to see the running screeens, <code class="highlighter-rouge">screen -l</code> to start a new login screen, <code class="highlighter-rouge">screen -r</code> to recover the connection to a detached screen).</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">if</span> <span class="o">[</span> <span class="nt">-z</span> <span class="s2">"</span><span class="nv">$STY</span><span class="s2">"</span> <span class="o">]</span> <span class="p">;</span> <span class="k">then
</span><span class="nb">echo</span> <span class="s2">"Please run in a screen ( screen -ls)"</span>
<span class="nb">exit </span>1
<span class="k">fi</span>
</code></pre></div></div>
<p>By naming the pool to backup with a name starting with bk_ and containing just letters and underscore I can easily get its name with</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">export </span><span class="nv">bkPool</span><span class="o">=</span><span class="si">$(</span>zpool list | <span class="nb">grep</span> <span class="nt">-E</span> <span class="s1">'^bk_'</span> | <span class="nb">head</span> <span class="nt">-n1</span> | <span class="nb">cut</span> <span class="nt">-d</span> <span class="s2">" "</span> <span class="nt">-f1</span><span class="si">)</span>
</code></pre></div></div>
<p>Whereas the pool to backup is passed as argument, so that in my case</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">export </span><span class="nv">poolToBackup</span><span class="o">=</span>swimmingpool
</code></pre></div></div>
<p>The strategy for the backup is to create a snapshot named exactly as the backup pool with</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">if </span>zfs list <span class="nt">-t</span> snapshot | <span class="nb">grep</span> <span class="k">${</span><span class="nv">poolToBackup</span><span class="k">}</span>@<span class="k">${</span><span class="nv">bkPool</span><span class="k">}</span><span class="se">\ </span> <span class="o">></span>& /dev/null <span class="p">;</span> <span class="k">then
</span><span class="nb">echo</span> <span class="s2">"WARNING a snapshot named </span><span class="k">${</span><span class="nv">bkPool</span><span class="k">}</span><span class="s2"> is already present in </span><span class="k">${</span><span class="nv">poolToBackup</span><span class="k">}</span><span class="s2"> we will not overwrite it, stopping."</span>
<span class="nb">exit </span>1
<span class="k">else
</span>zfs snapshot <span class="nt">-r</span> <span class="k">${</span><span class="nv">poolToBackup</span><span class="k">}</span>@<span class="k">${</span><span class="nv">bkPool</span><span class="k">}</span>
<span class="k">fi</span>
</code></pre></div></div>
<p>If on the backup disk and the disk to backup we have an old backup</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">if </span>zfs list <span class="nt">-t</span> snapshot | <span class="nb">grep</span> <span class="k">${</span><span class="nv">bkPool</span><span class="k">}</span>@<span class="k">${</span><span class="nv">bkPool</span><span class="k">}</span><span class="nt">-last</span><span class="se">\ </span> <span class="o">></span>& /dev/null <span class="o">&&</span> zfs list <span class="nt">-t</span> snapshot | <span class="nb">grep</span> <span class="k">${</span><span class="nv">poolToBackup</span><span class="k">}</span>@<span class="k">${</span><span class="nv">bkPool</span><span class="k">}</span><span class="nt">-last</span><span class="se">\ </span> <span class="o">></span>& /dev/null <span class="p">;</span> <span class="k">then</span>
</code></pre></div></div>
<p>then the backup can be sped up quite a bit by sending just what is missing (the old snapshot is kept on both pool with the name <code class="highlighter-rouge">${bkPool}-last</code>):</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code> zfs send <span class="nt">-Rce</span> <span class="nt">-i</span> <span class="k">${</span><span class="nv">poolToBackup</span><span class="k">}</span>@<span class="k">${</span><span class="nv">bkPool</span><span class="k">}</span><span class="nt">-last</span> <span class="k">${</span><span class="nv">poolToBackup</span><span class="k">}</span>@<span class="k">${</span><span class="nv">bkPool</span><span class="k">}</span> | zfs receive <span class="nt">-F</span> <span class="k">${</span><span class="nv">bkPool</span><span class="k">}</span>
</code></pre></div></div>
<p>(<code class="highlighter-rouge">receive -Fs $bkPool</code> should be resumable but I had issues with it, so I went with a plain <code class="highlighter-rouge">-F</code>).</p>
<p>otherwise, if there is no previous backup, send a full backup from scratch</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">else
</span>zfs send <span class="nt">-Rce</span> <span class="k">${</span><span class="nv">poolToBackup</span><span class="k">}</span>@<span class="k">${</span><span class="nv">bkPool</span><span class="k">}</span> | zfs receive <span class="nt">-F</span> <span class="k">${</span><span class="nv">bkPool</span><span class="k">}</span>
<span class="k">fi</span>
</code></pre></div></div>
<p>After the backup we should have a snapshot on the backup disk</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">if</span> <span class="o">!</span> zfs list <span class="nt">-t</span> snapshot | <span class="nb">grep</span> <span class="k">${</span><span class="nv">bkPool</span><span class="k">}</span>@<span class="k">${</span><span class="nv">bkPool</span><span class="k">}</span><span class="se">\ </span> <span class="o">></span>& /dev/null <span class="p">;</span> <span class="k">then
</span><span class="nb">echo</span> <span class="s2">"ERROR did not find snapshot on backup disk: </span><span class="k">${</span><span class="nv">bkPool</span><span class="k">}</span><span class="s2">@</span><span class="k">${</span><span class="nv">bkPool</span><span class="k">}</span><span class="s2">"</span>
<span class="nb">exit </span>1
<span class="k">fi</span>
</code></pre></div></div>
<p>Remove the old snapshots (to avoid keeping old files too long, and clean up for the next backup)
the snapshots are renamed</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">if </span>zfs list <span class="nt">-t</span> snapshot | <span class="nb">grep</span> <span class="k">${</span><span class="nv">poolToBackup</span><span class="k">}</span>@<span class="k">${</span><span class="nv">bkPool</span><span class="k">}</span><span class="nt">-last</span><span class="se">\ </span> <span class="o">></span>& /dev/null <span class="p">;</span> <span class="k">then
</span>zfs destroy <span class="nt">-r</span> <span class="k">${</span><span class="nv">poolToBackup</span><span class="k">}</span>@<span class="k">${</span><span class="nv">bkPool</span><span class="k">}</span><span class="nt">-last</span>
<span class="k">fi
if </span>zfs list <span class="nt">-t</span> snapshot | <span class="nb">grep</span> <span class="k">${</span><span class="nv">bkPool</span><span class="k">}</span>@<span class="k">${</span><span class="nv">bkPool</span><span class="k">}</span><span class="nt">-last</span><span class="se">\ </span> <span class="o">></span>& /dev/null <span class="p">;</span> <span class="k">then
</span>zfs destroy <span class="nt">-r</span> <span class="k">${</span><span class="nv">bkPool</span><span class="k">}</span>@<span class="k">${</span><span class="nv">bkPool</span><span class="k">}</span><span class="nt">-last</span>
<span class="k">fi</span>
</code></pre></div></div>
<p>Rename the snapshots to prepare the next backup</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>zfs rename -r ${bkPool}@${bkPool} ${bkPool}@${bkPool}-last
zfs rename -r ${poolToBackup}@${bkPool} ${poolToBackup}@${bkPool}-last
</code></pre></div></div>
<p>Ensure all writes are complete</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">sync</span>
</code></pre></div></div>
<p>Now the backup is complete and the disk can be removed using the GUI.</p>
<p class="notice notice-warn">IMPORTANT: make sure that you have the key otherwise the data will be lost</p>
<p><b>Storage > Pools</b>
choose the gear icon and <b>Export/Disconnect</b>.
Then select <i>Delete configuration of shares that used this pool</i> and <i>Confirm export/disconnect</i>.
Finally choose Export/Disconnect and you can remove the disk and store it away.</p>
<p>And let’s hope we do not need it…</p>
<p class="notice"><strong>2020-10-25 Update</strong> Fixed minor mistakes in code (together with a <a href="../2020-10-25-backup-script/">backuo script update</a>)</p>Fawzi MohamedMy Freenas setup Having a good network storage to share the files one has at home is a very useful thing. I have used for several years freenas, it is free, open source, built on the top of open zfs arguably one of the first new generation files systems with lots of nice enterprise features: snapshots, checksums for everything, growable volumes, configurable redundancy… It might be a bit more work, configuration than ready made solutions like synology, but it works well.Comments revisited..2020-01-27T00:00:00+01:002020-01-27T00:00:00+01:00https://kitabi.eu/blog/comments-revisited<h1 id="why">Why?</h1>
<p>I think that an important part of keeping the nice things of the internet, is to have small free spaces, and avoid bringing everything under the control of a few corporations.
That is also one of the reasons that I started my blog again, and I take care of the technical part myself.
Having a static site is about using simple future proof technologies so that it can be kept around with little effort, and migrate to various hosting options.
I think that <a href="https://jekyllb.com">jekyll</a> and the <a href="https://mademistakes.com/work/minimal-mistakes-jekyll-theme/">minimal mistakes jekyll theme</a> worked very well.
They did need some work to set up, but just worked, and allow one to have basically zero data collection/privacy issues.</p>
<p>I also wanted a way to enable some discussion on the site, i.e. comments.
I was a bit <a href="../2019-09-09-comments/">wary of spam problems</a>, but I wanted to see how it would go.
Spam is one of the things that makes an open internet a challenge.
Moderation, and especially moderation at scale <a href="https://www.techdirt.com/blog/?tag=content+moderation+at+scale">is difficult</a>, and one of the reasons to avoid having just a few players doing it: there are bound to have bad decisions, having several players is the best way to let a way out.
Decisions on <a href="https://www.techdirt.com/blog/?tag=eu+copyright+directive">copyright in the EU</a> made it more difficult to give free space to anonymous persons.</p>
<p>Still, I wanted to let a way for people to give feedback.
Unfortunately, spam did become a nuisance, I had always 5-10 spam messages a day, not totally overwhelming, but a big nuisance, especially as I do not want to dedicate much time to this, and a few skipped days quickly add up.
Luckily, others fight this same issue.</p>
<h1 id="solutions">Solutions</h1>
<h2 id="recaptcha">Recaptcha</h2>
<p>Google provides <a href="https://www.google.com/recaptcha">reCAPTCHA</a>, to help identifying humans (and avoid bots).
This is nice, and google does not do it out of pure altruism, it gets annotated images for its AI effort, so it is likely to continue providing it.</p>
<p>I was very happy that my site did basically no tracking, using reCAPTCHA changes that, and I did not want to force it on the bulk of the users that do not add comments, but would possibly be tracked through it by Google.
It is not fully clear when the tracking starts, and I did not want to trust google when not needed.
Thus, I modified the template to hide the form to add a comment, and added an “add a comment” button that shows the form and loads the reCAPTCHA script only when clicked.
This way any reCAPTCHA related tracking cannot start before the use expresses the intention to add a comment.</p>
<p>Doing it, I saw that the template did provide not just siteKey of reCAPTCHA, but also the private secret.
Looking at the staticman code I saw that indeed it requires it.
The situation the situation is not as bad as one might initially think: the secret isn’t in plain text, but is encrypted with staticman’s rsa key.
Still, I saw no point in having to give the encrypted secret in the form, it just makes it easy for a would be exploiter to get it.
So I submitted a patch to avoid sharing it (and be a bit more verbose on the failures/censoring).</p>
<h2 id="akismet">Akismet</h2>
<p>The other approach to avoid spam is to rely on a site that aggregates all posts and tries to then keep up to date filters that block it.
This comes from the fact that a spam message (of at least the links they try to share), have already been attempted on some other site, and thus filtering based on seeing large amounts of posts works.
<a href="http://akismet.com">Akismet.com</a> uses this approach, so I set it up, but I also added log messages describing the blocked posts to be able to evaluate if it works well or not.</p>
<h1 id="conclusions">Conclusions</h1>
<p>After these changes my spam went back to zero, I am very glad of it, and I am glad that I could keep my comments, even if they aren’t really used currently.
The solution worked, but it is still sad that one has to resort to these things to have a working site with comments.
I understand why some communities/forum went underground, removing public/open links to themselves, to keep a community that was working healthy.
Still, I think that there is value in making the effort of keeping things open.
Time will tell…</p>Fawzi MohamedWhy? I think that an important part of keeping the nice things of the internet, is to have small free spaces, and avoid bringing everything under the control of a few corporations. That is also one of the reasons that I started my blog again, and I take care of the technical part myself. Having a static site is about using simple future proof technologies so that it can be kept around with little effort, and migrate to various hosting options. I think that jekyll and the minimal mistakes jekyll theme worked very well. They did need some work to set up, but just worked, and allow one to have basically zero data collection/privacy issues.Meta Info Tools2019-12-22T00:00:00+01:002019-12-22T00:00:00+01:00https://kitabi.eu/blog/meta-info-tools<p>The meta info was at the hearth of the NOMAD Archive, as a way to make the data available.
The NOMAD meta info is an extensible language independent and format agnostic description of atomistic simulation data.</p>
<p>Recently I rewrote the tools to manage the meta info in python 3, improved them and generally tried to take advantage of the lesson learned.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>pip <span class="nb">install </span>meta_into_tools
python <span class="nt">-m</span> meta_info_tools.meta_tools <span class="nt">--help</span>
</code></pre></div></div>
<p>See the <a href="../../nomad/metainfo/">meta info page</a> for more details.</p>Fawzi MohamedThe meta info was at the hearth of the NOMAD Archive, as a way to make the data available. The NOMAD meta info is an extensible language independent and format agnostic description of atomistic simulation data.Directory compare2019-11-20T00:00:00+01:002019-11-20T00:00:00+01:00https://kitabi.eu/blog/directory-compare<h1 id="note-to-self">Note to self</h1>
<p>Quick directory compare:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>rsync <span class="nt">-rlDnv</span> <span class="nt">--delete</span> <inDir>/ <outDir> <span class="c"># using timestamps</span>
rsync <span class="nt">-rlDunv</span> <span class="nt">--delete</span> <inDir>/ <outDir> <span class="c"># skip newer files in outDir</span>
rsync <span class="nt">-rlDInv</span> <span class="nt">--delete</span> <inDir>/ <outDir> <span class="c"># ignoring timestamps (size only)</span>
rsync <span class="nt">-rlDcnv</span> <span class="nt">--delete</span> <inDir>/ <outDir> <span class="c"># using checksum</span>
</code></pre></div></div>
<p>with <code class="highlighter-rouge">-a</code> instead of <code class="highlighter-rouge">-rlD</code> checks also owner, group and permissions</p>
<p>do not forget the trailing slash to refer to its content.</p>Fawzi MohamedNote to self