Notetaking and Knowledge Management

Problem 1: Walled Gardens

I have two to three online services with notes, collections of bookmarks, code snippets and so on. Each of them has an easy way in and a hard way out - means I can add data easily, but to port my data to another service is painful (large JSON blob, archive with html files...).

Problem 2: Finding My Stuff

The problem I solved a year ago and the corrosponding website I bookmarked, how do I find this again? Go to each of my services and search, hoping I tagged or described it well enough and in the best case the search engine of the service is somehow smart (fuzzy) and reads my mind. What I end up with is a google-search finding solutions again in the wild and rarely hitting my own archive or services.

Solution

The solution comes down to One entry point to find my stuff and in the best case One entry point to record notes and bookmarks. Spreading my data over several services is easy, because most services offer an easy way to add bookmarks from the browser or code snippets from a context menu etc..

My Dream Setup

  • Storage is based on text files in my filesystem, so it's easy to keep that in sync via Dropbox or Nextcloud.
  • Notes and code snippets are saved as markdown files
  • Bookmarks can be added via browser or a service that is able to sync/backup my bookmarks to dropbox
  • All markdown files and the bookmark file are getting indexed and every time I search on google or any other search engine, a browser extensions inserts the results from a local search through my files.

My Actual Setup

Structure

I have one global /@docs/ folder in my Dropbox for general notes. That folder includes a variety of markdown files to different topics. The filename usually indicates what's in the files.

  • Logs.md Codesnippets and gotchas on how to parse logfiles
  • CMS.md all kind of information regarding CMS systems.
  • CMS-Wordpress.md or CMS Pico.md information about that specific CMS
  • and so on...

Projects are organized in folders, so every projects contains a @docs folder with a markdown file inside, containing notes, ideas, todo lists.

Most of the time there is a YAML header with a tags field.

Software and Services

I access notes with Obsidian, Sublime Text (which is open most of the time anyway) and experimented with Typora, Caret, Mark Text or Zettlr (I haven't settled on one yet, each of those editors has some unique features and lack others) on on my phone I use iA Writer, GTW - Markdown & Text Editoror Drafts 4+.

Copy as Markdown helps me to copy the link of a website, as markdown link.

Code snippets I'm torn about. I like Github Gist to share code, to fork other gists and so on. For that I use Lepton. I also wrote a script, that's backing up all my gists to my local hard drive. But I might save them as markdown as well as described here

Bookmarks I collect with Raindrop. The good thing about this service is, it saves a copy of the page, in case the site goes away (which happens too often) and I connect raindrops.io with Dropbox (via IFTTT), so there is a backup of all bookmarks in my Dropbox.

Search and Discovery

As mentioned before, it's easy to get data in, but hard to get out. The browser is open all the time, so I see that as my entry point for searching. Back in 2011, Google offered Google Desktop which indexed the hard-drive and displayed the results next to google search results. Unfortunately that is gone and I haven't found any good alternative.

Recoll

What I have setup right now is a little tool that's based on Xapian indexer called Recoll. There is a Windows version, which works ok. Recoll indexes files (pdf, txt, mailboxes) and you can run a fulltext fuzzy search with stemming. Unfortunately, the web interface doesn't work for Windows, so there is no way to get results injected into the google search results page. But I could see that as an viable option. Or I'm running this command and parse everything myself with php.


<?php

exec('recoll.exe')

Public Notes indexed by Google

At the end of the day, I have set up a working system, yet I still end up using google to search for the same stuff over and over again - even starting Recoll seems too much.

The consequence would be to publish my notes so they get publicly indexed. That's what I did with this wiki.

Everything 1.5

The best local filesystem search is Everything, and their 1.5 alpha version now has Content Indexing. This could be promising, but for displaying search results of text content, you need context, meaning text snippets.

Obsidian

The Obsidian editor seems to become the go to for note taking and building a second brain. The indexed search is great and really useful and working. For my approach I have to move everything in one vault to have it indexed. That might create the problem of losing focus, but I guess with the approach of Recoll or Everything indexing all my harddrive, I already have that problem.

Thoughts

  1. All Notes In One Folder (I do that for general notes)
  2. Notes Per Project (yep)
  3. One Single File With All Notes (I don't think that's very practical)
  4. Several Large Files About One Topic (that's what I do right now... see above!)
  5. A Mix Of The Above (yeah, whatever works)

See [[Markdown Editors and Note Taking]]

Other Peoples Thoughts

Knowledgebase

  • Markdown, plain text based.
  • Optional todo.txt org-mode
  • Bookmark management based on markdown or be able to export to markdown or plain text (even html in a bookmarks.html file)
  • dropbox, cloud
  • directory based

Personal File System

Theoretical concept of how to store files on a computer.

The act of using a computer (in general) for a purpose only materializes as an afterthought if ever. There is rarely a situation where one prepares the computer beforehand, or actively indicates "this is the purpose I'm using the computer now" before actual use. That is the root of the problem of organizing information. Using folders/directories means "folding N different meanings onto one single, rudimentary abstraction" and it's a shame that this is the only reliable primitive people can use to sort their digital information. What is proposed here is the need for a common model of "User Agent" (in the RFC 8890 sense) to interface humans and storage.

Promnesia

Really good writeup about the problem of how to record and access all movements across the web/research.

https://beepb00p.xyz/promnesia.html

https://github.com/karlicoss/promnesia

https://news.ycombinator.com/item?shownew&id=23668507

Semilattice

Concept of a Knowledgemanagement with Blocks

https://www.semilattice.xyz/

What's out there

Roamresearch

https://roamresearch.com/

DokuWiki

  • PHP based, Flat-file based
  • writes text files with *.txt extension
  • uses its own doku wiki markup
  • markdown extensions are available but not up to date and the integration is basic.

TiddlyWiki

Obsidian (or any Markdown App)

See Markdown Editors and Note Taking

Obsidian seems to be the best option in terms of linking docs and tags.

Notion.so

  • Webservice that aims to be everything...

Docsify

Similar to Mdwiki - html file + js = formatted markdown docs

https://docsify.js.org/