Note-taking and Knowledge Management

Problem 1: Walled Gardens

I have two to three online services with notes, collections of bookmarks, code snippets and so on. Each of them has an easy way in and a hard way out – means I can add data easily, but to port my data to another service is painful (large JSON blob, archive with HTML files…).

Problem 2: Finding My Stuff

The problem I solved a year ago and the corresponding website I bookmarked, how do I find this again? Go to each of my services and search, hoping I tagged or described it well enough and in the best case the search engine of the service is somehow smart (fuzzy) and reads my mind. What I end up with is a google-search finding solutions again in the wild and rarely hitting my own archive or services.

Solution

The solution comes down to One entry point to find my notes and in the best-case One entry point to record notes and bookmarks. Spreading my data over several services is easy because most services offer an easy way to add bookmarks from the browser or code snippets from a context menu, etc…

My Dream Setup

By now, all boxes are ticked with Obsidian and a new extension (https://github.com/jk-oster/obsidian-search-for-web) that injects Obsidian results into the SERP.

  • [x] Storage is based on text files in my file system, so it's easy to keep that in sync via Dropbox or Nextcloud.
  • [x] Notes and code snippets are saved as markdown files
  • [x] Bookmarks can be added via browser or a service that can sync/backup my bookmarks to Dropbox
  • [x] All markdown files and the bookmark file are getting indexed and every time I search on Google or any other search engine, a browser extension inserts the results from a local search through my files.

My Actual Setup

Structure

I have one global /@docs/ folder in my Dropbox for general notes. That folder includes a variety of markdown files to different topics. The filename usually indicates what's in the files.

  • Logs.md Code snippets and gotchas on how to parse log files
  • CMS.md all kind of information regarding CMS systems.
  • CMS-Wordpress.md or CMS Pico.md information about that specific CMS
  • and so on...

Work Projects are organized in folders, so all projects contains a @docs folder with a markdown file inside, containing notes, ideas, to-do lists.

Usually, there is a YAML header with a [[../programming/Taxonomy.md#Tags|tags]] field.

Software and Services

I access notes with Obsidian, Sublime Text (which is open most of the time anyway) and experimented with Typora, Caret, Mark Text or Zettlr in the past, but Obsidian it is. On my phone I use iA Writer, GTW - Markdown & Text Editor or Drafts 4+.

Copy as Markdown helps me to copy the link to a website or a collection of tabs, as Markdown link. With Obsidian Web exists an excellent tool for adding snippets of text or whole pages as Markdown file. And Obsidian Web Search queries the local Obsidian index when searching on any search engine. (https://www.reddit.com/r/ObsidianMD/comments/zy3wg2/obsidian_chrome_extension_search_your_notes_from/)

Code snippets I'm torn about. I like Github Gist to share code, to fork other gists and so on. For that, I use Lepton. I also wrote a script, that's backing up all my gists to my local hard drive. But I might save them as Markdown as well as described in Markdown and code snippet management - BrettTerpstra.com

Bookmarks I collect with Raindrop. The good thing about this service is, it saves a copy of the page, in case the site goes away (which happens too often) and I connect raindrops.io with Dropbox (via IFTTT), so there is a backup of all bookmarks in my Dropbox (in Markdown).

Search and Discovery

As mentioned before, it's easy to get data in, but hard to get out. The browser is open all the time, that's my entry point for searching. Back in 2011, Google offered Google Desktop, which indexed the hard-drive and displayed the results next to google search results. Unfortunately, that is gone, and I haven't found any suitable alternative (there are a few that I hated).

Recoll

What I have set up right now is a little tool that's based on Xapian indexer called Recoll. There is a Windows version, which works ok. Recoll indexes files (PDF, TXT, mailboxes) and you can run a full text fuzzy search with stemming. Unfortunately, the web interface doesn't work for Windows, so there is no way to get results injected into the Google search results page. But I could see that as a viable option. Or I'm running this command and parse everything myself with PHP.


<?php



exec('recoll.exe')

Public Notes indexed by Google

At the end of the day, I have set up a working system, yet I still end up using google to search for the same stuff over and over again – even starting Recoll seems too much.

The consequence would be to publish my notes online, so they get publicly indexed. That's what I did with this wiki. And as a bonus, I can set up a Programmable Search Engine by Google (former CSE)

Everything 1.5

The best and fastest local file system search is Everything, and their 1.5 alpha version now has Content Indexing capabilities. This could be promising, but for displaying search results of text content, I need context like text snippets.

Obsidian

The Obsidian editor seems to become the go to for note-taking and building a second brain. The indexed search is great and really useful and working. It's giving context with expandable snippets of surrounding text. For my approach, I have to move everything in one vault to have it indexed. That might create the problem of losing focus, but I guess with the approach of Recoll or Everything indexing all my hard drive, I already have that problem.

Thoughts

  1. All Notes In One Folder (I do that for general notes)
  2. Notes Per Project (yep)
  3. One Single File With All Notes (I doubt that's very practical)
  4. Several Large Files About One Topic (that's what I do right now... see above!)
  5. A Mix Of The Above (yeah, whatever works)

See [[Markdown Editors and Note Taking]]

Other Peoples Thoughts

Knowledge base

  • Markdown, plain text based.
  • Optional todo.txt org-mode (didn't work out for me)
  • Bookmark management based on Markdown or be able to export to markdown or plain text (even HTML in a bookmarks.html file)
  • Dropbox, Cloud (Dropbox is still the easiest compared to Gdrive, onedrive and nextcloud)
  • Directory based.

Personal File System

Theoretical concept of how to store files on a computer.

The act of using a computer (in general) for a purpose only materializes as an afterthought if ever. There is rarely a situation where one prepares the computer beforehand, or actively indicates "this is the purpose I'm using the computer now" before actual use. That is the root of the problem of organizing information. Using folders/directories means "folding N different meanings onto one single, rudimentary abstraction" and it's a shame that this is the only reliable primitive people can use to sort their digital information. What is proposed here is the need for a common model of "User Agent" (in the RFC 8890 sense) to interface humans and storage.

Johnny Decimal

A “numerical index system to note folders”

Promnesia

Really good write-up about the problem of how to record and access all movements across the web/research.

https://beepb00p.xyz/promnesia.html

https://github.com/karlicoss/promnesia

https://news.ycombinator.com/item?shownew&id=23668507

Semilattice

Concept of a Knowledge management with Blocks

https://www.semilattice.xyz/

What's out there

Roamresearch

https://roamresearch.com/

DokuWiki

  • PHP based, Flat-file based
  • writes text files with *.txt extension
  • uses its own doku wiki markup
  • markdown extensions are available but not up to date and the integration is basic.

TiddlyWiki

Obsidian (or any Markdown App)

See Markdown Editors and Note Taking

Obsidian seems to be the best option in terms of linking docs and tags.

Notion.so

  • Web service that aims to be everything… especially slow?

Anytype

[[#Notion.so]] offline first clone.

Some Pro's and Con's from the community: https://community.anytype.io/t/first-impressions-after-4-days-of-use/8732

Docsify

Similar to Mdwiki - html file + js = formatted markdown docs

https://docsify.js.org/

Bookmarks

Raindrop.io

  • Propitary cloud service
  • archive snapshot of webpage if possible
  • share groups of bookmarks
  • notes possible.
  • Export and REST API.

Braintool

  • Your data is stored in a continuously synced plain text file.
  • BrainTool uses structured org-mode syntax.
  • The file can be separately edited, backed-up or shared.
  • You can import and export to and from bookmarks or org-mode files.

https://braintool.org

Webcull

  • Bookmarkmanager similar to Raindrop.io
  • not tested yet

https://webcull.com

• #notes, #markdown