tumelune

a program to browse and search in mastodon/activitystreams and twitter archives

it can import a wide range of archive formats for both, merge them into a single database that you can browse with a web interface, and run precise queries over the whole set.

installation & usage

downloads for tumelune v0.17.0: (downloads archive)

• builds for windows: 64-bit exe, 32-bit exe
• builds for linux: 64-bit bin, 32-bit bin
• source code zip

once started, it will open your default browser or give you an url like http://127.0.0.1:29804/

on windows, just run the binary and leave it wherever you want.
on linux, you may run it with --install to copy it along with a .desktop file and icon in your ~/.local/ directory (or as set in XDG variables).

⚠ starting from v0.17, the database should be upgradable with no data loss.
however, always keep your archives separately.

command line overview

# the main interface is the web ui
  tumelune serve --listen 127.0.0.1:29804 --open
# or just
  tumelune

# some operations are available with a CLI:
  tumelune --help

# import an archive (faster with direct file access)
  tumelune import archive.tar.gz
  tumelune import archive.zip

# cli search (it's not great but it exists)
  tumelune search cats
# it supports the same query language as the web form, and returns JSON rows.
# maybe you'd like to pipe it into jq or a script

# import a whole folder maybe?
  for f in my-social-archives/*; do tumelune import $f; done

building and installing from source

# it needs rustc/cargo.
# you can install it with os packages:
#    arch: pacman -S rust
#    deb:  apt install cargo
# or through rustup:
#    url:  https://rustup.rs/
#    arch: pacman -S rustup
# with
#    rustup toolchain install stable

# this will build, run, and open your default web browser to it
cargo run --release

# this copies the binary to ~/.local/bin/ and and places
# a .desktop file and icon in ~/.local/share, or appropriate XDG variables
cargo run --release -- install

supported archive types

• twitter:

  • early 2010-era tweets ZIP with CSV and JS files by month (no images)
  • late 2010-era data export ZIP with JS data and media
  • 2020-era data export with a Your archive.html file at the root and JS data
  • CSV-only archives (unsure if they ever existed)

• mastodon-style activitypub archive: (outbox.json + media_attachments/ folder)

  • mastodon TAR.GZ (old) & ZIP
  • mastodon-style slurp archive compressed into a ZIP (tested with GoToSocial)

some technical notes

• the 'main interface' is a web application served by a rust package.
  it's meant to run locally and avoid strict dependencies on a gui library without being limited by a command line interface. archives are uploaded to the backend over http or read from a local file directly.
• the rust back end unpacks, stores, indexes data archives, and serves it with search and pagination.
• the web front end queries that and displays a result list.
  it's written at the interection of plainest and messiest javascript.
  i have never heard of this 'npm' and the whole thing has been written as organically as possible with a short term memory of about ten seconds
• key words from posts are extracted and indexed (in rust, with tantivy), but the source material is stored as is and parsed on display (in javascript). attachments are extracted and stored as individual files
• the two sides could be merged in wasm but dependencies arent ready for it yet.
  so it looks and works like web 2.0 in a complicated box and i find that neat.
• it does not bundle a browser because that makes builds and distribution a lot more complicated and brittle and demanding in resources. plus you most likely already have one or three good browsers around. maybe some day i will like tauri
• you could host a public instance too if you wanted. but everyone is sharing write access to the same index.

your database will be stored in:

• the TL_DB_PATH env variable or --db-path argument if set
• (linux) $XDG_DATA_HOME/tumelune or $HOME/.local/share/tumelune/
• (windows) your user's AppData\\Local\\tumelune\\
• (old versions, linux) $XDG_CONFIG_HOME/tumelune or $HOME/.config/tumelune
• (old versions, windows) your user's AppData\\Roaming\\tumelune\\

license & contributing

i hate software licenses. i don't want to pick one.
this is just my little personal project that i hope will be helpful to a few people.
feel free to download it, share it, modify it, and to send me your patches if you think they are helpful.
if you do modify and distribute it, alter the name, icon, and header canvas, at least a bit, and keep a link to the original project.
if it is ever brung up in a court of law i am allowed to eat you