#+title: Hyperdrive.el notes
#+TODO: TODO WAITING | DONE CANCELED
#+TODO: PROJECT | COMPLETED CANCELED
#+TODO: DECIDE | DECISION

* Reference

/Information for future reference./

** ~check-declare~ library does not account for symbol shorthands

So, e.g. ~makem~'s ~lint-declare~ rule, which uses ~check-declare~, gives false warnings.

*** TODO File bug report to Emacs about this
:PROPERTIES:
:assignee: Joseph
:END:
:LOGBOOK:
- State "TODO"       from              [2023-11-28 Tue 16:31]
:END:

* Depending on Org

** DECISION We will depend on a minimum version of Org.

We need to depend on a certain minimum version so we can effectively provide functionality related to remote files in a safe way without covering multiple versions of Org that each do so in a slightly different way.

** DECISION We will require Org, period. 

Our vision for hyperdrive involves Org and its functionality being a core part (while not necessarily requiring users to use Org files).  So we will not try to factor Org-related functionality into a separate package.

* Gateway version automatic installation/upgrading

+ Design :: As simple as possible.
  - We do /not/ attempt to automatically detect the latest gateway release on Sourcehut's Web page.
  - We include with ~hyperdrive.el~ the known-latest version, with URL and hash.
  - When we want to release a new version of the gateway, we will then release a new version of ~hyperdrive.el~ with the corresponding information.
  - The client will, upon first use, check the installed version of the gateway and prompt the user to upgrade it if necessary.

** Considerations

+ Gateway :: The status could be:
  - [ ] Gateway may or may not be running
  - [ ] Gateway may or may not have been downloaded at correct version
  - [ ] Gateway may or may not have been downloaded at different version
  - [ ] Gateway may be installed at location in PATH but not the one we expect

** Decisions

+ We don't want to delete or install the gateway anywhere other than ~~/.local/lib/hyperdrive.el~. :: In case the user has installed a different version at a different location, we want to respect that.
+  :: 

** Testing gateway commands

| Should test           | 👍 install | 👍 start | 👍 stop | gateway-in-expected-place | gateway-only-in-path | running-as-subprocess | running-outside-Emacs |
|-----------------------+------------+----------+---------+---------------------------+----------------------+-----------------------+-----------------------|
| no (impossible)       |            |          |         | yes                       | yes                  | yes                   | yes                   |
| no (impossible)       |            |          |         | yes                       | no                   | yes                   | yes                   |
| no (impossible)       |            |          |         | no                        | yes                  | yes                   | yes                   |
| no (impossible)       |            |          |         | no                        | no                   | yes                   | yes                   |
| no (unlikely)         |            |          |         | yes                       | no                   | no                    | yes                   |
| no (unlikely)         |            |          |         | no                        | no                   | no                    | yes                   |
| no (unlikely)         |            |          |         | no                        | no                   | yes                   | no                    |
| no (won't check PATH) |            |          |         | yes                       | yes                  | no                    | yes                   |
| no (won't check PATH) |            |          |         | yes                       | yes                  | yes                   | no                    |
| no (won't check PATH) |            |          |         | yes                       | yes                  | no                    | no                    |
|-----------------------+------------+----------+---------+---------------------------+----------------------+-----------------------+-----------------------|
| yes                   | yes        | yes      | yes     | no                        | no                   | no                    | no                    |
| yes                   | yes        | yes      | yes     | no                        | yes                  | no                    | no                    |
| yes                   | yes        | yes      | yes     | no                        | yes                  | no                    | yes                   |
| yes                   | yes        | yes      | yes     | no                        | yes                  | yes                   | no                    |
| yes                   | yes        | yes      | yes     | yes                       | no                   | no                    | no                    |
| yes                   | yes        | yes      | yes     | yes                       | no                   | yes                   | no                    |
* PROJECT Petnames
:PROPERTIES:
:ID:       e5b0c0f1-7ebc-4e8c-9712-cd2cd4a055ce
:END:
:LOGBOOK:
- State "PROJECT"    from              [2023-03-23 Thu 17:59]
:END:

** Design
:PROPERTIES:
:ID:       a7d0005c-a219-4cae-a32f-6bf7b2e7a8fb
:END:

+ [ ] Add appropriate sections of these design notes to the project docs.

+ *Terminology*
  + Nickname :: What we have been calling "public names" (the string loaded from the ~/.well-known/host-meta~ file) is now going to be called the "nickname."
  + Petname :: given by each local user to one and only one hyperdrive.
  + Public key :: the public key address of the hyperdrive.
  + Seed :: the value from which the public key is generated by hyper-gateway.
  + Domains :: One or more DNS names that are configured with DNSLink records to point to a hyperdrive.

+ *Internals*
  + [X] Add a ~petname~ field to the ~hyperdrive~ struct.
  + [X] Add /petnames/ support to ~hyperdrive--format-host~.
  + [X] Add petname to ~hyperdrive-preferred-formats~.
  + [X] Add petname face.
  + [X] Add faces for all types of hostname formats.
  + [X] Change ~public-name~ to ~nickname~ everywhere.
  + [X] Consider adding ~:with-label t~ to everywhere ~hyperdrive--format-host~ is called.
  + [X] Add a command to set a petname.

+ *UI*
  + The user must be able to clearly distinguish between petnames, nicknames, and public keys.  Each type should be visually distinctive and unambiguous.

  + [X] Empty string as a petname should be treated as an unset petname.
  + [ ] Have an option to automatically prompt the user to set a petname for newly visited hyperdrives.
  + [ ] Have an option to automatically prompt the user to set a nickname for newly visited hyperdrives.

  + *Considerations*
    + Privacy :: If /petnames/ leak (e.g. when sharing links, or in a file in ~/.emacs.d that gets committed to git and published to a public repo), it could reveal personal information (like what someone thinks of another person).
      + Is this a problem?  Is it okay for them to "leak"?  Should they even be considered private?

  + Where the /petname/ should be displayed:
    + [X] Directory header
    + [X] Buffer name
    + [X] describe-hyperdrive buffer
    + [X] Hyperdrive completion
    + [ ] When naming a link
    + [ ] When naming a bookmark

  + *Flows*

    + Creating a new hyperdrive ::
      1. Prompt the user for the /seed/.
      2. Prompt the user for the /petname/, using the /seed/ as the default value.
      3. Show the new hyperdrive's directory buffer.

    + Assigning petnames to existing hyperdrives ::
      1. Call a command to set the petname of a hyperdrive (defaulting to the one relating to the current buffer).
      2. Enter the new petname.
      3. Verify that it the petname isn't already assigned to another hyperdrive.
      4. Set the new petname in the struct.
      5. Persist the struct?

    + Removing petnames from hyperdrives ::
      1. Document that setting a petname to the empty string is considered unsetting the petname.

* PROJECT Versioning
:PROPERTIES:
:ID:       52bbabe0-dba6-4912-8f4d-0fa8b34babfa
:END:
:LOGBOOK:
- State "PROJECT"    from              [2023-04-04 Tue 14:27]
:END:

+ https://todo.sr.ht/~ushin/ushin/1

** Design

+ *Concepts*
  + Special folder :: Folder at root level of hyperdrive that is not returned in directory listing.  Named ~$~.  Requests for old versions are sent to ~$/version/~.  See: [[https://github.com/RangerMauve/hypercore-fetch#fetchhypernameversionversion_numberexampletxt][hypercore-fetch docs]].
  + ETag :: Version numbers are sent in responses in the ~ETag~ header.
    + NOTE: hyper-gateway's behavior in this regard may change soon.  See: [[https://github.com/RangerMauve/hypercore-fetch/issues/65][#65]].
    + Requests for directories always return the current version of the hyperdrive, unless the version is specified in the request.
  + URL :: URLs pointing at specific versions of a file will have the version number in the URL.
    + This is necessary so that URLs can link to specific versions of a file (i.e. there is no other way to specify the version in a request that would make this possible).

+ *Internals*
  + ~hyperdrive-entry~ already has an ~etag~ slot.
  + If a ~hyperdrive-entry-equalp~ function is needed in the future, it should compare the ~etag~ slot.
  + When viewing old versions of a directory or file, the buffer name should include the version number.
  + When an old version is not found, hyperdrive.el should not try to load

+ *UI*

  + *Flows*

    + Time machine :: Allowing the user to "go back in time" (and forward) and view various versions of entries.

      + ~hyperdrive-open-previous-version~ :: Show the previous version of the file.
        + [ ] If the current version of the file is the earliest, show a useful error.

      + ~hyperdrive-open-next-version~ :: Show the next version of the file.
        + *NOTE*: The hyper-gateway API doesn't seem to currently offer a way to retrieve the /next/ version of an entry, given an entry at a version.  Iterating backward from the latest version would be impractical.

        + [ ] If that version is also the current version, it should act as if the user requested the file without specifying a version.  See [[https://github.com/RangerMauve/hypercore-fetch/issues/72]].


** Examples

/This example is written at a point in time at which hyper-gateway returns the *previous* version of the file as the =ETag=.  When https://github.com/RangerMauve/hypercore-fetch/issues/65 is done, this example will match reality./

1. File A created at version 10.
2. File A modified at version 15.
3. File A modified at version 20.
4. Current version of hyperdrive is 25.
5. Request for hyper://PUBLIC-KEY/file-a returns an =ETag= of 20.
6. Request for hyper://PUBLIC-KEY/$/version/20/file-a returns an =ETag= of 20.
7. Request for hyper://PUBLIC-KEY/$/version/19/file-a returns an =ETag= of 15.
8. Request for hyper://PUBLIC-KEY/$/version/14/file-a returns an =ETag= of 10.
9. Request for hyper://PUBLIC-KEY/$/version/9/file-a returns HTTP 404 with =ETag= 0.


* Response matrix
:PROPERTIES:
:ID:       3381f0a8-7d75-4e1e-90e4-f4d9adf6015e
:END:

Currently observed behavior.

| Request for                                           | Returns         | # |
|-------------------------------------------------------+-----------------+---|
| URL to valid hyperdrive without content               | Etag == 1       |   |
| URL to unknown (i.e. network-inaccessible) hyperdrive | Etag == 1       | ! |
| URL to valid hyperdrive that has ever had content     | Etag > 1        |   |
| URL to valid hyperdrive directory but invalid file    | HTTP 404        |   |
| URL with too-short public key                         | HTTP 500        |   |

** Ideals?
:PROPERTIES:
:ID:       3f2fa76c-0a35-4b5f-bc8e-91a2d55c6db2
:END:

What we'd ideally like the behavior to be.

| Request for                           | HTTP            | Etag |
|---------------------------------------+-----------------+------|
| Obviously malformed URLs              | 400 Bad Request | N/A  |
| [[id:5951fa2d-6f95-41e0-9fd0-066ae06dbc3e][Unknown hyperdrive]]                    | 404 Not Found   | N/A  |
| [[id:aeb2dcd3-ec0f-42a6-aec2-9a7ba6eb163f][Known hyperdrive (never had content)]]  | 204 No Content  | N/A  |
| Known hyperdrive (has or had content) | 200 OK          | >= 1 |

To distinguish whether peers are available, ideally we would use another header, something like ~X-Hyperdrive-Peers~.

** Glossary

*** Unknown hyperdrive
:PROPERTIES:
:ID:       5951fa2d-6f95-41e0-9fd0-066ae06dbc3e
:END:

An unknown hyperdrive may or may not exist.  We don't know whether it does.  We have never received any information about it.

*** Known hyperdrive (never had content)
:PROPERTIES:
:ID:       aeb2dcd3-ec0f-42a6-aec2-9a7ba6eb163f
:END:

A hyperdrive that we know exists, and we know it is empty, and it has never yet had any content.  Possibly created by us.

* File interaction matrix

What Emacs does:

| Buffer backed by | Command      | buffer-file-name                        | buffer-name             |
|------------------+--------------+-----------------------------------------+-------------------------|
| nothing          | write-buffer | prompts for and changes to new filename | changes to new basename |
| nothing          | save-buffer  | prompts for and changes to new filename | changes to new basename |
| file on disk     | write-buffer | prompts for and changes to new filename | changes to new basename |
| file on disk     | save-buffer  | does not change                         | does not change         |

What hyperdrive does:

| Buffer backed by             | Command                 | Correct behavior? |
|------------------------------+-------------------------+-------------------|
| nothing                      | hyperdrive-write-buffer | yes               |
| nothing                      | hyperdrive-save-buffer  | yes               |
| file on disk                 | hyperdrive-write-buffer | yes               |
| file on disk                 | hyperdrive-save-buffer  | yes               |
| file on writable hyperdrive  | hyperdrive-write-buffer | yes               |
| file on writable hyperdrive  | hyperdrive-save-buffer  | yes               |
| file on read-only hyperdrive | hyperdrive-save-buffer  | yes               |
| file on read-only hyperdrive | hyperdrive-write-buffer | yes               |

** Overwriting existing Hyperdrive files

| Hyperdrive URL status | Command      | Should                      | Correct behavior? |
|-----------------------+--------------+-----------------------------+-------------------|
| Already exists        | write-buffer | Prompt to overwrite         |                   |
| Already exists        | save-buffer  | Overwrite without prompting |                   |

* DONE Conference presentation
:LOGBOOK:
- State "DONE"       from "PROJECT"    [2023-04-04 Tue 13:33] \\
  GREAT SUCCESS!
:END:

** DONE Screencast
DEADLINE: <2023-03-09 Thu>
:LOGBOOK:
- State "DONE"       from "PROJECT"    [2023-04-04 Tue 13:33]
- State "PROJECT"    from              [2023-02-28 Tue 18:13]
:END:

*** Issues to solve before screencast

+ [X] [[https://todo.sr.ht/~ushin/ushin/21][~ushin/ushin#21: Store link with search options in `hyperdrive-store-link` — sourcehut todo]]
+ [X] [[https://todo.sr.ht/~ushin/ushin/17][~ushin/ushin#17: Handle uri-encoding — sourcehut todo]] (maybe not absolutely necessary for screencast)

* Peer discovery (swarming)

** How to connect to a peer

~PUBLIC-KEY~ must be writable on both machines.

1. Alice ~GET hyper://PUBLIC-KEY/$/extensions/foo~
2. Alice ~POST hyper://PUBLIC-KEY/$/extensions/foo~ with body string
3. Send ~GET hyper://PUBLIC-KEY/$/extensions/~ to verify that the ~foo~
   extension is being advertised.
4. After Alice has ~GET hyper://PUBLIC-KEY/$/extensions/foo~

** How hyperdrive.el will use extension messages

Advertise on one or more topics. On connection with another peer, send
an extension message with a newline-delimited list of hyperdrive
public keys which are known to be relevant to that topic.

** Questions

*** TODO What difference should it make to send requests to different public keys (writable or non-writable)?

Currently, ~GET hyper://BAZ/$/extensions/foo~ and ~GET
hyper://BAR/$/extensions/foo~ give different results. (or does only
make a difference between writable and non-writable public keys?)

*** TODO Reverse DNS namespacing

Regarding the way we use extension messages, would it make sense to
use reverse DNS namespacing for extension message topics, e.g.,
.org.ushin.hyperdrive.topics.foo

*** TODO request.text

In hypercore-fetch's broadcastExtension() method, await request.text
is used, but in extensionToPeer(), await request.arrayBuffer is used.
Should we always use request.text?

*** TODO Return response headers immediately
~GET hyper://PUBLIC-KEY/$/extensions/~ with Accept: text/event-stream
header only returns response headers after first event is received.
Should it return response headers first?

*** TODO non-advertised topics appear in list

If already connected to another peer, ~GET
hyper://PUBLIC-KEY/$/extensions/not-advertising-this~ lists that peer
even when that peer is not advertising the topic ~not-advertising-this~.

*** TODO How to tell the gateway that you're no longer interested in a topic?

*** TODO We saw a "peer-remove" event when the gateway shut down, but no "peer-open".

*** TODO What topic do we have in common?

How to tell which extension caused a peer-open or peer-remove event?

* Org links 

+ [[file:tests/org links.org][org links.org]]

** Design

1. Basically, we want to always store a full ~hyper://...~ URL when the user calls ~org-store-link~.
2. Then, when the user calls ~org-insert-link~, we decide based on context (and maybe also on user input) what kind of a link to insert.

*** Terminology

**** Hyperdrive Org links :: Links to hyperdrive files/directories that are valid within Org documents.

***** With protocol prefix

    This link type or may not be surrounded by brackets. It may or may
    not contain a search option. Path and search option must be
    URL-encoded and separated by a decoded ~#~.

    * No search option :: e.g. ~hyper://deadbeef/foo/bar%20quux.org~, which decodes to ~hyper://deadbeef/foo/bar quux.org~

    * ~CUSTOM_ID~ :: e.g.
      ~hyper://deadbeef/foo/bar%20quux.org#%3A%3A%23baz%20zot~, which decodes to ~hyper://deadbeef/foo/bar quux.org#::#baz zot~

    * Heading text search option :: With or without ~*~ (actually [[elisp:(rx "*" (0+ space))]]) prefix, e.g.

      - ~hyper://deadbeef/foo/bar%20quux.org#%3A%3A%2AHeading%20A~, which decodes to ~hyper://deadbeef/foo/bar quux.org#::*Heading A~
      - ~hyper://deadbeef/foo/bar%20quux.org#%3A%3A%2A%20%20Heading%20A~, which decodes to ~hyper://deadbeef/foo/bar quux.org#::*  Heading A~
      - ~hyper://deadbeef/foo/bar%20quux.org#%3A%3AHeading%20A~, which decodes to ~hyper://deadbeef/foo/bar quux.org#::Heading A~

***** Without protocol prefix

    This link type must be surrounded by brackets.  It has no
    URL-encoding in any part. It may or may not contain a path:

    + With path :: A link pointing to a file at a path, starting with ~/~ or ~.~, with or without search option:

      - No search option :: ~[[/foo/bar quux.org]]~

      - ~CUSTOM_ID~ :: e.g. ~[[/foo/bar quux.org::#CUSTOM_ID]]~

      - Heading text search option :: With or without ~*~ (actually [[elisp:(rx "*" (0+ space))]]) prefix, e.g.

        + ~[[/foo/bar quux.org::*Heading A]]~
        + ~[[/foo/bar quux.org::*  Heading A]]~
        + ~[[/foo/bar quux.org::Heading A]]~

    + Without path :: A link pointing to a heading in the same file with search option alone:

      - ~CUSTOM_ID~ :: e.g. ~#CUSTOM_ID~

      - Heading text search option :: With or without ~*~ (actually [[elisp:(rx "*" (0+ space))]]) prefix, e.g.

        + ~*Heading A~
        + ~*  Heading A~
        + ~Heading A~

*** Limitations

+ Because hyperdrive buffers are not considered file-backed by Emacs, and Org refuses to follow ~ID~-based links in non-file-backed buffers, in a hyperdrive-backed Org buffer, ~ID~-based links /will not/ be followable.  This is a limitation of ~org-id.el~, and it seems unlikely that we would be able to overcome it (unless and until we support hyperdrive using TRAMP's infrastructure).

**** Org does not distinguish between ~file://~-prefixed and non-prefixed paths/URLs

In Org, these two paths/URLs are equivalent:

- ~file://./tmp/setupfile.org~
- ~./tmp/setupfile.org~

Therefore, when opening these two types of links, we should treat them
the same way: as both pointing to files on the same filesystem as the
document in which they are written, i.e.  ~file:~ and
protocol-prefix-less links within a hyperdrive file should be treated
as links to files within that hyperdrive--never as links to files on
the local filesystem.  (This is akin to connecting to a remote system
via TRAMP and viewing Org files that way.)

However, as described in [[*Without protocol prefix][Without protocol prefix]], when inserting links
to files within the same hyperdrive, we optionally insert a
non-prefixed path, but never a ~file://~-prefixed URL.

*** Use cases

+ We want users to be able to take one or more Org files and have links between them continue working regardless of whether the files are on a hyperdrive or not.

*** Storing links

+ All examples below assume point is on a heading titled ~Heading A~.

**** To a heading with a ~CUSTOM_ID~

Assuming the ~CUSTOM_ID~ is ~heading-a~:

+ We'll follow Org's example by storing two links:
  + ~hyper://public-key/foo.org#heading-a~
  + ~hyper://public-key/foo.org#::*Heading A~

**** To a heading with only an ~ID~

+ Because of the [[*Limitations][Limitations]], if the user stores a link to a heading that does not have a ~CUSTOM_ID~ but only has an ~ID~, we ignore the ~ID~.
+ We store: [[hyper://public-key/foo.org#::*Heading A]]

**** To a heading with neither ~CUSTOM_ID~ nor ~ID~

+ We store: [[hyper://public-key/foo.org#::*Heading A]]
+ Internally, the entry's ~etc~ map's ~target~ key will have the value ~"::*Heading A"~.

*** Inserting links

+ *NOTE*: After several attempts at using ~:override~ advice, integrating, and modifying ~org-insert-link~, we concluded that those approaches are impractical.  Instead, we'll use ~:after~ advice to allow ~org-insert-link~ to behave normally, and then read the link it inserted and modify it as appropriate for the hyperdrive-backed buffer.

+ We first distinguish between whether the buffer we're inserting the link into is within the same hyperdrive, or not.

**** Inserting into a buffer that is in the same hyperdrive

+ Like Org mode, we'll use a relative link.

**** Inserting into a buffer that is not in the same hyperdrive

**** Inserting full ~hyper://~ -prefixed links

+ These links may point only to files, or headings in files which have ~CUSTOM_ID~ properties--not headings which have ~ID~ properties, nor by using ~:search-option~-style heading matches.
+ Like in Org, a link using a ~#~-prefixed "fragment" may point only to a heading which has a ~CUSTOM_ID~ property; the ~ID~ property is not considered, nor is heading text.

+ This type of link may point to:
  * A file (i.e. not to a heading within a file)
  * A heading within a file, by:
    * ~CUSTOM_ID~ (by placing the ID in the URL fragment, i.e. ~hyper://.../foo.org#CUSTOM_ID~ )
    * Heading text (by placing the Org search option in the URL fragment, URL-encoded, i.e. ~hyper://.../foo.org/#%3A%3A%2AHeading%20A~ to encode the search option ~::*Heading A~).
    # TODO: Consider also supporting ~ID~ properties.

**** Inserting relative links

+ 

* Markdown links

** Angle-bracketed links (i.e. ~<URL>~)

~markdown-follow-link-at-point~ does not allow the user to follow links in angle brackets, so we don't support them.

** Path encoding

Depending on the resolution of [[https://github.com/jrblevin/markdown-mode/issues/805][Should filename links be URI-decoded? · Issue #805 · jrblevin/markdown-mode · GitHub]], we may want to URL-decode paths.  For now, we won't.

** Targets

Targets are not currently implemented.

* UI

** Menu bar

*** DECISION Don't show status indicators in menu bar

Since the menu bar's items can't be updated while the menu is displayed, showing an indicator could be misleading to the user, since it could become outdated while the menu is displayed (e.g. while starting the gateway).

So we won't show such indicators in the menu bar.  Instead, the Transient ~hyperdrive-menu~ UI is a suitable "dashboard" for such indicators, because the Transient can be refreshed while it's displayed.

** DECISION No uninstall command

+ [X] Should we add an uninstall command? :: Probably not, due to issues with potentially shared data and data loss.  Probably sufficient to document the file/directory locations in a manual section and allow the user to delete them if desired.

