dPay

dPack Core

Features

  • High-level glue for common dPack and dDrive modules.
  • Sane defaults and consistent management of storage & secret keys across applications, using @dpack/storage.
  • Easily connect to the dPack network, using flock-revelation
  • Import files from the file system, using dPack Mirror
  • Serve dPacks over http with http
  • Access APIs to lower level modules with a single require!

dApp Support

Many of our dependencies work in the browser, but dpack-core is tailored for file system applications. See dpackJS if you want to build browser-friendly dWeb dApps.

Example

To send files via dPack:

  1. Tell dpack-core where the files are.
  2. Import the files.
  3. Share the files on the dWeb network! (And share the link)
var DPack = require('@dpack/core')

// 1. My files are in /dpack-example/cancer-analysis
DPack('/dpack-example/cancer-analysis', function (err, dpack) {
  if (err) throw err

  // 2. Import the files
  dpack.importFiles()

  // 3. Share the files on the network!
  dpack.joinNetwork()
  // (And share the link)
  console.log('My dPack link is: dweb://', dpack.key.toString('hex'))
})

These files are now available to share over the dWeb network via the key printed in the console.

To download the files, you can make another dpack-core instance in a different folder. This time we also have three steps:

  1. Tell dPack where I want to download the files.
  2. Tell dPack what the link is.
  3. Join the dWeb network and download the dPack!
var DPack = require('@dpack/core')

// 1. Tell DPack where to download the files
DPack('/download/cat-analysis', {
  // 2. Tell DPack what link I want
  key: '' // (a 64 character hash from above)
}, function (err, dpack) {
  if (err) throw err

  // 3. Join the network & download (files are automatically downloaded)
  dpack.joinNetwork()
})

Thats it! By default, all files are automatically downloaded when you connect to the other users.

Usage

All dpack-core applications have a similar structure around three main elements:

  1. Storage - where the files and metadata are stored.
  2. Network - connecting to other users to upload or download data.
  3. Adding Files - adding files from the file system to the dDrive vault.

We'll go through what these are for and a few of the common usages of each element.

Storage

Every dPack vault has storage, this is the required first argument for dpack-core. By default, we use @dpack/storage which stores the secret key in ~/.dpack/ and the rest of the ddata in dir/.dpack. Other common options are:

  • Persistent storage: Stored files in /dpack-dir and metadata in dpack-dir/.dpack by passing /dpack-dir as the first argument.
  • Temporary Storage: Use the temp: true option to keep metadata stored in memory.
// Permanent Storage
DPack('/dpack-dir', function (err, dpack) {
  // Do DPack Stuff
})

// Temporary Storage
DPack('/dpack-dir', {temp: true}, function (err, dpack) {
  // Do DPack Stuff
})

Both of these will import files from /dpack-dir when doing dpack.importFiles() but only the first will make a .dpack folder and keep the metadata on disk.

The storage argument can also be passed through to dDrive for more advanced storage use cases.

Network

DPack is all about the network! You'll almost always want to join the network right after you create your DPack:

DPack('/dpack-dir', function (err, dpack) {
  dpack.joinNetwork()
  dpack.network.on('connection', function () {
    console.log('I connected to someone!')
  })
})

Downloading Files

Remember, if you are downloading - metadata and file downloads will happen automatically once you join the network!

dPack runs on a peer to peer network, sometimes there may not be anyone online for a particular key. You can make your application more user friendly by using the callback in joinNetwork:

// Downloading  with joinNetwork callback
DPack('/dpack-dir', {key: ''}, function (err, dpack) {
  dpack.joinNetwork(function (err) {
    if (err) throw err

    // After the first round of network checks, the callback is called
    // If no one is online, you can exit and let the user know.
    if (!dpack.network.connected || !dpack.network.connecting) {
      console.error('No users currently online for that key.')
      process.exit(1)
    }
  })
})
Download on Demand

If you want to control what files and metadata(dpack.json) are downloaded, you can use the thin option:

// Downloading  with thin option
DPack('/dpack-dir', {key: '', thin: true}, function (err, dpack) {
  dpack.joinNetwork()

  // Manually download files via the dDrive API:
  dpack.vault.readFile('/cat-locations.txt', function (err, content) {
    console.log(content) // prints cat-locations.txt file!
  })
})

dPack will only download metadata and content for the parts you request with thin mode!

Importing Files

There are many ways to get files imported into an vault! dPack node provides a few basic methods. If you need more advanced imports, you can use the vault.createWriteStream() methods directly.

By default, just call dpack.importFiles() to import from the directory you initialized with. You can watch that folder for changes by setting the watch option:

DPack('/my-data', function (err, dpack) {
  if (err) throw err

  var progress = dpack.importFiles({watch: true}) // with watch: true, there is no callback
  progress.on('put', function (src, dest) {
    console.log('Importing ', src.name, ' into vault')
  })
})

You can also import from another directory:

DPack('/my-data', function (err, dpack) {
  if (err) throw err

  dpack.importFiles('/another-dir', function (err) {
    console.log('done importing another-dir')
  })
})

That covers some of the common use cases, let us know if there are more to add! Keep reading for the full API docs.

API

DPack(dir|storage, [opts], callback(err, dpack))

Initialize a dPack Vault in dir. If there is an existing dPack Vault, the vault will be resumed.

Storage

  • dir (Default) - Use @dpack/storage inside dir. This stores files as files, sleep files inside .dpack, and the secret key in the user's home directory.
  • dir with opts.latest: false - Store as SLEEP files, including storing the content as a content.data file. This is useful for storing all history in a single flat file.
  • dir with opts.temp: true - Store everything in memory (including files).
  • storage function - pass a custom storage function along to dDrive, see @dpack/storage for an example.

Most options are passed directly to the module you're using (e.g. dpack.importFiles(opts). However, there are also some initial opts can include:

opts = {
  key: '', // existing key to create vault with or resume
  temp: false, // Use dwrem as the storage.

  // dDrive options
  thin: false // download only files you request
}

The callback, cb(err, dpack), includes a dpack object that has the following properties:

  • dpack.key: key of the dPack (this will be set later for non-live vaults)
  • dpack.vault: dDrive vault instance.
  • dpack.path: Path of the dPack Vault
  • dpack.live: vault.live
  • dpack.writable: Is the vault writable?
  • dpack.resumed: true if the vault was resumed from an existing database
  • dpack.options: All options passed to dPack and the other submodules

Module Interfaces

dpack-core provides an easy interface to common dPack modules for the created dPack Vault on the dPack object provided in the callback:

var network = dpack.joinNetwork([opts], [cb])

Join the network to start transferring data for dpack.key, using flock-revelation. You can also use dpack.join([opts], [cb]).

If you specify cb, it will be called when the first round of revelation has completed. This is helpful to check immediately if peers are available and if not fail gracefully, more similar to http requests.

Returns a network object with properties:

  • network.connected - number of peers connected
  • network.on('listening') - emitted with network is listening
  • network.on('connection', connection, info) - Emitted when you connect to another peer. Info is an object that contains info about the connection
Network Options

opts are passed to flock-revelation, which can include:

opts = {
  upload: true, // announce and upload data to other peers
  download: true, // download data from other peers
  port: 3282, // port for revelation flock
  utp: true, // use utp in revelation flock
  tcp: true // use tcp in revelation flock
}

//Defaults from @dweb/flock-presets can also be overwritten:

opts = {
  dns: {
    server: // DNS server
    domain: // DNS domain
  }
  dht: {
    bootstrap: // distributed hash table bootstrapping nodes
  }
}

Returns a flock-revelation instance.

dpack.leaveNetwork() or dpack.leave()

Leaves the network for the vault.

var importer = dpack.importFiles([src], [opts], [cb])

Vault must be writable to import.

Import files to your dPack Vault from the directory using dPack Mirror.

  • src - By default, files will be imported from the folder where the vault was initiated. Import files from another directory by specifying src.
  • opts - options passed to mirror-folder (see below).
  • cb - called when import is finished.

Returns a importer object with properties:

  • importer.on('error', err)
  • importer.on('put', src, dest) - file put started. src.live is true is file was added by file watch event.
  • importer.on('put-data', chunk) - chunk of file added
  • importer.on('put-end', src, dest) - end of file write stream
  • importer.on('del', dest) - file deleted from dest
  • importer.on('end') - Emits when mirror is done (not emitted in watch mode)
  • If opts.count is true:
    • importer.on('count', {files, bytes}) - Emitted after initial scan of src directory. See import progress section for details.
    • importer.count will be {files, bytes} to import after initial scan.
    • importer.putDone will track {files, bytes} for imported files.
Importer Options

Options include:

var opts = {
  count: true, // do an initial dry run import for rendering progress
  ignoreHidden: true, // ignore hidden files  (if false, .dpack will still be ignored)
  ignoreDirs: true, // do not import directories (dDrive does not need them and it pollutes metadata)
  useDPackIgnore: true, // ignore entries in the `.dpackignore` file from import dir target.
  ignore: // (see below for default info) anymatch expression to ignore files
  watch: false, // watch files for changes & import on change (vault must be live)
}
Ignoring Files

You can use a .dpackignore file in the imported directory, src, to ignore any the user specifies. This is done by default.

dpack-core uses @dpack/ignore to provide a default ignore option, ignoring the .dpack folder and all hidden files or directories. Use opts.ignoreHidden = false to import hidden files or folders, except the .dpack directory.

It's important that the .dpack folder is not imported because it contains a private key that allows the owner to write to the vault.

var stats = dpack.trackStats()

stats.on('update')

Emitted when vault stats are updated. Get new stats with stats.get().

var st = dpack.stats.get()

dpack.trackStats() adds a stats object to dpack. Get general vault stats for the latest version:

{
  files: 12,
  byteLength: 1234,
  length: 4, // number of blocks for latest files
  version: 6, // vault.version for these stats
  downloaded: 4 // number of downloaded blocks for latest
}
stats.network

Get upload and download speeds: stats.network.uploadSpeed or stats.network.downloadSpeed. Transfer speeds are tracked using @ddrive/netspeed.

var peers = stats.peers
  • peers.total - total number of connected peers
  • peers.complete - connected peers with all the content data

var server = dpack.serveHttp(opts)

Serve files over http via @ddrive/http. Returns a node http server instance.

opts = {
  port: 8080, // http port
  live: true, // live update directory index listing
  footer: 'Served via DPack.', // Set a footer for the index listing
  exposeHeaders: false // expose dPack key in headers
}

dpack.pause()

Pause all upload & downloads. Currently, this is the same as dpack.leaveNetwork(), which leaves the network and destroys the flock. Revelation will happen again on resume().

dpack.resume()

Resume network activity. Current, this is the same as dpack.joinNetwork().

dpack.close(cb)

Stops replication and closes all the things opened for dpack-core, including:

  • dpack.vault.close(cb)
  • dpack.network.close(cb)
  • dpack.importer.destroy() (file watcher)