crawl

@twine-protocol/twine-core v0.1.0 • Docs

---

twine-js / @twine-protocol/twine-core / crawl

Function: crawl()

crawl(inputs, direct?, visited?): AsyncIterable<CrawlPending>

Crawl the tapestry

This is the main function to use for crawling the tapestry. The idea is to specify a starting pulse (or pulses) and a guide to direct the crawl. There are several built-in guides that can be used, or you can create your own.

The crawl won't load pulses until you call the load function on the pending crawl result. This allows you to control the loading of pulses and only load what you need.

The crawl will also track visited pulses so that it doesn't visit the same pulse twice. Although loops are impossible in the tapestry, different chains can contain links to the same pulse, so it's possible to visit the same pulse without tracking visited pulses.

It's recommended to use an asyncIterable library (like streaming-iterables) to handle the async nature of the crawl. The crawl function returns an async iterable that can use to iterate over the pending crawl results.

Parameters

Parameter	Type	Description
inputs	MaybeIterable<Awaitable<IntoResolvePulseQuery>>	The starting pulse or pulses
direct?	CrawlGuide	The guide to use for crawling
visited?	Set<string>	The Set object to use for tracking visited pulses

Returns

AsyncIterable<CrawlPending>

• An async iterable of pending crawl results

Example

import { crawl, across } from '@twine-protocol/twine-core'
import { HttpResolver } from '@twine-protocol/twine-http-store'

const resolver = new HttpStore('https://someapi.io')
const start = { chain: 'chaincid', pulse: 'pulsecid' }

const limit = 100
const chains = new Map()
for await (const pending of crawl(start, across())) {
  if (path.length < limit) {
    const result = await pending.load(resolver)
    chains.set(result.chain.cid.toString(), result.chain)
  }
}
// we now have a list of chains within 100 steps
console.log(Array.from(chains.values()))

Defined in

packages/twine-core/src/crawl.ts:284