Source file¶

One file per source at sources/<id>.yaml. The file name must match id. Print the JSON Schema for editor completion with gleanyard json-schema source.

id: aachen-city-events
name: Veranstaltungskalender der Stadt Aachen
schema: event
enabled: true
delivery: ical
url: https://www.aachen.de/kalender/veranstaltungskalender-alle-termine/event.ics
homepage: https://www.aachen.de/services/veranstaltungskalender/
reader:
  timezone: Europe/Berlin
extract:
  mode: fields
  map: {title: SUMMARY, start_date: DTSTART, source_url: URL}
  defaults: {city: Aachen}
fetch: {max_requests: 20, delay_seconds: 1}
expect_empty: never
coverage: {country: DE, city: Aachen, scope_type: municipality, scope_name: Aachen}
rights: {effective_tier: internal_only, allowed_actions: {private_store: true, normalize: true}}
retention: {raw_days: 30}

Key	Meaning
`id`	Lower-case letters, digits and `-`. Must equal the file name.
`schema`	Must equal the project schema's `name`.
`enabled`	Default `false`. Only enabled sources run by default, and an enabled source must allow `private_store` and `normalize`.
`delivery`	`html`, `ical`, `json`, `rss`, `csv`, `pdf` or `sitemap`.
`url`	What to fetch.
`homepage`	Optional. Relative URLs resolve against it, and its host is also allowed for requests.
`reader`	Options for the delivery; see below. Unknown keys are errors.
`extract`	How to map onto the schema; see below.
`pagination`	Optional. Walk a paged source; see below.
`filter`	Optional. Check that a filter in the URL is still applied; see below.
`fetch`	Request limits.
`expect_empty`	`never` (default) or `seasonal`. A seasonal source with no records is healthy, not failing.
`keep_removed`	Optional, default false. Keep stored records that a later run no longer finds, for sources that publish only recent values. See keeping history.
`expect_change_days`	Optional. Mark the source `degraded` when its records have not changed for this many days. See Health.
`coverage`	Where the source applies. Informational.
`rights`	See Rights and export.
`evidence`	Optional. Folder under `evidence/` holding this source's terms review; default the source id. Sources sharing one site name the same folder. See watching the terms.
`retention`	`raw_days` (default 30) and `normalized_days` (default 730).
`recipe`	Set by `gleanyard add`: the registry recipe's `id`, its `digest` and the `registry` it came from. `status.json` reports it, so the registry can count how the recipe fares. See Source registry.

`reader` options¶

Delivery	Option	Default
`ical`	`timezone`	`UTC`
	`max_occurrences`	5000
	`past_days`	7
	`horizon_days`	366
`json`	`items_path` (dot path to the list)	the top level
	`unpivot` (one item per listed column; see wide tables)	off
`csv`	`delimiter`	`,`
	`encoding`	`utf-8`
	`unpivot`	off
`sitemap`	`include` (regular expression a URL must match)	all
	`max_pages`	50
`rss`	`include` (regular expression the item's title or text must match; other items are skipped, not rejected)	all
`html`, `pdf`	none

`extract` options¶

mode picks the shape; each mode accepts defaults, constant values for schema fields the source does not carry. fields and selectors also accept explode, a dot path to a list in each item: every element becomes its own candidate record (see one record per list element).

Mode	Keys
`fields`	`map`: schema field to source key or dot path; a list of keys takes the first with a value; optional `cases` (constant fields chosen by an item's values), `rewrite` (a regex substitution per field), `where` (keep items whose value matches a regex) and `join` (lookup tables from CSV files in the repo); see values chosen by what an item holds
`jsonld`	`types`: schema.org types to read; optional `map`
`selectors`	`selectors`: path to a crawl4ai JSON CSS schema, relative to the project
`llm`	`llm.model`, `llm.max_pages_per_run`, `llm.max_cost_per_run_eur`, `llm.only_if_changed`, `llm.input_price_per_mtok_eur`, `llm.output_price_per_mtok_eur`, `llm.max_input_chars`; optional `instructions`

Each mode must fit the delivery: feeds (ical, json, rss, csv) use fields; html and sitemap use jsonld, selectors or llm; pdf uses llm.

`pagination`¶

For sources that serve records in pages through one query parameter (json, html, rss and csv deliveries).

pagination: {param: page, start: 1, step: 1, max_pages: 10}

Key	Default
`param`	required; the query parameter that selects the page, such as `page` or `offset`
`start`	1; value on the first page
`step`	1; use the page size for offsets
`max_pages`	10; at least 2. `fetch.max_requests` must allow one more request than this

Pages are fetched until one has no records. A run fails with pagination_changed when a page repeats an earlier document or only records already seen, or when records still arrive after max_pages. collect --from-cache replays every cached page.

`filter`¶

When the URL narrows the data (?city=aachen), a source can quietly stop honouring it and return everything. filter states what should be inside and is checked on the extracted records.

filter: {field: city, one_of: [Aachen], max_outside_share: 0.2}

Key	Meaning
`field`	Schema field the URL filter constrains.
`one_of`	Values that count as inside; strings ignore case.
`max_outside_share`	Default 0.2. When more of the records fall outside, the run fails with `filter_ignored`; fewer are moved to the review file.

Records with no value for the field are not judged.

`fetch` limits¶

Key	Default
`max_requests`	20 (includes `robots.txt`)
`max_response_bytes`	5 MiB
`delay_seconds`	1.0
`timeout_seconds`	30
`accept`	none; the `Accept` header, for APIs that choose the format by content negotiation

Source file¶

reader options¶

extract options¶

pagination¶

filter¶

fetch limits¶

`reader` options¶

`extract` options¶

`pagination`¶

`filter`¶

`fetch` limits¶