repomatic.metrics module

Accumulate what forges say about a set of repositories, one reading at a time.

Every reading is one row of one table: which repository, which metric, on which date, what it said, and where the figure came from. A new metric is a Metric entry and one line in the forge reader, not a new file, a new schema or a new command.

Note

How long a reading is kept is a property of the metric, not of the caller.

A counter accrues: its whole point is the curve, so every dated reading is kept and charted. A star count is the one that motivated all this.

An attribute does not: the date of a project’s newest commit is a fact about today, nothing reads it chronologically, and a hundred subjects sampled weekly would pile up thousands of rows a year that no page ever opens. Only the newest reading is kept, dated when the value last moved, so a quiet week leaves the file untouched rather than restamping every row.

Retention is where that choice lives, and upsert() is the only code that has to know about it.

Note

The star history replaces the third-party charts a project used to embed. On 2026-06-30 GitHub restricted the REST stargazer endpoints to a repository’s own admins and collaborators, and closed the equivalent GraphQL field on 2026-07-17, which left every such embed on the web rendering an error card.

What survived is the aggregate count on the repository object, which stays public for everyone. Sampled on a schedule it accumulates into a history nobody can revoke.

Warning

A reconstruction and a sample do not measure the same thing, and the difference is deliberate rather than a defect.

The stargazers API lists only the accounts that still have the repository starred, so a reconstruction attributes today’s surviving stars to the dates they were given: it understates every past date by the number of stars since withdrawn, converging on the true figure at the present day. Kept on purpose, since a curve that sags where a project shed followers carries a signal a monotonic one hides. Each row therefore names its SOURCES, so a reader can always tell which question a point answers.

repomatic.metrics.GITHUB_EPOCH = datetime.date(2008, 1, 1)

No star predates GitHub, so nothing earlier can be a real reading.

The guard that tells a star-history.com calendar export from its by-age sibling: the latter measures each curve from epoch zero, so its rows land in the 1970s and would otherwise enter the store as genuine points four decades before the repository existed.

repomatic.metrics.MAX_RETRY_DELAY = 15.0

Ceiling on fetch()’s exponential backoff, in seconds.

Doubling without a bound spends the whole attempt budget waiting, which is the wrong trade against a service that fails most requests but recovers within seconds on the next one.

repomatic.metrics.METRIC_HEADERS = ('repo', 'metric', 'date', 'value', 'source')

Columns of the committed store, in file order.

The three key columns first, then the payload, so the file reads top to bottom as one repository at a time, one metric at a time, chronologically. That is also the sort order, which is what makes a scheduled commit an append per subject rather than a reshuffle.

repomatic.metrics.PREDECESSOR_SUFFIX = ':prior'

Marks a predecessor’s series key, appended to the subject it belongs to.

Keeps the configured subject list exactly the curves a chart plots, while still letting the collectors and the renderer address the extra one through the same code paths.

repomatic.metrics.SAMPLE_HEADER_DEFS: tuple[tuple[str, str], ...] = (('Subject', 'subject'), ('Phase', 'phase'), ('Repository', 'repository'), ('Stars', 'stars'), ('Rows', 'rows'), ('Note', 'note'))

Column definitions for the repomatic sample-metrics table.

Lives beside the rows’ domain model so the columns and the fields they render cannot drift apart; the CLI derives its --sort-by choices from it.

repomatic.metrics.SOURCE_RANK: dict[str, int] = {'created': 3, 'github': 3, 'sample': 2, 'star-history': 1, 'wayback': 1}

How authoritative each provenance is, for resolving two readings of a day.

An exact reconstruction supersedes a mined or imported count; a contemporaneous sample supersedes both, since it was taken by this collector against the live API. A backfill never overwrites something stronger, which is what lets a one-off import run against an already-populated store without degrading it.

repomatic.metrics.SOURCES: dict[str, str] = {'created': 'Repository creation, the one date a star count is known to be 0.', 'github': 'Exact per-star timestamps, surviving stars only (admin token).', 'sample': "Read from the forge's own API, contemporaneous.", 'star-history': 'Count at a date, imported from a star-history.com export.', 'wayback': 'Contemporaneous count mined from an archived GitHub page.'}

Provenance vocabulary, recorded per row.

A chart may mix methodologies it cannot reconcile, so it records which one each point came from rather than presenting a uniform curve it cannot honestly claim.

created is the outlier: not a measurement but a fact, and the only origin every series shares. A repository backfilled from the archives has no knowable first star, since its earliest capture already shows a count, so its curve would otherwise begin in mid-air. It is also what a by-age chart aligns on.

repomatic.metrics.USER_AGENT = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120 Safari/537.36'

Sent to the Wayback Machine, which serves robots a reduced index.

repomatic.metrics.WAYBACK_PAGE_TRIES = 8

Attempts per archived page.

Sized against a measurement rather than a guess: 25 requests for one capture known to exist returned 23 plain 503 responses and 2 truncated bodies, and no clean response at all. Since a truncated body still carries the counter, the per-try success rate that matters was 2 in 25, and eight tries is the point past which more attempts cost more than the captures they recover.

repomatic.metrics.WAYBACK_REFUSAL_LIMIT = 10

Consecutive refused captures tolerated before the run abandons the archive.

A served page proves the archive healthy whatever it holds, so only refusals extend the streak, and any payload resets it. Sized against the healthy success rate WAYBACK_PAGE_TRIES buys: with eight tries a capture lands about half the time, so ten misses in a row happens by luck roughly once in a thousand runs. Past it the per-IP budget is spent for a while, and every further capture only burns a full retry schedule proving it again.

repomatic.metrics.WAYBACK_REQUEST_DELAY = 3.0

Seconds to wait between two archived pages.

The backfill is a one-off that nobody watches, so trading minutes for a higher completion rate is free. Its counterpart is the retry backoff in fetch(), which handles a single hiccup; this handles the sustained budget.

repomatic.metrics.WAYBACK_STAR_PATTERNS = (re.compile('id="repo-stars-counter-star"[^>]*title="([\\d,]+)"', re.IGNORECASE), re.compile('title="([\\d,]+)"[^>]*id="repo-stars-counter-star"', re.IGNORECASE), re.compile('aria-label="([\\d,]+) users? starred', re.IGNORECASE), re.compile('href="/[^"]+/stargazers"[^>]*class="social-count[^"]*"[^>]*>\\s*([\\d,]+)', re.IGNORECASE), re.compile('class="social-count[^"]*"[^>]*href="/[^"]+/stargazers"[^>]*>\\s*([\\d,]+)', re.IGNORECASE))

Star-counter markups GitHub has shipped over the years, newest first.

An archived page states the exact figure in an attribute rather than the abbreviated 4.4k shown to readers, so a capture yields an integer, not an estimate. The layout was reworked twice in the window these mine, hence the alternatives.

class repomatic.metrics.Retention(*values)

Bases: Enum

How long the store keeps a metric’s readings.

HISTORY = 1

Every dated reading, forever. For a counter, whose curve is the point.

LATEST = 2

Only the newest reading, dated when the value last moved.

For an attribute, which describes today rather than accruing. Nothing reads it chronologically, and keeping every sample would bury the file in rows restating what the previous one already said.

class repomatic.metrics.Metric(id, retention, label, description)

Bases: object

One thing a forge can be asked about a repository.

id: str

Value of the store’s metric column, and the name a chart selects on.

retention: Retention

Which of Retention governs this metric’s rows.

label: str

Human-readable name, for a rendered table or a chart axis.

description: str

What the reading means, and what it deliberately does not.

property accrues: bool

Whether this metric’s past readings are kept and can be charted.

repomatic.metrics.METRICS: tuple[Metric, ...] = (Metric(id='commit', retention=<Retention.LATEST: 2>, label='Last commit', description='Date of the newest commit on the default branch, which stays true for a rolling repository that never tags a release.'), Metric(id='release', retention=<Retention.LATEST: 2>, label='Last release', description='Date of the newest release or tag, whichever is more recent.'), Metric(id='release_source', retention=<Retention.LATEST: 2>, label='Release kind', description='Whether the release date came from a release the project announced, or from the newest tag it merely labelled.'), Metric(id='stars', retention=<Retention.HISTORY: 1>, label='Stars', description='Accounts following the repository on its own forge.'))

Every metric the sampler collects, sorted by ID.

The extension point: a new counter is one entry here plus one yield in readings(). Nothing else changes, because the store, the retention rule and the chart all read this registry.

repomatic.metrics.METRICS_BY_ID: dict[str, Metric] = {'commit': Metric(id='commit', retention=<Retention.LATEST: 2>, label='Last commit', description='Date of the newest commit on the default branch, which stays true for a rolling repository that never tags a release.'), 'release': Metric(id='release', retention=<Retention.LATEST: 2>, label='Last release', description='Date of the newest release or tag, whichever is more recent.'), 'release_source': Metric(id='release_source', retention=<Retention.LATEST: 2>, label='Release kind', description='Whether the release date came from a release the project announced, or from the newest tag it merely labelled.'), 'stars': Metric(id='stars', retention=<Retention.HISTORY: 1>, label='Stars', description='Accounts following the repository on its own forge.')}

Index for O(1) metric lookup by ID.

repomatic.metrics.CHARTABLE_METRICS: tuple[str, ...] = ('stars',)

Metrics a chart can plot, since only an accruing one has a curve.

class repomatic.metrics.MetricRecord(repo, metric, day, value, source)

Bases: object

One reading: what a forge said about one repository on one date.

repo: str

Canonical https://host/owner/name URL of the subject.

metric: str

Which METRICS entry this reading is of.

day: str

The reading’s date, in YYYY-MM-DD form.

For an accruing metric, when the reading was taken. For an attribute, when its value last changed.

value: str

What the forge answered, as text.

CSV carries no types, so a consumer wanting a number coerces it. The store keeps the forge’s own answer rather than a parsed one, since a metric added later may not be numeric at all.

source: str

Which key of SOURCES produced the figure.

property key: tuple[str, str, str]

Deduplication identity: one reading per subject, metric and day.

property subject_key: tuple[str, str]

What an attribute keeps only one row of.

property count: int

The reading as an integer, for a counter metric.

Raises:

ValueError – When the value is not a number, which means a chart was pointed at an attribute.

as_row()

Flatten to one CSV row, in METRIC_HEADERS order.

Return type:

tuple[str, ...]

classmethod from_row(row)

Rebuild a record from one parsed CSV row.

Parameters:

row (Mapping[str, str]) – The row, keyed by column name.

Return type:

MetricRecord

Returns:

The corresponding record.

Raises:

KeyError – When a column is missing.

class repomatic.metrics.SampleOutcome(subject, repo, phase, stars=None, rows=0, note='')

Bases: object

What one subject’s sample produced, for the CLI to report.

subject: str

Name the repository gives this subject.

repo: str

Canonical URL it read from.

phase: str

Sampling lane that produced this outcome.

One of forward, reconstruct, import or wayback. The CLI reports one row per subject per lane, and the columns mean different things in each, so the row names the lane whose semantics it carries.

stars: int | None = None

Its current star count, when the collector read one.

rows: int = 0

How many stored rows this collector added or moved.

note: str = ''

Why nothing was collected, empty when something was.

repomatic.metrics.collected_subjects(subjects, predecessors=None)

Every repository a collector touches, keyed by its subject name.

Parameters:

• subjects (Mapping[str, str]) – Tracked subjects, mapping each name to a slug or URL.

• predecessors (Mapping[str, str] | None) – Retired forerunners, mapping the name of the subject they belong to onto their own slug or URL.

Return type:

dict[str, str]

Returns:

The subjects, plus one entry per forerunner whose key carries PREDECESSOR_SUFFIX so a caller can tell the two apart. Every value is a canonical URL.

Raises:

ValueError – When a declared subject parses as neither a slug nor a URL.

repomatic.metrics.last_fetch_failure()

Summarize why the most recent fetch() gave up.

Return type:

str

Returns:

A tally like 6x HTTP 503, 2x truncated, or no response when nothing was recorded.

repomatic.metrics.load_metrics(path)

Read the committed store, keyed by subject, metric and date.

Parameters:

path (Path) – Path to the CSV store.

Return type:

dict[tuple[str, str, str], MetricRecord]

Returns:

The records, empty when the file does not exist.

Raises:

ValueError – When the file exists but cannot be parsed. Loud on purpose: a corrupt store must never be silently clobbered by the next save_metrics() write.

repomatic.metrics.save_metrics(path, records)

Write the store back, sorted by subject, metric and date.

Merges whatever is on disk under the caller’s own records rather than overwriting the file wholesale. A slow backfill flushes after every point across a run lasting hours, so it holds a snapshot that goes stale the moment anything else records a reading: without the merge its next flush would silently drop those rows.

Caution

The merge is additive, so it cannot express a deletion. An attribute whose older rows upsert() just pruned would come back from disk. The prune therefore happens against a store that was loaded from that same file, which is what every collector here does; a caller assembling records from nothing must write with a store it loaded first.

Parameters:

• path (Path) – Path to the CSV store.

• records (Mapping[tuple[str, str, str], MetricRecord]) – The records to write.

Return type:

bool

Returns:

True when the file content changed.

repomatic.metrics.upsert(records, record)

Record one reading, returning whether it changed anything.

Re-running on the same day overwrites rather than appends, which is what keeps the scheduled job idempotent. Beyond that the metric’s Retention decides:

• An accruing metric keeps every day, and a more authoritative source wins over a weaker one for the same day, per SOURCE_RANK.

• An attribute keeps one row. An unchanged value leaves the stored date alone, so a quiet week rewrites nothing; a moved value replaces the row and takes the new date, which is therefore when the value last changed rather than when it was last confirmed.

Parameters:

• records (dict[tuple[str, str, str], MetricRecord]) – The in-memory store, mutated in place.

• record (MetricRecord) – The reading to store.

Return type:

bool

Returns:

True when the store moved.

Raises:

KeyError – When the metric is not in METRICS_BY_ID.

repomatic.metrics.gunzip(blob)

Decompress a gzip payload, tolerating one cut short mid-stream.

gzip.GzipFile needs the trailer to finish, so it raises on the truncated bodies a degraded archive delivers, discarding the megabyte that did arrive. Feeding the same bytes to a raw decompressor returns everything decodable before the cut and simply never reports the end of stream.

Parameters:

blob (bytes) – The compressed payload.

Return type:

bytes

Returns:

Everything that could be decoded, empty on an undecodable blob.

repomatic.metrics.fetch(url, tries=3, timeout=45, user_agent='Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120 Safari/537.36')

Fetch a URL with capped backoff, returning None once every try failed.

Deliberately separate from repomatic.http, whose single-retry policy is right for an API that either answers or does not. The Wayback Machine’s replay service is frequently only partly healthy: its load balancer answers 503 for most requests while a minority succeed, with neighbouring requests for the same capture landing on different backends. A failure therefore says nothing about whether the capture exists, and repeating the request is the lever that works. Pacing is not: the whole service is degraded, not this client’s budget.

Parameters:

• url (str) – The URL to fetch.

• tries (int) – How many attempts to make before giving up.

• timeout (int) – Socket timeout in seconds.

• user_agent (str) – Identity to send, defaulting to the browser one the archive wants and every forge refuses.

Return type:

bytes | None

Returns:

The body, or None once every attempt failed. Consult last_fetch_failure() for why.

repomatic.metrics.sample_subject(records, subject, repo, extra_forges=None, day=None)

Read every metric of one subject, through whichever forge hosts it.

The scheduled collector, and the only one that works for a repository the token does not administer, or that lives outside GitHub entirely.

Parameters:

• records (dict[tuple[str, str, str], MetricRecord]) – The in-memory store, mutated in place.

• subject (str) – Name the repository gives this subject.

• repo (str) – Its canonical URL.

• extra_forges (Mapping[str, str] | None) – Host-to-forge entries for self-hosted instances.

• day (str | None) – Reading date in YYYY-MM-DD form. Today (UTC) when None.

Return type:

SampleOutcome

Returns:

What the sample produced.

repomatic.metrics.reconstruct_from_github(records, subject, repo)

Reconstruct one repository’s star curve from per-star timestamps.

Only works on GitHub, and only where the token administers the repository; GitHub answers 404 rather than 403 on the restricted endpoint for every other. Collapses to one cumulative reading per day on which the count moved, rather than one per star.

Pagination is all-or-nothing on purpose. A transient failure halfway through would otherwise write a truncated cumulative curve over a correct one, and every point of it would look exactly as legitimate as the rest.

Parameters:

• records (dict[tuple[str, str, str], MetricRecord]) – The in-memory store, mutated in place once the whole walk succeeded.

• subject (str) – Name the repository gives this subject.

• repo (str) – Its canonical URL.

Return type:

SampleOutcome

Returns:

What the reconstruction produced.

repomatic.metrics.wayback_captures(path)

List one archived capture per month of a repository’s GitHub page.

Parameters:

path (str) – The repository’s owner/name path.

Return type:

list[str] | None

Returns:

The capture timestamps, or None when the index itself could not be read. That is not the same answer as an empty list and must not be reported as one: the archive fails this query as readily as any other, and a run treating the outage as “never archived” skips the repository silently and for good.

repomatic.metrics.backfill_wayback(records, subject, repo, store=None, on_status=None, on_row=None)

Mine contemporaneous star counts from archived copies of a GitHub page.

The only route to the past of a repository the token cannot administer, and the only one reporting what the counter actually read on the day rather than what survives today.

Parameters:

• records (dict[tuple[str, str, str], MetricRecord]) – The in-memory store, mutated in place.

• subject (str) – Name the repository gives this subject.

• repo (str) – Its canonical URL.

• store (Path | None) – Store to flush to after every recovered point, since a run spans many minutes of a flaky remote. Skipped when None.

• on_status (Callable[[str], None] | None) – Called with whatever the backfill is reaching for next, so a caller can animate a live label. One subject is a single call spanning minutes, and a watcher hears nothing at all without this.

• on_row (Callable[[str], None] | None) – Called with each recovered point, for a caller keeping a persistent line per result. Misses stay on the INFO log instead: the archive refuses far more captures than it serves, and a line each would bury the handful that landed.

Return type:

SampleOutcome

Returns:

What the backfill produced. When the archive refuses WAYBACK_REFUSAL_LIMIT captures in a row, the run is abandoned with a retry-later note and every later subject is skipped: the budget is per IP, so no following subject stands a better chance.

repomatic.metrics.read_star_counter(html)

Read the exact star count out of an archived GitHub repository page.

Parameters:

html (str) – The archived page’s markup.

Return type:

int | None

Returns:

The count, or None when no known counter markup matched.

repomatic.metrics.parse_csv_day(stamp)

Read the UTC calendar day out of a star-history.com CSV timestamp.

Parameters:

stamp (str) – A JavaScript Date.toString() stamp.

Return type:

date | None

Returns:

The day in UTC, or None when the stamp does not parse.

repomatic.metrics.import_star_history_csv(records, path, repos=None)

Import the calendar export a star-history.com user downloaded.

That service reconstructed its curves from the same stargazer endpoint GitHub has since closed, so an export taken while it worked is the only surviving record of the past for a repository nobody administers and the archives never captured.

Caution

A replacement export cannot be obtained today. The service now inherits the restriction it reports: asked for a repository the visitor neither owns nor collaborates on, it answers that star history is unavailable instead of exporting anything. So a file reaching this function was either downloaded before the endpoints closed, or covers a repository its downloader administers, which reconstruct_from_github() already rebuilds exactly and at finer resolution. For a competitor, backfill_wayback() and forward sampling are what is left.

Its by-age export is refused rather than imported: that variant measures every curve from epoch zero, so its rows land in the 1970s and would enter the store as readings four decades before the repository existed.

Parameters:

• records (dict[tuple[str, str, str], MetricRecord]) – The in-memory store, mutated in place.

• path (Path) – The exported CSV.

• repos (Iterable[str] | None) – Only import rows whose repository canonicalizes into this set. Every row when None.

Return type:

list[SampleOutcome]

Returns:

One outcome per repository the file covered.

Raises:

ValueError – When the file carries no usable row, naming the by-age export as the likely cause.

repomatic.metrics.series(records, subjects, metric='stars', predecessors=None)

Group one metric’s readings into a chronological series per subject.

Parameters:

• records (Mapping[tuple[str, str, str], MetricRecord]) – The store.

• subjects (Mapping[str, str]) – Tracked subjects, mapping each name to a slug or URL.

• metric (str) – Which accruing metric to plot.

• predecessors (Mapping[str, str] | None) – Retired forerunners, keyed by the subject they precede.

Return type:

dict[str, list[tuple[date, int]]]

Returns:

One sorted list of (day, value) per subject that has any reading, forerunners under their PREDECESSOR_SUFFIX key.

Raises:

ValueError – When metric does not accrue, so has no curve to plot.