Rewriting Surge-Geosite

The original Geosite project had been running reliably for a year and a half, with only a handful of issues about request limits and false-positive matches. Request limits were easier to deal with, and a few workarounds provided a temporary fix. The false positives, however, remained unresolved. With some free time recently, I decided to rewrite the whole project.

The principles and implementation of the old version were covered in the previous article; this one focuses on the new version.

Why Rewrite It?

Both false-positive issues came from incorrect regex conversion. #3 occurred because the private list's original ^[a-z]([a-z0-9-]{0,61}[a-z0-9])?$ was directly converted to DOMAIN-WILDCARD,?*, causing any domain to match. #4 involved category-porn, where (^|\.)[a-z][1-9][0-9][a-z]\.com$ was converted into *????.com, which could even classify google.com as porn.

These problems were actually expected. When designing the conversion algorithm, it was clear that character classes all become ?, quantifiers and alternations all become *, and constraints in the regex such as "the first character must be a letter", "at most 62 characters", and "must not end with a hyphen" all disappear after conversion, leaving only very broad wildcards. After #3 appeared, I temporarily added a workaround that directly skipped any result consisting only of ? and *, but when #4 appeared, *????.com still slipped through.

Moreover, when rechecking the code, I found an even more serious problem. Geosite's include: can be followed by attributes. For example, category-ads contains a line like this.

include:adobe @ads

This means only importing rules from adobe that have the @ads attribute. But when the old version handled include, it extracted the list name like this.

const subContentName = line.split(" ")[0].replace("include:", "");

These problems could not be solved by changing a few lines of code, so I decided to rewrite the entire project. The new version is on the repository's next branch; it and main have completely independent Git histories. All existing URLs remain supported.

Overall Architecture

The new version is a pnpm monorepo split into four packages: core, cli, worker, and panel.

core consists entirely of pure functions: Geosite text goes in and Surge rules come out, with no file reads or network requests. It works in four stages. parser turns each list into typed entries with line numbers, so errors can identify the exact list and line. resolver expands includes, filters attributes, and removes duplicates; circular references cause an immediate error. regex handles regular expressions separately. Finally, surge emits the rule text.

The include problem above can be handled by a single function in resolver. include:xxx @a @-b is split by parser into two sets of conditions, “must have a” and “must not have b”; resolver checks them against each rule.

function matchesAttrFilters(rule: DomainRule, inclusion: Inclusion): boolean {
  if (inclusion.mustAttrs.length === 0 && inclusion.banAttrs.length === 0) return true;
  if (rule.attrs.length === 0) return inclusion.mustAttrs.length === 0;
  for (const attr of inclusion.mustAttrs) if (!rule.attrs.includes(attr)) return false;
  for (const attr of inclusion.banAttrs) if (rule.attrs.includes(attr)) return false;
  return true;
}

Alongside the converted rules, core produces a report showing how each regex was handled: converted equivalently, broadened, or skipped, with the reason.

The CLI is a thin wrapper around core. Point it at the data directory of a local domain-list-community checkout, and it converts every list in every mode, writing the rules, the intermediate results after include expansion, and per-list statistics. After changing the conversion logic, a local run shows exactly which rules changed.

Conversion Modes

There is no single answer for regex conversion: some people want complete equivalence and would rather have fewer rules, while others want maximum coverage and can accept some extra matches, so I divided conversion into three modes.

strict only outputs conversions that can be proven equivalent, for example, ^a\.b$ corresponds to DOMAIN, (^|\.)a\.b$ corresponds to DOMAIN-SUFFIX; anything else is omitted. balanced is the default mode and uses heuristics to convert to DOMAIN-WILDCARD. It skips zero-width assertions, backreferences and other constructs that wildcards cannot express, as well as results that would be meaningless. full, on top of balanced, tries to extract a literal domain suffix from any regex that could not be converted; as a last resort, it outputs *.

I tested the upstream data available at the time: there were 181 distinct regexes. strict retained none, because none could be converted into a strictly equivalent Surge rule. For that dataset, strict is basically equivalent to discarding all regexes. balanced converted 180, and full converted all 181. For example, .+\.dkr\.ecr\.[^\.]+\.amazonaws\.com$ under balanced becomes *.dkr.ecr.*.amazonaws.com. For the regex from #3, ^[a-z]([a-z0-9-]{0,61}[a-z0-9])?$, balanced skips it; only full turns it into *.

Worker

The old version fetched upstream, decompressed, and converted only after a request came in. The new version moves all of that into scheduled jobs, so requests only need to read from R2.

A Cron job runs every 5 minutes, sends a HEAD request for the upstream ZIP, and compares the ETag. If it has not changed, nothing happens. Otherwise, the job downloads the archive once, extracts every list in the data directory, and fully parses and resolves them with core. Only if that succeeds does it write the snapshot and index to R2, then switch state/latest.json. Each version's directory is named after the upstream ETag, so old and new snapshots can coexist.

state/latest.json
snapshots/{etag}/sources.json.gz
snapshots/{etag}/index/geosite.json
artifacts/{etag}/{mode}/{name[@filter]}.txt

The flow after a request comes in is as follows.

Converted rule sets are stored as artifacts by the combination of "list, mode, attribute"; they are generated and written to R2 only when someone first requests them, and are read directly afterward. If the current version does not yet have this artifact but the previous version does, the old one is returned first, while using waitUntil to generate the new one in the background, so the request does not need to wait. The response headers include ETag and x-upstream-etag, and when returning an old artifact there is an additional x-stale: 1. The Cache-Control is normally public, max-age=300, s-maxage=1800, stale-while-revalidate=86400, but when returning an old artifact it is shortened to max-age=60, s-maxage=120.

This way, the request path no longer touches GitHub. If upstream is slow or unstable, it only affects the next refresh; existing rule sets are unaffected.

For this domain, /geosite* is handled by this Worker, and all other paths are handled by the panel.

Panel

The old version only offered rule-set conversion and a basic list. To find available lists, you had to scan the very long index I periodically updated in the README, or dig through the upstream data directory on GitHub—a page large enough to freeze the browser. I wanted a frontend panel where, while configuring Surge, you could quickly find a list, inspect its converted rules, compare modes, and see what remains after applying @cn. That was the reason for adding a panel.

The project runs entirely on Cloudflare, but people opening the panel to configure rules may not have a proxy enabled. Surge can fetch rules through a proxy, whereas the browser may connect directly. The panel therefore needs to be lightweight, with a small frontend bundle—exactly where Svelte shines.

The panel uses SvelteKit, plus Tailwind 4 and shadcn-svelte, and has only one runtime dependency: the icon library @lucide/svelte.

For a good initial load, SSR was essential. But upstream has more than 1.4k lists, and shipping the full index immediately would be large even with gzip. I compromised: server-side rendering includes just the first 40 list names, about 7 KB, so the page is usable immediately; the browser fetches the rest after loading. The panel server also caches the 64 most recent previews and checks the index every 20 seconds using If-None-Match. It calls the Worker through a service binding, not over the public internet.

The interface also has Chinese and English versions, and automatically redirects according to Accept-Language. The search box is debounced, and when switching lists quickly, late-arriving old responses are discarded.

The result is that surge.bojin.co opens almost instantly and is ready to use.