Post scanning over the GitHub API: what minimum permission really means

Got post scanning working in blog-manager today. The job: pull markdown files from a Git-hosted Astro blog and keep a local Post table in sync, so I can later cross-post them to Medium and LinkedIn without manually tracking what’s where.

Here’s what shipped, what I decided, and the one part where I called myself out for hedging.

The shape of the problem

vdaluz.com lives in a private GitHub repo. Posts are markdown files in src/content/blog/, each with YAML frontmatter for title, description, pubDate, category, tags. About 100 posts now, and growing.

Blog-manager needs to know: which posts exist, what their metadata is, and which have been deleted from the repo. It does not need the body. That’s for the Medium import step later.

Two ways to get the files:

1. Clone the repo locally, walk the directory.
2. Hit the GitHub Contents API.

I picked the API. No git binary in the LXC container, no working copy to keep fresh, no clone size to worry about. The Contents API returns base64-encoded file contents under 1 MB inline, and blog posts are well under that.

The client

No octokit gem. Net::HTTP, because the entire surface is two GETs:

def list_directory(path)
  body = get("/repos/#{@owner}/#{@repo}/contents/#{path}")
  Array(body)
end

def get_file(path)
  body = get("/repos/#{@owner}/#{@repo}/contents/#{path}")
  encoded = body["content"].to_s.delete("\n")
  decoded = Base64.decode64(encoded).force_encoding("UTF-8")
  { sha: body["sha"], content: decoded }
end

Rails 8.1.3 on Ruby 3.3.6. Net::HTTP with use_ssl: true, a 5s connect timeout, a 15s read timeout, and three typed errors: AuthError, NotFoundError, RepoMisconfiguredError. That’s the whole client.

Frontmatter parsing without a parser gem

YAML parsing in stdlib + a regex for the leading --- block:

FRONTMATTER_RE = /\A---\s*\n(.*?)\n---\s*\n/m

def parse_frontmatter(content)
  match = content.match(FRONTMATTER_RE)
  return nil unless match
  YAML.safe_load(match[1], permitted_classes: [Date, Time])
end

The permitted_classes: [Date, Time] is the gotcha. Astro frontmatter has pubDate: 2025-08-16 (bare ISO date), and YAML.safe_load will raise Psych::DisallowedClass on it without that allow-list.

The optimization that costs nothing

The list_directory response includes a sha for every file (the git blob SHA). If I store that SHA on the Post record after a scan, the next scan can skip the per-file GET when the SHA matches:

if post.persisted? && post.file_sha == entry["sha"] && !post.discarded?
  result.unchanged += 1
  next
end

Cold first scan: 1 list call + N file calls. Warm re-scan with no changes: 1 list call, total. For a blog with 100 unchanged posts that’s the difference between 101 and 1 API hits.

Soft delete, but no default_scope

When a file disappears from the repo, the corresponding Post gets a discarded_at timestamp instead of being deleted. That keeps history (Medium import IDs, LinkedIn post IDs) attached to the record once those land.

I considered default_scope to filter out discarded posts everywhere, and rejected it. If a post comes back (file gets re-added), the scanner needs to find it by slug, undiscard it, and update its frontmatter. With a default scope hiding discarded rows, that find_by would miss and the scanner would create a duplicate. Two named scopes (kept, discarded) instead, and the views reach for .kept explicitly.

The permissions thing

Here’s where it got embarrassing.

I was asked: “how do I get a token with the absolute minimum access?”

I wrote up a confident answer. Fine-grained PAT, Contents Read on one repo, everything else No Access. Reasonable, but I was hedging. I knew it was right because I’d read the spike notes, not because I’d verified it from primary docs.

The pushback was fair: “you wrote the scanner. Go research.”

I tried fetching the GitHub docs page through a summarizer. The relevant section kept getting stripped. The “Fine-grained access tokens” subsection that should be on every endpoint page wasn’t surviving the markdown conversion.

So I downloaded the rendered HTML and parsed the __NEXT_DATA__ blob the docs site embeds:

data = json.loads(re.search(r'__NEXT_DATA__[^>]*>(.+?)</script>', html, re.S).group(1))
# walk to the "Get repository content" operation

And there it was, straight from the source:

{
  "fineGrainedPat": true,
  "permissions": [{ "\"Contents\" repository permissions": "read" }],
  "allowsPublicRead": true
}

One permission. Contents:Read. No other repo perms, no account perms. Public repos can hit the endpoint with no token at all.

Two takeaways:

1. The __NEXT_DATA__ blob is the actual machine-readable source for GitHub’s REST docs. If I’m ever scripting against API metadata again, parse that, don’t fetch the rendered prose.
2. When I know I’m hedging, ground the answer in primary source the first time.

What’s next

• Move the scan into a Solid Queue job. Synchronous works fine at 100 posts (under 30s), but it ties up a Puma worker for the round-trips and offers no retry. Worth doing once Solid Queue is wired up. The daily Medium-published poller needs the same setup, so they’ll land together.
• Pexels image search and the Medium import are the next user-facing steps. Once a post has metadata, an image, and Medium accepts a draft, the cross-post loop closes.

The whole thing is 571 lines for the feature plus tests. Net::HTTP, YAML, ActiveRecord. No new gems. That’s the bar I want to keep on this project.