A tool that turns Git into queryable 'code entities' — and the '2.3x more accurate' claim the thread shredded

The idea

Sem (from Ataraxy Labs) pitches a "new primitive for code understanding": instead of relying on LSPs, it builds semantic entities on top of Git, so you can run something like sem impact <function> and see what it depends on, what calls it, how many entities and tests are transitively affected, and AST-level (not line-level) diffs of commits. The landing page leads with a benchmark claim: "AI agents are 2.3x more accurate when given sem output vs raw line diffs."

The thread largely liked the entities-over-Git concept. It went after everything wrapped around it.

The benchmark claim gets shredded

"Okay, this looks great, but for the love of God... please cut this out: 'AI agents are 2.3x more accurate when given sem output vs raw line diffs' … You're trying to pretend like your tool makes AI agents 2.3x better at coding or bug fixing. It doesn't." — a developer challenging the marketing claim

Another reader explained why the number is hollow — it measures the tool's own concept:

"The benchmarks aren't great, they're super specific to sem's output: why would I ask Claude how many 'entities' were modified by a commit … Note that an 'entity' is a sem-specific concept." — a developer calling the benchmark circular

A benchmark that scores models on producing sem's own vocabulary isn't measuring accuracy; it's measuring compliance.

The installer raises eyebrows

"The 'Try it. 10 seconds.' section at the bottom of the page hijacks an existing tool (git diff) and installs a pre-commit hook. But there are no instructions for how to reverse those actions if you don't like the tool. Feels a little user-hostile to me." — a developer flagging the install behavior

Where it could actually shine

The most generous comment found a concrete, real use:

"This may help jujutsu auto split a large revision into small orthogonal revs. Sometimes agent makes a monolithic commit and it's a lot of work to manually split code you didn't write." — a commenter with a jujutsu/agent-commit use case

The pattern

Sem is a good example of a genre: a real technical idea undercut by AI-era marketing reflexes. Entity-level, AST-aware views layered on Git is a legitimately useful primitive — for agents that make monolithic commits, for impact analysis, for splitting changes. But a circular "2.3x" benchmark and an installer that quietly rewrites your git diff with no off-switch teach experienced developers to distrust exactly the thing worth trusting. The thread's message to the author was, in effect: cut the number, ship the primitive.