Duplicate-invoice detection benchmark

The short answer. On this ledger, the duplicate rules the accounting systems document find between 0% and 57% of the planted duplicates, and none of them finds these kinds: sent again with a new number, within 20 days; same number with a space, dash, case or leading zero changed. Our free check finds 86% with its default 30-day window and 97% with a 60-day window, at 86% to 87% precision. Its only false alarms are new orders for the same amount a few days apart, which a person clears from the hold list. The ledger, the key and the scorer are public, so anyone, including the vendors named below, can rerun it and send a result.

The scores

Precision: of the rows a check flagged, the share that were planted duplicates. Recall: of the planted duplicates, the share it found. A false alarm is a flagged row that is not a planted duplicate.

By kind of duplicate

What the ledger holds

• 1924 rows, one synthetic year, 40 invented suppliers with three billing habits: ad hoc invoices, a fixed monthly fee, a fixed weekly charge. Every name and amount is made up.
• 140 planted duplicates, 20 of each of the six kinds in the table above.
• 336 planted traps that look like duplicates and are not: a credit note (20), the replacement invoice after a credit note (20), a new order for the same amount, days apart (20), another supplier, same amount, same day (20), a fixed monthly fee (48), a fixed weekly charge (208). A check that flags them makes work for a person.
• Four columns, as most exports have them: supplier, invoice number, invoice date, total. Credit notes are negative totals.
• The file is regenerated from a seed, so the committed ledger and key are reproducible to the byte.

How the vendor rules were scored

We did not run the vendors' software. Each "as documented" row runs the rule the vendor's own help page describes, exactly and no more, on this ledger. If a page says the check compares the supplier and the bill number, the row flags a later bill whose supplier and number match an earlier one to the letter. The quotes:

• Xero, as documented. Xero Central, on reviewing duplicate bills: "Xero compares the contact, reference and amount of all the bills and credit notes" (source) We read that as all three matching; the page does not say so in those words.
• QuickBooks Online, as answered on Intuit's community. An answer on Intuit's QuickBooks community, not a help article, names the setting under Account and Settings that warns when someone enters a "bill number that’s already been used for that vendor" (source)
• NetSuite, as documented. Oracle's NetSuite help, on the Duplicate Number Warnings preference: a warning or a block when a transaction has "the same document number as another record of the same type" (source) The preference page does not limit the check to one vendor, so the row matches the number across vendors.
• Sage Intacct, as documented. Sage Intacct's Accounts Payable configuration help, on duplicate numbers: "For bills and adjustments, this action applies per vendor only." (source)
• Oracle Fusion Cloud Payables, invoice import, as documented. Oracle's Fusion Payables documentation, on the invoice number in the import interface: "Must be a unique number for the supplier." (source)
• Tipalti, as documented. Tipalti's help centre lists what makes a duplicate: "Same payee, Same invoice number" (source)
• Sage 50 Accounts (UK), as documented. Sage's UK knowledge base, on the batch entry duplicate check: flagged transactions must have the same "Accounts reference, Date, Reference, Net amount" (source)
• Sage 200 Professional, single invoice entry, as documented. Sage 200 Professional help, on entering a purchase invoice: it checks that "you haven't already entered an invoice for the same date and reference number" (source)
• SAP ERP 6.0 Logistics Invoice Verification, as documented. SAP's help on the duplicate invoice check lists the fields it compares: "Vendor, Currency, Company code, Gross invoice amount, Reference document number, Invoice doc. date" (source)
• Dynamics 365 Business Central, as documented: no check. Microsoft's Business Central help, on external document numbers: "Business Central doesn't check if these external document numbers are unique or duplicates." (source)

Not scored, because no public page states what their check compares: Exact Online, Xelix, FISCAL Technologies, AppZen, Stampli, Medius, Yooz. Several say their check uses more than exact matches, which is exactly what this ledger tests. Any of them can run the ledger and send the result; it goes on this page as sent.

Run it yourself

Everything is in the open repository, under an MIT licence: the generator, the ledger, the key, the scorer and our check.

• Score your own tool: flag rows in the ledger, write the row numbers to a file, run the scorer against the key.
• Score ours: run the check on the ledger with the command-line tool and score the output.
• Send a result: email the run file and the name of the tool. We publish it on this page, as sent, with the date it arrived.

Repository: github.com/stackforagents/duplicate-invoice-check. The free check that scored above runs in your browser on your own export: duplicate invoice check.

What this benchmark does not say

• It scores rules on one synthetic ledger. A real ledger has its own habits, and the right date window is a choice.
• It says nothing about how a vendor's software behaves beyond the rule its help page documents. Vendors often do more than they document; that is why the ledger is public.
• A check with no false alarms at all is usually too loose. The hold list exists so a person decides.

The Proof runs the same checks on your real ledger, with the duplicates already paid, the rule that let them through and a hold list every week: from 1,500 euro, 5 working days, in shadow mode. The sample day-5 output shows what you get.

Send us the job in the form below and a plan with a fixed price comes back within 24 hours.