Manual contract collection breaks at the same points every time: multiple portals, changing filters, attachments, and duplicate records. Procurement contract scraping on Twin.so can collect public records and return them in a consistent structure, but only when the workflow has strict source and review rules.
Twin.so isn’t a substitute for a procurement analyst. It is an autonomous agent that can use APIs first, then operate an embedded browser when a suitable API isn’t available. Start with one permitted source, one record schema, and a small test batch. Scale only after the output survives manual review.
How procurement contract scraping works on Twin.so
Twin’s browser automation can open websites, move through multi-step pages, read tables, fill fields, and extract records. Its no-API browser automation is useful for procurement portals that lack a usable export or expose information through dynamic pages.
The platform also supports scheduled agents and connected workflows. Its browser automation learning materials describe use cases that include public pages, portals, dashboards, and downloaded files.
Twin should receive a narrow job. A prompt such as “collect all contracts” leaves too many decisions to the agent. Define the source, filters, fields, stopping rules, and destination before the first run.
Start with one approved source
Choose a source that matches the procurement question. For federal research, SAM.gov contracting provides access to contracting information, opportunities, awards, and related reports. USAspending.gov provides official federal spending data, including contract awards.
These sources don’t always contain the complete executed contract document. Your workflow may need to collect award data first, then follow an approved attachment or agency source URL. Store the distinction between an award record, a solicitation, a modification, and a signed contract file.
Set the access boundary
Public visibility doesn’t remove every restriction. Review the source’s terms, robots.txt instructions, storage rules, redistribution limits, attribution requirements, and any data license before running the workflow.
Don’t bypass authentication, paywalls, CAPTCHAs, download controls, or technical restrictions. If an employee can access a private portal, that doesn’t automatically authorize automated collection. Obtain written approval when the source owner or your organization requires it.
Define the contract record before collection
A reliable workflow starts with a data contract. This is a written definition of each field, its format, and the conditions for accepting or rejecting a record.
Keep source values alongside normalized values. For example, preserve the original date string while storing a standard ISO date for filtering. This gives an analyst enough evidence to check an unexpected conversion.
Use a field-level schema
A practical output schema can include the following fields:
| Field | Example | Rule |
|---|---|---|
contract_id | 47QRAA24F0012 | Preserve the source value exactly |
record_type | award | Separate awards, notices, modifications, and attachments |
agency | Department of Energy | Store the displayed agency name |
title | Facility maintenance services | Keep the full source title |
award_date | 2026-07-14 | Store the original date and normalized date |
period | 2026-08-01 to 2029-07-31 | Keep start and end dates separate |
value | 1250000 | Store the numeric amount separately from currency |
currency | USD | Never assume the currency from a symbol |
status | active | Preserve the source label |
source_url | Canonical record URL | Required for review |
attachment_url | Permitted file URL | Leave blank when no file is available |
collected_at | Timestamp with timezone | Record when Twin retrieved the record |
This structure prevents a common error: treating every page as the same kind of contract record. A modification can change value or dates without creating a new base contract.
Build a stable identity
Use the strongest identifier available. A contract number combined with an agency code and modification number is usually better than the title alone.
If a source has no stable ID, combine fields such as the canonical URL, title, agency, award date, and attachment URL. Store a content hash when possible. This helps detect a changed contract page without creating a second record.
Keep the original page or permitted file reference with every row. A clean table without source evidence isn’t ready for procurement analysis.
Build the Twin.so workflow step by step
Twin’s quickstart documentation shows how agents can use plain-language instructions, schedules, browser actions, and connected services. Use that same outcome-first structure for contract collection.
Give the agent these input fields:
- The approved source URL and allowed URL patterns.
- Agencies, jurisdictions, categories, and date filters.
- The maximum number of pages or records per run.
- The required contract fields and accepted empty states.
- The attachment policy, including whether files may be downloaded.
- The destination for raw results, exceptions, and approved records.
Use a prompt with explicit stopping rules:
Open the approved procurement source. Apply the saved agency, category, and date filters. Collect only records visible without bypassing access controls. For each record, return contract ID, record type, agency, title, award date, period, value, currency, status, source URL, attachment URL, and collection timestamp. Continue through every permitted result page until the next-page control is unavailable. Return incomplete records as exceptions. Do not invent missing values. Do not submit forms or change source data.
Add a start-of-run deduplication check. Then separate the workflow into three stages:
- Collect the raw page values, source links, page number, and retrieval time.
- Validate required fields, dates, amounts, duplicate keys, and attachment status.
- Publish only accepted records to the database, spreadsheet, or review queue.
A failed validation should create an exception. It shouldn’t replace the last trusted dataset with an empty result.
Troubleshoot PDFs, pagination, and source changes
Procurement data often fails at the document layer rather than the page layer. Test each document type before scheduling the workflow.
Handle text PDFs and scanned documents
Twin’s public materials focus on browser pages, portals, dashboards, and browser actions. They don’t establish universal first-class PDF extraction or OCR support.
Treat PDF handling as a separate step. If an authorized workflow can download a file, preserve the original file, page count, source URL, and retrieval timestamp. Send text-based PDFs through an approved parser. Send scanned PDFs through approved OCR, then route low-confidence fields to human review.
Store the page number for important values. A reviewer should be able to find the stated amount, period, clause, or modification without searching the entire document again.
Test pagination and dynamic loading
A browser workflow can finish without an error and still miss the second page. Ask Twin to return the page number and record count for each batch. Add a maximum page limit and a clear reason when the workflow stops.
For dynamic tables, wait for visible evidence such as the expected headers or a known record ID. A fixed delay alone is unreliable. Save progress by page, cursor, record ID, or file name when the source supports it.
Detect duplicates and changed layouts
Duplicate contracts can appear across an agency portal, an aggregator, and a downloadable file index. Upsert records with a stable identity instead of appending every result.
A blank field may mean the label changed, the page loaded the wrong section, or the source returned incomplete data. Review the page manually and update the instruction with the exact visible label. Use bounded retries with increasing waits for temporary failures. Stop retrying when the failure is consistent.
Track missing records, new duplicates, changed labels, row-count changes, and failed pages on every rerun.
Validate records before procurement decisions
Automation should prepare evidence for a buyer or analyst. It shouldn’t make an unsupported decision about supplier selection, award validity, or contract compliance.
Apply field and consistency checks
Reject or flag records when:
- The contract ID is missing or changes format unexpectedly.
- The award date falls outside the requested range.
- The amount is non-numeric, negative, or missing its currency.
- The source labels an award as modified, cancelled, expired, or pending.
- The same contract identity appears more than once.
- The attachment link points to a different record.
- A required PDF is unavailable, password protected, or unreadable.
Don’t turn missing data into zero. Return null, an empty state, or an exception reason. A missing contract value has a different meaning from a contract with a value of zero.
Compare a small known sample with the live source. Check titles, agencies, dates, amounts, status labels, URLs, and attachments one by one. Measure field accuracy, missing-field rate, duplicate rate, source coverage, failed-run rate, and review time.
Keep humans in the approval loop
Use separate storage for raw results, validated results, and approved records. A reviewer should see the extracted value, source URL, page number or document reference, and any quality flag.
Don’t let Twin write directly into an official procurement system until the acceptance rules pass. Human review is required when sources conflict, contract language is unclear, OCR confidence is low, or a record could affect a bid, renewal, payment, or compliance decision.
Control permissions, rate limits, and Twin.so cost
Responsible procurement contract scraping is part of the system design. Add an allowlist of domains and URL patterns. Exclude account settings, unrelated customer data, administration pages, and paths containing personal information.
The Robots Exclusion Protocol provides a standard way for site owners to publish crawler instructions. Robots rules aren’t a replacement for authorization, terms review, or a written permission boundary.
Use low request rates and bounded retries. Don’t repeatedly open a failing page. A source change may require an instruction update, not more requests.
Twin usage depends on browser actions, searches, API calls, retries, record volume, and generated output. Published planning examples place a 100-item scraping job around 20 to 70 credits. A browser session with about 20 steps may use 100 to 200 credits. Treat those figures as estimates, not a fixed quote.
Run 20 to 50 approved records first. Record credits consumed, accepted records, exceptions, and review minutes. Calculate credits per accepted record, then forecast monthly usage from real runs. A cheap workflow that creates duplicates or corrections isn’t saving money.
For a multi-portal implementation, Book A Call can help map permissions, destinations, exception paths, and approval points before a wider rollout.
Conclusion
Twin.so can reduce the manual work behind public contract collection when the source, fields, permissions, and failure rules are explicit. The strongest workflow uses APIs where they fit and browser automation for approved pages that require interaction.
Start with one source and a small sample. Preserve the original evidence, test PDFs and pagination, deduplicate by contract identity, and hold incomplete records for review.
The useful result isn’t the largest dataset. It’s a traceable contract record that a procurement professional can verify before using it.
