SEC filings are public, but turning them into reliable rows isn’t a copy-and-paste task. If you scrape SEC filings, you need the correct document, the correct version, the relevant section, and a source URL another analyst can verify.
Twin.so can help convert SEC EDGAR searches into repeatable research workflows. It can search filings by company, ticker, or keyword, return structured results, track Form 4 activity, and export findings to Google Sheets. Accuracy still depends on how you define the workflow and validate every result.
What it means to scrape SEC filings
SEC filing retrieval and filing extraction are separate jobs.
Retrieval means finding and downloading the right document from EDGAR. This includes identifying the company, CIK, form type, filing date, reporting period, accession number, and filing URL.
Extraction means reading that document and returning a specific piece of information. You might extract a risk factor, a revenue figure, an executive transaction, a debt maturity, or a statement from the Management’s Discussion and Analysis section.
A keyword search can locate a filing that mentions “supply chain risk.” It doesn’t automatically tell you whether the reference appears in Item 1A, a footnote, an exhibit, or a quoted historical disclosure. The extraction step needs rules.
| Task | Output | Main risk |
|---|---|---|
| Filing retrieval | Filing metadata and source document | Selecting the wrong filing version |
| Text extraction | Section, passage, or quote | Returning a nearby but irrelevant reference |
| Numeric extraction | Value, unit, period, and context | Losing scale, sign, or reporting date |
| Monitoring | New filings or transactions | Missing amendments or duplicate results |
The SEC provides official access to company submissions and extracted XBRL data through its developer APIs. Review the SEC’s EDGAR API documentation before building a production process around public filing data.
Twin fits above that source layer. Its SEC EDGAR integration is designed for agent-based searches and downstream workflows. You describe the task, Twin assembles the workflow, and the result can move into a spreadsheet or another connected system.
Prepare the filing scope before you scrape SEC filings
A vague instruction produces a vague dataset. Start by defining the filing universe.
Choose the companies, forms, date range, and information target. A workflow for quarterly earnings should not search every document filed by every company. Limit it to the relevant CIKs and forms, such as 10-Q, 10-K, and selected 8-K filings.
The same rule applies to insider monitoring. Form 4 filings contain transaction disclosures, but the workflow should identify the reporting person, issuer, transaction date, transaction code, share count, price, and ownership after the transaction. A simple search for “buy” can return commentary, footnotes, or unrelated text.
Set the output fields before you connect the workflow. A practical filing record includes:
- Company name and CIK
- Form type, filing date, and reporting period
- Accession number
- Filing URL
- Section or item number
- Extracted value or passage
- Units, currency, and scale
- Amendment status
- A short evidence quote
The accession number is especially useful. Company names and tickers can change. An accession number gives each filing a stable identifier for deduplication and review.
Your prompt should also define what Twin must do when it finds no answer. Instruct it to return “not found” instead of guessing. Require a citation for every extracted value. Ask it to separate reported facts from interpretation.
A useful filing record contains the answer and enough evidence to locate that answer in the original document.
This approach makes the output easier to audit. It also prevents a spreadsheet from becoming a collection of unsupported summaries.
How to scrape SEC filings with Twin.so
Twin’s setup is designed for users who want to describe a workflow in plain language instead of writing every API request manually. The process still needs a clear build plan.
1. Connect the SEC EDGAR source
Twin describes its SEC setup as an OAuth connection. You authorize the connection once, then the agent can use the integration for later runs. Check the access scope, connected account, and destination before sending sensitive workflow results into a shared workspace.
The connection is not the same as selecting a dataset. You still need to define which companies, forms, and fields the agent should use.
2. Describe the search and extraction task
Write the task as an operating instruction. Include the filing types, date range, target terms, output columns, and evidence requirements.
For example:
Search new 10-K and 10-Q filings for the selected CIKs. Return the filing date, accession number, reporting period, Item 1A risk factors that mention supplier concentration, the exact supporting quote, and the original filing URL. Exclude exhibits and return “not found” when no matching passage exists.
This instruction does more than ask Twin to scrape SEC filings. It defines how the result should be filtered and documented.
3. Separate discovery from extraction
For large research jobs, use two stages.
The first stage finds candidate filings. It should return metadata and a reason for inclusion. The second stage opens each candidate filing and extracts the requested fields.
This separation reduces false positives. A filing can match a keyword without containing the financial disclosure you need. A second pass can check the section, reporting period, and surrounding text.
4. Choose the destination
Twin advertises Google Sheets export for tickers, filing summaries, balances, and portfolio data. Sheets works well for review queues and small research teams. Use stable columns and avoid placing several unrelated facts into one cell.
A useful layout separates:
- Filing metadata
- Extracted data
- Evidence text
- Validation status
- Reviewer notes
Treat the spreadsheet as an output surface, not automatically as your system of record. Store the original source reference and the run date with every row.
5. Run on demand before scheduling
Run a small test set first. Use a few companies and one or two filing types. Inspect the results manually. Correct the prompt when the agent confuses filing dates with period-end dates, mixes sections, or returns a summary without evidence.
Twin supports workflows that run on demand or on a schedule. Scheduling only makes sense after the manual sample passes review. Otherwise, automation multiplies the same error every day.
Build extraction rules for text and numbers
Text extraction and numeric extraction need different instructions.
For text, name the filing section when possible. Use terms such as Item 1A, Item 7, Note 8, or “controls and procedures.” Ask for the exact passage and a short context window. This gives a reviewer enough information to confirm that the text supports the result.
For numbers, require the value, unit, scale, currency, reporting period, and statement type. “Revenue: 4.2” is not usable without knowing whether the value means $4.2 million, $4.2 billion, or another unit.
Watch for common filing problems:
- Parentheses can indicate negative values.
- Tables may use thousands or millions.
- A column can compare current and prior periods.
- A value may be restated in a later filing.
- XBRL tags can differ across issuers.
- Similar labels can describe different measurements.
The SEC’s XBRL data can support structured financial extraction, but tagged data doesn’t remove the need for context. A reported value still needs its period, unit, and statement location.
For section-level extraction, compare your workflow design with tools such as the SEC filings extraction API. That type of service focuses on pulling defined sections and filing content. Twin’s value is the agent workflow around search, filtering, transformation, and delivery.
A useful rule is to store both the normalized value and the original display value. The normalized value supports calculations. The original value helps reviewers compare the output with the filing.
Validate every result before it enters analysis
Never treat an AI-extracted filing row as verified because it looks plausible.
Start with source validation. Open the filing URL and confirm that the accession number, form type, issuer, and filing date match the row. Then locate the cited section or passage.
Next, perform content validation. Check that the quote supports the extracted value. A passage can mention revenue without reporting the company’s revenue. It can also describe a future risk without stating that the risk occurred.
Use period validation for financial figures. Confirm the reporting period and compare it with the column heading in the filing. Store whether the amount covers a quarter, six months, a fiscal year, or a point-in-time balance.
Handle amendments as separate records. A 10-K/A is not the same filing as the original 10-K. The same applies to 10-Q/A. Preserve both accession numbers, mark the amended version clearly, and define whether your final dataset uses the latest version or retains the original disclosure.
A production review can use a simple status field:
pendingfor new outputverifiedfor manually confirmed rowsrejectedfor unsupported or incorrect resultssupersededfor data replaced by an amendment
Keep a run timestamp, prompt version, and workflow version. When an output changes, you need to know whether the filing changed, the prompt changed, or the extraction logic changed.
A detailed guide to the SEC EDGAR API and database structure can help your team understand how filing metadata and company submissions fit together.
Follow responsible SEC access practices
Public access doesn’t mean unlimited access. Keep requests controlled and identify your client where the SEC requires it. Use caching, avoid unnecessary repeated downloads, and handle rate-limit or server-error responses without aggressive retries.
If Twin manages the EDGAR connection, monitor failed runs and unexpected output. An agent may adapt to a failed request, but your process still needs logging and alerts. A missing filing should be visible. It shouldn’t look like a clean result set with no rows.
Store source URLs, accession numbers, and evidence passages with the data. Don’t remove those fields when sending results to another system. They are part of the dataset’s audit trail.
SEC filing extraction supports research and workflow automation. It doesn’t replace legal review, accounting review, or investment judgment.
Build a filing workflow you can audit
Twin.so can reduce the manual work involved in searching SEC EDGAR, extracting filing content, tracking Form 4 activity, and sending structured results to Google Sheets. The strongest results come from narrow instructions, explicit fields, evidence quotes, and careful handling of amendments.
Use Twin as the workflow layer, not as a reason to skip source verification. When every row retains its filing identity, citation, period, and validation status, you can scrape SEC filings at a useful scale without losing control of the underlying evidence.
