How We Unified 4,934 Australian Healthcare Jobs Across 7 Job Boards Into One Clean Weekly Feed

A healthcare recruitment firm needed weekly intelligence on the Australian nursing and aged care job market: every active listing across every major platform, deduplicated and formatted for Monday-morning delivery to their recruiter network. Their existing process was manual. Staff visited each job board, copied listings into a spreadsheet, and assembled a report by hand. Several hours every week, prone to missed listings and inconsistent state codes.

We delivered 4,934 unique, deduplicated healthcare jobs in a single weekly run: 7 platforms, fully automated, zero manual steps.

---

Why This Replaces the Manual Process for Good

The old report wasn't just slow, it was fragile. A staffer on leave, a job board that quietly changed its layout, the same role copied in twice: any of these put errors into the recruiters' Monday feed, and nobody noticed until a client did.

The pipeline removes that risk in three ways the manual process never could:

• Complete coverage. Every platform is checked on every run, so nothing slips through because someone was busy.
• No duplicates. The same role posted on three different boards shows up once. In one weekly run, more than half of everything collected — 6,952 of 13,474 records — was duplicate postings, all removed automatically.
• It survives change. Each board is handled independently. If one changes or goes offline, the other six still deliver, and we're alerted to fix the one that broke before it ever reaches the feed.

That last point is the real difference between a one-off scraper and a feed you can build a business on. Job boards change constantly. The question is never if one breaks, but whether your Monday report quietly goes stale when it does. This one doesn't.

---

The Output

Three files, in their inbox before Monday standup: a clean CSV of every deduplicated listing, a facility-level rollup, and a written weekly report. No spreadsheets to assemble. If any of the seven integrated platforms becomes unreachable during a run, the others continue and the failure surfaces at exit, so a single broken board never means a missing Monday feed.

---

What This Means for You

If your team pulls candidate or market data from SEEK, Indeed, or any spread of job boards, you already know the real problem is not writing one scraper. It is that every platform hides its data somewhere different, and a pipeline that is not built to survive one of them changing will quietly go stale, usually right when you are relying on it most.

Everything we pull comes from publicly accessible listing pages, the same data a recruiter sees browsing the site, just gathered reliably and at scale. After the sample, going live is straightforward: we scope an ongoing feed to your platforms and delivery cadence, stand up the pipeline, and monitor it so a board changing never means a stale report.

We run free 500-record proof-of-concept extractions: give us the platform your team is struggling with. We extract a clean sample, format it to your spec, and deliver overnight.

No invoice. Just proof our infrastructure works.

Request Your Free Sample →

---

We Were Asked to Cover Ten Boards. Seven Made the Cut.

The brief was ten Australian job boards, delivered as one clean CSV every Monday. Three of them genuinely couldn't be reached, and we'd rather say that up front than pad a number:

• HealthcareLink locks every listing behind a user login.
• Queensland Health SmartJobs runs on an enterprise system that returns nothing without a live browser session.
• HealthKite turned out not to be a real site at all. Its domain resolves to an empty placeholder page.

The remaining seven became the feed. Every one of them hid its data in a completely different place, which is exactly why a single off-the-shelf scraper never works here, and why the manual process was the only thing that had worked until now.

---

Under the Hood: How Each Board Was Integrated

Before writing a single line of code, the rule was simple: open the browser's Network tab on each platform and watch what it actually requests. The HTML is never the data source. Every one of the seven hid its data somewhere different.

SEEK: the clean one. SEEK looks like a standard job board, and for once the inside matches. The Network tab shows a clean JSON call to /api/jobsearch/v5/search: versioned, structured, reliable. Each response includes full job objects with listingDate, advertiser details, and a classifications array carrying 30+ sub-classification labels. Those labels needed mapping. SEEK uses strings like "Nursing - A&E, Critical Care & ICU" and "Physiotherapy, OT & Rehabilitation", all of which had to be collapsed into the 4 canonical categories the client could actually work with.

EthicalJobs: the data hides inside the page. EthicalJobs is built on Next.js, and the Network tab shows no obvious API call, because there isn't one. All job data arrives embedded inside self.__next_f.push([1, "..."]) script chunks in the initial HTML response. On the search-results page, each chunk is prefixed with a hex stream identifier that has to be stripped before the JSON inside can be parsed. Worse, the posting date, approvedAt, doesn't exist in the listing payload at all. It only appears on the individual job detail page. That means one additional HTTP request per job, at 0.5-second intervals, to recover a single field the client's 7-day window depends on.

Jora: page two fights back. Jora page 1 loads cleanly using the canonical keyword slug URL, au.jora.com/{Keyword}-jobs. The obvious instinct for page 2 is to try the /j mobile endpoint with ?p=2. That returns an HTTP 403 — gated behind Cloudflare and a per-page token. The slug URL, it turns out, paginates freely using ?p=N with no token at all. Two smaller bugs surfaced alongside it. Jora duplicates the job title text inside the link's inner DOM, producing "Registered Nurse Registered Nurse", so every title needed a word-split deduplication check before reaching the output. And dates arrive as relative strings ("2d ago", "1w ago") that had to be converted to ISO dates at parse time.

CareerOne: a full page with zero jobs in it. CareerOne returns a complete, well-formed HTML page containing zero job listings. The page bootstraps a Nuxt.js JavaScript runtime, and listings are injected client-side only after that JS executes. The real data comes from a REST API endpoint visible in the Network tab. You call GET /csrf-cookie first, which automatically sets a seekerapi_session cookie and an XSRF-TOKEN. The token is extracted from the cookie jar and attached as an x-xsrf-token header on every subsequent search request. Two additional custom headers are required that no documentation mentions: platform-code: careerone and site-code: careerone.

Indeed AU: the mobile app's private API. Indeed AU routes all job search through an internal GraphQL API, the same one their iOS app uses. The indeed-co: AU header is required and case-sensitive: lowercase returns HTTP 400. The API key is the public mobile app key, widely documented in the open-source scraping community. Date fields arrive as millisecond Unix timestamps that need dividing by 1000 before conversion. SSL certificate verification has to be disabled to reach the endpoint at all — set only on that single GraphQL call, not anywhere else in the pipeline.

WorkforceAustralia: government-clean, with quirks. WorkforceAustralia is the Australian government's jobs platform: clean JSON API, no authentication required. The non-obvious detail is pagination, which uses a skip= offset parameter rather than a page number, so skip=0, skip=20, skip=40. Every suburb field arrives in ALL CAPS ("GERALDTON", "WAGGA WAGGA") and needs title-casing before it is usable. The X-Requested-With: XMLHttpRequest header is required even though no browser session is involved.

Adzuna AU: the one with real documentation. Adzuna AU has a registered developer API with actual docs, the one straightforward integration in the set. State is extracted from location.area[1], position 1 in a hierarchical array: ["Australia", "Queensland", "Brisbane Region", "Brisbane", "South Brisbane"].

The three boards that didn't make the cut failed for concrete technical reasons: HealthcareLink locks every listing behind an authenticated session (a PHPSESSID cookie) that can't be obtained without logging in, Queensland Health SmartJobs runs on Oracle Taleo, a fully JavaScript-rendered enterprise ATS where every API path returns 404 without an active browser session, and HealthKite's domain resolves to a 114-byte placeholder page. That site does not exist.

That left seven.

---

Seven Scrapers. One Pipeline.

Each platform got its own scraper class: different endpoint, different auth flow, different pagination pattern, different response schema. None shared a data structure with any other. All seven inherit from a shared base class that handles retry logic, 3 attempts with exponential backoff between 2 and 10 seconds, plus raw dump persistence.

The EthicalJobs extraction is the most technically unusual of the seven. The RSC payload embeds raw HTML strings inside JSON string values: full job descriptions with newlines, angle brackets, and escaped sequences. json.loads() fails because those embedded newlines break the JSON parser. json.JSONDecoder().raw_decode() locates the job object boundary by finding the "job":{ marker directly in the payload string, then decodes forward from that position regardless of what sits inside the string values:

# EthicalJobs RSC payload embeds raw HTML inside JSON string values.
# json.loads() fails on embedded newlines. raw_decode() finds the
# object boundary correctly regardless of what's inside the string.
marker = '"job":{'
idx = payload.find(marker)
if idx != -1:
    decoder = json.JSONDecoder()
    job_obj, _ = decoder.raw_decode(payload, idx + len(marker) - 1)
    if isinstance(job_obj, dict) and 'approvedAt' in job_obj:
        return job_obj

EthicalJobs also produces false positives, because the platform covers every ethical sector rather than healthcare alone. A job title filter runs at the processor stage: 40+ healthcare keywords checked against every EthicalJobs title before it reaches the output. Roles like "Community Fundraising Manager" that matched on the "disability support" keyword but are not healthcare positions get dropped there.

The SEEK category mapping needed the same care. SEEK returns a subclassification.description field that can hold one of 30+ distinct strings. The common ones are mapped explicitly to the client's 4 output categories; anything unmapped falls back to a title-keyword check and is dropped only if the title isn't healthcare at all. Some SEEK results carried healthcare-adjacent titles filed under non-healthcare classifications, and those were reclassified by title keyword rather than dropped.

Every scraper writes the complete raw API response to disk before the processor runs. The processor is fully idempotent: if normalisation logic changes, new fields need adding, or the window needs widening, it re-runs against cached data without touching a single platform. Each scraper also runs inside an isolated try/except block, so if one platform goes unreachable, the other six finish their run and the error surfaces at exit.

---

13,474 Records. 4,934 Jobs.

A single weekly run pulled 13,474 raw job records. After deduplication and the 7-day window, 4,934 unique jobs remained. Of the 8,540 records that dropped out, 6,952 were the same roles appearing in different wrappers across platforms at the same time; the other 1,588 were simply older than seven days.

The same position gets posted on SEEK, Adzuna, and CareerOne on the same morning: same facility, same title, completely different URLs on each platform. URL deduplication alone never catches it.

So the pipeline runs two passes.

Pass 1 strips search artifact parameters (keywords, ref, src, UTM tags) before comparing URLs, while preserving identity parameters like Indeed's jk= job key. The same job scraped under several different keywords on one platform drops here.

Pass 2 builds a composite key from Facility Name + Job Title + State, lowercased (the fields themselves are trimmed when each record is built). When the same role appears on multiple platforms, the first occurrence survives.

# Pass 1: strip tracking params, preserve identity params like jk=
df['_url_key'] = df['Job URL'].apply(_url_key)
df = df.drop_duplicates(subset=['_url_key'])

# Pass 2: cross-platform composite key, case-insensitive
df['_composite'] = (
    df['Facility Name'] + '_' + df['Job Title'] + '_' + df['State']
).str.lower()
df = df.drop_duplicates(subset=['_composite'])

13,474 raw records in. 4,934 unique jobs out. Roughly 52% of everything collected was the same listings in different wrappers — a further 1,588 dropped out only because they were older than seven days.

---

Seven Location Formats. One Standard.

Every platform formats location differently, and none of them do it consistently.

SEEK returns a combined string: "Hawthorn, Melbourne VIC". EthicalJobs returns named regions from a fixed vocabulary: "Regional NT", "Canberra & ACT", "Flexible / Work from Home". WorkforceAustralia returns the suburb in ALL CAPS with a separate state field. Indeed returns location.admin1Code, a two-letter state code when it is present at all. CareerOne returns location_state_code. Adzuna returns a hierarchical array where state always sits at index 1.

Each scraper handles its own location extraction before saving raw data. A shared extract_state() regex then runs across all records inside the processor. A city_to_state() fallback dictionary covering roughly 60 common Australian cities, from wagga wagga to kalgoorlie to belconnen, recovers the state for platforms that return only a suburb name with no code attached.

The 7-day rolling window filter runs last. Its cutoff date recalculates at runtime on every Monday execution, so the weekly feed is always current no matter when the cron fires. pd.to_datetime(errors='coerce') handles malformed dates gracefully: they fall outside the window and drop out without crashing the pipeline.

---

4,934 unique jobs · 7 platforms unified into one feed · delivered in under 48 hours