1. The anatomy of a task module

A task module is a plain .lua file in the modules/ folder next to the Ergon binary. The script runs once per work item; you call the bot API and finish with a terminal call:

-- example_task.lua
local item = get_data()          -- the current work item
sleep(0.1)                      -- simulated work
hit("processed: " .. item)      -- terminal: store a hit, stop the script

That is the whole module. The run options (bots, timeout, retries, ...) are not part of the file — you pick them in the GUI start form.

No file but the ones you call hit/custom/ retry/fail with is ever stored. If the script simply ends, the item is recorded as kind none.

2. Terminal results

hit, custom, retry, fail and skip stop the script immediately — code after them never runs:

hit("done")
print("never printed")  -- "print" is not exported either; use bot_log()

An unhandled Lua error (e.g. an HTTP network error) is not a crash — it is treated as a retry with the error message:

http.get("https://example.com/api")   -- raises on network error -> auto retry

With pcall you can catch the error yourself and decide manually what happens — a retry, a fail, a fallback value, or anything else. pcall(fn, ...) runs fn and returns true plus the return values on success, or false plus the error message on failure:

-- retry only on network errors, but NOT on a normal 4xx response
local ok, resp = pcall(http.get, "https://example.com/api")
if not ok then
    retry("request failed: " .. tostring(resp))   -- manual retry, exact error text
end
if resp.status == 404 then fail("not found") end   -- no retry for 404s
if resp.status >= 500 then retry("server error " .. resp.status) end

This works with every raising call — HTTP requests, json.decode on invalid input, url_parse on a malformed URL, ... . A manual retry(msg) behaves exactly like the automatic one: the item is queued for another attempt (limited by max_retries), the error message lands in the task log. If you don't catch the error, the automatic retry kicks in with the same message — pcall is purely a way to take control yourself.

An attempt that ends without any terminal call is stored as kind none.

3. The bot API (globals)

These are available in every module. All arguments are plain Lua values.

get_data() -> string

Returns the current work item.

local url = get_data()

get_attempt() -> number

Returns the current attempt number (1-based).

if get_attempt() > 1 then bot_log("retrying " .. get_data()) end

get_proxy() -> string

Returns a random proxy from the task's Proxy List (empty string when none is configured). Use it as the proxy option of http.get():

local url = get_data()
local r = http.get(url, { proxy = get_proxy(), timeout = 10 })

Proxies that keep failing network requests are temporarily banned and skipped; once every proxy is banned, the whole list is reactivated and handed out again.

hit(capture, ...) -> (stops the script)

Stores the item as a hit with the given capture text(s) and ends the attempt immediately. Multiple captures can be saved with one call:

hit(resp.body)
hit("a", "b", "c")   -- three hit rows, same item

custom(text) -> (stops the script)

Like hit, but stored with kind custom — for alternative results you want to keep separate in the Hits tab.

retry(msg) -> (stops the script)

Queues the item for another attempt later. max_retries limits the number of retries after the first attempt (0 = no retries, -1 = retry forever).

if resp.status >= 500 then retry("server error " .. resp.status) end

fail(msg) -> (stops the script)

Drops the item permanently (not stored in hits, counted as failed).

if resp.status == 404 then fail("not found") end

skip(msg) -> (stops the script)

Like fail, but counted as skipped instead of failed — e.g. for items that are irrelevant.

if not resp.body:find("expected") then skip("not applicable") end

status(msg) -> nil

Sets the bot's live status shown in the GUI bot table.

status("GET " .. url)

bot_log(...) / bot_print(...) -> nil

Writes a line to the task log (visible in the task detail view). Both names are aliases.

bot_log("fetched:", url, "->", resp.status)

sleep(seconds) -> nil

Blocks this bot for the given seconds.

sleep(0.5)

4. HTTP — the full reference

Response object

Every request returns the resp table. On a network failure / invalid URL / TLS error the request raises a Lua error with the request's error message. Catch it with pcall when you want custom handling:

local ok, resp = pcall(http.get, "https://example.com/api")
if not ok then fail("request failed: " .. tostring(resp)) end  -- resp = errmsg

Request options (all optional)

The last argument of every request is a table; every field is optional and may also be passed to http.new_client() as session defaults:

One-shot requests

Each call uses a short-lived client that is automatically closed after the item — you don't need to manage it:

body is always a string (it is sent as-is). Only POST/PUT carry a body argument — GET, DELETE, HEAD and OPTIONS do not (use params instead). A failed request raises and retries automatically:

-- one-shot GET with options (raises -> auto retry on failure)
local resp = http.get("https://api.example.com/search", {
    params        = { q = "lua", page = 2 },
    headers       = { ["User-Agent"] = "ergon/1.0", Accept = "application/json" },
    timeout       = 10,
    follow_redirects = false,
})

-- one-shot POST with a JSON body
local r2 = http.post("https://api.example.com/items", json.encode({name = "x"}),
    { headers = { ["Content-Type"] = "application/json" } })
if r2.status >= 500 then retry("server error " .. r2.status) end

Session clients (http.new_client)

Create a client once per item with session-wide defaults (auth, proxy, TLS, timeout, redirects) and reuse it for many requests. Options can still be overridden per request. The client is registered on the runner and auto-closed when the item ends — calling client:close() is optional:

local client = http.new_client({
    proxy       = "http://user:pass@proxy.example:823",
    insecure    = false,
    user        = "alice",
    pass        = "s3cret",
    timeout     = 20,
    follow_redirects = true,
    headers     = { ["User-Agent"] = "ergon/1.0" },
})
local r = client:get("https://api.example.com/users/me")   -- raises -> retry
-- per-request overrides
local r2 = client:get("https://api.example.com/search", {
    params = { q = "lua" }, timeout = 5, headers = { Accept = "application/json" },
})

Complete example — every method, session + one-shot, every option

-- http_full.lua: exercises every request function and every option.
local url = get_data()

-- ---- session client with all session-wide options ----
local client = http.new_client({
    timeout          = 20,
    follow_redirects = true,
    user             = "alice",
    pass             = "wonderland",
    insecure         = false,
    proxy            = "",                -- e.g. "http://proxy:8080" or "socks5://proxy:1080"
    headers          = { ["User-Agent"] = "ergon/1.0" },
})

status("session GET")
local r = client:get(url .. "/get", { params = { a = "1", b = "2" } })

status("session POST")
local p = client:post(url .. "/post", "name=lua&kind=demo",
    { headers = { ["Content-Type"] = "application/x-www-form-urlencoded" } })

status("session PUT")
local pu = client:put(url .. "/put", json.encode({ ok = true }),
    { headers = { ["Content-Type"] = "application/json" }, timeout = 5 })

status("session DELETE")
local d = client:delete(url .. "/delete", { timeout = 10 })

status("session HEAD")
local h = client:head(url .. "/head")

status("session OPTIONS")
local o = client:options(url .. "/options")

status("session request(PATCH)")
local pa = client:request("PATCH", url .. "/patch", "partial",
    { headers = { ["Content-Type"] = "text/plain" } })

-- ---- one-shot requests (short-lived client, auto-closed) ----
status("one-shot GET")
local g1 = http.get(url .. "/get", { params = { q = "lua" } })

status("one-shot POST")
local g2 = http.post(url .. "/post", "x=1", { timeout = 15 })

status("one-shot request")
local g3 = http.request("DELETE", url .. "/delete")

-- arbitrary request method with a body via http.request
local g4 = http.request("PATCH", url .. "/patch", "data", { follow_redirects = false })

-- ---- evaluate ----
local last = r
if p then last = p end
if last.status >= 500 then retry("server error " .. last.status) end
if last.status >= 400 then fail("HTTP " .. last.status) end

status("done")
bot_log("last status:", last.status)
hit(last.status .. " " .. string.sub(last.body, 1, 200))

5. Mail — IMAP / POP3

Fetch emails straight from a mailbox. Both protocols expose the same API: login() returns a session client, fetch() reads messages, close() ends the session (optional — like the HTTP clients, a session is automatically closed when the item ends).

imap.login(server, user, pass, opts?) / pop3.login(...) -> client | nil, err

The fourth argument is optional: a port number, or a table with port (default: IMAP 993, POP3 995), timeout (seconds, default 20) and insecure (skip TLS verification).

local c, err = imap.login("imap.example.com", "alice", "s3cret")
if err then fail("imap login: " .. err) end

local p, err = pop3.login("pop.example.com", "alice", "s3cret",
    { port = 995, timeout = 30 })

client:fetch(opts?) -> emails | nil, err

Options (all optional):

local mails, err = c:fetch({ folder = "INBOX", limit = 10, since = "3d", unread = true })
if err then fail("fetch: " .. err) end
for _, m in ipairs(mails) do
    bot_log(m.date, "|", m.from, "|", m.subject)
    -- m.id, m.date, m.from, m.subject, m.body
end

Fetch everything

limit = 0 fetches the whole mailbox (every message, every field). Be careful with big mailboxes — the bodies of all messages are returned at once, so combine it with since or a reasonable limit unless you really want the entire archive:

-- fetch ALL messages of INBOX, full bodies included
local all, err = c:fetch({ limit = 0, body = true })
if err then fail("fetch all: " .. err) end
bot_log("fetched " .. #all .. " messages")

-- practical alternative: everything, but only the newest 5000
local many, err = c:fetch({ limit = 5000, since = "30d" })
for _, m in ipairs(many) do
    hit(m.subject)   -- or process / store each message
end

The body is the plain-text part when present, otherwise the HTML part. login and fetch return nil, err on failure — catch them with pcall or handle them like any raised error.

6. Browser automation

Drive a real browser from Lua through a WebDriver binary. Pass the path to the driver (geckodriver for Firefox, chromedriver for Chrome), not the browser itself — the driver kind is auto-detected from the filename:

local b = Browser("/usr/bin/geckodriver")   -- or "C:\\tools\\chromedriver.exe"
b:Get("https://example.com")
local title = b:Title()                    -- the page title
local el = b:Find(b.selector.id, "login")  -- find an element by id
el:Click()                                 -- click it
el:Type("user@example.com")                -- type into it
local txt = b:Find(b.selector.css, "h1"):Text()  -- read its text
b:Screenshot("/tmp/shot.png")              -- save a screenshot
b:Quit()                                   -- close the browser + driver

Instead of magic strings, Ergon exposes the selector kinds as constants on the browser object:

b:Find(b.selector.id, "email")               -- <input id="email">
b:Find(b.selector.name, "password")          -- <input name="password">
b:Find(b.selector.tag, "button")             -- first <button>
b:Find(b.selector.class, "alert")            -- first .alert
b:Find(b.selector.link_text, "Sign up")      -- <a>Sign up</a>
b:Find(b.selector.partial_link_text, "Sign") -- <a>Sign in</a>
b:Find(b.selector.xpath, "//div[@id='menu']//a")   -- full XPath
b:Find(b.selector.css, "form input[type='text']")   -- CSS selector

The plain string values work too ("id", "name", "tag name", "class name", "link text", "partial link text", "xpath", "css selector"), but the constants catch typos immediately — an unknown selector kind raises an error right away instead of waiting.

Every Find waits for the element: it polls until it appears (with a default timeout of 10 s). If the element never appears, Find raises an error and the item is retried. Pass a different timeout as the third argument, or a state as the fourth ("visible" / "enabled"):

local el = b:Find(b.selector.id, "result")                  -- wait up to 10s
local el = b:Find(b.selector.id, "result", 30)              -- wait up to 30s
local btn = b:Find(b.selector.id, "submit", 10, "enabled")  -- waits up to 10s for it to be enabled
btn:Click()

Browser(path) only prepares the object; the driver starts on the first command. The browser is closed automatically at the end of the item — even when an error causes a retry — so no driver process is ever left behind. Calling Quit() explicitly just closes it sooner.

Options

An optional second argument configures the browser. Everything is optional:

local b = Browser("/usr/bin/geckodriver", {
    user_agent  = "Mozilla/5.0 ...",      -- override the browser user agent
    proxy       = "socks5://127.0.0.1:9050",  -- or "host:port"
    headless    = true,                   -- no visible window
    window_size = "1920,1080",
    browser_path = "/usr/bin/firefox",    -- use a specific browser binary
    no_sandbox  = true,                   -- add Chrome --no-sandbox flags
    args        = { "--enable-unsafe-swiftshader" },  -- extra CLI args
})

Methods

• Get(url) — navigate to a URL.
• Find(by, value, timeout?, state?) — wait for and return an element (default timeout 10s; optional state "visible"/"enabled"; raises on timeout.
• Title() — the page title.
• Url() — the current URL.
• Click(by?, value?) — click an element (or the page).
• Type(by, value, text) — type into an element.
• Text(by, value) — read an element's text.
• Screenshot(file?) — save a PNG (or return the bytes).
• Sleep(ms) — wait (only use when a short delay is enough).
• Close(), Quit() — end the session / driver.

Elements returned by Find/Wait have :Click(), :Type(text) and :Text().

7. JSON

json.encode(value) -> string

Encodes any Lua value (numbers, strings, booleans, nil, nested tables — array-like tables become JSON arrays).

json.encode({name = "ergon", tags = {"a", "b"}})
-- "{\"name\":\"ergon\",\"tags\":[\"a\",\"b\"]}"

json.decode(s) -> any

Parses a JSON string into Lua values (raises an error on invalid JSON — which becomes a retry).

local ip_json = {"ip": "127.0.0.1."}
local data = json.decode(ip_json)   -- table
local ip = data.ip -- "127.0.0.1"

8. Hashing & encoding

The Raw variants return the raw bytes (Lua string with binary data); the plain ones return a lowercase hex string.

alg is one of "md5", "sha1" (default), "sha256", "sha512", "sha3".

sha256("hello")                   -- 2cf24dba5fb0a30e26e83b2ac5b9e29e...
md5_raw("hello")                  -- "\x5d\x41\x40..."
base64_encode(hex_encode("abc"))  -- base64 of the hex of "abc"
base64_decode("aGVsbG8=")         -- "hello"
hmac("sha256", "key", "msg")      -- hex HMAC-SHA256

9. Helpers

parse_string(s, left, right, recursive?) -> string | table

Extracts the substring(s) between left and right. Without recursive the first match is returned as a string; with recursive=true all matches are returned as a table:

parse_string("x=<a> y=<b>", "<", ">")            -- "a"
parse_string("x=<a> y=<b>", "<", ">", true)   -- {"a", "b"}

split_string(s, splitter) -> table

Splits a string by a separator into a Lua array table (empty separator splits per rune):

split_string("a,b,c", ",")                    -- {"a", "b", "c"}
split_string("name=lua&kind=demo", "&")    -- {"name=lua", "kind=demo"}
local host, port = split_string("host:8080", ":")[1], split_string("host:8080", ":")[2]

contains_any(s, substrings) -> boolean

Returns true when s contains any of the substrings (stops at the first match):

contains_any("hello world", {"foo", "world"})  -- true
if contains_any(resp.body, {"captcha", "blocked"}) then retry("blocked") end

contains_all(s, substrings) -> boolean

Returns true only when s contains every one of the substrings:

contains_all("hello world", {"hello", "world"}) -- true
 contains_all("hello world", {"hello", "mars"})   -- false
 if contains_all(resp.body, {"status", "ok"}) then hit("all present") end

user_agent(browser?, os?, platform?) -> string

Returns a realistic user-agent string from the built-in database of real user agents. All arguments are optional and narrow the pool; pass nil or "any" to leave a dimension open:

user_agent()                             -- random UA
user_agent("chrome")                     -- a Chrome UA (incl. mobile)
user_agent("firefox", "android")         -- Firefox on Android
user_agent("chrome", "windows", "desktop")  -- desktop Chrome on Windows
local ua = user_agent("edge", "macos", "desktop")
client = http.new_client({ headers = { ["User-Agent"] = ua } })

Browsers: chrome, firefox, edge, opera, safari, android, samsung, yandex, twitter, facebook, miui, whale, amazon. OS: windows, linux, ubuntu, chromeos, macos, android, ios. Platform: desktop, tablet, mobile.

url_encode(s) -> string

url_decode(s) -> string | nil, err

Percent-encode / decode a string for URL query values. Useful when you build a query yourself instead of using params:

url_encode("a b/c")               -- "a+b%2Fc"
local q = url_encode(get_data())
local resp = http.get("https://api.example.com/search?q=" .. q)
url_decode("a+b%2Fc")              -- "a b/c"

url_parse(s) -> table | nil, err

Splits a URL into its components. Returns a table with the fields scheme, host, port, path, query, params (decoded query as a table), fragment, user, pass — on an invalid URL it returns nil, err:

local u = url_parse("https://user:secret@api.example.com:8443/x?q=lua#top")
u.scheme      -- "https"
u.host        -- "api.example.com"
u.port        -- "8443"
u.path        -- "/x"
u.params.q    -- "lua"
u.fragment    -- "top"
u.user        -- "user"
u.pass        -- "secret"

html_unescape(s) -> string

Decodes HTML entities back to plain text (  & < > " ' …) — the typical last step after extracting text from a scraped page:

html_unescape("Tom & Jerry   42 < 100")
-- "Tom & Jerry  42 < 100"
html_unescape(resp.body)  -- decode an entire HTML page's entities

html_to_text(s) -> string

Converts an HTML fragment or page to readable plain text — tags are stripped, text is kept, block elements become line breaks. The natural complement to html_unescape when you want the visible text of a scraped page (and it's what the mail feature returns for HTML-only message bodies):

html_to_text("<h1>Hi</h1><p>Hello <b>world</b> &amp; more</p>")
-- "Hi
-- Hello world & more"

local text = html_to_text(resp.body)  -- visible text of a page

UUIDs

namespace is a UUID string; it defaults to the DNS namespace. All return a canonical 36-character lowercase string.

uuid4()                     -- "1d2f4a5e-...-..."
uuid3("bob@example.com")    -- deterministic for the same name

random:random(...) / random_string(chars, len)

random is a global object. random:random(...) picks a random value — an element from a table, an integer, or a float; the plain random(...) call does the same:

random:random({"a", "b", "c"})    -- one of the three
random:random(1, 6)                -- die roll: 1..6
random:random(0.1, 1.0)            -- float 0.1..1.0
random:random(3)                   -- 1..3
random(1, 6)                       -- same as random:random(1, 6)

random:random_string(chars, length) builds a random string of the given length drawn from the characters in chars. Ready-made character sets are available as fields, exactly like Python's string module — and you can combine them with ..:

random:random_string("ABC", 3)                          -- e.g. "AAC"
random:random_string(random.digits, 6)                  -- e.g. "485021"
random:random_string(random.ascii_letters, 16)          -- random letters
random:random_string(random.ascii_letters .. random.digits, 30)  -- mix

-- available sets: ascii_letters, ascii_lowercase, ascii_uppercase,
-- digits, hexdigits, punctuation
random.ascii_letters   -- "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ"
random.digits          -- "0123456789"

10. File I/O

write_file(path, string|table) -> true | nil, err

Writes a string, or a table as one line per entry, to a file. Parent directories are created automatically.

read_file(path, remove_duplicates?) -> table | nil

Reads a file and returns a table of trimmed lines, or nil when the file is missing or empty.

A bare/relative path is resolved against the folder the binary was started from; an absolute path is used as-is:

write_file("results.txt", {"a", "b", "c"})   -- one line per entry
write_file("log.txt", "some text")            -- raw string

local lines = read_file("results.txt")        -- {"a", "b", "c"}
local uniq  = read_file("results.txt", true)  -- duplicates removed
local none  = read_file("missing.txt")        -- nil

11. Run options (GUI start form)

You don't write these anywhere — they are the fields of the start form when you create a task. In plain words:

• Bots — how many workers run in parallel. More bots = more speed, but more load on the target. Start with 5.
• Timeout (s) — how long a single attempt may take. If it takes longer, the attempt is aborted and the item retried.
• Max retries — the number of retries after the first attempt before an item is given up and recorded as skipped. 0 = no retries at all, -1 = retry forever.
• Retry delay (s) — a small pause between retries, so a struggling server isn't hammered (there is always a minimum of 0.05 s).
• Proxy List — an optional proxy list for get_proxy(). The optional Ban after field temporarily removes a proxy after that many failed requests (0 = ban after the first failure, -1 = never ban).
• Skip items already in hits — re-running the same list skips everything that already produced a hit. The optional task field checks only against one task's hits (empty = any hit counts).

A task always runs against a work list: in the Work Lists tab you register a text file (one item per line) by name and path. Nothing is imported — the file stays the source of truth, and when a task starts it streams the lines from that file into its queue, so even a list with millions of lines starts immediately without loading it into memory.

Module options (the OPTIONS table)

If your module defines a global OPTIONS table with simple literal values (numbers, strings, booleans), it is shown as editable fields in the start form. The edited values override the defaults in the module — the module just reads OPTIONS:

OPTIONS = {
    min_anon_level = 2,
    proxy_test_url = "https://httpbin.org/anything",
    proxy_location_url = "https://ipinfo.io/json",
}

-- read the (possibly edited) values anywhere in the module
local min = OPTIONS.min_anon_level

OPTIONS must be global, must be a table, and its values must be simple literals. Nested tables are preserved. Modules without an OPTIONS table show no extra fields.

Module description (block comment)

Above the options, the start form shows the module's description if the file starts with a --[[ ... ]] block comment (before anything else). Use it to explain what the module does and how it reports results:

--[[
Sends a GET to the target and reports the HTTP status as a hit.
]]

OPTIONS = { my_option = "Hello" }

The comment must start at the first non-whitespace byte of the file and be closed with ]]. Modules without one simply show no description box.

12. Debug tab

The Debug → Run tab executes your module once with a single work item (the "data" field) and shows:

• the terminal result (kind + capture),
• every status event and bot_log line in chronological order,
• every HTTP request/response — method, URL, request headers, status, response headers and body — because all HTTP flows through the Go exports. This works for one-shot requests and session clients alike.

The Editor tab lets you edit and save modules.

13. Complete example module

-- fetch.lua: one item = one URL. GET it, validate, store a hit.
-- If a request fails or the server reports 5xx, the item is retried
-- automatically (Lua error / retry()).
-- 404s and 4xx are dropped as fails.

local url = get_data()

status("GET " .. url)
local client = http.new_client({ timeout = 15, follow_redirects = true })
local resp = client:get(url)   -- raises -> auto retry on failure

if resp.status >= 500 then retry("server error " .. resp.status) end
if resp.status >= 400 then fail("HTTP " .. resp.status) end

status("done")
bot_log("fetched:", url, "->", resp.status)
hit(resp.status .. " " .. string.sub(resp.body, 1, 300))

That module (plus this tutorial) ships in the build folder. For editor autocompletion, modules/stubs/ergon.lua contains an EmmyLua-style stub of every exported function.