Run a raw OpenSearch DSL query and return its hits plus aggregations.
Use this for full DSL control over the query and aggregation bodies. When you
only need a match count and not the documents, use count. For Malcolm's
simpler field-filter syntax instead of raw DSL, use malcolm_search.
Aggregations honor the time filter inside the DSL body, so there is no hidden
default time window. Returns the raw OpenSearch _search response.
Both input guards run before any request leaves this server: malformed
query_dsl, and an index containing /, ? or .., are refused as input
errors rather than costing an upstream scan. When the query is easier to
say as an Arkime expression, compile it with arkime_build_query and hand
the index and query_dsl it returns straight to this tool — serialise its
query_dsl object to a JSON string first, which is what this parameter
declares.
Count documents matching a DSL query clause, without returning the documents.
Use this instead of search_dsl when you only need the number of matches, not
the documents themselves. Note the query_dsl shape differs from search_dsl's —
the schema says how. Returns the raw OpenSearch _count response
({"count": N, ...}).
This tool takes no time arguments and applies no default window, so a
bare call counts everything the index still holds, which on any real
capture is millions of documents. Bound it with a range clause inside
query_dsl, use malcolm_search when you want a human-readable time range,
or arkime_sessions_summary when you want byte and packet totals beside
the count.
List indices with their health, status, and document count.
Use this to discover which indices exist before querying one. For the field
schema (field names and types) of a single index, use index_mapping instead;
for cluster-wide health rather than per-index status, use cluster_health.
Returns a JSON array, one object per index, with name, health, status, and doc
count.
This reads OpenSearch's index list directly, so Malcolm's own internals
come back beside the traffic, and most of what is listed holds no
network data at all (.kibana_1, .opendistro_security, the arkime_*_v*
config indices, top_queries-*). The traffic is in the arkime_sessions3-*
indices alone; Arkime opens a new one per day, so their number grows and
the newest is usually still empty — read "docs.count" rather than the
name to find the one carrying the capture. A pattern matching nothing
returns an empty array, not an error. "health" is a shard-replication
fact and says nothing about whether capture is still arriving —
malcolm_data_coverage answers that.
Return one index's field mapping: every field name and its OpenSearch type.
Use this to learn what fields an index holds and how they are typed before
writing a DSL query against it. To list which indices exist rather than inspect
one index's schema, use list_indices. For Malcolm's non-standard field names
across all indices, malcolm_field_search is easier than reading raw mappings.
Returns the raw OpenSearch _mapping response; a non-existent index yields an
OpenSearch error in the response body.
A wildcard returns one mapping block per matching index rather than a
merged one, and each block repeats the whole schema: "arkime_sessions3-*"
costs roughly a megabyte of JSON, growing by another block every day
Arkime opens a new index. Name ONE index when you only need the schema —
the blocks are near-identical. The types it reports are OpenSearch's own
(keyword, long, text), while malcolm_field_search reports Malcolm's names
for the same fields (string, integer) — so come here only when the
OpenSearch type is what you need.
Report OpenSearch cluster health: green/yellow/red status plus node and shard counts.
This checks the storage backend (OpenSearch) itself, cluster-wide. To check
whether the Malcolm API is reachable, use malcolm_ping; for the readiness of
Malcolm's individual services, use malcolm_service_status; for per-index
status rather than the whole cluster, use list_indices. Returns the raw
OpenSearch _cluster/health document.
This is a storage-layer answer only: every shard allocated says nothing
about whether packets are still being captured or parsed. Measured on
Malcolm v26.07.1 (single node) the steady state is green with
number_of_nodes=1 and unassigned_shards=0, so treat yellow as something
to explain rather than as normal. For whether data is still arriving use
malcolm_data_coverage; for whether a capture node is dropping packets use
arkime_node_stats.
Search Malcolm's indexed network traffic using Malcolm's simple filter dict.
Use this for field-based filtering with human-readable time ranges. To
search with Arkime expression syntax instead, or when you need a session
id to feed arkime_session_pcap / arkime_add_tags afterward, use
arkime_sessions (only its rows carry that id). For raw OpenSearch DSL,
use search_dsl. Confirm field names with malcolm_field_search first —
Malcolm uses non-standard names. Returns the raw Malcolm /mapi/document
response (matching documents); when nothing matched and a filter names a
field Malcolm does not index, the correct field name is reported above
the response.
Two defaults to know before the first call: with no time_from this
searches ALL retained history, where malcolm_aggregate covers only the
last 24 hours; and filter values are matched exactly, so any wildcard or
substring has to go to search_dsl instead.
Aggregate network traffic into top-N value buckets for one or more fields.
Use this to count distinct values (top talkers, protocol distribution)
rather than fetch documents — for the documents themselves use
malcolm_search. For distinct values of a single field with less setup,
malcolm_field_values is simpler. Returns the raw Malcolm /mapi/agg
response (bucket keys with doc counts); when no buckets came back and an
aggregated or filtered field is not one Malcolm indexes, the correct
field name is reported above the response.
With no time_from this covers only the LAST 24 HOURS, unlike
malcolm_search which covers all history. Against a capture older than a
day that returns an empty bucket list, which reads as "no such traffic"
when it means "nothing in the last day" — suspect the window before the
data.
Search Suricata alerts with structured parameters, no field knowledge needed.
Use this instead of malcolm_search when hunting Suricata alerts: it maps
each argument to the correct Malcolm field for you (you don't need to
know whether it's suricata.alert.signature or rule.name). It always
filters event.dataset=alert. These are Suricata IDS alerts, signature
matches on the wire; three other things on this server are also called
alerts and are different mechanisms — malcolm_alerting_monitors and
malcolm_alerting_alerts are the OpenSearch alerting plugin's standing
rules and their firings, malcolm_anomaly_detectors is its machine-learning
baseline, and malcolm_create_alert (alerting write class) records a
finding of your own.
Behavior: `signature` and `category` are substring searches, which Malcolm
cannot express in a filter (its filters are exact terms), so this tool
resolves the substring against the field's 500 most common values first
and filters on the matches. A substring that matches no recorded value
returns a message saying so rather than an empty result set — that is the
difference between "no such signature here" and "no alerts fired". That
pre-scan is the one place the time range bites: it reads only the last 24
hours, while the alert search itself covers ALL history when time_from is
empty, so on a capture older than a day every signature reads as
unrecorded until you pass time_from.
Returns the raw Malcolm /mapi/document response (matching alert documents).
Discover which field NAMES exist in Malcolm's index, by keyword, prefix, or type.
Use this first, before any query, to confirm a field name exists — Malcolm uses
non-standard names (e.g. http.useragent, NOT http.user_agent). To then see the
VALUES a field holds, use malcolm_field_values; to see which datasets contain
it, use malcolm_field_profile. Do NOT source an arkime_* argument from here:
these are the names malcolm_* and search_dsl take, and Arkime has its own
spelling for the same field (ip.src, srcIp) that arkime_field_search reports.
Pass at least one argument. Returns a text list of "name (type)" lines,
sorted alphabetically.
Arguments narrow (AND), they never widen, and the mapping is big enough
that one keyword rarely lands: it runs to thousands of fields, and a
keyword as common as "ip" matches over a thousand of them on its own.
The header line counts every match but only the first 100 names are
printed, so add a prefix or a field_type rather than reading the printed
list as the whole answer.
List a single field's distinct VALUES with per-value document counts.
Use this to see what values a field actually holds before filtering on it, so
you don't invent values. To confirm the field NAME exists first, use
malcolm_field_search; to see which datasets carry the field, use
malcolm_field_profile. For multi-field or nested bucketing, use
malcolm_aggregate. A "-" in the output is Malcolm's placeholder for
documents where the field is absent, not a value you can filter on.
Returns a text list of "value (N docs)" lines.
With no time range this reads only the last 24 hours, so a value that
exists only in older data is missing here and reads as invalid —
measured on Malcolm v26.07.1, network.protocol lists nothing at the
default window while its top value carries millions of documents once
time_from reaches the capture. Pass time_from before concluding a value
is not in this Malcolm.
Show which event.dataset types actually contain a given field, with doc counts.
Use this to learn where a field lives (e.g. whether it only appears in SSL or DNS
records) before scoping a query. To confirm the field NAME first, use
malcolm_field_search; to list its distinct VALUES, use malcolm_field_values.
Behavior: first resolves the name against the index mapping, then aggregates over
event.dataset. Three distinct text outcomes — (1) unknown field → a "not found"
message with close-name suggestions (no profile); (2) known field but no matching
documents in the time window → an "exists but no documents" message; (3) a
per-dataset "event.dataset=<name> (N docs)" list. The dataset counts honor the
time window: with no range it uses the last 24 hours, so a field that only has
old data can resolve as known yet profile as empty — pass time_from/time_to to
reach historical data. Returns plain text, not JSON.
Report readiness of each Malcolm service plus Malcolm version and OpenSearch health.
Call this before a hunt to confirm the whole stack is up. For a bare
is-the-API-alive check use malcolm_ping; for the OpenSearch cluster's
green/yellow/red detail alone use cluster_health; for data freshness and
per-dataset counts use malcolm_data_coverage. Returns a JSON summary with
malcolm_version, mode, opensearch_health, a per-service readiness map, and an
"N/total services ready" line. One probe failing adds an `errors` entry and
keeps the rest; both failing is reported as an error, since there is then no
status at all to report.
The readiness map is also where the optional subsystems declare
themselves — measured on Malcolm v26.07.1, 15 keys, netbox, filescan and
extracted_files among them. Read the relevant key here before taking an
empty answer from malcolm_netbox_lookup or malcolm_file_scans as "no
such asset" when it may mean "that subsystem is not deployed".
+39 more tools listed on main page