% ASNCOUNTER(1) | collect hits per ASN and netblock

# NAME

**asncounter** — collect hits per ASN and netblock

# DESCRIPTION

Count the number of hits (HTTP, packets, etc) per autonomous system
number (ASN) and related network blocks.

This is useful when you get a lot of traffic on a server to figure out
which network is responsible for the traffic, to direct abuse
complaints or block whole networks, or on core routers to figure out
who your peers are and who you might want to seek particular peering
agreements with.

# SYNOPSIS

| **asncounter** [OPTIONS] [ADDRESS ...]

# OPTIONS

\-h, --help

:   show this help message and exit

\--cache-directory, -C CACHE_DIRECTORY

:   where to store pyasn cache files, default: ~/.cache/pyasn

\--no-prefixes

:   disable prefix count

\--no-asn

:   disable ASN count

\--no-resolve-asn

:   disable ASN to name resolution in output

\--top, -t N

:   only show top N entries, default: 10

\--input, -i INPUT

:   input file, default: stdin

\--input-format, -I {line,tuple,tcpdump,scapy}

:   input format, default: line

\--scapy-filter SCAPY_FILTER

:   BPF filter to apply to incoming packets, default: ip and not src
    host 0.0.0.0 and not src net 192.168.0.0/24

\--interface [INTERFACE]

:   open an interface instead of stdin for packets, implies
    -I scapy, auto-detects by default

\--output, -o OUTPUT

:   write stats or final prometheus metrics to the given file,
    default: stdout

\--output-format, -O {tsv,prometheus,null}

:   output format, choices: tsv, prometheus, null, default: tsv

\--port, -p [PORT]

:   start a prometheus server on the given port, default disabled,
    port 8999 if unspecified

\--refresh, -R

:    download a recent RIB cache file and exit

\--dump-prefixes, -D ASN

:    dump the aggregated prefixes of the given ASN to stdout and exit

\--repl

:    run a REPL thread in main loop

\--manhole

:    setup a REPL socket with manhole

\--debug

:    more debugging output

ADDRESS

:    zero or more IP addresses to parse directly from the command line,
     before the input stream is read. disables the default stdin
     reading and `--input-format` cannot be changed.

# INPUT FORMATS

The `--input-formats` option warrants more discussion.

## line

The `line` input formats treats each line in the stream as an IP
address, counting as one hit.

Empty lines are skipped, and comments -- whatever follows the pound
(`#`) sign -- are trimmed. Whatever cannot be parsed as an IP address
is loggd as a warning and skipped.

This, for example, counts as a hit on two different IP addresses, for
a total of two hits. It will also yield a warning:

```
192.0.2.1 # comment
2001:DB8::1
# comment
garbage that generates a warning
```

## tuple

Same as the `line` input format, except the count is specified in a
second, whitespace-separated field.

This, for example, will count one hit for the first IP address, and
two for the second one, and will generate a warning.

```
192.0.2.1 1 # comment
2001:DB8::1 2
# comment
garbage that generates a warning
```

The "count" field can be anything: to represent a count, but also
sizes, timings, `asncounter` doesn't care.

The counts are actually parsed as floats, as Python understand them.

The default output format (`tsv`) will [round](https://docs.python.org/3/library/functions.html#round) the numbers to the
nearest even integer. This, for example, adds up to 5, which might be
surprising to some (because Python rounds `2.5` to `2` and not `3`):

```
192.0.2.2 3.4
192.0.2.2 2.5
```

This is known as the ["rounding half to even"](https://en.wikipedia.org/wiki/Rounding#Rounding_half_to_even) rule or the [IEEE
754 standard](https://en.wikipedia.org/wiki/IEEE_754).

If the `--output-format` is set to `prometheus` floats will be
recorded as accurately as Python allows. In that context, the above
correctly sums up to `5.9`.

## tcpdump

The tcpdump format is a bit of an odd ball: it parses a tcpdump(1)
line with a regular expression to extract the source IP address, and
counts that.

It *could* be extended to count the packets *sizes* but currently does
not do so. Likewise, it only tracks the *left* (source) side of
packets, and not the destination, but could be extended to track both.

This approach likely can't deal with a multi-gigabit per second small
packet attack (2 million packets per second or more). But in a real
production environment, it could easily deal with regular the 100-200
megabit per second traffic, where tcpdump and asncounter each took
about 2% of one core to handle about 3-5 thousand packets per second.

## scapy

The `scapy` input format is also special: instead of parsing text
lines, it parses packets.

With the `--interface` flag, it will open the default interface unless
one is provided (e.g. `--interface` is generally equivalent to
`--interface eth0` if `eth0` is the primary interface). This requires
elevated privileges.

This is much slower than the tcpdump parser (close to full 100% CPU
usage) in a 100-200mbps scenario like above, but could eventually be
leveraged to implement byte counts, which are harder to extract from
tcpdump because of the variability of its output.

This only counts packets, regardless of direction, and, like
`tcpdump`, only keeps track of source IP addresses. Like `tcpdump`, it
could also be improved by tracking sizes instead of counts, but does
not currently do so.

# OUTPUT FORMATS

The `--output-format` argument also warrants a little more discussion.

## tsv

TSV stands for Tab-Separated Values. It's a poorly designed output
formats that dumps two tables where rows are separated by newlines and
columns by tabs. One table shows per ASN counts, the other shows per
prefix counts.

As mentioned in the above `tuple` section, counts are rounded when
recorded in `tsv` mode. This is to simplify the display, in theory,
the underlying [Counter](https://docs.python.org/3/library/collections.html#collections.Counterfl) supports floats as well.

If more precision, long term storage, or alerting are needed, the
`prometheus` output format is preferred.

This format is useful because it doesn't require any dependency
outside of the standard library (and, obviously, `pyasn`).

## prometheus

The `prometheus` output format keeps tracks of counters inside
Prometheus data structures. With the `--port` flag, it will open up a
port (defaulting to `8999`) where metrics will be exposed over HTTP,
without any special security, on all interfaces.

Otherwise, upon completion, results will be written in a [textfile
collector](https://github.com/prometheus/node_exporter#textfile-collector)-compatible format.

## null

The `null` output formats doesn't display anything. It can be used for
debugging, but internally uses the same recorder as the `tsv` format.

# EXAMPLES

## Simple web log counter

This extracts the IP addresses from current access logs and reports ratios:

```
> awk '{print $2}' /var/log/apache2/*access*.log | asncounter
INFO: using datfile ipasn_20250527.1600.dat.gz
INFO: collecting addresses from <stdin>
INFO: loading datfile /home/anarcat/.cache/pyasn/ipasn_20250527.1600.dat.gz...
INFO: finished reading data
INFO: loading /home/anarcat/.cache/pyasn/asnames.json
count	percent	ASN	AS
12779	69.33	66496	SAMPLE, CA
3361	18.23	None	None
366	1.99	66497	EXAMPLE, FR
337	1.83	16276	OVH, FR
321	1.74	8075	MICROSOFT-CORP-MSN-AS-BLOCK, US
309	1.68	14061	DIGITALOCEAN-ASN, US
128	0.69	16509	AMAZON-02, US
77	0.42	48090	DMZHOST, GB
56	0.3	136907	HWCLOUDS-AS-AP HUAWEI CLOUDS, HK
53	0.29	17621	CNCGROUP-SH China Unicom Shanghai network, CN
total: 18433
count	percent	prefix	ASN	AS
12779	69.33	192.0.2.0/24	66496	SAMPLE, CA
3361	18.23	None		
298	1.62	178.128.208.0/20	14061	DIGITALOCEAN-ASN, US
289	1.57	51.222.0.0/16	16276	OVH, FR
272	1.48	2001:DB8::/48	66497	EXAMPLE, FR
235	1.27	172.160.0.0/11	8075	MICROSOFT-CORP-MSN-AS-BLOCK, US
94	0.51	2001:DB8:1::/48	66497	EXAMPLE, FR
72	0.39	47.128.0.0/14	16509	AMAZON-02, US
69	0.37	93.123.109.0/24	48090	DMZHOST, GB
53	0.29	27.115.124.0/24	17621	CNCGROUP-SH China Unicom Shanghai network, CN
```

This can also be done in real time of course:

```
tail -F /var/log/apache2/*access*.log | awk '{print $2}' | asncounter
```

The above report will be generated when the process is killed. Send
`SIGHUP` to show a report without interrupting the parser:

    pkill -HUP asncounter

You can count sizes with `--input-format=tuple` as well. Assuming the
size field is in the 10th column, this will sum sizes instead of just
number of hits:

```
tail -F /var/log/apache2/*access*.log | awk '{print $1 $10}' |
asncounter --input-format=tuple
```

If logs hold that information, you can also add up processing times,
for example.

## tcpdump parser

Extract IP addresses from incoming TCP/UDP packets on `eth0` and
report the top 5:

```sh
> tcpdump -c 10000 -q -i eth0 -n -Q in "(udp or tcp)" | asncounter --top 5 --input-format tcpdump
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
INFO: collecting IPs from stdin, using datfile ipasn_20250523.1600.dat.gz
INFO: loading datfile /root/.cache/pyasn/ipasn_20250523.1600.dat.gz...
INFO: loading /root/.cache/pyasn/asnames.json
ASN     count   AS
136907  7811    HWCLOUDS-AS-AP HUAWEI CLOUDS, HK
8075    254     MICROSOFT-CORP-MSN-AS-BLOCK, US
62744   164     QUINTEX, US
24940   114     HETZNER-AS, DE
14618   82      AMAZON-AES, US
prefix  count
166.108.192.0/20        1294
188.239.32.0/20 1056
166.108.224.0/20        970
111.119.192.0/20        951
124.243.128.0/18        667
```

A query similar to the HTTP log parser might be:

```sh
tcpdump -i eth0 -n -Q in "tcp and (port 80 or port 443)" | grep 'Flags \[S\]' | asncounter --input-format=tcpdump --repl
```

... otherwise you will get different results from a pure packet count,
as various connections will yield different number of packets! The
above counts connection attempts, which is still different than an
actual HTTP hit, as the connection could be refused before it reaches
the webserver or aborted before it gets logged properly.

It's still a good estimate, and is especially useful if you do not log
IP addresses, for example on high traffic caching servers.

Note that we use `grep` above because tcpdump's `tcp[tcpflags] &
tcp-syn != 0` only works for IPv4 packets, a disappointing (but
understandable) limitation.

## scapy parser

Extract IP addresses directly from the network interface, bypassing
tcpdump entirely:

```sh
asncounter --interface
```

## REPL

With `--repl`, you will drop into a Python shell where you can
interactively get real-time statistics:

```sh
> awk '{print $2}' /var/log/apache2/*access*.log | asncounter --repl --top 2
INFO: using datfile ipasn_20250527.1600.dat.gz
INFO: collecting addresses from <stdin>
INFO: starting interactive console, use recorder.display_results() to show current results
INFO: recorder.asn_counter and .prefix_counter dictionaries have the full data
Python 3.11.2 (main, Apr 28 2025, 14:11:48) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> INFO: loading datfile /home/anarcat/.cache/pyasn/ipasn_20250527.1600.dat.gz...
INFO: finished reading data

>>> recorder.display_results()
INFO: loading /home/anarcat/.cache/pyasn/asnames.json
count	percent	ASN	AS
13008	69.38	66496	SAMPLE, CA
3422	18.25	None	None
total: 18748
count	percent	prefix	ASN	AS
13008	69.38	192.0.2.0/24	66496	SAMPLE, CA
3422	18.25	None		
total: 18748
>>> recorder.asn_counter
Counter({66496: 13008, None: 3422, [...]})
>>> recorder.prefix_counter
Counter({'192.0.2.0/24': 13008, None: 3422, [...]})
```

So you can get the actual number of hits for an AS, even if it's not
listed in the `--top` entries with:

```python
>>> recorder.asn_counter.get(66496)
13008
```

## Blocking whole networks

asncounter does not block anything: it only counts. Another mechanism
needs to be used to actually block attackers or act on the collected
data.

If you want to block the network blocks, you can use the shown
netblocks directly in (say) Linux's netfilter firewall, or Nginx's
[access module](https://nginx.org/en/docs/http/ngx_http_access_module.html) [geo module](https://nginx.org/en/docs/http/ngx_http_geo_module.html). For example, this will reject
traffic from a network with iptables:

    iptables -I INPUT -s 192.0.2.0/24 -j REJECT 

or with nftables:

    nft insert rule inet filter INPUT 'ip saddr 192.0.2.0/24 reject'

This will likely become impractical with large number of networks,
look into [IP sets](https://wiki.nftables.org/wiki-nftables/index.php/Sets) to scale that up.

With Nginx, you can block a network with the `deny` directive:

    deny 192.0.2.0/24;

This will return a `403` status code. If you want to be fancier, you
can return a tailored status code and build a larger list with the
`geo` module:

    geo $geo_map_deny {
        default 0;

        192.0.2.0/24 1;
    }

    if ($geo_map_deny) {
      return 429;
    }

Many networks can be listed in the `geo` block relatively effectively.

[pyasn][] doesn't ([unfortunately](https://github.com/hadiasghari/pyasn/issues/82)) provide an easy command line
interface to extract the data you need to block an *entire* AS. 

`asncounter` provides you with the `--dump-prefixes` flag for that
purpose:

    asncounter --dump-prefixes 64496

You can also Python, from inside the `--repl` loop:

```python
print("\n".join(sorted(recorder.asn_all_prefixes(64496))))
```

This will give you the list of ALL prefixes associated with AS64496,
which is actually empty in this case, as AS64496 is an example AS from
[RFC5398][].

Note the list of prefixes is not aggregated by default in that Python
code, but it is in `--dump-prefixes`, if `netaddr` is installed.  You
can pass `aggregate=True` to reduce the set from Python in that case.

 [RFC5398]: https://www.rfc-editor.org/rfc/rfc5398.html

### Aggregating results

It might be worth aggregating large number of netblocks for
performance reasons. Network block announcements can be spread in
multiple contiguous blocks for various reasons and can often be
unified in smaller sets. For IPv4-only, [iprange](https://github.com/firehol/iprange/) is good (and
fast) enough:

```
> grep -v :: networks > networks-ipv4
> iprange < networks-ipv4 > networks-ipv4-filtered
> wc -l networks*
  588 networks
  495 networks-ipv4
  181 networks-ipv4-filtered
```

If you have it installed, the [netaddr](https://netaddr.readthedocs.io/en/latest/reference/cli.html) Python package can also do
that for you, and it supports IPv6:

```python
import netaddr
print("\n".join([str(n) for n in netaddr.cidr_merge(recorder.asndb.get_as_prefixes(64496))]))
```

Note that asncounter can aggregate those results directly now, for
example:


```python
print(recorder.asn_all_prefixes_str(66496, aggregate=True))
```

... but, as above, it requires the `netaddr` package to be
available. This is also done automatically by the `--dump-prefixes`
flag, as mentioned above:

    asncounter --dump-prefixes 64496

### Selective blocking

A more delicate approach is to block all network blocks from a
specific ASN that have been found in the result sets, instead of
blocking the entire netblock.

The `recorder.as_prefixes` and `recorder.as_prefixes_str` functions
can do this for you, merging multiple ASNs and aggregating with
netaddr as well:

```python
print(recorder.asn_prefixes_str(66496, 66497, aggregate=True))
```

Note that the `asn_prefixes` selectors are not implemented in
Prometheus mode.

Remember you can extract the list of current ASNs and prefixes just by
looking at the dictionary keys as well:

```python
print("\n".join(recorder.asn_counter.keys()))
print("\n".join(recorder.prefix_counter.keys()))
```

This can't be done with the `--dump-prefixes` flag, which starts from
scratch and dumps the entire AS, not just the networks encountered
during parsing.

# ENVIRONMENT

- `$XDG_CACHE_HOME/.cache/pyasn`: where to store the pyasn cache files
- `$XDG_RUNTIME_DIR` or `$XDG_STATE_HOME/.local/state`: where the
  manhole socket will be created if --manhole is used without arguments

# SIGNALS

- `SIGHUP`: display the current statistics

# FILES

*~/.cache/pyasn/*

:   Default storage location for pyasn cache files.

*/run/\$UID/asncounter-manhole-\$PID* or *~/.local/.state/asncounter-manhole-\$PID*

:   Default location for the debugging manhole socket, if enabled.

# LIMITATIONS

- only counts, does not calculate bandwidth, but could be extended to
  do so
- does not actually do any sort of mitigation or blocking, purely an
  analysis tool; if you want such mitigation, hook up `asncount` in
  Prometheus and AlertManager with web hooks, this is not a `fail2ban`
  rewrite
- test coverage is relatively low, 37% as of this writing. most
  critical paths are covered, although not the scapy parser or the
  RIB file download procedures
- requires downloading RIB files, could be improved by talking
  directly with a BGP router daemon like Bird or FRR
- only a small set of tcpdump outputs have been tested
- the REPL shell does not have proper readline support (keyboard
  arrows, control characters like "control a" do not work)

Note that this documentation and test code uses sample AS numbers from
[RFC5398][], IPv4 addresses from [RFC5737](https://www.rfc-editor.org/rfc/rfc5737.html), and IPv6 addresses from
[RFC3849](https://www.rfc-editor.org/rfc/rfc3849.html). Some more well known entities (e.g. Amazon, Facebook)
have not been redacted from the output for clarity.

## Performance considerations

As mentioned above, this will unlikely tolerate multi-gigabit denial
of service attacks. The tcpdump parser, however, is pretty fast and
should be able to sustain a saturated gigabit link under normal
conditions. The scapy parser is slower.

Memory usage seems reasonable: on startup, it uses about 250MB of
memory, and a long-running process with about 40 000 blocks was using
about 400MB.

By extrapolation, it is expect that data on the full routing table
(currently 1.2 million entries) could be held within 12 GB of memory,
although that would be a rare condition, only occurring on a core
router with traffic from literally the entire internet.

## Security considerations

There's an unknown in the form of the C implementation of a Radix tree
in pyasn. asncounter itself should be fairly safe: it does not trust
its inputs, and the worse it can do is likely a resource exhaustion
attack on high traffic.

It can run completely unprivileged as long as it has access to the
input files, although in many scenarios people will not bother to drop
privileges before calling it and it will not, itself, attempt to do
so.

Privileges can be dropped with `systemd-run`, for example:

    systemd-run --pipe --property=DynamicUser=yes \
       --property=CacheDirectory=asncounter \
       --setenv=XDG_CACHE_HOME=/var/cache/asncounter \
       -- asncounter

This interacts poorly with `--repl` option, as it tries to reopen the
`tty` for stdin. You might have better luck with sharing a debug
socket with `--manhole`:

    systemd-run --pipe --property=DynamicUser=yes \
       --property=CacheDirectory=asncounter \
       --setenv=XDG_CACHE_HOME=/var/cache/asncounter \
       -- asncounter --manhole=/var/cache/asncounter/asncounter-manhole

Then you can open a Python debugging shell for further diagnostics
with:

    nc -U /var/cache/asncounter/asncounter-manhole

Note that the source distribution ships with a more advanced and
locked down `asncounter-exporter.service` which exposes a Prometheus
exporter but generally provides a good way to start `asncounter`
securely under systemd. If `asncounter` was installed through the
Debian package, the unit is not enabled by default. It can be enabled
and started with:

    systemctl enable --now asncounter-exporter.service

Metrics are exposed over the port 8999 and a "manhole" is opened in
`/var/cache/asncounter/asncounter-manhole` if available.

# AUTHOR

Antoine Beaupré <anarcat@debian.org>

# SEE ALSO

**tcpdump(8)**, **fail2ban(1)**, **netaddr(1)**, **grepcidr(1)**
