Edit on GitHub

BGP monitor

Definitions

Autonomous Systems

The Internet is a network of networks, and autonomous systems are the big networks that make up the Internet.
Every computer that has access to the Internet is linked to an Autonomous System (AS).

More specifically, an autonomous system (AS) is a large network or group of networks that are all managed, controlled and supervised by a single entity or organization. It can be your Internet Service Provider (ISP), a private society (eg. Google, Facebook, etc), a government etc.
Each AS are connected to their neighbors called peers, which receives and advertise different BGP messages.
Also, these connections form a network graph built using AS Paths.

Routing policy

Autonomous Systems are connected to their neighbors called peers, which receives and sends different BGP messages.
A routing policy is also defined for each AS:

  • has its own list of prefixes (IP ranges).
  • has a list of its peer AS

IP Address prefixes

While networking, each Autonomous System announce or withdraw ranges of IP addresses. Also, a range of IP Addresses is defined in CIDR format.

A prefix is a CIDR (Classless Inter-Domain Routing) format. It is composed as follow as "network/subnet" mask where Network is a normal ipv4 or ipv6 address and Subnet mask is a number between 0 and 32 which is the mask.

For example, 10.10.0.0/24 represents IP addresses 10.10.0.0 to 10.10.0.255

BGP (Border Gateway Protocol)

Border Gateway Protocol (BGP) is a standardized gateway protocol designed to exchange routing and reachability information among autonomous systems (AS) on the Internet.

BGP is classified as a path-vector routing protocol and it makes routing decisions based on paths, network policies, or rulesets configured by a network administrator.

BGP AS Path

In every prefix announcement, an AS-path is specified to know the route to the announced prefix.
This path is a list of AS numbers. Also, when a prefix is withdrawn, all AS-paths (or routes) to the prefix are removed

BGP Messages

There are 3 types of BGP messages:

  • Open:
    • Establish a TCP connection
    • Exchange information such as BGP Version number, AS Number, Hold Down Time value, BGP Identifier
  • Keepalive: If no update or keepalive message is sent for more than 30 second, then the connection is closed.
  • Update:
    • Announce a new prefix (message can contain an AS-path)
    • Withdraw a prefix
    • Peer State change By default, update messages contains peers' IP address, asn, etc
  • Notification:
    Used to send error codes
  • Route-Refresh

References and more sources

For more details, you can consult the following links:

Installation

From source

  1. First, you must install the following C libraries :
apt install cython
  1. Clone repo and Install required Python 3.9+ (venv recommended) packages:
git clone https://github.com/D4-project/bgp-monitor.git
pip3 install -r requirements.txt

You can export the path to the repo if you want to execute it from anywhere:

chmod +x /path/to/repo/bgp-monitor/bin/monitor.py
export PATH=$PATH:/path/to/repo/bgp-monitor/bin/
  1. Database

Therefore, you can install the desired database:

Don't forget to uncomment the corresponding lines in the config file

Docker install

From source

You can run bgp-monitor without database and run your own instance separately :

git clone https://github.com/D4-project/bgp-monitor.git
docker build -f docker/Dockerfile -t bgp-monitor . # Build bgp-monitor image

Database

You run an other docker image with pre-installed database (require bgp-monitor:latest / previous step) :

docker build -f docker/{dbname}/Dockerfile -t bgp-monitor:{dbname} . # Generate an other image from the previous
docker run -it bgp-monitor:{dbname}

Available dbname :

  • kvrocks
  • clickhouse
  • quest

From DockerHub

You can install generated images from Dockerhub:

docker run -it ustaenes/bgp-monitor:latest

docker run -it -P ustaenes/bgp-monitor:kvrocks

docker run -it -P ustaenes/bgp-monitor:quest

docker run -it -P ustaenes/bgp-monitor:clickhouse

Usage

Examples of command-line usage

Default stream testing (No filtering, massive print):

monitor.py --verbose

Filter ip addresses 84.205.67.0 through 84.205.67.255:

monitor.py -pf 84.205.67.0/24 --verbose

You can filter many AS numbers and/or prefixes in etc/filter_file.cfg.sample instead of using long command line:

monitor.py --filter_file etc/filter_file.cfg.sample --verbose

Retrieve past records instead of live stream

monitor.py --record --start "2022-01-01 10:00:00" --stop "2022-01-01 10:10:00" --verbose

Specify a project / list of collectors:

monitor.py -p routeviews --collectors route-views.bdix --start "2022-01-01 10:00:00" --stop "2022-01-01 10:10:00" --verbose

Retrieve records from single file as source instead of a broker:

monitor.py --input_data ../datasets/updates.20220425.1215 --verbose

Output

BGP records are reformatted and presented as follows:

  • type
    • R: RIB table entry
    • A: prefix announcement
    • W: prefix withdrawal
    • S: peer state change
  • time : Timestamp
  • peer : The IP address of the peer that this element was received from.
  • peer_asn: The ASN of the peer that this element was received from.
  • collector: The name of the collector that generated this element.
  • prefix : The prefix of the source ASN that this element was generated from. [A and W types]
  • country_code: The country of the source ASN that this element was generated from. [A and W types]
  • next-hop: The next-hop IP address [A type]
  • as-path: String list separated by spaces of AS numbers, ordered by the near peer ASN to the source ASN. Therefore, Source ASN is at the end of the string. [A type]

Testing

To test different filters, you can download some datasets here :

It will be easier to work with static data instead of ris-live stream:

./monitor.py --input_data ../datasets/updates.20220425.1215 --verbose

Note that you can use options like --json_out (to save the output) or --expected_result (check if json_out is equal to the specified file)

License

Contribution

This repository is open to any contribution.

Add your own Database

To process data in you own way, here are the steps:

  1. Copy SampleDatabase.py and implement it in your own way It must be in the Databases folder
  2. To load and use it, you must then add your class name to etc/config.cfg
  3. Be careful used technologies and implementation are important for performances

How it's made?

To update or upgrade a specific part of the code, check the following diagram

1"""
2.. include:: ../docs/home.md
3"""
4
5__docformat__ = "restructuredtext"
6__pdoc__ = {}
7__pdoc__[".__pycache__"] = False
classDiagram class BGPFilter{ -isRecord : Boolean -start_time : String -end_time : String -country_file : String ~get/set~ -f_country : MaxmindDB -countries_filter : List[String] ~get/set~ -asn_filter : String ~get/set~ -ipversion : String ~get/set~ -prefix_filter : String ~get~ -asn_list : String -prefix_match_type_filter : String ~get~ -project : String ~get/set~ -collectors : Array ~get/set~ -data_source : String #_stream : BGPStream +out : BGPOut +record_mode(isRecord, start, end) +data_source(record_type: Boolean, file_format: String, file_path: String) +prefix_filter(prefix_list: List[String], match_type: String) +start() +stop() } class BGPOut{ -json_out : File ~get/set~ +expected_result : File ~get/set~ +iteration(e: BGPElem) +closeFile(file: File) +start() +stop() } class Database{ String name$ start()* stop()* save(record: BGPElement)* get(as_numbers, prefixes, match_type, start_time, end_time, countries)* } class BGPDatabases{ -databases : ~List~Databases~ get/ Array set~ +start() +stop() +save(record: BGPElement) } class BGPGraph{ +graph +filter_edge() +get() +update(record: BGPElement) } class ClickHouseDB~Database~{ +BATCH_SIZE : Integer +started : Boolean +queue : Queue +client get_data() : Iterable[dict] insert_batches() } class QuestDB~Database~{ sock : socket connection conf : dict send_utf8(msg) } class KvrocksDB~Database~{ client : Redis } BGPFilter "1" --> "0" BGPOut : iterate BGPOut "1" *--> BGPDatabases : send data BGPOut *--> BGPGraph : send data BGPDatabases "0..*" *-- "" Database Database <|.. KvrocksDB Database <|.. QuestDB Database <|.. ClickHouseDB