Skip to content

Commit

Permalink
Merge pull request #4 from entropic-security/v0.5.0
Browse files Browse the repository at this point in the history 
v0.5.0
  • Loading branch information
@tnballo
tnballo authored Feb 6, 2021
2 parents 0948f6a + 4143b32 commit 0cc9211
Show file tree
Hide file tree
Showing 38 changed files with 2,923 additions and 807 deletions.
27 changes: 16 additions & 11 deletions Cargo.toml
View file
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[package]
name = "xgadget"
version = "0.4.0"
version = "0.5.0"
authors = ["Tiemoko Ballo"]
edition = "2018"
license = "MIT"
Expand All @@ -20,51 +20,56 @@ include = [
]

[dependencies]
iced-x86 = "1.9.1"
iced-x86 = "1"
goblin = "0.2"
rayon = "1"
bitflags = "1"
colored = "2"
rustc-hash = "1"
lazy_static = { version = "1", optional = true }
structopt = { version = "0.3", default-features = false, optional = true }
num_cpus = { version = "1", optional = true }
regex = { version = "1", optional = true }
strip-ansi-escapes = { version = "0.1", optional = true }
lazy_static = { version = "1.4", optional = true }
term_size = { version = "0.3.2", optional = true }
checksec = { version = "0.0.7", features = ["elf", "pe", "color"], optional = true }
memmap = { version = "0.7.0", optional = true }
term_size = { version = "0.3", optional = true }
checksec = { version = "0.0.8", features = ["elf", "pe", "color"], optional = true }
memmap = { version = "0.7", optional = true }

[dev-dependencies]
pprof = { version = "0.4", features = ["flamegraph"] }
criterion = "0.3"
rand = "0.7"
dirs = "3"
predicates = "1"
assert_cmd = "1"
tempfile = "3"
regex = "1"

[features]
cli-bin = ["structopt", "num_cpus", "regex", "strip-ansi-escapes", "lazy_static", "term_size", "checksec", "memmap"]
cli-bin = ["lazy_static", "structopt", "num_cpus", "regex", "term_size", "checksec", "memmap"]

[lib]
name = "xgadget"
path = "src/lib.rs"

[[bin]]
name = "xgadget"
path = "src/cli/cli.rs"
path = "src/cli/main.rs"
required-features = ["cli-bin"]

[[bench]]
name = "bench_1_misc"
harness = false

[[bench]]
name = "bench_2_elf_userspace"
name = "bench_2_fmt"
harness = false

[[bench]]
name = "bench_3_elf_kernels"
name = "bench_3_elf_userspace"
harness = false

[[bench]]
name = "bench_4_elf_kernels"
harness = false

[profile.release]
Expand Down
140 changes: 89 additions & 51 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -13,22 +13,23 @@ Uses the [iced-x86 disassembler library](https://github.com/0xd4d/iced).
To the best of my knowledge, `xgadget` is the first gadget search tool to have these features:

* JOP search uses instruction semantics - not hardcoded regex for individual encodings
* Optionally filter to JOP "dispatcher" gadgets with flag `--dispatcher`
* Optionally filter to JOP "dispatcher" gadgets with flag `--dispatcher`
* Finds gadgets that work across multiple variants of a binary (e.g. different program or compiler versions)
* **Full-match** - Same instruction sequence, same program counter: gadget fully re-usable.
* E.g. `pop rsp; add [rax-0x77], cl; ret ------------------------------------- [ 0xc748d ]`
* **Partial-match** - Same instruction sequence, different program counter: gadget logic portable.
* E.g. `pop rsp; add [rax-0x77], cl; ret; --- [ 'bin_v1.1': 0xc748d, 'bin_v1.2': 0xc9106 ]`
* This is entirely optional, you're free to run this tool on a single binary.
* **Full-match** - Same instruction sequence, same program counter: gadget fully re-usable.
* E.g. `pop rsp; add [rax-0x77], cl; ret ------------------------------------- [ 0xc748d ]`
* **Partial-match** - Same instruction sequence, different program counter: gadget logic portable.
* E.g. `pop rsp; add [rax-0x77], cl; ret; --- [ 'bin_v1.1': 0xc748d, 'bin_v1.2': 0xc9106 ]`
* This is entirely optional, you're free to run this tool on a single binary.
* The stack pointer is explicitly colored in terminal output, for workflow convenience.

Other features include:

* Both library API and CLI tool
* Supports ELF32, ELF64, PE32, PE32+ [1], and raw files
* Parallel across available cores [2], whether searching a single binary or multiple variants
* CI/CD for automated integration test and binary releases (Linux, 64-bit) [3]
* Statistical benchmark harness for performance tuning [4]
* 8086/x86/x64 only, uses a speed-optimized disassembly backend [5]
* Supports ELF32, ELF64, PE32, PE32+ \[1\], and raw files
* Parallel across available cores \[2\], whether searching a single binary or multiple variants
* CI/CD for automated integration test and binary releases (Linux, 64-bit) \[3\]
* Statistical benchmark harness for performance tuning \[4\]
* 8086/x86/x64 only, uses a speed-optimized disassembly backend \[5\]

### API Usage

Expand All @@ -38,60 +39,91 @@ Find gadgets:
use xgadget;

let max_gadget_len = 5;
let search_config = xgadget::SearchConfig::DEFAULT;

// Search single binary
let search_config = xgadget::SearchConfig::DEFAULT;
let bin_1 = xgadget::Binary::from_path_str("/path/to/bin_v1").unwrap();
let bins = vec![bin_1];
let gadgets = xgadget::find_gadgets(&bins, max_gadget_len, search_config).unwrap();
let stack_pivot_gadgets = xgadget::filter_stack_pivot(&gadgets);

// Search for cross-variant gadgets
// Search for cross-variant gadgets, including partial matches
let search_config = xgadget::SearchConfig::DEFAULT | xgadget::SearchConfig::PART;
let bin_1 = xgadget::Binary::from_path_str("/path/to/bin_v1").unwrap();
let bin_2 = xgadget::Binary::from_path_str("/path/to/bin_v2").unwrap();
let bins = vec![bin_1, bin_2];
let cross_gadgets = xgadget::find_gadgets(&bins, max_gadget_len, search_config).unwrap();
let cross_reg_write_gadgets = xgadget::filter_stack_set_regs(&cross_gadgets);
let cross_reg_pop_gadgets = xgadget::filter_reg_pop_only(&cross_gadgets);
```

Custom filters can be created using the [`GadgetAnalysis`](crate::gadget::GadgetAnalysis) object and/or functions from the [`semantics`](crate::semantics) module.
How the above [`filter_stack_pivot`](crate::filters::filter_stack_pivot) function is implemented:

```rust
use rayon::prelude::*;
use iced_x86;
use xgadget::{Gadget, GadgetAnalysis};

/// Parallel filter to gadgets that write the stack pointer
pub fn filter_stack_pivot<'a>(gadgets: &[Gadget<'a>]) -> Vec<Gadget<'a>> {
gadgets
.par_iter()
.filter(|g| {
let regs_overwritten = GadgetAnalysis::new(&g).regs_overwritten();
if regs_overwritten.contains(&iced_x86::Register::RSP)
|| regs_overwritten.contains(&iced_x86::Register::ESP)
|| regs_overwritten.contains(&iced_x86::Register::SP)
{
return true;
}
false
})
.cloned()
.collect()
}
```

### CLI Usage

Run `xgadget --help`:

```
xgadget v0.4.0
xgadget v0.5.0
About: Fast, parallel, cross-variant ROP/JOP gadget search for x86/x64 binaries.
Cores: 8 logical, 8 physical
About: Fast, parallel, cross-variant ROP/JOP gadget search for x86/x64 binaries.
Cores: 8 logical, 8 physical
USAGE:
xgadget [FLAGS] [OPTIONS] <FILE(S)>...
xgadget [FLAGS] [OPTIONS] <FILE(S)>...
FLAGS:
-t, --att Display gadgets using AT&T syntax [default: Intel syntax]
-c, --check-sec Run checksec on the 1+ binaries instead of gadget search
-d, --dispatcher Filter to potential JOP 'dispatcher' gadgets [default: all gadgets]
-e, --extended-fmt Print in terminal-wide format [default: only used for partial match search]
-h, --help Prints help information
--inc-call Include gadgets containing a call [default: don't include]
--inc-imm16 Include '{ret, ret far} imm16' (e.g. add to stack ptr) [default: don't include]
-j, --jop Search for JOP gadgets only [default: ROP, JOP, and SYSCALL]
-n, --no-color Don't color output, useful for UNIX piping [default: color output]
-m, --partial-match Include cross-variant partial matches [default: full matches only]
-w, --reg-write Filter to 'pop {reg} * 1+, {ret or ctrl-ed jmp/call}' gadgets [default: all gadgets]
-r, --rop Search for ROP gadgets only [default: ROP, JOP, and SYSCALL]
-p, --stack-pivot Filter to gadgets that write the stack ptr [default: all gadgets]
-s, --sys Search for SYSCALL gadgets only [default: ROP, JOP, and SYSCALL]
-V, --version Prints version information
-t, --att Display gadgets using AT&T syntax [default: Intel syntax]
-c, --check-sec Run checksec on the 1+ binaries instead of gadget search
-d, --dispatcher Filter to potential JOP 'dispatcher' gadgets [default: all]
-e, --extended-fmt Print in terminal-wide format [default: only used for partial match search]
-h, --help Prints help information
--inc-call Include gadgets containing a call [default: don't include]
--inc-imm16 Include '{ret, ret far} imm16' (e.g. add to stack ptr) [default: don't include]
-j, --jop Search for JOP gadgets only [default: ROP, JOP, and SYSCALL]
-n, --no-color Don't color output [default: color output]
--param-ctrl Filter to gadgets that control function parameters [default: all]
-m, --partial-match Include cross-variant partial matches [default: full matches only]
--reg-pop Filter to 'pop {reg} * 1+, {ret or ctrl-ed jmp/call}' gadgets [default: all]
-r, --rop Search for ROP gadgets only [default: ROP, JOP, and SYSCALL]
-p, --stack-pivot Filter to gadgets that write the stack ptr [default: all]
-s, --sys Search for SYSCALL gadgets only [default: ROP, JOP, and SYSCALL]
-V, --version Prints version information
OPTIONS:
-a, --arch <ARCH> For raw (no header) files: specify arch ('x8086', 'x86', or 'x64') [default: x64]
-b, --bad-bytes <BYTE(S)>... Filter to gadgets whose addrs don't contain given bytes [default: all gadgets]
-l, --max-len <LEN> Gadgets up to LEN instrs long. If 0: all gadgets, any length [default: 5]
-f, --regex-filter <EXPR> Filter to gadgets matching a regular expression
-a, --arch <ARCH> For raw (no header) files: specify arch ('x8086', 'x86', or 'x64') [default: x64]
-b, --bad-bytes <BYTE(S)>... Filter to gadgets whose addrs don't contain given bytes [default: all]
-l, --max-len <LEN> Gadgets up to LEN instrs long. If 0: all gadgets, any length [default: 5]
--no-deref <OPT_REG> Filter to gadgets that don't deref any regs or a specific reg [default: all]
--reg-ctrl <OPT_REG> Filter to gadgets that control any reg or a specific reg [default: all]
-f, --regex-filter <EXPR> Filter to gadgets matching a regular expression
ARGS:
<FILE(S)>... 1+ binaries to gadget search. If > 1: gadgets common to all
<FILE(S)>... 1+ binaries to gadget search. If > 1: gadgets common to all
```

### CLI Build and Install (Recommended)
Expand All @@ -105,33 +137,39 @@ cargo install xgadget --features cli-bin # Build on host (pre-req: https://ww
### CLI Binary Releases for Linux

Commits to this repo's `master` branch automatically run integration tests and build a statically-linked binary for 64-bit Linux.
You can [download it here](https://github.com/entropic-security/xgadget/releases) and use the CLI immediately, instead of building from source.
You can [download it here](https://github.com/entropic-security/xgadget/releases) to try out the CLI immediately, instead of building from source.
Static binaries for Windows may also be supported in the future.

The statically-linked binary is about 8x slower, presumably due to the built-in memory allocator for target `x86_64-unknown-linux-musl`.
Building a dynamically-linked binary from source with the above `cargo install` command is *highly* recommended.
Unfortunately the statically-linked binary is several times slower on an i7-9700K, likely due to the built-in memory allocator for target `x86_64-unknown-linux-musl`.
So building a dynamically-linked binary from source with the above `cargo install` command is *highly* recommended for performance (links against your system's allocator).

### Why No Chain Generation?

Tools that attempt to automate ROP chain generation require heavyweight analysis - typically symbolic execution of an intermediate representation.
While this works well for small binaries and CTF problems, it tends to be slow and difficult to scale for large, real-world programs.
At present, `xgadget` has a different goal: enable an expert user to manually craft stable exploits by providing fast, accurate gadget discovery.

### ~~Yeah, but can it do 10 OS kernels under 10 seconds?!~~ Repeatable Benchmark Harness

```bash
bash ./benches/bench_setup_ubuntu.sh # Ubuntu-specific, download/build 10 kernel versions
cargo bench # Grab a coffee, this'll take a while...
bash ./benches/bench_setup_ubuntu.sh # Ubuntu-specific, download/build 10 kernel versions
cargo bench # Grab a coffee, this'll take a while...
```

* `bench_setup_ubuntu.sh` downloads and builds 10 consecutive Linux kernels (versions `5.0.1` to `5.0.10` - with `x86_64_defconfig`).
* `cargo bench`, among other benchmarks, searches all 10 kernels for common gadgets.

On an i7-9700K (8C/8T, 3.6GHz base, 4.9 GHz max) machine with `gcc` version 8.4.0: the average runtime, to process *all ten 54MB kernels simultaneously* with a max gadget length of 5 instructions and full-match search for all gadget types (ROP, JOP, and syscall gadgets), is *only 5.8 seconds*! Including partial matches as well takes *just 7.2 seconds*.
On an i7-9700K (8C/8T, 3.6GHz base, 4.9 GHz max) machine with `gcc` version 8.4.0: the average runtime, to process *all ten 54MB kernels simultaneously* with a max gadget length of 5 instructions and full-match search for all gadget types (ROP, JOP, and syscall gadgets), is *only 6.3 seconds*! Including partial matches as well takes *just 7.9 seconds*.

### Acknowledgements

This project started as an optimized solution to Chapter 8, exercise 3 of "Practical Binary Analysis" by Dennis Andreisse [6], and builds on the design outlined therein.
This project started as an optimized solution to Chapter 8, exercise 3 of "Practical Binary Analysis" by Dennis Andreisse \[6\], and builds on the design outlined therein.

### References

* [1] [`goblin` crate by Lzu Tao, m4b, Philip Craig, seu, Will Glynn](https://crates.io/crates/goblin)
* [2] [`rayon` crate by Josh Stone, Niko Matsakis](https://crates.io/crates/rayon)
* [3] [`xgadget/.github/workflows`](https://github.com/entropic-security/xgadget/tree/master/.github/workflows)
* [4] [`criterion` crate by Brook Heisler, Jorge Aparicio](https://crates.io/crates/criterion)
* [5] [`iced-x86` crate by 0xd4d](https://crates.io/crates/iced-x86)
* [6] ["Practical Binary Analysis" by Dennis Andreisse](https://practicalbinaryanalysis.com/)
* \[1\] [`goblin` crate by Lzu Tao, m4b, Philip Craig, seu, Will Glynn](https://crates.io/crates/goblin)
* \[2\] [`rayon` crate by Josh Stone, Niko Matsakis](https://crates.io/crates/rayon)
* \[3\] [`xgadget/.github/workflows`](https://github.com/entropic-security/xgadget/tree/master/.github/workflows)
* \[4\] [`criterion` crate by Brook Heisler, Jorge Aparicio](https://crates.io/crates/criterion)
* \[5\] [`iced-x86` crate by 0xd4d](https://crates.io/crates/iced-x86)
* \[6\] ["Practical Binary Analysis" by Dennis Andreisse](https://practicalbinaryanalysis.com/)
6 changes: 6 additions & 0 deletions README.tpl
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
# {{crate}}

![crates.io](https://img.shields.io/crates/v/xgadget.svg)
![GitHub Actions](https://github.com/entropic-security/xgadget/workflows/test/badge.svg)

{{readme}}
20 changes: 20 additions & 0 deletions benches/README.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
### Run all benchmarks

```
cargo bench
```

### Review flamegraph for a specific benchmark set

```
cargo bench --bench <name_of_file> -- --profile-time=20
find . -iname "*flame*.svg"
```

E.g. `<name_of_file>` == `bench_2_fmt`

### Troubleshooting if flamegraphs don't work

```
echo -1 | sudo tee /proc/sys/kernel/perf_event_paranoid
```
Loading

0 comments on commit 0cc9211

Please sign in to comment.