Skip to content

Commit

Permalink
Add AnsiHtmlFormatter
Browse files Browse the repository at this point in the history
  • Loading branch information
@kdeldycke
kdeldycke committed May 18, 2024
1 parent 25d62c5 commit 82887bd
Show file tree
Hide file tree
Showing 5 changed files with 394 additions and 122 deletions.
307 changes: 197 additions & 110 deletions README.md
View file
Original file line number Diff line number Diff line change
@@ -1,73 +1,116 @@
pygments-ansi-color
-------------------
# pygments-ansi-color

[![build status](https://github.com/chriskuehl/pygments-ansi-color/actions/workflows/main.yml/badge.svg)](https://github.com/chriskuehl/pygments-ansi-color/actions/workflows/main.yml)
[![pre-commit.ci status](https://results.pre-commit.ci/badge/github/chriskuehl/pygments-ansi-color/main.svg)](https://results.pre-commit.ci/latest/github/chriskuehl/pygments-ansi-color/main)
[![PyPI version](https://badge.fury.io/py/pygments-ansi-color.svg)](https://pypi.python.org/pypi/pygments-ansi-color)

An ANSI color-code highlighting lexer for Pygments.
This project adds **parsing and coloring of ANSI sequences to Pygments**.

![](https://i.fluffy.cc/nHPkL3gfBtj5Kt4H3RR51T9TJLh6rtv2.png)


### Basic usage
## Installation

1. Install `pygments-ansi-color`:
The project is [available on PyPi under the `pygments-ansi-color` name
](https://pypi.org/project/pygments-ansi-color/):

```shell-session
$ pip install pygments-ansi-color
```
```shell-session
$ pip install pygments-ansi-color
```

2. `pygments-ansi-color` is not magic (yet?), so you need to [choose an exising
Pygments style](https://pygments.org/styles/), which will be used as a base
for your own style.

For example, let's choose `pygments.styles.xcode.XcodeStyle`, which looks
great to use. And then we will augment this reference style with
`pygments-ansi-color`'s color tokens thanks to the `color_tokens` function,
to make our final `MyStyle` custom style.
## Basic usage

Here is how the code looks like:
Once installed, you can highlight your content with Pygments' regular API. All
you need is to use the dedicated ANSI lexer and formatter provided by
`pygments-ansi-color`:

```python
from pygments_ansi_color import color_tokens
```python
from pygments import highlight
from pygments_ansi_color import AnsiHtmlFormatter, AnsiColorLexer

class MyStyle(pygments.styles.xcode.XcodeStyle):
styles = dict(pygments.styles.xcode.XcodeStyle.styles)
styles.update(color_tokens())
```
code = "\x1b[0m\x1b[34mA\x1b[0m\x1b[35mN\x1b[0m\x1b[36mS\x1b[0m\x1b[31mI\x1b[0m\x1b[32m"
print(highlight(code, AnsiColorLexer(), AnsiHtmlFormatter()))
```

This produce the following HTML code, with ANSI codes properly interpreted and
linked to their color:

```html
<div class="highlight">
<pre>
<span></span>
<span class=" -Color -Color-Blue -C-Blue">A</span>
<span class=" -Color -Color-Magenta -C-Magenta">N</span>
<span class=" -Color -Color-Cyan -C-Cyan">S</span>
<span class=" -Color -Color-Red -C-Red">I</span>
</pre>
</div>
```

That's all the custom code you need to integrate with `pygments-ansi-color`.
And here are is the corresponding CSS style:

3. Now you can highlight your content with the dedicated ANSI lexer and your
custom style, with the Pygments regular API:
```python
print(AnsiHtmlFormatter().get_style_defs('.highlight'))
```

```python
import pygments
import pygments.formatters
import pygments.lexers
```css
pre { line-height: 125%; }
(...)
.highlight .-Color-Blue { color: #3465a4 } /* Color.Blue */
.highlight .-Color-Cyan { color: #34e2e2 } /* Color.Cyan */
.highlight .-Color-Magenta { color: #c509c5 } /* Color.Magenta */
(...)
```

lexer = pygments.lexers.get_lexer_by_name('ansi-color')
formatter = pygments.formatters.HtmlFormatter(style=MyStyle)
print(pygments.highlight('your text', lexer, formatter))
```

### Design
## Design

We had to configure above a custom Pygments style with the appropriate color
tokens. That's because existing Pygments lexers are built around contextual
tokens (think `Comment` or `Punctuation`) rather than actual colors.
In the code above, we rely on a custom `AnsiColorLexer`. Its role is to produce
custom color tokens for the highlighter. That's because existing Pygments
lexers are built around contextual tokens (think `Comment` or `Punctuation`)
rather than actual colors.

In the case of ANSI escape sequences, colors have no context beyond the color
themselves; we'd always want a `red` rendered as `red`, regardless of your
particular theme.
particular theme. Hence these custom tokens.

Now for the rendering part, we again need a custom `AnsiHtmlFormatter`, so we
have a way to interpret these color tokens, and produce the corresponding
custom CSS classes with the right style.


## Pygments integration

### Custom theme
`pygments-ansi-color` is properly integrated to Pygments, so you do not need
custom code to integrate with it.

By default, `pygments-ansi-color` maps ANSI codes to its own set of colors.
They have been carefully crafted for readability, and are [loosely based on the
color scheme used by iTerm2
At intallation, `pygments-ansi-color` registers its custom lexers and
formatters. You can fetch them from their own IDs:

```pycon
>>> from pygments.lexers import get_lexer_by_name
>>> get_lexer_by_name('ansi-color')
<pygments.lexers.AnsiColorLexer>
```

```pycon
>>> from pygments.formatters import get_formatter_by_name
>>> get_formatter_by_name('ansi-html')
<pygments_ansi_color.AnsiHtmlFormatter object at 0x1044cbf10>
```


## Default ANSI colors

By default, `pygments-ansi-color` renders [the 8 basic ANSI colors and their
bright variants
](https://en.wikipedia.org/wiki/ANSI_escape_code#3-bit_and_4-bit), for both
foreground and background.

But the default colors are not set to the primary hues from the VGA-era.
Instead, the set has been carefully crafted for readability, and are [loosely
based on the color scheme used by iTerm2
](https://github.com/chriskuehl/pygments-ansi-color/pull/27#discussion_r1113790011).

Default colors are hard-coded by the `pygments_ansi_color.DEFAULT_STYLE`
Expand All @@ -89,86 +132,130 @@ constant as such:
- ![#5ffdff](https://placehold.co/15/5ffdff/5ffdff) `BrightCyan`: `#5ffdff`
- ![#feffff](https://placehold.co/15/feffff/feffff) `BrightWhite`: `#feffff`


## Custom ANSI theme

Still, you may want to use your own colors, to tweak the rendering to your
background color, or to match your own theme.
background, or to match your own theme.

For that you can override each color individually, by passing them as
arguments to the `color_tokens` function:
arguments to `AnsiHtmlFormatter`:

```python
from pygments_ansi_color import color_tokens

class MyStyle(pygments.styles.xcode.XcodeStyle):
styles = dict(pygments.styles.xcode.XcodeStyle.styles)
styles.update(color_tokens(
fg_colors={'Cyan': '#00ffff', 'BrightCyan': '#00ffff'},
bg_colors={'BrightWhite': '#000000'},
))
AnsiHtmlFormatter(
fg_colors={'Cyan': '#00ffff', 'BrightCyan': '#00ffff'},
bg_colors={'BrightWhite': '#000000'},
)
```


### Used by
## Pygments style

You can see an example [on fluffy][fluffy-example], the project that this lexer
was originally developed for.
You can [use any Pygments style you want](https://pygments.org/styles/) with
`AnsiHtmlFormatter`:

The colors are defined as part of your Pygments style and can be changed.


### Optional: Enable "256 color" support

This library supports rendering terminal output using [256 color
(8-bit)][256-color] ANSI color codes. However, because of limitations in
Pygments tokens, this is an opt-in feature which requires patching the
formatter you're using.

The reason this requires patching the Pygments formatter is that Pygments does
not support multiple tokens on a single piece of text, requiring us to
"compose" a single state (which is a tuple of `(bold enabled, fg color, bg
color)`) into a single token like `Color.Bold.FGColor.BGColor`. We then need to
output the styles for this token in the CSS.

In the default mode where we only support the standard 8 colors (plus 1 for no
color), we need 2 × 9 × 9 - 1 = 161 tokens, which is reasonable to contain in
one CSS file. With 256 colors (plus the standard 8, plus 1 for no color),
though, we'd need 2 × 265 × 265 - 1 = 140,449 tokens defined in CSS. This makes
the CSS too large to be practical.

To make 256-color support realistic, we patch Pygments' HTML formatter so that
it places a class for each part of the state tuple independently. This means
you need only 1 + 265 + 265 = 531 CSS classes to support all possibilities.

If you'd like to enable 256-color support, you'll need to do two things:

1. When calling `color_tokens`, pass `enable_256color=True`:

```python
styles.update(color_tokens(enable_256color=True))
```

This change is what causes your CSS to have the appropriate classes in it.

2. When constructing your formatter, use the `ExtendedColorHtmlFormatterMixin`
mixin, like this:

```python
from pygments.formatters import HtmlFormatter
from pygments_ansi_color import ExtendedColorHtmlFormatterMixin

...

class MyFormatter(ExtendedColorHtmlFormatterMixin, HtmlFormatter):
pass

...
```python
AnsiHtmlFormatter(style="monokai")
```

formatter = pygments.formatter.HtmlFormatter(style=MyStyle)
```
Behind the scene, `AnsiHtmlFormatter` will create a custom, local copy of the
style you choose, and augment it with our custom directives, so we can apply
styles to the custom color tokens produced by `AnsiColorLexer`.

> **Note**: Custom style
>
> For some reasons, you might want to replicate this behavior and manually
> manage your style. Here is how to do it.
>
> First, choose an exising Pygments style to be used as a base. Let's do that
> with `pygments.styles.xcode.XcodeStyle` as reference and make our final
> `MyStyle` custom style:
>
> ```python
> from pygments.styles.xcode import XcodeStyle
>
> from pygments_ansi_color import color_tokens
>
> class MyStyle(XcodeStyle):
> styles = dict(XcodeStyle.styles)
> styles.update(color_tokens())
> ```
>
> You can customize the styling further as `color_tokens` takes the same
> `fg_colors` and `bg_colors` parameters as `AnsiHtmlFormatter` (see above).
## `256-color` support
This library supports rendering terminal output using the [`256-color` (8-bit)
mode](https://en.wikipedia.org/wiki/ANSI_escape_code#8-bit) of ANSI codes.
This is **not active by default** and is an opt-in feature, because of its
underlaying complexity.
To enable `256-color` support, you only need to pass `enable_256color=True`,
and then you can use Pygments as normal:
This change is what causes the rendered HTML to have the right class names.
```python
from pygments import highlight
from pygments_ansi_color import AnsiHtmlFormatter, AnsiColorLexer
Once these two changes have been made, you can use pygments-ansi-color as normal.
formatter = AnsiHtmlFormatter(enable_256color=True)
code = "\x1b[0m\x1b[34mA\x1b[0m\x1b[35mN\x1b[0m\x1b[36mS\x1b[0m\x1b[31mI\x1b[0m\x1b[32m"
print(highlight(code, AnsiColorLexer(), formatter))
```
[fluffy-example]: https://i.fluffy.cc/3Gq7Fg86mv3dX30Qx9LHMWcKMqsQLCtd.html
[256-color]: https://en.wikipedia.org/wiki/ANSI_escape_code#8-bit
> **Note**: `256-color` design
>
> Pygments' tokens are not allowed to overlap. Which means we cannot
> accumulates the effects of multiple tokens over a segment of text. So we use
> a trick which consist in the composition of a single state into a single
> token. That way, the state tuple of `(bold_state, fgcolor, bgcolor)` is
> encoded as the `Color.Bold.FGColor.BGColor` token.
>
> In the default mode where we only support the standard 8 colors (plus 1 for
> no color), we need `2 × 9 × 9 - 1 = 161` tokens, which is reasonable to have
> them all contained within one CSS file. With 256 colors (plus the standard 8,
> plus 1 for no color) though, we'd need `2 × 265 × 265 - 1 = 140,449`
> individual tokens defined in CSS. This makes the CSS too large to be
> practical.
>
> To make the `256-color` mode support realistic, we patch Pygments' HTML
> formatter so that it places a class for each part of the state tuple
> independently. This means you only need `1 + 265 + 265 = 531` CSS classes to
> support all possibilities.
## Language lexers with ANSI output
With this project, you can provide strings of ANSI codes and have them rendered
in HTML by Pygments. Which means you can parse and render pure ANSI content
like ANSI art files. Or build your own lexer that produces ANSI output.
Now there are some [languages supported by Pygments
](https://pygments.org/languages/) that produce ANSI output. For example, the
[`console` lexer can be used to highlight shell sessions
](https://pygments.org/docs/terminal-sessions/). If the general structure of
the shell session will be highlighted, the ANSI codes in the output will not be
interpreted and will be rendered as-is.
To fix that, you need [ANSI-capable lexers from Click Extra
](https://kdeldycke.github.io/click-extra/pygments.html#lexers). These can
parse both the language syntax and the ANSI codes.
With Click Extra, you will be able to use the `ansi-console` lexer to highlight
shell sessions with ANSI support.
## Used by
- [fluffy](https://fluffy.cc) - A file-sharing web app that doesn't suck, and
the project that [this lexer was originally developed for
](https://i.fluffy.cc/3Gq7Fg86mv3dX30Qx9LHMWcKMqsQLCtd.html).
- [Click Extra](https://github.com/kdeldycke/click-extra) - A ready-to-use
wrapper for Click, with extra colorization and configuration loading.
- [PrairieLearn](https://github.com/PrairieLearn/PrairieLearn) - Online
problem-driving learning system.
- [pygments-pytest](https://github.com/asottile/pygments-pytest) - A pygments
lexer for pytest output.
Loading

0 comments on commit 82887bd

Please sign in to comment.