v2/README: make a definitive decision on v2 import path

This change makes it clear that the v2 import path
is github.com/russross/blackfriday/v2, and updates
various links accordingly.

See https://github.com/russross/blackfriday/issues/587#issuecomment-703393820 for details.

This change also converges the README for v1 and v2
to be consistent, as they've started to drift apart.

For #587.

GitHub-Pull-Request: #675

.travis.yml

@@ -1,18 +1,17 @@
-# Travis CI (http://travis-ci.org/) is a continuous integration service for
-# open source projects. This file configures it to run unit tests for
-# blackfriday.
-
+sudo: false
 language: go
-
 go:
-    - 1.2
-    - 1.3
-    - 1.4
-    - 1.5
-
+  - "1.10.x"
+  - "1.11.x"
+  - tip
+matrix:
+  fast_finish: true
+  allow_failures:
+    - go: tip
 install:
-    - go get -d -t -v ./...
-    - go build -v ./...
-
+  - # Do nothing. This is needed to prevent default install action "go get -t -v ./..." from happening here (we want it to happen inside script step).
 script:
-    - go test -v ./...
+  - go get -t -v ./...
+  - diff -u <(echo -n) <(gofmt -d -s .)
+  - go tool vet .
+  - go test -v ./...

README.md

@@ -1,4 +1,6 @@
-Blackfriday [![Build Status](https://travis-ci.org/russross/blackfriday.svg?branch=master)](https://travis-ci.org/russross/blackfriday)
+Blackfriday
+[![Build Status][BuildV2SVG]][BuildV2URL]
+[![PkgGoDev][PkgGoDevV2SVG]][PkgGoDevV2URL]
 ===========
 
 Blackfriday is a [Markdown][1] processor implemented in [Go][2]. It
@@ -8,7 +10,7 @@ punctuation substitutions, etc.), and it is safe for all utf-8
 (unicode) input.
 
 HTML output is currently supported, along with Smartypants
-extensions. An experimental LaTeX output engine is also included.
+extensions.
 
 It started as a translation from C of [Sundown][3].
 
@@ -16,63 +18,98 @@ It started as a translation from C of [Sundown][3].
 Installation
 ------------
 
-Blackfriday is compatible with Go 1. If you are using an older
-release of Go, consider using v1.1 of blackfriday, which was based
-on the last stable release of Go prior to Go 1. You can find it as a
-tagged commit on github.
+Blackfriday is compatible with modern Go releases in module mode.
+With Go installed:
 
-With Go 1 and git installed:
+    go get github.com/russross/blackfriday/v2
 
-    go get github.com/russross/blackfriday
+will resolve and add the package to the current development module,
+then build and install it. Alternatively, you can achieve the same
+if you import it in a package:
 
-will download, compile, and install the package into your `$GOPATH`
-directory hierarchy. Alternatively, you can achieve the same if you
-import it into a project:
-
-    import "github.com/russross/blackfriday"
+    import "github.com/russross/blackfriday/v2"
 
 and `go get` without parameters.
 
+Legacy GOPATH mode is unsupported.
+
+
+Versions
+--------
+
+Currently maintained and recommended version of Blackfriday is `v2`. It's being
+developed on its own branch: https://github.com/russross/blackfriday/tree/v2 and the
+documentation is available at
+https://pkg.go.dev/github.com/russross/blackfriday/v2.
+
+It is `go get`-able in module mode at `github.com/russross/blackfriday/v2`.
+
+Version 2 offers a number of improvements over v1:
+
+* Cleaned up API
+* A separate call to [`Parse`][4], which produces an abstract syntax tree for
+  the document
+* Latest bug fixes
+* Flexibility to easily add your own rendering extensions
+
+Potential drawbacks:
+
+* Our benchmarks show v2 to be slightly slower than v1. Currently in the
+  ballpark of around 15%.
+* API breakage. If you can't afford modifying your code to adhere to the new API
+  and don't care too much about the new features, v2 is probably not for you.
+* Several bug fixes are trailing behind and still need to be forward-ported to
+  v2. See issue [#348](https://github.com/russross/blackfriday/issues/348) for
+  tracking.
+
+If you are still interested in the legacy `v1`, you can import it from
+`github.com/russross/blackfriday`. Documentation for the legacy v1 can be found
+here: https://pkg.go.dev/github.com/russross/blackfriday.
+
+
 Usage
 -----
 
-For basic usage, it is as simple as getting your input into a byte
-slice and calling:
+For the most sensible markdown processing, it is as simple as getting your input
+into a byte slice and calling:
 
-    output := blackfriday.MarkdownBasic(input)
+```go
+output := blackfriday.Run(input)
+```
 
-This renders it with no extensions enabled. To get a more useful
-feature set, use this instead:
+Your input will be parsed and the output rendered with a set of most popular
+extensions enabled. If you want the most basic feature set, corresponding with
+the bare Markdown specification, use:
 
-    output := blackfriday.MarkdownCommon(input)
+```go
+output := blackfriday.Run(input, blackfriday.WithNoExtensions())
+```
 
 ### Sanitize untrusted content
 
 Blackfriday itself does nothing to protect against malicious content. If you are
-dealing with user-supplied markdown, we recommend running blackfriday's output
-through HTML sanitizer such as
-[Bluemonday](https://github.com/microcosm-cc/bluemonday).
+dealing with user-supplied markdown, we recommend running Blackfriday's output
+through HTML sanitizer such as [Bluemonday][5].
 
-Here's an example of simple usage of blackfriday together with bluemonday:
+Here's an example of simple usage of Blackfriday together with Bluemonday:
 
-``` go
+```go
 import (
     "github.com/microcosm-cc/bluemonday"
-    "github.com/russross/blackfriday"
+    "github.com/russross/blackfriday/v2"
 )
 
 // ...
-unsafe := blackfriday.MarkdownCommon(input)
+unsafe := blackfriday.Run(input)
 html := bluemonday.UGCPolicy().SanitizeBytes(unsafe)
 ```
 
 ### Custom options
 
-If you want to customize the set of options, first get a renderer
-(currently either the HTML or LaTeX output engines), then use it to
-call the more general `Markdown` function. For examples, see the
-implementations of `MarkdownBasic` and `MarkdownCommon` in
-`markdown.go`.
+If you want to customize the set of options, use `blackfriday.WithExtensions`,
+`blackfriday.WithRenderer` and `blackfriday.WithRefOverride`.
+
+### `blackfriday-tool`
 
 You can also check out `blackfriday-tool` for a more complete example
 of how to use it. Download and install it using:
@@ -84,7 +121,7 @@ markdown file using a standalone program.  You can also browse the
 source directly on github if you are just looking for some example
 code:
 
-* <http://github.com/russross/blackfriday-tool>
+* <https://github.com/russross/blackfriday-tool>
 
 Note that if you have not already done so, installing
 `blackfriday-tool` will be sufficient to download and install
@@ -93,6 +130,22 @@ installed in `$GOPATH/bin`.  This is a statically-linked binary that
 can be copied to wherever you need it without worrying about
 dependencies and library versions.
 
+### Sanitized anchor names
+
+Blackfriday includes an algorithm for creating sanitized anchor names
+corresponding to a given input text. This algorithm is used to create
+anchors for headings when `AutoHeadingIDs` extension is enabled. The
+algorithm has a specification, so that other packages can create
+compatible anchor names and links to those anchors.
+
+The specification is located at https://pkg.go.dev/github.com/russross/blackfriday/v2#hdr-Sanitized_Anchor_Names.
+
+[`SanitizedAnchorName`](https://pkg.go.dev/github.com/russross/blackfriday/v2#SanitizedAnchorName) exposes this functionality, and can be used to
+create compatible links to the anchor names generated by blackfriday.
+This algorithm is also implemented in a small standalone package at
+[`github.com/shurcooL/sanitized_anchor_name`](https://pkg.go.dev/github.com/shurcooL/sanitized_anchor_name). It can be useful for clients
+that want a small package and don't need full functionality of blackfriday.
+
 
 Features
 --------
@@ -114,7 +167,7 @@ All features of Sundown are supported, including:
     know and send me the input that does it.
 
     NOTE: "safety" in this context means *runtime safety only*. In order to
-    protect yourself agains JavaScript injection in untrusted content, see
+    protect yourself against JavaScript injection in untrusted content, see
     [this example](https://github.com/russross/blackfriday#sanitize-untrusted-content).
 
 *   **Fast processing**. It is fast enough to render on-demand in
@@ -160,7 +213,7 @@ implements the following extensions:
     and supply a language (to make syntax highlighting simple). Just
     mark it like this:
 
-        ``` go
+        ```go
         func getTrue() bool {
             return true
         }
@@ -169,12 +222,21 @@ implements the following extensions:
     You can use 3 or more backticks to mark the beginning of the
     block, and the same number to mark the end of the block.
 
+    To preserve classes of fenced code blocks while using the bluemonday
+    HTML sanitizer, use the following policy:
+
+    ```go
+    p := bluemonday.UGCPolicy()
+    p.AllowAttrs("class").Matching(regexp.MustCompile("^language-[a-zA-Z0-9]+$")).OnElements("code")
+    html := p.SanitizeBytes(unsafe)
+    ```
+
 *   **Definition lists**. A simple definition list is made of a single-line
     term followed by a colon and the definition for that term.
 
         Cat
         : Fluffy animal everyone likes
-        
+
         Internet
         : Vector of transmission for pictures of cats
 
@@ -185,7 +247,7 @@ implements the following extensions:
     end of the document. A footnote looks like this:
 
         This is a footnote.[^1]
-        
+
         [^1]: the footnote text.
 
 *   **Autolinking**. Blackfriday can find URLs that have not been
@@ -194,10 +256,8 @@ implements the following extensions:
 *   **Strikethrough**. Use two tildes (`~~`) to mark text that
     should be crossed out.
 
-*   **Hard line breaks**. With this extension enabled (it is off by
-    default in the `MarkdownBasic` and `MarkdownCommon` convenience
-    functions), newlines in the input translate into line breaks in
-    the output.
+*   **Hard line breaks**. With this extension enabled newlines in the input
+    translate into line breaks in the output. This extension is off by default.
 
 *   **Smart quotes**. Smartypants-style punctuation substitution is
     supported, turning normal double- and single-quote marks into
@@ -222,9 +282,9 @@ Other renderers
 Blackfriday is structured to allow alternative rendering engines. Here
 are a few of note:
 
-*   [github_flavored_markdown](https://godoc.org/github.com/shurcooL/github_flavored_markdown):
+*   [github_flavored_markdown](https://pkg.go.dev/github.com/shurcooL/github_flavored_markdown):
     provides a GitHub Flavored Markdown renderer with fenced code block
-    highlighting, clickable header anchor links.
+    highlighting, clickable heading anchor links.
 
     It's not customizable, and its goal is to produce HTML output
     equivalent to the [GitHub Markdown API endpoint](https://developer.github.com/v3/markdown/#render-a-markdown-document-in-raw-mode),
@@ -233,25 +293,28 @@ are a few of note:
 *   [markdownfmt](https://github.com/shurcooL/markdownfmt): like gofmt,
     but for markdown.
 
-*   LaTeX output: renders output as LaTeX. This is currently part of the
-    main Blackfriday repository, but may be split into its own project
-    in the future. If you are interested in owning and maintaining the
-    LaTeX output component, please be in touch.
+*   [LaTeX output](https://gitlab.com/ambrevar/blackfriday-latex):
+    renders output as LaTeX.
 
-    It renders some basic documents, but is only experimental at this
-    point. In particular, it does not do any inline escaping, so input
-    that happens to look like LaTeX code will be passed through without
-    modification.
+*   [bfchroma](https://github.com/Depado/bfchroma/): provides convenience
+    integration with the [Chroma](https://github.com/alecthomas/chroma) code
+    highlighting library. bfchroma is only compatible with v2 of Blackfriday and
+    provides a drop-in renderer ready to use with Blackfriday, as well as
+    options and means for further customization.
+
+*   [Blackfriday-Confluence](https://github.com/kentaro-m/blackfriday-confluence): provides a [Confluence Wiki Markup](https://confluence.atlassian.com/doc/confluence-wiki-markup-251003035.html) renderer.
+
+*   [Blackfriday-Slack](https://github.com/karriereat/blackfriday-slack): converts markdown to slack message style
 
 
-Todo
+TODO
 ----
 
 *   More unit testing
-*   Improve unicode support. It does not understand all unicode
+*   Improve Unicode support. It does not understand all Unicode
     rules (about what constitutes a letter, a punctuation symbol,
     etc.), so it may fail to detect word boundaries correctly in
-    some instances. It is safe on all utf-8 input.
+    some instances. It is safe on all UTF-8 input.
 
 
 License
@@ -260,6 +323,13 @@ License
 [Blackfriday is distributed under the Simplified BSD License](LICENSE.txt)
 
 
-   [1]: http://daringfireball.net/projects/markdown/ "Markdown"
-   [2]: http://golang.org/ "Go Language"
+   [1]: https://daringfireball.net/projects/markdown/ "Markdown"
+   [2]: https://golang.org/ "Go Language"
    [3]: https://github.com/vmg/sundown "Sundown"
+   [4]: https://pkg.go.dev/github.com/russross/blackfriday/v2#Parse "Parse func"
+   [5]: https://github.com/microcosm-cc/bluemonday "Bluemonday"
+
+   [BuildV2SVG]: https://travis-ci.org/russross/blackfriday.svg?branch=v2
+   [BuildV2URL]: https://travis-ci.org/russross/blackfriday
+   [PkgGoDevV2SVG]: https://pkg.go.dev/badge/github.com/russross/blackfriday/v2
+   [PkgGoDevV2URL]: https://pkg.go.dev/github.com/russross/blackfriday/v2

block_test.go

@@ -18,66 +18,8 @@ import (
 	"testing"
 )
 
-func runMarkdownBlockWithRenderer(input string, extensions int, renderer Renderer) string {
-	return string(Markdown([]byte(input), renderer, extensions))
-}
-
-func runMarkdownBlock(input string, extensions int) string {
-	htmlFlags := 0
-	htmlFlags |= HTML_USE_XHTML
-
-	renderer := HtmlRenderer(htmlFlags, "", "")
-
-	return runMarkdownBlockWithRenderer(input, extensions, renderer)
-}
-
-func runnerWithRendererParameters(parameters HtmlRendererParameters) func(string, int) string {
-	return func(input string, extensions int) string {
-		htmlFlags := 0
-		htmlFlags |= HTML_USE_XHTML
-
-		renderer := HtmlRendererWithParameters(htmlFlags, "", "", parameters)
-
-		return runMarkdownBlockWithRenderer(input, extensions, renderer)
-	}
-}
-
-func doTestsBlock(t *testing.T, tests []string, extensions int) {
-	doTestsBlockWithRunner(t, tests, extensions, runMarkdownBlock)
-}
-
-func doTestsBlockWithRunner(t *testing.T, tests []string, extensions int, runner func(string, int) string) {
-	// catch and report panics
-	var candidate string
-	defer func() {
-		if err := recover(); err != nil {
-			t.Errorf("\npanic while processing [%#v]: %s\n", candidate, err)
-		}
-	}()
-
-	for i := 0; i+1 < len(tests); i += 2 {
-		input := tests[i]
-		candidate = input
-		expected := tests[i+1]
-		actual := runner(candidate, extensions)
-		if actual != expected {
-			t.Errorf("\nInput   [%#v]\nExpected[%#v]\nActual  [%#v]",
-				candidate, expected, actual)
-		}
-
-		// now test every substring to stress test bounds checking
-		if !testing.Short() {
-			for start := 0; start < len(input); start++ {
-				for end := start + 1; end <= len(input); end++ {
-					candidate = input[start:end]
-					_ = runMarkdownBlock(candidate, extensions)
-				}
-			}
-		}
-	}
-}
-
 func TestPrefixHeaderNoExtensions(t *testing.T) {
+	t.Parallel()
 	var tests = []string{
 		"# Header 1\n",
 		"<h1>Header 1</h1>\n",
@@ -147,6 +89,7 @@ func TestPrefixHeaderNoExtensions(t *testing.T) {
 }
 
 func TestPrefixHeaderSpaceExtension(t *testing.T) {
+	t.Parallel()
 	var tests = []string{
 		"# Header 1\n",
 		"<h1>Header 1</h1>\n",
@@ -203,10 +146,11 @@ func TestPrefixHeaderSpaceExtension(t *testing.T) {
 		"<ul>\n<li><p>List</p>\n\n<ul>\n<li><p>Nested list</p>\n\n" +
 			"<h1>Nested header</h1></li>\n</ul></li>\n</ul>\n",
 	}
-	doTestsBlock(t, tests, EXTENSION_SPACE_HEADERS)
+	doTestsBlock(t, tests, SpaceHeadings)
 }
 
 func TestPrefixHeaderIdExtension(t *testing.T) {
+	t.Parallel()
 	var tests = []string{
 		"# Header 1 {#someid}\n",
 		"<h1 id=\"someid\">Header 1</h1>\n",
@@ -263,10 +207,11 @@ func TestPrefixHeaderIdExtension(t *testing.T) {
 		"<ul>\n<li><p>List</p>\n\n<ul>\n<li><p>Nested list</p>\n\n" +
 			"<h1 id=\"someid\">Nested header</h1></li>\n</ul></li>\n</ul>\n",
 	}
-	doTestsBlock(t, tests, EXTENSION_HEADER_IDS)
+	doTestsBlock(t, tests, HeadingIDs)
 }
 
 func TestPrefixHeaderIdExtensionWithPrefixAndSuffix(t *testing.T) {
+	t.Parallel()
 	var tests = []string{
 		"# header 1 {#someid}\n",
 		"<h1 id=\"PRE:someid:POST\">header 1</h1>\n",
@@ -306,15 +251,20 @@ func TestPrefixHeaderIdExtensionWithPrefixAndSuffix(t *testing.T) {
 			"<h1 id=\"PRE:someid:POST\">Nested header</h1></li>\n</ul></li>\n</ul>\n",
 	}
 
-	parameters := HtmlRendererParameters{
-		HeaderIDPrefix: "PRE:",
-		HeaderIDSuffix: ":POST",
+	parameters := HTMLRendererParameters{
+		HeadingIDPrefix: "PRE:",
+		HeadingIDSuffix: ":POST",
 	}
 
-	doTestsBlockWithRunner(t, tests, EXTENSION_HEADER_IDS, runnerWithRendererParameters(parameters))
+	doTestsParam(t, tests, TestParams{
+		extensions:             HeadingIDs,
+		HTMLFlags:              UseXHTML,
+		HTMLRendererParameters: parameters,
+	})
 }
 
 func TestPrefixAutoHeaderIdExtension(t *testing.T) {
+	t.Parallel()
 	var tests = []string{
 		"# Header 1\n",
 		"<h1 id=\"header-1\">Header 1</h1>\n",
@@ -362,10 +312,11 @@ func TestPrefixAutoHeaderIdExtension(t *testing.T) {
 		"# Header\n\n# Header 1\n\n# Header\n\n# Header",
 		"<h1 id=\"header\">Header</h1>\n\n<h1 id=\"header-1\">Header 1</h1>\n\n<h1 id=\"header-1-1\">Header</h1>\n\n<h1 id=\"header-1-2\">Header</h1>\n",
 	}
-	doTestsBlock(t, tests, EXTENSION_AUTO_HEADER_IDS)
+	doTestsBlock(t, tests, AutoHeadingIDs)
 }
 
 func TestPrefixAutoHeaderIdExtensionWithPrefixAndSuffix(t *testing.T) {
+	t.Parallel()
 	var tests = []string{
 		"# Header 1\n",
 		"<h1 id=\"PRE:header-1:POST\">Header 1</h1>\n",
@@ -414,23 +365,116 @@ func TestPrefixAutoHeaderIdExtensionWithPrefixAndSuffix(t *testing.T) {
 		"<h1 id=\"PRE:header:POST\">Header</h1>\n\n<h1 id=\"PRE:header-1:POST\">Header 1</h1>\n\n<h1 id=\"PRE:header-1-1:POST\">Header</h1>\n\n<h1 id=\"PRE:header-1-2:POST\">Header</h1>\n",
 	}
 
-	parameters := HtmlRendererParameters{
-		HeaderIDPrefix: "PRE:",
-		HeaderIDSuffix: ":POST",
+	parameters := HTMLRendererParameters{
+		HeadingIDPrefix: "PRE:",
+		HeadingIDSuffix: ":POST",
 	}
 
-	doTestsBlockWithRunner(t, tests, EXTENSION_AUTO_HEADER_IDS, runnerWithRendererParameters(parameters))
+	doTestsParam(t, tests, TestParams{
+		extensions:             AutoHeadingIDs,
+		HTMLFlags:              UseXHTML,
+		HTMLRendererParameters: parameters,
+	})
+}
+
+func TestPrefixHeaderLevelOffset(t *testing.T) {
+	t.Parallel()
+	var offsetTests = []struct {
+		offset int
+		tests  []string
+	}{{
+		offset: 0,
+		tests: []string{
+			"# Header 1\n",
+			"<h1>Header 1</h1>\n",
+
+			"## Header 2\n",
+			"<h2>Header 2</h2>\n",
+
+			"### Header 3\n",
+			"<h3>Header 3</h3>\n",
+
+			"#### Header 4\n",
+			"<h4>Header 4</h4>\n",
+
+			"##### Header 5\n",
+			"<h5>Header 5</h5>\n",
+
+			"###### Header 6\n",
+			"<h6>Header 6</h6>\n",
+
+			"####### Header 7\n",
+			"<h6># Header 7</h6>\n",
+		},
+	}, {
+		offset: 1,
+		tests: []string{
+			"# Header 1\n",
+			"<h2>Header 1</h2>\n",
+
+			"## Header 2\n",
+			"<h3>Header 2</h3>\n",
+
+			"### Header 3\n",
+			"<h4>Header 3</h4>\n",
+
+			"#### Header 4\n",
+			"<h5>Header 4</h5>\n",
+
+			"##### Header 5\n",
+			"<h6>Header 5</h6>\n",
+
+			"###### Header 6\n",
+			"<h6>Header 6</h6>\n",
+
+			"####### Header 7\n",
+			"<h6># Header 7</h6>\n",
+		},
+	}, {
+		offset: -1,
+		tests: []string{
+			"# Header 1\n",
+			"<h1>Header 1</h1>\n",
+
+			"## Header 2\n",
+			"<h1>Header 2</h1>\n",
+
+			"### Header 3\n",
+			"<h2>Header 3</h2>\n",
+
+			"#### Header 4\n",
+			"<h3>Header 4</h3>\n",
+
+			"##### Header 5\n",
+			"<h4>Header 5</h4>\n",
+
+			"###### Header 6\n",
+			"<h5>Header 6</h5>\n",
+
+			"####### Header 7\n",
+			"<h5># Header 7</h5>\n",
+		},
+	}}
+	for _, offsetTest := range offsetTests {
+		offset := offsetTest.offset
+		tests := offsetTest.tests
+		doTestsParam(t, tests, TestParams{
+			HTMLRendererParameters: HTMLRendererParameters{HeadingLevelOffset: offset},
+		})
+	}
 }
 
 func TestPrefixMultipleHeaderExtensions(t *testing.T) {
+	t.Parallel()
 	var tests = []string{
 		"# Header\n\n# Header {#header}\n\n# Header 1",
 		"<h1 id=\"header\">Header</h1>\n\n<h1 id=\"header-1\">Header</h1>\n\n<h1 id=\"header-1-1\">Header 1</h1>\n",
 	}
-	doTestsBlock(t, tests, EXTENSION_AUTO_HEADER_IDS|EXTENSION_HEADER_IDS)
+	doTestsBlock(t, tests, AutoHeadingIDs|HeadingIDs)
 }
 
 func TestUnderlineHeaders(t *testing.T) {
+	t.Parallel()
 	var tests = []string{
 		"Header 1\n========\n",
 		"<h1>Header 1</h1>\n",
@@ -481,6 +525,7 @@ func TestUnderlineHeaders(t *testing.T) {
 }
 
 func TestUnderlineHeadersAutoIDs(t *testing.T) {
+	t.Parallel()
 	var tests = []string{
 		"Header 1\n========\n",
 		"<h1 id=\"header-1\">Header 1</h1>\n",
@@ -527,10 +572,11 @@ func TestUnderlineHeadersAutoIDs(t *testing.T) {
 		"Header 1\n========\n\nHeader 1\n========\n",
 		"<h1 id=\"header-1\">Header 1</h1>\n\n<h1 id=\"header-1-1\">Header 1</h1>\n",
 	}
-	doTestsBlock(t, tests, EXTENSION_AUTO_HEADER_IDS)
+	doTestsBlock(t, tests, AutoHeadingIDs)
 }
 
 func TestHorizontalRule(t *testing.T) {
+	t.Parallel()
 	var tests = []string{
 		"-\n",
 		"<p>-</p>\n",
@@ -596,6 +642,7 @@ func TestHorizontalRule(t *testing.T) {
 }
 
 func TestUnorderedList(t *testing.T) {
+	t.Parallel()
 	var tests = []string{
 		"* Hello\n",
 		"<ul>\n<li>Hello</li>\n</ul>\n",
@@ -707,6 +754,7 @@ func TestUnorderedList(t *testing.T) {
 }
 
 func TestOrderedList(t *testing.T) {
+	t.Parallel()
 	var tests = []string{
 		"1. Hello\n",
 		"<ol>\n<li>Hello</li>\n</ol>\n",
@@ -803,6 +851,7 @@ func TestOrderedList(t *testing.T) {
 }
 
 func TestDefinitionList(t *testing.T) {
+	t.Parallel()
 	var tests = []string{
 		"Term 1\n:   Definition a\n",
 		"<dl>\n<dt>Term 1</dt>\n<dd>Definition a</dd>\n</dl>\n",
@@ -901,10 +950,23 @@ func TestDefinitionList(t *testing.T) {
 			"</dl>\n" +
 			"\n<p>Text 2</p>\n",
 	}
-	doTestsBlock(t, tests, EXTENSION_DEFINITION_LISTS)
+	doTestsBlock(t, tests, DefinitionLists)
+}
+
+func TestConsecutiveLists(t *testing.T) {
+	t.Parallel()
+	var tests = []string{
+		"1. Hello\n\n* Hello\n\nTerm 1\n:   Definition a\n",
+		"<ol>\n<li>Hello</li>\n</ol>\n\n<ul>\n<li>Hello</li>\n</ul>\n\n<dl>\n<dt>Term 1</dt>\n<dd>Definition a</dd>\n</dl>\n",
+
+		"1. Not nested\n2. ordered list\n\n\t1. nested\n\t2. ordered list\n\n\t* nested\n\t* unordered list\n* Not nested\n* unordered list",
+		"<ol>\n<li><p>Not nested</p></li>\n\n<li><p>ordered list</p>\n\n<ol>\n<li>nested</li>\n<li>ordered list</li>\n</ol>\n\n<ul>\n<li>nested</li>\n<li>unordered list</li>\n</ul></li>\n</ol>\n\n<ul>\n<li>Not nested</li>\n<li>unordered list</li>\n</ul>\n",
+	}
+	doTestsBlock(t, tests, DefinitionLists)
 }
 
 func TestPreformattedHtml(t *testing.T) {
+	t.Parallel()
 	var tests = []string{
 		"<div></div>\n",
 		"<div></div>\n",
@@ -958,6 +1020,7 @@ func TestPreformattedHtml(t *testing.T) {
 }
 
 func TestPreformattedHtmlLax(t *testing.T) {
+	t.Parallel()
 	var tests = []string{
 		"Paragraph\n<div>\nHere? >&<\n</div>\n",
 		"<p>Paragraph</p>\n\n<div>\nHere? >&<\n</div>\n",
@@ -977,14 +1040,18 @@ func TestPreformattedHtmlLax(t *testing.T) {
 		"Paragraph\n\n<div>\nHow about here? >&<\n</div>\n\nAnd here?\n",
 		"<p>Paragraph</p>\n\n<div>\nHow about here? >&<\n</div>\n\n<p>And here?</p>\n",
 	}
-	doTestsBlock(t, tests, EXTENSION_LAX_HTML_BLOCKS)
+	doTestsBlock(t, tests, LaxHTMLBlocks)
 }
 
 func TestFencedCodeBlock(t *testing.T) {
+	t.Parallel()
 	var tests = []string{
 		"``` go\nfunc foo() bool {\n\treturn true;\n}\n```\n",
 		"<pre><code class=\"language-go\">func foo() bool {\n\treturn true;\n}\n</code></pre>\n",
 
+		"``` go foo bar\nfunc foo() bool {\n\treturn true;\n}\n```\n",
+		"<pre><code class=\"language-go\">func foo() bool {\n\treturn true;\n}\n</code></pre>\n",
+
 		"``` c\n/* special & char < > \" escaping */\n```\n",
 		"<pre><code class=\"language-c\">/* special &amp; char &lt; &gt; &quot; escaping */\n</code></pre>\n",
 
@@ -1062,11 +1129,18 @@ func TestFencedCodeBlock(t *testing.T) {
 
 		"Some text before a fenced code block\n``` oz\ncode blocks breakup paragraphs\n```\nSome text in between\n``` oz\nmultiple code blocks work okay\n```\nAnd some text after a fenced code block",
 		"<p>Some text before a fenced code block</p>\n\n<pre><code class=\"language-oz\">code blocks breakup paragraphs\n</code></pre>\n\n<p>Some text in between</p>\n\n<pre><code class=\"language-oz\">multiple code blocks work okay\n</code></pre>\n\n<p>And some text after a fenced code block</p>\n",
+
+		"```\n[]:()\n```\n",
+		"<pre><code>[]:()\n</code></pre>\n",
+
+		"```\n[]:()\n[]:)\n[]:(\n[]:x\n[]:testing\n[:testing\n\n[]:\nlinebreak\n[]()\n\n[]:\n[]()\n```",
+		"<pre><code>[]:()\n[]:)\n[]:(\n[]:x\n[]:testing\n[:testing\n\n[]:\nlinebreak\n[]()\n\n[]:\n[]()\n</code></pre>\n",
 	}
-	doTestsBlock(t, tests, EXTENSION_FENCED_CODE)
+	doTestsBlock(t, tests, FencedCode)
 }
 
 func TestFencedCodeInsideBlockquotes(t *testing.T) {
+	t.Parallel()
 	cat := func(s ...string) string { return strings.Join(s, "\n") }
 	var tests = []string{
 		cat("> ```go",
@@ -1175,10 +1249,11 @@ okay
 	tests = append(tests, forms[0], want)
 	tests = append(tests, forms[1], want)
 
-	doTestsBlock(t, tests, EXTENSION_FENCED_CODE)
+	doTestsBlock(t, tests, FencedCode)
 }
 
 func TestTable(t *testing.T) {
+	t.Parallel()
 	var tests = []string{
 		"a | b\n---|---\nc | d\n",
 		"<table>\n<thead>\n<tr>\n<th>a</th>\n<th>b</th>\n</tr>\n</thead>\n\n" +
@@ -1222,10 +1297,11 @@ func TestTable(t *testing.T) {
 		"a|b\\|c|d\n---|---|---\nf|g\\|h|i\n",
 		"<table>\n<thead>\n<tr>\n<th>a</th>\n<th>b|c</th>\n<th>d</th>\n</tr>\n</thead>\n\n<tbody>\n<tr>\n<td>f</td>\n<td>g|h</td>\n<td>i</td>\n</tr>\n</tbody>\n</table>\n",
 	}
-	doTestsBlock(t, tests, EXTENSION_TABLES)
+	doTestsBlock(t, tests, Tables)
 }
 
 func TestUnorderedListWith_EXTENSION_NO_EMPTY_LINE_BEFORE_BLOCK(t *testing.T) {
+	t.Parallel()
 	var tests = []string{
 		"* Hello\n",
 		"<ul>\n<li>Hello</li>\n</ul>\n",
@@ -1333,10 +1409,11 @@ func TestUnorderedListWith_EXTENSION_NO_EMPTY_LINE_BEFORE_BLOCK(t *testing.T) {
 		"* List\n\n    * sublist\n\n    normal text\n\n    * another sublist\n",
 		"<ul>\n<li><p>List</p>\n\n<ul>\n<li>sublist</li>\n</ul>\n\n<p>normal text</p>\n\n<ul>\n<li>another sublist</li>\n</ul></li>\n</ul>\n",
 	}
-	doTestsBlock(t, tests, EXTENSION_NO_EMPTY_LINE_BEFORE_BLOCK)
+	doTestsBlock(t, tests, NoEmptyLineBeforeBlock)
 }
 
 func TestOrderedList_EXTENSION_NO_EMPTY_LINE_BEFORE_BLOCK(t *testing.T) {
+	t.Parallel()
 	var tests = []string{
 		"1. Hello\n",
 		"<ol>\n<li>Hello</li>\n</ol>\n",
@@ -1429,14 +1506,18 @@ func TestOrderedList_EXTENSION_NO_EMPTY_LINE_BEFORE_BLOCK(t *testing.T) {
 		"1. numbers\n1. are ignored\n",
 		"<ol>\n<li>numbers</li>\n<li>are ignored</li>\n</ol>\n",
 	}
-	doTestsBlock(t, tests, EXTENSION_NO_EMPTY_LINE_BEFORE_BLOCK)
+	doTestsBlock(t, tests, NoEmptyLineBeforeBlock)
 }
 
 func TestFencedCodeBlock_EXTENSION_NO_EMPTY_LINE_BEFORE_BLOCK(t *testing.T) {
+	t.Parallel()
 	var tests = []string{
 		"``` go\nfunc foo() bool {\n\treturn true;\n}\n```\n",
 		"<pre><code class=\"language-go\">func foo() bool {\n\treturn true;\n}\n</code></pre>\n",
 
+		"``` go foo bar\nfunc foo() bool {\n\treturn true;\n}\n```\n",
+		"<pre><code class=\"language-go\">func foo() bool {\n\treturn true;\n}\n</code></pre>\n",
+
 		"``` c\n/* special & char < > \" escaping */\n```\n",
 		"<pre><code class=\"language-c\">/* special &amp; char &lt; &gt; &quot; escaping */\n</code></pre>\n",
 
@@ -1500,10 +1581,52 @@ func TestFencedCodeBlock_EXTENSION_NO_EMPTY_LINE_BEFORE_BLOCK(t *testing.T) {
 		"    ``` oz\nleading spaces\n    ```\n",
 		"<pre><code>``` oz\n</code></pre>\n\n<p>leading spaces</p>\n\n<pre><code>```\n</code></pre>\n",
 	}
-	doTestsBlock(t, tests, EXTENSION_FENCED_CODE|EXTENSION_NO_EMPTY_LINE_BEFORE_BLOCK)
+	doTestsBlock(t, tests, FencedCode|NoEmptyLineBeforeBlock)
+}
+
+func TestListWithFencedCodeBlock(t *testing.T) {
+	t.Parallel()
+	var tests = []string{
+		"1. one\n\n    ```\n    code\n    ```\n\n2. two\n",
+		"<ol>\n<li><p>one</p>\n\n<pre><code>code\n</code></pre></li>\n\n<li><p>two</p></li>\n</ol>\n",
+		// https://github.com/russross/blackfriday/issues/239
+		"1. one\n\n    ```\n    - code\n    ```\n\n2. two\n",
+		"<ol>\n<li><p>one</p>\n\n<pre><code>- code\n</code></pre></li>\n\n<li><p>two</p></li>\n</ol>\n",
+	}
+	doTestsBlock(t, tests, FencedCode)
+}
+
+func TestListWithMalformedFencedCodeBlock(t *testing.T) {
+	t.Parallel()
+	// Ensure that in the case of an unclosed fenced code block in a list,
+	// no source gets ommitted (even if it is malformed).
+	// See russross/blackfriday#372 for context.
+	var tests = []string{
+		"1. one\n\n    ```\n    code\n\n2. two\n",
+		"<ol>\n<li>one\n```\ncode\n2. two</li>\n</ol>\n",
+
+		"1. one\n\n    ```\n    - code\n\n2. two\n",
+		"<ol>\n<li>one\n```\n- code\n2. two</li>\n</ol>\n",
+	}
+	doTestsBlock(t, tests, FencedCode)
+}
+
+func TestListWithFencedCodeBlockNoExtensions(t *testing.T) {
+	t.Parallel()
+	// If there is a fenced code block in a list, and FencedCode is not set,
+	// lists should be processed normally.
+	var tests = []string{
+		"1. one\n\n    ```\n    code\n    ```\n\n2. two\n",
+		"<ol>\n<li><p>one</p>\n\n<p><code>\ncode\n</code></p></li>\n\n<li><p>two</p></li>\n</ol>\n",
+
+		"1. one\n\n    ```\n    - code\n    ```\n\n2. two\n",
+		"<ol>\n<li><p>one</p>\n\n<p>```</p>\n\n<ul>\n<li>code\n```</li>\n</ul></li>\n\n<li><p>two</p></li>\n</ol>\n",
+	}
+	doTestsBlock(t, tests, 0)
 }
 
 func TestTitleBlock_EXTENSION_TITLEBLOCK(t *testing.T) {
+	t.Parallel()
 	var tests = []string{
 		"% Some title\n" +
 			"% Another title line\n" +
@@ -1511,13 +1634,14 @@ func TestTitleBlock_EXTENSION_TITLEBLOCK(t *testing.T) {
 		"<h1 class=\"title\">" +
 			"Some title\n" +
 			"Another title line\n" +
-			"Yep, more here too\n" +
-			"</h1>",
+			"Yep, more here too" +
+			"</h1>\n",
 	}
-	doTestsBlock(t, tests, EXTENSION_TITLEBLOCK)
+	doTestsBlock(t, tests, Titleblock)
 }
 
 func TestBlockComments(t *testing.T) {
+	t.Parallel()
 	var tests = []string{
 		"Some text\n\n<!-- comment -->\n",
 		"<p>Some text</p>\n\n<!-- comment -->\n",
@@ -1530,3 +1654,265 @@ func TestBlockComments(t *testing.T) {
 	}
 	doTestsBlock(t, tests, 0)
 }
+
+func TestTOC(t *testing.T) {
+	t.Parallel()
+	var tests = []string{
+		"# Title\n\n##Subtitle1\n\n##Subtitle2",
+		//"<nav>\n<ul>\n<li><a href=\"#toc_0\">Title</a>\n<ul>\n<li><a href=\"#toc_1\">Subtitle1</a></li>\n<li><a href=\"#toc_2\">Subtitle2</a></li>\n</ul></li>\n</ul>\n</nav>\n\n<h1 id=\"toc_0\">Title</h1>\n\n<h2 id=\"toc_1\">Subtitle1</h2>\n\n<h2 id=\"toc_2\">Subtitle2</h2>\n",
+		`<nav>
+
+<ul>
+<li><a href="#toc_0">Title</a>
+<ul>
+<li><a href="#toc_1">Subtitle1</a></li>
+
+<li><a href="#toc_2">Subtitle2</a></li>
+</ul></li>
+</ul>
+
+</nav>
+
+<h1 id="toc_0">Title</h1>
+
+<h2 id="toc_1">Subtitle1</h2>
+
+<h2 id="toc_2">Subtitle2</h2>
+`,
+
+		"# Title\n\n##Subtitle\n\n#Title2",
+		//"<nav>\n<ul>\n<li><a href=\"#toc_0\">Title</a>\n<ul>\n<li><a href=\"#toc_1\">Subtitle</a></li>\n</ul></li>\n<li><a href=\"#toc_2\">Title2</a></li>\n</ul>\n</nav>\n\n<h1 id=\"toc_0\">Title</h1>\n\n<h2 id=\"toc_1\">Subtitle</h2>\n\n<h1 id=\"toc_2\">Title2</h1>\n",
+		`<nav>
+
+<ul>
+<li><a href="#toc_0">Title</a>
+<ul>
+<li><a href="#toc_1">Subtitle</a></li>
+</ul></li>
+
+<li><a href="#toc_2">Title2</a></li>
+</ul>
+
+</nav>
+
+<h1 id="toc_0">Title</h1>
+
+<h2 id="toc_1">Subtitle</h2>
+
+<h1 id="toc_2">Title2</h1>
+`,
+
+		"## Subtitle\n\n# Title",
+		`<nav>
+
+<ul>
+<li>
+<ul>
+<li><a href="#toc_0">Subtitle</a></li>
+</ul></li>
+
+<li><a href="#toc_1">Title</a></li>
+</ul>
+
+</nav>
+
+<h2 id="toc_0">Subtitle</h2>
+
+<h1 id="toc_1">Title</h1>
+`,
+
+		"# Title 1\n\n## Subtitle 1\n\n### Subsubtitle 1\n\n# Title 2\n\n### Subsubtitle 2",
+		`<nav>
+
+<ul>
+<li><a href="#toc_0">Title 1</a>
+<ul>
+<li><a href="#toc_1">Subtitle 1</a>
+<ul>
+<li><a href="#toc_2">Subsubtitle 1</a></li>
+</ul></li>
+</ul></li>
+
+<li><a href="#toc_3">Title 2</a>
+<ul>
+<li>
+<ul>
+<li><a href="#toc_4">Subsubtitle 2</a></li>
+</ul></li>
+</ul></li>
+</ul>
+
+</nav>
+
+<h1 id="toc_0">Title 1</h1>
+
+<h2 id="toc_1">Subtitle 1</h2>
+
+<h3 id="toc_2">Subsubtitle 1</h3>
+
+<h1 id="toc_3">Title 2</h1>
+
+<h3 id="toc_4">Subsubtitle 2</h3>
+`,
+
+		"# Title with `code`",
+		`<nav>
+
+<ul>
+<li><a href="#toc_0">Title with <code>code</code></a></li>
+</ul>
+
+</nav>
+
+<h1 id="toc_0">Title with <code>code</code></h1>
+`,
+
+		// Trigger empty TOC
+		"#",
+		"",
+	}
+	doTestsParam(t, tests, TestParams{
+		HTMLFlags: UseXHTML | TOC,
+	})
+}
+
+func TestCompletePage(t *testing.T) {
+	t.Parallel()
+	var tests = []string{
+		"*foo*",
+		`<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
+<html xmlns="http://www.w3.org/1999/xhtml">
+<head>
+  <title></title>
+  <meta name="GENERATOR" content="Blackfriday Markdown Processor v2.0" />
+  <meta charset="utf-8" />
+</head>
+<body>
+
+<p><em>foo</em></p>
+
+</body>
+</html>
+`,
+	}
+	doTestsParam(t, tests, TestParams{HTMLFlags: UseXHTML | CompletePage})
+}
+
+func TestIsFenceLine(t *testing.T) {
+	t.Parallel()
+	tests := []struct {
+		data          []byte
+		infoRequested bool
+		wantEnd       int
+		wantMarker    string
+		wantInfo      string
+	}{
+		{
+			data:       []byte("```"),
+			wantEnd:    3,
+			wantMarker: "```",
+		},
+		{
+			data:       []byte("```\nstuff here\n"),
+			wantEnd:    4,
+			wantMarker: "```",
+		},
+		{
+			data:          []byte("```\nstuff here\n"),
+			infoRequested: true,
+			wantEnd:       4,
+			wantMarker:    "```",
+		},
+		{
+			data:    []byte("stuff here\n```\n"),
+			wantEnd: 0,
+		},
+		{
+			data:          []byte("```"),
+			infoRequested: true,
+			wantEnd:       3,
+			wantMarker:    "```",
+		},
+		{
+			data:          []byte("``` go"),
+			infoRequested: true,
+			wantEnd:       6,
+			wantMarker:    "```",
+			wantInfo:      "go",
+		},
+		{
+			data:          []byte("``` go foo bar"),
+			infoRequested: true,
+			wantEnd:       14,
+			wantMarker:    "```",
+			wantInfo:      "go foo bar",
+		},
+		{
+			data:          []byte("``` go foo bar  "),
+			infoRequested: true,
+			wantEnd:       16,
+			wantMarker:    "```",
+			wantInfo:      "go foo bar",
+		},
+	}
+
+	for _, test := range tests {
+		var info *string
+		if test.infoRequested {
+			info = new(string)
+		}
+		end, marker := isFenceLine(test.data, info, "```")
+		if got, want := end, test.wantEnd; got != want {
+			t.Errorf("got end %v, want %v", got, want)
+		}
+		if got, want := marker, test.wantMarker; got != want {
+			t.Errorf("got marker %q, want %q", got, want)
+		}
+		if test.infoRequested {
+			if got, want := *info, test.wantInfo; got != want {
+				t.Errorf("got info string %q, want %q", got, want)
+			}
+		}
+	}
+}
+
+func TestSanitizedAnchorName(t *testing.T) {
+	tests := []struct {
+		text string
+		want string
+	}{
+		{
+			text: "This is a header",
+			want: "this-is-a-header",
+		},
+		{
+			text: "This is also          a header",
+			want: "this-is-also-a-header",
+		},
+		{
+			text: "main.go",
+			want: "main-go",
+		},
+		{
+			text: "Article 123",
+			want: "article-123",
+		},
+		{
+			text: "<- Let's try this, shall we?",
+			want: "let-s-try-this-shall-we",
+		},
+		{
+			text: "        ",
+			want: "",
+		},
+		{
+			text: "Hello, 世界",
+			want: "hello-世界",
+		},
+	}
+	for _, test := range tests {
+		if got := SanitizedAnchorName(test.text); got != test.want {
+			t.Errorf("SanitizedAnchorName(%q):\ngot %q\nwant %q", test.text, got, test.want)
+		}
+	}
+}

doc.go

@@ -0,0 +1,46 @@
+// Package blackfriday is a markdown processor.
+//
+// It translates plain text with simple formatting rules into an AST, which can
+// then be further processed to HTML (provided by Blackfriday itself) or other
+// formats (provided by the community).
+//
+// The simplest way to invoke Blackfriday is to call the Run function. It will
+// take a text input and produce a text output in HTML (or other format).
+//
+// A slightly more sophisticated way to use Blackfriday is to create a Markdown
+// processor and to call Parse, which returns a syntax tree for the input
+// document. You can leverage Blackfriday's parsing for content extraction from
+// markdown documents. You can assign a custom renderer and set various options
+// to the Markdown processor.
+//
+// If you're interested in calling Blackfriday from command line, see
+// https://github.com/russross/blackfriday-tool.
+//
+// Sanitized Anchor Names
+//
+// Blackfriday includes an algorithm for creating sanitized anchor names
+// corresponding to a given input text. This algorithm is used to create
+// anchors for headings when AutoHeadingIDs extension is enabled. The
+// algorithm is specified below, so that other packages can create
+// compatible anchor names and links to those anchors.
+//
+// The algorithm iterates over the input text, interpreted as UTF-8,
+// one Unicode code point (rune) at a time. All runes that are letters (category L)
+// or numbers (category N) are considered valid characters. They are mapped to
+// lower case, and included in the output. All other runes are considered
+// invalid characters. Invalid characters that precede the first valid character,
+// as well as invalid character that follow the last valid character
+// are dropped completely. All other sequences of invalid characters
+// between two valid characters are replaced with a single dash character '-'.
+//
+// SanitizedAnchorName exposes this functionality, and can be used to
+// create compatible links to the anchor names generated by blackfriday.
+// This algorithm is also implemented in a small standalone package at
+// github.com/shurcooL/sanitized_anchor_name. It can be useful for clients
+// that want a small package and don't need full functionality of blackfriday.
+package blackfriday
+
+// NOTE: Keep Sanitized Anchor Name algorithm in sync with package
+//       github.com/shurcooL/sanitized_anchor_name.
+//       Otherwise, users of sanitized_anchor_name will get anchor names
+//       that are incompatible with those generated by blackfriday.

esc.go

@@ -0,0 +1,70 @@
+package blackfriday
+
+import (
+	"html"
+	"io"
+)
+
+var htmlEscaper = [256][]byte{
+	'&': []byte("&"),
+	'<': []byte("&lt;"),
+	'>': []byte("&gt;"),
+	'"': []byte("&quot;"),
+}
+
+func escapeHTML(w io.Writer, s []byte) {
+	escapeEntities(w, s, false)
+}
+
+func escapeAllHTML(w io.Writer, s []byte) {
+	escapeEntities(w, s, true)
+}
+
+func escapeEntities(w io.Writer, s []byte, escapeValidEntities bool) {
+	var start, end int
+	for end < len(s) {
+		escSeq := htmlEscaper[s[end]]
+		if escSeq != nil {
+			isEntity, entityEnd := nodeIsEntity(s, end)
+			if isEntity && !escapeValidEntities {
+				w.Write(s[start : entityEnd+1])
+				start = entityEnd + 1
+			} else {
+				w.Write(s[start:end])
+				w.Write(escSeq)
+				start = end + 1
+			}
+		}
+		end++
+	}
+	if start < len(s) && end <= len(s) {
+		w.Write(s[start:end])
+	}
+}
+
+func nodeIsEntity(s []byte, end int) (isEntity bool, endEntityPos int) {
+	isEntity = false
+	endEntityPos = end + 1
+
+	if s[end] == '&' {
+		for endEntityPos < len(s) {
+			if s[endEntityPos] == ';' {
+				if entities[string(s[end:endEntityPos+1])] {
+					isEntity = true
+					break
+				}
+			}
+			if !isalnum(s[endEntityPos]) && s[endEntityPos] != '&' && s[endEntityPos] != '#' {
+				break
+			}
+			endEntityPos++
+		}
+	}
+
+	return isEntity, endEntityPos
+}
+
+func escLink(w io.Writer, text []byte) {
+	unesc := html.UnescapeString(string(text))
+	escapeHTML(w, []byte(unesc))
+}

esc_test.go

@@ -0,0 +1,49 @@
+package blackfriday
+
+import (
+	"bytes"
+	"testing"
+)
+
+func TestEsc(t *testing.T) {
+	t.Parallel()
+	tests := []string{
+		"abc", "abc",
+		"a&c", "a&c",
+		"<", "&lt;",
+		"[]:<", "[]:&lt;",
+		"Hello <!--", "Hello &lt;!--",
+	}
+	for i := 0; i < len(tests); i += 2 {
+		var b bytes.Buffer
+		escapeHTML(&b, []byte(tests[i]))
+		if !bytes.Equal(b.Bytes(), []byte(tests[i+1])) {
+			t.Errorf("\nInput   [%#v]\nExpected[%#v]\nActual  [%#v]",
+				tests[i], tests[i+1], b.String())
+		}
+	}
+}
+
+func BenchmarkEscapeHTML(b *testing.B) {
+	tests := [][]byte{
+		[]byte(""),
+		[]byte("AT&T has an ampersand in their name."),
+		[]byte("AT&amp;T is another way to write it."),
+		[]byte("This & that."),
+		[]byte("4 < 5."),
+		[]byte("6 > 5."),
+		[]byte("Here's a [link] [1] with an ampersand in the URL."),
+		[]byte("Here's a link with an ampersand in the link text: [AT&T] [2]."),
+		[]byte("Here's an inline [link](/script?foo=1&bar=2)."),
+		[]byte("Here's an inline [link](</script?foo=1&bar=2>)."),
+		[]byte("[1]: http://example.com/?foo=1&bar=2"),
+		[]byte("[2]: http://att.com/  \"AT&T\""),
+	}
+	var buf bytes.Buffer
+	for n := 0; n < b.N; n++ {
+		for _, t := range tests {
+			escapeHTML(&buf, t)
+			buf.Reset()
+		}
+	}
+}

go.mod

@@ -0,0 +1 @@
+module github.com/russross/blackfriday/v2

helpers_test.go

@@ -0,0 +1,208 @@
+//
+// Blackfriday Markdown Processor
+// Available at http://github.com/russross/blackfriday
+//
+// Copyright © 2011 Russ Ross <russ@russross.com>.
+// Distributed under the Simplified BSD License.
+// See README.md for details.
+//
+
+//
+// Helper functions for unit testing
+//
+
+package blackfriday
+
+import (
+	"fmt"
+	"io/ioutil"
+	"path/filepath"
+	"regexp"
+	"strings"
+	"testing"
+)
+
+type TestParams struct {
+	extensions        Extensions
+	referenceOverride ReferenceOverrideFunc
+	HTMLFlags
+	HTMLRendererParameters
+}
+
+func execRecoverableTestSuite(t *testing.T, tests []string, params TestParams, suite func(candidate *string)) {
+	// Catch and report panics. This is useful when running 'go test -v' on
+	// the integration server. When developing, though, crash dump is often
+	// preferable, so recovery can be easily turned off with doRecover = false.
+	var candidate string
+	const doRecover = true
+	if doRecover {
+		defer func() {
+			if err := recover(); err != nil {
+				t.Errorf("\npanic while processing [%#v]: %s\n", candidate, err)
+			}
+		}()
+	}
+	suite(&candidate)
+}
+
+func runMarkdown(input string, params TestParams) string {
+	params.HTMLRendererParameters.Flags = params.HTMLFlags
+	renderer := NewHTMLRenderer(params.HTMLRendererParameters)
+	return string(Run([]byte(input), WithRenderer(renderer),
+		WithExtensions(params.extensions),
+		WithRefOverride(params.referenceOverride)))
+}
+
+// doTests runs full document tests using MarkdownCommon configuration.
+func doTests(t *testing.T, tests []string) {
+	doTestsParam(t, tests, TestParams{
+		extensions: CommonExtensions,
+		HTMLRendererParameters: HTMLRendererParameters{
+			Flags: CommonHTMLFlags,
+		},
+	})
+}
+
+func doTestsBlock(t *testing.T, tests []string, extensions Extensions) {
+	doTestsParam(t, tests, TestParams{
+		extensions: extensions,
+		HTMLFlags:  UseXHTML,
+	})
+}
+
+func doTestsParam(t *testing.T, tests []string, params TestParams) {
+	execRecoverableTestSuite(t, tests, params, func(candidate *string) {
+		for i := 0; i+1 < len(tests); i += 2 {
+			input := tests[i]
+			*candidate = input
+			expected := tests[i+1]
+			actual := runMarkdown(*candidate, params)
+			if actual != expected {
+				t.Errorf("\nInput   [%#v]\nExpected[%#v]\nActual  [%#v]",
+					*candidate, expected, actual)
+			}
+
+			// now test every substring to stress test bounds checking
+			if !testing.Short() {
+				for start := 0; start < len(input); start++ {
+					for end := start + 1; end <= len(input); end++ {
+						*candidate = input[start:end]
+						runMarkdown(*candidate, params)
+					}
+				}
+			}
+		}
+	})
+}
+
+func doTestsInline(t *testing.T, tests []string) {
+	doTestsInlineParam(t, tests, TestParams{})
+}
+
+func doLinkTestsInline(t *testing.T, tests []string) {
+	doTestsInline(t, tests)
+
+	prefix := "http://localhost"
+	params := HTMLRendererParameters{AbsolutePrefix: prefix}
+	transformTests := transformLinks(tests, prefix)
+	doTestsInlineParam(t, transformTests, TestParams{
+		HTMLRendererParameters: params,
+	})
+	doTestsInlineParam(t, transformTests, TestParams{
+		HTMLFlags:              UseXHTML,
+		HTMLRendererParameters: params,
+	})
+}
+
+func doSafeTestsInline(t *testing.T, tests []string) {
+	doTestsInlineParam(t, tests, TestParams{HTMLFlags: Safelink})
+
+	// All the links in this test should not have the prefix appended, so
+	// just rerun it with different parameters and the same expectations.
+	prefix := "http://localhost"
+	params := HTMLRendererParameters{AbsolutePrefix: prefix}
+	transformTests := transformLinks(tests, prefix)
+	doTestsInlineParam(t, transformTests, TestParams{
+		HTMLFlags:              Safelink,
+		HTMLRendererParameters: params,
+	})
+}
+
+func doTestsInlineParam(t *testing.T, tests []string, params TestParams) {
+	params.extensions |= Autolink | Strikethrough
+	params.HTMLFlags |= UseXHTML
+	doTestsParam(t, tests, params)
+}
+
+func transformLinks(tests []string, prefix string) []string {
+	newTests := make([]string, len(tests))
+	anchorRe := regexp.MustCompile(`<a href="/(.*?)"`)
+	imgRe := regexp.MustCompile(`<img src="/(.*?)"`)
+	for i, test := range tests {
+		if i%2 == 1 {
+			test = anchorRe.ReplaceAllString(test, `<a href="`+prefix+`/$1"`)
+			test = imgRe.ReplaceAllString(test, `<img src="`+prefix+`/$1"`)
+		}
+		newTests[i] = test
+	}
+	return newTests
+}
+
+func doTestsReference(t *testing.T, files []string, flag Extensions) {
+	params := TestParams{extensions: flag}
+	execRecoverableTestSuite(t, files, params, func(candidate *string) {
+		for _, basename := range files {
+			filename := filepath.Join("testdata", basename+".text")
+			inputBytes, err := ioutil.ReadFile(filename)
+			if err != nil {
+				t.Errorf("Couldn't open '%s', error: %v\n", filename, err)
+				continue
+			}
+			input := string(inputBytes)
+
+			filename = filepath.Join("testdata", basename+".html")
+			expectedBytes, err := ioutil.ReadFile(filename)
+			if err != nil {
+				t.Errorf("Couldn't open '%s', error: %v\n", filename, err)
+				continue
+			}
+			expected := string(expectedBytes)
+
+			actual := string(runMarkdown(input, params))
+			if actual != expected {
+				t.Errorf("\n" + doTestDiff(basename, expected, actual))
+			}
+
+			// now test every prefix of every input to check for
+			// bounds checking
+			if !testing.Short() {
+				start, max := 0, len(input)
+				for end := start + 1; end <= max; end++ {
+					*candidate = input[start:end]
+					runMarkdown(*candidate, params)
+				}
+			}
+		}
+	})
+}
+
+func doTestDiff(name, expected, actual string) string {
+	expectedLines := strings.Split(expected, "\n")
+	actualLines := strings.Split(actual, "\n")
+	d := "file: " + name + "\n"
+	for i, line := range expectedLines {
+		// Allow the actualLines indexing to panic because we're in tests where
+		// that's okay and we probably want to know about it if this input is wrong
+		// somehow.
+		if line != actualLines[i] {
+			d += fmt.Sprintf(`
+line: %d
+
+-%s
++%s
+`, i, line, actualLines[i])
+		}
+	}
+
+	return d
+}

inline.go

@@ -22,6 +22,23 @@ import (
 var (
 	urlRe    = `((https?|ftp):\/\/|\/)[-A-Za-z0-9+&@#\/%?=~_|!:,.;\(\)]+`
 	anchorRe = regexp.MustCompile(`^(<a\shref="` + urlRe + `"(\stitle="[^"<>]+")?\s?>` + urlRe + `<\/a>)`)
+
+	// https://www.w3.org/TR/html5/syntax.html#character-references
+	// highest unicode code point in 17 planes (2^20): 1,114,112d =
+	// 7 dec digits or 6 hex digits
+	// named entity references can be 2-31 characters with stuff like &lt;
+	// at one end and &CounterClockwiseContourIntegral; at the other. There
+	// are also sometimes numbers at the end, although this isn't inherent
+	// in the specification; there are never numbers anywhere else in
+	// current character references, though; see &frac34; and &blk12;, etc.
+	// https://www.w3.org/TR/html5/syntax.html#named-character-references
+	//
+	// entity := "&" (named group | number ref) ";"
+	// named group := [a-zA-Z]{2,31}[0-9]{0,2}
+	// number ref := "#" (dec ref | hex ref)
+	// dec ref := [0-9]{1,7}
+	// hex ref := ("x" | "X") [0-9a-fA-F]{1,6}
+	htmlEntityRe = regexp.MustCompile(`&([a-zA-Z]{2,31}[0-9]{0,2}|#([0-9]{1,7}|[xX][0-9a-fA-F]{1,6}));`)
 )
 
 // Functions to parse text within a block
@@ -29,87 +46,89 @@ var (
 // data is the complete block being rendered
 // offset is the number of valid chars before the current cursor
 
-func (p *parser) inline(out *bytes.Buffer, data []byte) {
-	// this is called recursively: enforce a maximum depth
-	if p.nesting >= p.maxNesting {
+func (p *Markdown) inline(currBlock *Node, data []byte) {
+	// handlers might call us recursively: enforce a maximum depth
+	if p.nesting >= p.maxNesting || len(data) == 0 {
 		return
 	}
 	p.nesting++
-
-	i, end := 0, 0
-	for i < len(data) {
-		// copy inactive chars into the output
-		for end < len(data) && p.inlineCallback[data[end]] == nil {
+	beg, end := 0, 0
+	for end < len(data) {
+		handler := p.inlineCallback[data[end]]
+		if handler != nil {
+			if consumed, node := handler(p, data, end); consumed == 0 {
+				// No action from the callback.
+				end++
+			} else {
+				// Copy inactive chars into the output.
+				currBlock.AppendChild(text(data[beg:end]))
+				if node != nil {
+					currBlock.AppendChild(node)
+				}
+				// Skip past whatever the callback used.
+				beg = end + consumed
+				end = beg
+			}
+		} else {
 			end++
 		}
-
-		p.r.NormalText(out, data[i:end])
-
-		if end >= len(data) {
-			break
-		}
-		i = end
-
-		// call the trigger
-		handler := p.inlineCallback[data[end]]
-		if consumed := handler(p, out, data, i); consumed == 0 {
-			// no action from the callback; buffer the byte for later
-			end = i + 1
-		} else {
-			// skip past whatever the callback used
-			i += consumed
-			end = i
-		}
 	}
-
+	if beg < len(data) {
+		if data[end-1] == '\n' {
+			end--
+		}
+		currBlock.AppendChild(text(data[beg:end]))
+	}
 	p.nesting--
 }
 
 // single and double emphasis parsing
-func emphasis(p *parser, out *bytes.Buffer, data []byte, offset int) int {
+func emphasis(p *Markdown, data []byte, offset int) (int, *Node) {
 	data = data[offset:]
 	c := data[0]
-	ret := 0
 
 	if len(data) > 2 && data[1] != c {
 		// whitespace cannot follow an opening emphasis;
 		// strikethrough only takes two characters '~~'
 		if c == '~' || isspace(data[1]) {
-			return 0
+			return 0, nil
 		}
-		if ret = helperEmphasis(p, out, data[1:], c); ret == 0 {
-			return 0
+		ret, node := helperEmphasis(p, data[1:], c)
+		if ret == 0 {
+			return 0, nil
 		}
 
-		return ret + 1
+		return ret + 1, node
 	}
 
 	if len(data) > 3 && data[1] == c && data[2] != c {
 		if isspace(data[2]) {
-			return 0
+			return 0, nil
 		}
-		if ret = helperDoubleEmphasis(p, out, data[2:], c); ret == 0 {
-			return 0
+		ret, node := helperDoubleEmphasis(p, data[2:], c)
+		if ret == 0 {
+			return 0, nil
 		}
 
-		return ret + 2
+		return ret + 2, node
 	}
 
 	if len(data) > 4 && data[1] == c && data[2] == c && data[3] != c {
 		if c == '~' || isspace(data[3]) {
-			return 0
+			return 0, nil
 		}
-		if ret = helperTripleEmphasis(p, out, data, 3, c); ret == 0 {
-			return 0
+		ret, node := helperTripleEmphasis(p, data, 3, c)
+		if ret == 0 {
+			return 0, nil
 		}
 
-		return ret + 3
+		return ret + 3, node
 	}
 
-	return 0
+	return 0, nil
 }
 
-func codeSpan(p *parser, out *bytes.Buffer, data []byte, offset int) int {
+func codeSpan(p *Markdown, data []byte, offset int) (int, *Node) {
 	data = data[offset:]
 
 	nb := 0
@@ -131,7 +150,7 @@ func codeSpan(p *parser, out *bytes.Buffer, data []byte, offset int) int {
 
 	// no matching delimiter?
 	if i < nb && end >= len(data) {
-		return 0
+		return 0, nil
 	}
 
 	// trim outside whitespace
@@ -147,39 +166,36 @@ func codeSpan(p *parser, out *bytes.Buffer, data []byte, offset int) int {
 
 	// render the code span
 	if fBegin != fEnd {
-		p.r.CodeSpan(out, data[fBegin:fEnd])
+		code := NewNode(Code)
+		code.Literal = data[fBegin:fEnd]
+		return end, code
 	}
 
-	return end
-
+	return end, nil
 }
 
 // newline preceded by two spaces becomes <br>
-// newline without two spaces works when EXTENSION_HARD_LINE_BREAK is enabled
-func lineBreak(p *parser, out *bytes.Buffer, data []byte, offset int) int {
-	// remove trailing spaces from out
-	outBytes := out.Bytes()
-	end := len(outBytes)
-	eol := end
-	for eol > 0 && outBytes[eol-1] == ' ' {
-		eol--
-	}
-	out.Truncate(eol)
-
-	precededByTwoSpaces := offset >= 2 && data[offset-2] == ' ' && data[offset-1] == ' '
-	precededByBackslash := offset >= 1 && data[offset-1] == '\\' // see http://spec.commonmark.org/0.18/#example-527
-	precededByBackslash = precededByBackslash && p.flags&EXTENSION_BACKSLASH_LINE_BREAK != 0
-
-	// should there be a hard line break here?
-	if p.flags&EXTENSION_HARD_LINE_BREAK == 0 && !precededByTwoSpaces && !precededByBackslash {
-		return 0
+func maybeLineBreak(p *Markdown, data []byte, offset int) (int, *Node) {
+	origOffset := offset
+	for offset < len(data) && data[offset] == ' ' {
+		offset++
 	}
 
-	if precededByBackslash && eol > 0 {
-		out.Truncate(eol - 1)
+	if offset < len(data) && data[offset] == '\n' {
+		if offset-origOffset >= 2 {
+			return offset - origOffset + 1, NewNode(Hardbreak)
+		}
+		return offset - origOffset, nil
 	}
-	p.r.LineBreak(out)
-	return 1
+	return 0, nil
+}
+
+// newline without two spaces works when HardLineBreak is enabled
+func lineBreak(p *Markdown, data []byte, offset int) (int, *Node) {
+	if p.extensions&HardLineBreak != 0 {
+		return 1, NewNode(Hardbreak)
+	}
+	return 0, nil
 }
 
 type linkType int
@@ -198,27 +214,43 @@ func isReferenceStyleLink(data []byte, pos int, t linkType) bool {
 	return pos < len(data)-1 && data[pos] == '[' && data[pos+1] != '^'
 }
 
+func maybeImage(p *Markdown, data []byte, offset int) (int, *Node) {
+	if offset < len(data)-1 && data[offset+1] == '[' {
+		return link(p, data, offset)
+	}
+	return 0, nil
+}
+
+func maybeInlineFootnote(p *Markdown, data []byte, offset int) (int, *Node) {
+	if offset < len(data)-1 && data[offset+1] == '[' {
+		return link(p, data, offset)
+	}
+	return 0, nil
+}
+
 // '[': parse a link or an image or a footnote
-func link(p *parser, out *bytes.Buffer, data []byte, offset int) int {
+func link(p *Markdown, data []byte, offset int) (int, *Node) {
 	// no links allowed inside regular links, footnote, and deferred footnotes
 	if p.insideLink && (offset > 0 && data[offset-1] == '[' || len(data)-1 > offset && data[offset+1] == '^') {
-		return 0
+		return 0, nil
 	}
 
 	var t linkType
 	switch {
 	// special case: ![^text] == deferred footnote (that follows something with
 	// an exclamation point)
-	case p.flags&EXTENSION_FOOTNOTES != 0 && len(data)-1 > offset && data[offset+1] == '^':
+	case p.extensions&Footnotes != 0 && len(data)-1 > offset && data[offset+1] == '^':
 		t = linkDeferredFootnote
 	// ![alt] == image
-	case offset > 0 && data[offset-1] == '!':
+	case offset >= 0 && data[offset] == '!':
 		t = linkImg
+		offset++
 	// ^[text] == inline footnote
 	// [^refId] == deferred footnote
-	case p.flags&EXTENSION_FOOTNOTES != 0:
-		if offset > 0 && data[offset-1] == '^' {
+	case p.extensions&Footnotes != 0:
+		if offset >= 0 && data[offset] == '^' {
 			t = linkInlineFootnote
+			offset++
 		} else if len(data)-1 > offset && data[offset+1] == '^' {
 			t = linkDeferredFootnote
 		}
@@ -231,7 +263,7 @@ func link(p *parser, out *bytes.Buffer, data []byte, offset int) int {
 
 	var (
 		i                       = 1
-		noteId                  int
+		noteID                  int
 		title, link, altContent []byte
 		textHasNl               = false
 	)
@@ -246,7 +278,7 @@ func link(p *parser, out *bytes.Buffer, data []byte, offset int) int {
 		case data[i] == '\n':
 			textHasNl = true
 
-		case data[i-1] == '\\':
+		case isBackslashEscaped(data, i):
 			continue
 
 		case data[i] == '[':
@@ -261,11 +293,12 @@ func link(p *parser, out *bytes.Buffer, data []byte, offset int) int {
 	}
 
 	if i >= len(data) {
-		return 0
+		return 0, nil
 	}
 
 	txtE := i
 	i++
+	var footnoteNode *Node
 
 	// skip any amount of whitespace or newline
 	// (this is much more lax than original markdown syntax)
@@ -301,7 +334,7 @@ func link(p *parser, out *bytes.Buffer, data []byte, offset int) int {
 		}
 
 		if i >= len(data) {
-			return 0
+			return 0, nil
 		}
 		linkE := i
 
@@ -326,7 +359,7 @@ func link(p *parser, out *bytes.Buffer, data []byte, offset int) int {
 			}
 
 			if i >= len(data) {
-				return 0
+				return 0, nil
 			}
 
 			// skip whitespace after title
@@ -378,7 +411,7 @@ func link(p *parser, out *bytes.Buffer, data []byte, offset int) int {
 			i++
 		}
 		if i >= len(data) {
-			return 0
+			return 0, nil
 		}
 		linkE := i
 
@@ -408,7 +441,7 @@ func link(p *parser, out *bytes.Buffer, data []byte, offset int) int {
 		// find the reference with matching id
 		lr, ok := p.getRef(string(id))
 		if !ok {
-			return 0
+			return 0, nil
 		}
 
 		// keep link and title from reference
@@ -445,9 +478,10 @@ func link(p *parser, out *bytes.Buffer, data []byte, offset int) int {
 			}
 		}
 
+		footnoteNode = NewNode(Item)
 		if t == linkInlineFootnote {
 			// create a new reference
-			noteId = len(p.notes) + 1
+			noteID = len(p.notes) + 1
 
 			var fragment []byte
 			if len(id) > 0 {
@@ -458,14 +492,15 @@ func link(p *parser, out *bytes.Buffer, data []byte, offset int) int {
 				}
 				copy(fragment, slugify(id))
 			} else {
-				fragment = append([]byte("footnote-"), []byte(strconv.Itoa(noteId))...)
+				fragment = append([]byte("footnote-"), []byte(strconv.Itoa(noteID))...)
 			}
 
 			ref := &reference{
-				noteId:   noteId,
+				noteID:   noteID,
 				hasBlock: false,
 				link:     fragment,
 				title:    id,
+				footnote: footnoteNode,
 			}
 
 			p.notes = append(p.notes, ref)
@@ -476,11 +511,12 @@ func link(p *parser, out *bytes.Buffer, data []byte, offset int) int {
 			// find the reference with matching id
 			lr, ok := p.getRef(string(id))
 			if !ok {
-				return 0
+				return 0, nil
 			}
 
 			if t == linkDeferredFootnote {
-				lr.noteId = len(p.notes) + 1
+				lr.noteID = len(p.notes) + 1
+				lr.footnote = footnoteNode
 				p.notes = append(p.notes, lr)
 			}
 
@@ -488,27 +524,13 @@ func link(p *parser, out *bytes.Buffer, data []byte, offset int) int {
 			link = lr.link
 			// if inline footnote, title == footnote contents
 			title = lr.title
-			noteId = lr.noteId
+			noteID = lr.noteID
 		}
 
 		// rewind the whitespace
 		i = txtE + 1
 	}
 
-	// build content: img alt is escaped, link content is parsed
-	var content bytes.Buffer
-	if txtE > 1 {
-		if t == linkImg {
-			content.Write(data[1:txtE])
-		} else {
-			// links cannot contain other links, so turn off link parsing temporarily
-			insideLink := p.insideLink
-			p.insideLink = true
-			p.inline(&content, data[1:txtE])
-			p.insideLink = insideLink
-		}
-	}
-
 	var uLink []byte
 	if t == linkNormal || t == linkImg {
 		if len(link) > 0 {
@@ -518,49 +540,54 @@ func link(p *parser, out *bytes.Buffer, data []byte, offset int) int {
 		}
 
 		// links need something to click on and somewhere to go
-		if len(uLink) == 0 || (t == linkNormal && content.Len() == 0) {
-			return 0
+		if len(uLink) == 0 || (t == linkNormal && txtE <= 1) {
+			return 0, nil
 		}
 	}
 
 	// call the relevant rendering function
+	var linkNode *Node
 	switch t {
 	case linkNormal:
+		linkNode = NewNode(Link)
+		linkNode.Destination = normalizeURI(uLink)
+		linkNode.Title = title
 		if len(altContent) > 0 {
-			p.r.Link(out, uLink, title, altContent)
+			linkNode.AppendChild(text(altContent))
 		} else {
-			p.r.Link(out, uLink, title, content.Bytes())
+			// links cannot contain other links, so turn off link parsing
+			// temporarily and recurse
+			insideLink := p.insideLink
+			p.insideLink = true
+			p.inline(linkNode, data[1:txtE])
+			p.insideLink = insideLink
 		}
 
 	case linkImg:
-		outSize := out.Len()
-		outBytes := out.Bytes()
-		if outSize > 0 && outBytes[outSize-1] == '!' {
-			out.Truncate(outSize - 1)
+		linkNode = NewNode(Image)
+		linkNode.Destination = uLink
+		linkNode.Title = title
+		linkNode.AppendChild(text(data[1:txtE]))
+		i++
+
+	case linkInlineFootnote, linkDeferredFootnote:
+		linkNode = NewNode(Link)
+		linkNode.Destination = link
+		linkNode.Title = title
+		linkNode.NoteID = noteID
+		linkNode.Footnote = footnoteNode
+		if t == linkInlineFootnote {
+			i++
 		}
 
-		p.r.Image(out, uLink, title, content.Bytes())
-
-	case linkInlineFootnote:
-		outSize := out.Len()
-		outBytes := out.Bytes()
-		if outSize > 0 && outBytes[outSize-1] == '^' {
-			out.Truncate(outSize - 1)
-		}
-
-		p.r.FootnoteRef(out, link, noteId)
-
-	case linkDeferredFootnote:
-		p.r.FootnoteRef(out, link, noteId)
-
 	default:
-		return 0
+		return 0, nil
 	}
 
-	return i
+	return i, linkNode
 }
 
-func (p *parser) inlineHtmlComment(out *bytes.Buffer, data []byte) int {
+func (p *Markdown) inlineHTMLComment(data []byte) int {
 	if len(data) < 5 {
 		return 0
 	}
@@ -579,44 +606,75 @@ func (p *parser) inlineHtmlComment(out *bytes.Buffer, data []byte) int {
 	return i + 1
 }
 
+func stripMailto(link []byte) []byte {
+	if bytes.HasPrefix(link, []byte("mailto://")) {
+		return link[9:]
+	} else if bytes.HasPrefix(link, []byte("mailto:")) {
+		return link[7:]
+	} else {
+		return link
+	}
+}
+
+// autolinkType specifies a kind of autolink that gets detected.
+type autolinkType int
+
+// These are the possible flag values for the autolink renderer.
+const (
+	notAutolink autolinkType = iota
+	normalAutolink
+	emailAutolink
+)
+
 // '<' when tags or autolinks are allowed
-func leftAngle(p *parser, out *bytes.Buffer, data []byte, offset int) int {
+func leftAngle(p *Markdown, data []byte, offset int) (int, *Node) {
 	data = data[offset:]
-	altype := LINK_TYPE_NOT_AUTOLINK
-	end := tagLength(data, &altype)
-	if size := p.inlineHtmlComment(out, data); size > 0 {
+	altype, end := tagLength(data)
+	if size := p.inlineHTMLComment(data); size > 0 {
 		end = size
 	}
 	if end > 2 {
-		if altype != LINK_TYPE_NOT_AUTOLINK {
+		if altype != notAutolink {
 			var uLink bytes.Buffer
 			unescapeText(&uLink, data[1:end+1-2])
 			if uLink.Len() > 0 {
-				p.r.AutoLink(out, uLink.Bytes(), altype)
+				link := uLink.Bytes()
+				node := NewNode(Link)
+				node.Destination = link
+				if altype == emailAutolink {
+					node.Destination = append([]byte("mailto:"), link...)
+				}
+				node.AppendChild(text(stripMailto(link)))
+				return end, node
 			}
 		} else {
-			p.r.RawHtmlTag(out, data[:end])
+			htmlTag := NewNode(HTMLSpan)
+			htmlTag.Literal = data[:end]
+			return end, htmlTag
 		}
 	}
 
-	return end
+	return end, nil
 }
 
 // '\\' backslash escape
 var escapeChars = []byte("\\`*_{}[]()#+-.!:|&<>~")
 
-func escape(p *parser, out *bytes.Buffer, data []byte, offset int) int {
+func escape(p *Markdown, data []byte, offset int) (int, *Node) {
 	data = data[offset:]
 
 	if len(data) > 1 {
+		if p.extensions&BackslashLineBreak != 0 && data[1] == '\n' {
+			return 2, NewNode(Hardbreak)
+		}
 		if bytes.IndexByte(escapeChars, data[1]) < 0 {
-			return 0
+			return 0, nil
 		}
 
-		p.r.NormalText(out, data[1:2])
+		return 2, text(data[1:2])
 	}
 
-	return 2
+	return 2, nil
 }
 
 func unescapeText(ob *bytes.Buffer, src []byte) {
@@ -642,7 +700,7 @@ func unescapeText(ob *bytes.Buffer, src []byte) {
 
 // '&' escaped when it doesn't belong to an entity
 // valid entities are assumed to be anything matching &#?[A-Za-z0-9]+;
-func entity(p *parser, out *bytes.Buffer, data []byte, offset int) int {
+func entity(p *Markdown, data []byte, offset int) (int, *Node) {
 	data = data[offset:]
 
 	end := 1
@@ -658,25 +716,70 @@ func entity(p *parser, out *bytes.Buffer, data []byte, offset int) int {
 	if end < len(data) && data[end] == ';' {
 		end++ // real entity
 	} else {
-		return 0 // lone '&'
+		return 0, nil // lone '&'
 	}
 
-	p.r.Entity(out, data[:end])
+	ent := data[:end]
+	// undo &amp; escaping or it will be converted to &amp;amp; by another
+	// escaper in the renderer
+	if bytes.Equal(ent, []byte("&amp;")) {
+		ent = []byte{'&'}
+	}
 
-	return end
+	return end, text(ent)
 }
 
 func linkEndsWithEntity(data []byte, linkEnd int) bool {
-	entityRanges := htmlEntity.FindAllIndex(data[:linkEnd], -1)
+	entityRanges := htmlEntityRe.FindAllIndex(data[:linkEnd], -1)
 	return entityRanges != nil && entityRanges[len(entityRanges)-1][1] == linkEnd
 }
 
-func autoLink(p *parser, out *bytes.Buffer, data []byte, offset int) int {
-	// quick check to rule out most false hits on ':'
-	if p.insideLink || len(data) < offset+3 || data[offset+1] != '/' || data[offset+2] != '/' {
-		return 0
+// hasPrefixCaseInsensitive is a custom implementation of
+//     strings.HasPrefix(strings.ToLower(s), prefix)
+// we rolled our own because ToLower pulls in a huge machinery of lowercasing
+// anything from Unicode and that's very slow. Since this func will only be
+// used on ASCII protocol prefixes, we can take shortcuts.
+func hasPrefixCaseInsensitive(s, prefix []byte) bool {
+	if len(s) < len(prefix) {
+		return false
 	}
+	delta := byte('a' - 'A')
+	for i, b := range prefix {
+		if b != s[i] && b != s[i]+delta {
+			return false
+		}
+	}
+	return true
+}
 
+var protocolPrefixes = [][]byte{
+	[]byte("http://"),
+	[]byte("https://"),
+	[]byte("ftp://"),
+	[]byte("file://"),
+	[]byte("mailto:"),
+}
+
+const shortestPrefix = 6 // len("ftp://"), the shortest of the above
+
+func maybeAutoLink(p *Markdown, data []byte, offset int) (int, *Node) {
+	// quick check to rule out most false hits
+	if p.insideLink || len(data) < offset+shortestPrefix {
+		return 0, nil
+	}
+	for _, prefix := range protocolPrefixes {
+		endOfHead := offset + 8 // 8 is the len() of the longest prefix
+		if endOfHead > len(data) {
+			endOfHead = len(data)
+		}
+		if hasPrefixCaseInsensitive(data[offset:endOfHead], prefix) {
+			return autoLink(p, data, offset)
+		}
+	}
+	return 0, nil
+}
+
+func autoLink(p *Markdown, data []byte, offset int) (int, *Node) {
 	// Now a more expensive check to see if we're not inside an anchor element
 	anchorStart := offset
 	offsetFromAnchor := 0
@@ -687,8 +790,9 @@ func autoLink(p *parser, out *bytes.Buffer, data []byte, offset int) int {
 
 	anchorStr := anchorRe.Find(data[anchorStart:])
 	if anchorStr != nil {
-		out.Write(anchorStr[offsetFromAnchor:])
-		return len(anchorStr) - offsetFromAnchor
+		anchorClose := NewNode(HTMLSpan)
+		anchorClose.Literal = anchorStr[offsetFromAnchor:]
+		return len(anchorStr) - offsetFromAnchor, anchorClose
 	}
 
 	// scan backward for a word boundary
@@ -697,14 +801,14 @@ func autoLink(p *parser, out *bytes.Buffer, data []byte, offset int) int {
 		rewind++
 	}
 	if rewind > 6 { // longest supported protocol is "mailto" which has 6 letters
-		return 0
+		return 0, nil
 	}
 
 	origData := data
 	data = data[offset-rewind:]
 
 	if !isSafeLink(data) {
-		return 0
+		return 0, nil
 	}
 
 	linkEnd := 0
@@ -781,19 +885,17 @@ func autoLink(p *parser, out *bytes.Buffer, data []byte, offset int) int {
 		}
 	}
 
-	// we were triggered on the ':', so we need to rewind the output a bit
-	if out.Len() >= rewind {
-		out.Truncate(len(out.Bytes()) - rewind)
-	}
-
 	var uLink bytes.Buffer
 	unescapeText(&uLink, data[:linkEnd])
 
 	if uLink.Len() > 0 {
-		p.r.AutoLink(out, uLink.Bytes(), LINK_TYPE_NORMAL)
+		node := NewNode(Link)
+		node.Destination = uLink.Bytes()
+		node.AppendChild(text(uLink.Bytes()))
+		return linkEnd, node
 	}
 
-	return linkEnd - rewind
+	return linkEnd, nil
 }
 
 func isEndOfLink(char byte) bool {
@@ -826,17 +928,17 @@ func isSafeLink(link []byte) bool {
 }
 
 // return the length of the given tag, or 0 is it's not valid
-func tagLength(data []byte, autolink *int) int {
+func tagLength(data []byte) (autolink autolinkType, end int) {
 	var i, j int
 
 	// a valid tag can't be shorter than 3 chars
 	if len(data) < 3 {
-		return 0
+		return notAutolink, 0
 	}
 
 	// begins with a '<' optionally followed by '/', followed by letter or number
 	if data[0] != '<' {
-		return 0
+		return notAutolink, 0
 	}
 	if data[1] == '/' {
 		i = 2
@@ -845,11 +947,11 @@ func tagLength(data []byte, autolink *int) int {
 	}
 
 	if !isalnum(data[i]) {
-		return 0
+		return notAutolink, 0
 	}
 
 	// scheme test
-	*autolink = LINK_TYPE_NOT_AUTOLINK
+	autolink = notAutolink
 
 	// try to find the beginning of an URI
 	for i < len(data) && (isalnum(data[i]) || data[i] == '.' || data[i] == '+' || data[i] == '-') {
@@ -858,21 +960,20 @@ func tagLength(data []byte, autolink *int) int {
 
 	if i > 1 && i < len(data) && data[i] == '@' {
 		if j = isMailtoAutoLink(data[i:]); j != 0 {
-			*autolink = LINK_TYPE_EMAIL
-			return i + j
+			return emailAutolink, i + j
 		}
 	}
 
 	if i > 2 && i < len(data) && data[i] == ':' {
-		*autolink = LINK_TYPE_NORMAL
+		autolink = normalAutolink
 		i++
 	}
 
 	// complete autolink test: no whitespace or ' or "
 	switch {
 	case i >= len(data):
-		*autolink = LINK_TYPE_NOT_AUTOLINK
-	case *autolink != 0:
+		autolink = notAutolink
+	case autolink != notAutolink:
 		j = i
 
 		for i < len(data) {
@@ -887,24 +988,20 @@ func tagLength(data []byte, autolink *int) int {
 		}
 
 		if i >= len(data) {
-			return 0
+			return autolink, 0
 		}
 		if i > j && data[i] == '>' {
-			return i + 1
+			return autolink, i + 1
 		}
 
 		// one of the forbidden chars has been found
-		*autolink = LINK_TYPE_NOT_AUTOLINK
+		autolink = notAutolink
 	}
-
-	// look for something looking like a tag end
-	for i < len(data) && data[i] != '>' {
-		i++
+	i += bytes.IndexByte(data[i:], '>')
+	if i < 0 {
+		return autolink, 0
 	}
-	if i >= len(data) {
-		return 0
-	}
-	return i + 1
+	return autolink, i + 1
 }
 
 // look for the address part of a mail autolink and '>'
@@ -928,9 +1025,8 @@ func isMailtoAutoLink(data []byte) int {
 		case '>':
 			if nb == 1 {
 				return i + 1
-			} else {
-				return 0
 			}
+			return 0
 		default:
 			return 0
 		}
@@ -993,9 +1089,8 @@ func helperFindEmphChar(data []byte, c byte) int {
 			if data[i] != '[' && data[i] != '(' { // not a link
 				if tmpI > 0 {
 					return tmpI
-				} else {
-					continue
 				}
+				continue
 			}
 			cc := data[i]
 			i++
@@ -1014,7 +1109,7 @@ func helperFindEmphChar(data []byte, c byte) int {
 	return 0
 }
 
-func helperEmphasis(p *parser, out *bytes.Buffer, data []byte, c byte) int {
+func helperEmphasis(p *Markdown, data []byte, c byte) (int, *Node) {
 	i := 0
 
 	// skip one symbol if coming from emph3
@@ -1025,11 +1120,11 @@ func helperEmphasis(p *parser, out *bytes.Buffer, data []byte, c byte) int {
 	for i < len(data) {
 		length := helperFindEmphChar(data[i:], c)
 		if length == 0 {
-			return 0
+			return 0, nil
 		}
 		i += length
 		if i >= len(data) {
-			return 0
+			return 0, nil
 		}
 
 		if i+1 < len(data) && data[i+1] == c {
@@ -1039,52 +1134,46 @@ func helperEmphasis(p *parser, out *bytes.Buffer, data []byte, c byte) int {
 
 		if data[i] == c && !isspace(data[i-1]) {
 
-			if p.flags&EXTENSION_NO_INTRA_EMPHASIS != 0 {
+			if p.extensions&NoIntraEmphasis != 0 {
 				if !(i+1 == len(data) || isspace(data[i+1]) || ispunct(data[i+1])) {
 					continue
 				}
 			}
 
-			var work bytes.Buffer
-			p.inline(&work, data[:i])
-			p.r.Emphasis(out, work.Bytes())
-			return i + 1
+			emph := NewNode(Emph)
+			p.inline(emph, data[:i])
+			return i + 1, emph
 		}
 	}
 
-	return 0
+	return 0, nil
 }
 
-func helperDoubleEmphasis(p *parser, out *bytes.Buffer, data []byte, c byte) int {
+func helperDoubleEmphasis(p *Markdown, data []byte, c byte) (int, *Node) {
 	i := 0
 
 	for i < len(data) {
 		length := helperFindEmphChar(data[i:], c)
 		if length == 0 {
-			return 0
+			return 0, nil
 		}
 		i += length
 
 		if i+1 < len(data) && data[i] == c && data[i+1] == c && i > 0 && !isspace(data[i-1]) {
-			var work bytes.Buffer
-			p.inline(&work, data[:i])
-
-			if work.Len() > 0 {
-				// pick the right renderer
-				if c == '~' {
-					p.r.StrikeThrough(out, work.Bytes())
-				} else {
-					p.r.DoubleEmphasis(out, work.Bytes())
-				}
+			nodeType := Strong
+			if c == '~' {
+				nodeType = Del
 			}
-			return i + 2
+			node := NewNode(nodeType)
+			p.inline(node, data[:i])
+			return i + 2, node
 		}
 		i++
 	}
-	return 0
+	return 0, nil
 }
 
-func helperTripleEmphasis(p *parser, out *bytes.Buffer, data []byte, offset int, c byte) int {
+func helperTripleEmphasis(p *Markdown, data []byte, offset int, c byte) (int, *Node) {
 	i := 0
 	origData := data
 	data = data[offset:]
@@ -1092,7 +1181,7 @@ func helperTripleEmphasis(p *parser, out *bytes.Buffer, data []byte, offset int,
 	for i < len(data) {
 		length := helperFindEmphChar(data[i:], c)
 		if length == 0 {
-			return 0
+			return 0, nil
 		}
 		i += length
 
@@ -1104,30 +1193,36 @@ func helperTripleEmphasis(p *parser, out *bytes.Buffer, data []byte, offset int,
 		switch {
 		case i+2 < len(data) && data[i+1] == c && data[i+2] == c:
 			// triple symbol found
-			var work bytes.Buffer
-
-			p.inline(&work, data[:i])
-			if work.Len() > 0 {
-				p.r.TripleEmphasis(out, work.Bytes())
-			}
-			return i + 3
+			strong := NewNode(Strong)
+			em := NewNode(Emph)
+			strong.AppendChild(em)
+			p.inline(em, data[:i])
+			return i + 3, strong
 		case (i+1 < len(data) && data[i+1] == c):
 			// double symbol found, hand over to emph1
-			length = helperEmphasis(p, out, origData[offset-2:], c)
+			length, node := helperEmphasis(p, origData[offset-2:], c)
 			if length == 0 {
-				return 0
-			} else {
-				return length - 2
+				return 0, nil
 			}
+			return length - 2, node
 		default:
 			// single symbol found, hand over to emph2
-			length = helperDoubleEmphasis(p, out, origData[offset-1:], c)
+			length, node := helperDoubleEmphasis(p, origData[offset-1:], c)
 			if length == 0 {
-				return 0
-			} else {
-				return length - 1
+				return 0, nil
 			}
+			return length - 1, node
 		}
 	}
-	return 0
+	return 0, nil
+}
+
+func text(s []byte) *Node {
+	node := NewNode(Text)
+	node.Literal = s
+	return node
+}
+
+func normalizeURI(s []byte) []byte {
+	return s // TODO: implement
 }

inline_test.go

@@ -20,91 +20,8 @@ import (
 	"strings"
 )
 
-func runMarkdownInline(input string, opts Options, htmlFlags int, params HtmlRendererParameters) string {
-	opts.Extensions |= EXTENSION_AUTOLINK
-	opts.Extensions |= EXTENSION_STRIKETHROUGH
-
-	htmlFlags |= HTML_USE_XHTML
-
-	renderer := HtmlRendererWithParameters(htmlFlags, "", "", params)
-
-	return string(MarkdownOptions([]byte(input), renderer, opts))
-}
-
-func doTestsInline(t *testing.T, tests []string) {
-	doTestsInlineParam(t, tests, Options{}, 0, HtmlRendererParameters{})
-}
-
-func doLinkTestsInline(t *testing.T, tests []string) {
-	doTestsInline(t, tests)
-
-	prefix := "http://localhost"
-	params := HtmlRendererParameters{AbsolutePrefix: prefix}
-	transformTests := transformLinks(tests, prefix)
-	doTestsInlineParam(t, transformTests, Options{}, 0, params)
-	doTestsInlineParam(t, transformTests, Options{}, commonHtmlFlags, params)
-}
-
-func doSafeTestsInline(t *testing.T, tests []string) {
-	doTestsInlineParam(t, tests, Options{}, HTML_SAFELINK, HtmlRendererParameters{})
-
-	// All the links in this test should not have the prefix appended, so
-	// just rerun it with different parameters and the same expectations.
-	prefix := "http://localhost"
-	params := HtmlRendererParameters{AbsolutePrefix: prefix}
-	transformTests := transformLinks(tests, prefix)
-	doTestsInlineParam(t, transformTests, Options{}, HTML_SAFELINK, params)
-}
-
-func doTestsInlineParam(t *testing.T, tests []string, opts Options, htmlFlags int,
-	params HtmlRendererParameters) {
-	// catch and report panics
-	var candidate string
-	/*
-		defer func() {
-			if err := recover(); err != nil {
-				t.Errorf("\npanic while processing [%#v] (%v)\n", candidate, err)
-			}
-		}()
-	*/
-
-	for i := 0; i+1 < len(tests); i += 2 {
-		input := tests[i]
-		candidate = input
-		expected := tests[i+1]
-		actual := runMarkdownInline(candidate, opts, htmlFlags, params)
-		if actual != expected {
-			t.Errorf("\nInput   [%#v]\nExpected[%#v]\nActual  [%#v]",
-				candidate, expected, actual)
-		}
-
-		// now test every substring to stress test bounds checking
-		if !testing.Short() {
-			for start := 0; start < len(input); start++ {
-				for end := start + 1; end <= len(input); end++ {
-					candidate = input[start:end]
-					_ = runMarkdownInline(candidate, opts, htmlFlags, params)
-				}
-			}
-		}
-	}
-}
-
-func transformLinks(tests []string, prefix string) []string {
-	newTests := make([]string, len(tests))
-	anchorRe := regexp.MustCompile(`<a href="/(.*?)"`)
-	imgRe := regexp.MustCompile(`<img src="/(.*?)"`)
-	for i, test := range tests {
-		if i%2 == 1 {
-			test = anchorRe.ReplaceAllString(test, `<a href="`+prefix+`/$1"`)
-			test = imgRe.ReplaceAllString(test, `<img src="`+prefix+`/$1"`)
-		}
-		newTests[i] = test
-	}
-	return newTests
-}
-
 func TestEmphasis(t *testing.T) {
+	t.Parallel()
 	var tests = []string{
 		"nothing inline\n",
 		"<p>nothing inline</p>\n",
@@ -161,6 +78,7 @@ func TestEmphasis(t *testing.T) {
 }
 
 func TestReferenceOverride(t *testing.T) {
+	t.Parallel()
 	var tests = []string{
 		"test [ref1][]\n",
 		"<p>test <a href=\"http://www.ref1.com/\" title=\"Reference 1\">ref1</a></p>\n",
@@ -183,11 +101,11 @@ func TestReferenceOverride(t *testing.T) {
 		"test [ref5][]\n",
 		"<p>test <a href=\"http://www.ref5.com/\" title=\"Reference 5\">Moo</a></p>\n",
 	}
-	doTestsInlineParam(t, tests, Options{
-		ReferenceOverride: func(reference string) (rv *Reference, overridden bool) {
+	doTestsInlineParam(t, tests, TestParams{
+		referenceOverride: func(reference string) (rv *Reference, overridden bool) {
 			switch reference {
 			case "ref1":
-				// just an overriden reference exists without definition
+				// just an overridden reference exists without definition
 				return &Reference{
 					Link:  "http://www.ref1.com/",
 					Title: "Reference 1"}, true
@@ -214,10 +132,12 @@ func TestReferenceOverride(t *testing.T) {
 				}, true
 			}
 			return nil, false
-		}}, 0, HtmlRendererParameters{})
+		},
+	})
 }
 
 func TestStrong(t *testing.T) {
+	t.Parallel()
 	var tests = []string{
 		"nothing inline\n",
 		"<p>nothing inline</p>\n",
@@ -277,6 +197,7 @@ func TestStrong(t *testing.T) {
 }
 
 func TestEmphasisMix(t *testing.T) {
+	t.Parallel()
 	var tests = []string{
 		"***triple emphasis***\n",
 		"<p><strong><em>triple emphasis</em></strong></p>\n",
@@ -306,6 +227,7 @@ func TestEmphasisMix(t *testing.T) {
 }
 
 func TestEmphasisLink(t *testing.T) {
+	t.Parallel()
 	var tests = []string{
 		"[first](before) *text[second] (inside)text* [third](after)\n",
 		"<p><a href=\"before\">first</a> <em>text<a href=\"inside\">second</a>text</em> <a href=\"after\">third</a></p>\n",
@@ -323,6 +245,7 @@ func TestEmphasisLink(t *testing.T) {
 }
 
 func TestStrikeThrough(t *testing.T) {
+	t.Parallel()
 	var tests = []string{
 		"nothing inline\n",
 		"<p>nothing inline</p>\n",
@@ -352,6 +275,7 @@ func TestStrikeThrough(t *testing.T) {
 }
 
 func TestCodeSpan(t *testing.T) {
+	t.Parallel()
 	var tests = []string{
 		"`source code`\n",
 		"<p><code>source code</code></p>\n",
@@ -390,6 +314,7 @@ func TestCodeSpan(t *testing.T) {
 }
 
 func TestLineBreak(t *testing.T) {
+	t.Parallel()
 	var tests = []string{
 		"this line  \nhas a break\n",
 		"<p>this line<br />\nhas a break</p>\n",
@@ -424,12 +349,12 @@ func TestLineBreak(t *testing.T) {
 		"this has an   \nextra space\n",
 		"<p>this has an<br />\nextra space</p>\n",
 	}
-	doTestsInlineParam(t, tests, Options{
-		Extensions: EXTENSION_BACKSLASH_LINE_BREAK},
-		0, HtmlRendererParameters{})
+	doTestsInlineParam(t, tests, TestParams{
+		extensions: BackslashLineBreak})
 }
 
 func TestInlineLink(t *testing.T) {
+	t.Parallel()
 	var tests = []string{
 		"[foo](/bar/)\n",
 		"<p><a href=\"/bar/\">foo</a></p>\n",
@@ -547,6 +472,7 @@ func TestInlineLink(t *testing.T) {
 }
 
 func TestRelAttrLink(t *testing.T) {
+	t.Parallel()
 	var nofollowTests = []string{
 		"[foo](http://bar.com/foo/)\n",
 		"<p><a href=\"http://bar.com/foo/\" rel=\"nofollow\">foo</a></p>\n",
@@ -566,8 +492,9 @@ func TestRelAttrLink(t *testing.T) {
 		"[foo](../bar)\n",
 		"<p><a href=\"../bar\">foo</a></p>\n",
 	}
-	doTestsInlineParam(t, nofollowTests, Options{}, HTML_SAFELINK|HTML_NOFOLLOW_LINKS,
-		HtmlRendererParameters{})
+	doTestsInlineParam(t, nofollowTests, TestParams{
+		HTMLFlags: Safelink | NofollowLinks,
+	})
 
 	var noreferrerTests = []string{
 		"[foo](http://bar.com/foo/)\n",
@@ -576,21 +503,35 @@ func TestRelAttrLink(t *testing.T) {
 		"[foo](/bar/)\n",
 		"<p><a href=\"/bar/\">foo</a></p>\n",
 	}
-	doTestsInlineParam(t, noreferrerTests, Options{}, HTML_SAFELINK|HTML_NOREFERRER_LINKS,
-		HtmlRendererParameters{})
+	doTestsInlineParam(t, noreferrerTests, TestParams{
+		HTMLFlags: Safelink | NoreferrerLinks,
+	})
 
-	var nofollownoreferrerTests = []string{
+	var noopenerTests = []string{
 		"[foo](http://bar.com/foo/)\n",
-		"<p><a href=\"http://bar.com/foo/\" rel=\"nofollow noreferrer\">foo</a></p>\n",
+		"<p><a href=\"http://bar.com/foo/\" rel=\"noopener\">foo</a></p>\n",
 
 		"[foo](/bar/)\n",
 		"<p><a href=\"/bar/\">foo</a></p>\n",
 	}
-	doTestsInlineParam(t, nofollownoreferrerTests, Options{}, HTML_SAFELINK|HTML_NOFOLLOW_LINKS|HTML_NOREFERRER_LINKS,
-		HtmlRendererParameters{})
+	doTestsInlineParam(t, noopenerTests, TestParams{
+		HTMLFlags: Safelink | NoopenerLinks,
+	})
+
+	var nofollownoreferrernoopenerTests = []string{
+		"[foo](http://bar.com/foo/)\n",
+		"<p><a href=\"http://bar.com/foo/\" rel=\"nofollow noreferrer noopener\">foo</a></p>\n",
+
+		"[foo](/bar/)\n",
+		"<p><a href=\"/bar/\">foo</a></p>\n",
+	}
+	doTestsInlineParam(t, nofollownoreferrernoopenerTests, TestParams{
+		HTMLFlags: Safelink | NofollowLinks | NoreferrerLinks | NoopenerLinks,
+	})
 }
 
 func TestHrefTargetBlank(t *testing.T) {
+	t.Parallel()
 	var tests = []string{
 		// internal link
 		"[foo](/bar/)\n",
@@ -614,10 +555,13 @@ func TestHrefTargetBlank(t *testing.T) {
 		"[foo](http://example.com)\n",
 		"<p><a href=\"http://example.com\" target=\"_blank\">foo</a></p>\n",
 	}
-	doTestsInlineParam(t, tests, Options{}, HTML_SAFELINK|HTML_HREF_TARGET_BLANK, HtmlRendererParameters{})
+	doTestsInlineParam(t, tests, TestParams{
+		HTMLFlags: Safelink | HrefTargetBlank,
+	})
 }
 
 func TestSafeInlineLink(t *testing.T) {
+	t.Parallel()
 	var tests = []string{
 		"[foo](/bar/)\n",
 		"<p><a href=\"/bar/\">foo</a></p>\n",
@@ -651,6 +595,7 @@ func TestSafeInlineLink(t *testing.T) {
 }
 
 func TestReferenceLink(t *testing.T) {
+	t.Parallel()
 	var tests = []string{
 		"[link][ref]\n",
 		"<p>[link][ref]</p>\n",
@@ -681,11 +626,15 @@ func TestReferenceLink(t *testing.T) {
 
 		"[ref]\n   [ref]: ../url/ \"title\"\n",
 		"<p><a href=\"../url/\" title=\"title\">ref</a></p>\n",
+
+		"[link][ref]\n   [ref]: /url/",
+		"<p><a href=\"/url/\">link</a></p>\n",
 	}
 	doLinkTestsInline(t, tests)
 }
 
 func TestTags(t *testing.T) {
+	t.Parallel()
 	var tests = []string{
 		"a <span>tag</span>\n",
 		"<p>a <span>tag</span></p>\n",
@@ -703,6 +652,7 @@ func TestTags(t *testing.T) {
 }
 
 func TestAutoLink(t *testing.T) {
+	t.Parallel()
 	var tests = []string{
 		"http://foo.com/\n",
 		"<p><a href=\"http://foo.com/\">http://foo.com/</a></p>\n",
@@ -808,21 +758,25 @@ func TestAutoLink(t *testing.T) {
 
 		"http://foo.com/viewtopic.php?param=&quot;18&quot;",
 		"<p><a href=\"http://foo.com/viewtopic.php?param=&quot;18&quot;\">http://foo.com/viewtopic.php?param=&quot;18&quot;</a></p>\n",
+
+		"<a href=\"https://fancy.com\">https://fancy.com</a>\n",
+		"<p><a href=\"https://fancy.com\">https://fancy.com</a></p>\n",
 	}
 	doLinkTestsInline(t, tests)
 }
 
 var footnoteTests = []string{
 	"testing footnotes.[^a]\n\n[^a]: This is the note\n",
-	`<p>testing footnotes.<sup class="footnote-ref" id="fnref:a"><a rel="footnote" href="#fn:a">1</a></sup></p>
+	`<p>testing footnotes.<sup class="footnote-ref" id="fnref:a"><a href="#fn:a">1</a></sup></p>
+
 <div class="footnotes">
 
 <hr />
 
 <ol>
-<li id="fn:a">This is the note
-</li>
+<li id="fn:a">This is the note</li>
 </ol>
+
 </div>
 `,
 
@@ -838,9 +792,10 @@ var footnoteTests = []string{
 
 No longer in the footnote
 `,
-	`<p>testing long<sup class="footnote-ref" id="fnref:b"><a rel="footnote" href="#fn:b">1</a></sup> notes.</p>
+	`<p>testing long<sup class="footnote-ref" id="fnref:b"><a href="#fn:b">1</a></sup> notes.</p>
 
 <p>No longer in the footnote</p>
+
 <div class="footnotes">
 
 <hr />
@@ -854,9 +809,9 @@ No longer in the footnote
 some code
 </code></p>
 
-<p>Paragraph 3</p>
-</li>
+<p>Paragraph 3</p></li>
 </ol>
+
 </div>
 `,
 
@@ -874,26 +829,28 @@ what happens here
 [note]: /link/c
 
 `,
-	`<p>testing<sup class="footnote-ref" id="fnref:c"><a rel="footnote" href="#fn:c">1</a></sup> multiple<sup class="footnote-ref" id="fnref:d"><a rel="footnote" href="#fn:d">2</a></sup> notes.</p>
+	`<p>testing<sup class="footnote-ref" id="fnref:c"><a href="#fn:c">1</a></sup> multiple<sup class="footnote-ref" id="fnref:d"><a href="#fn:d">2</a></sup> notes.</p>
 
 <p>omg</p>
 
 <p>what happens here</p>
+
 <div class="footnotes">
 
 <hr />
 
 <ol>
-<li id="fn:c">this is <a href="/link/c">note</a> c
-</li>
-<li id="fn:d">this is note d
-</li>
+<li id="fn:c">this is <a href="/link/c">note</a> c</li>
+
+<li id="fn:d">this is note d</li>
 </ol>
+
 </div>
 `,
 
 	"testing inline^[this is the note] notes.\n",
-	`<p>testing inline<sup class="footnote-ref" id="fnref:this-is-the-note"><a rel="footnote" href="#fn:this-is-the-note">1</a></sup> notes.</p>
+	`<p>testing inline<sup class="footnote-ref" id="fnref:this-is-the-note"><a href="#fn:this-is-the-note">1</a></sup> notes.</p>
+
 <div class="footnotes">
 
 <hr />
@@ -901,11 +858,13 @@ what happens here
 <ol>
 <li id="fn:this-is-the-note">this is the note</li>
 </ol>
+
 </div>
 `,
 
 	"testing multiple[^1] types^[inline note] of notes[^2]\n\n[^2]: the second deferred note\n[^1]: the first deferred note\n\n\twhich happens to be a block\n",
-	`<p>testing multiple<sup class="footnote-ref" id="fnref:1"><a rel="footnote" href="#fn:1">1</a></sup> types<sup class="footnote-ref" id="fnref:inline-note"><a rel="footnote" href="#fn:inline-note">2</a></sup> of notes<sup class="footnote-ref" id="fnref:2"><a rel="footnote" href="#fn:2">3</a></sup></p>
+	`<p>testing multiple<sup class="footnote-ref" id="fnref:1"><a href="#fn:1">1</a></sup> types<sup class="footnote-ref" id="fnref:inline-note"><a href="#fn:inline-note">2</a></sup> of notes<sup class="footnote-ref" id="fnref:2"><a href="#fn:2">3</a></sup></p>
+
 <div class="footnotes">
 
 <hr />
@@ -913,12 +872,13 @@ what happens here
 <ol>
 <li id="fn:1"><p>the first deferred note</p>
 
-<p>which happens to be a block</p>
-</li>
+<p>which happens to be a block</p></li>
+
 <li id="fn:inline-note">inline note</li>
-<li id="fn:2">the second deferred note
-</li>
+
+<li id="fn:2">the second deferred note</li>
 </ol>
+
 </div>
 `,
 
@@ -928,7 +888,8 @@ what happens here
 
     may be multiple paragraphs.
 `,
-	`<p>This is a footnote<sup class="footnote-ref" id="fnref:1"><a rel="footnote" href="#fn:1">1</a></sup><sup class="footnote-ref" id="fnref:and-this-is-an-i"><a rel="footnote" href="#fn:and-this-is-an-i">2</a></sup></p>
+	`<p>This is a footnote<sup class="footnote-ref" id="fnref:1"><a href="#fn:1">1</a></sup><sup class="footnote-ref" id="fnref:and-this-is-an-i"><a href="#fn:and-this-is-an-i">2</a></sup></p>
+
 <div class="footnotes">
 
 <hr />
@@ -936,21 +897,22 @@ what happens here
 <ol>
 <li id="fn:1"><p>the footnote text.</p>
 
-<p>may be multiple paragraphs.</p>
-</li>
+<p>may be multiple paragraphs.</p></li>
+
 <li id="fn:and-this-is-an-i">and this is an inline footnote</li>
 </ol>
+
 </div>
 `,
 
 	"empty footnote[^]\n\n[^]: fn text",
-	"<p>empty footnote<sup class=\"footnote-ref\" id=\"fnref:\"><a rel=\"footnote\" href=\"#fn:\">1</a></sup></p>\n<div class=\"footnotes\">\n\n<hr />\n\n<ol>\n<li id=\"fn:\">fn text\n</li>\n</ol>\n</div>\n",
+	"<p>empty footnote<sup class=\"footnote-ref\" id=\"fnref:\"><a href=\"#fn:\">1</a></sup></p>\n\n<div class=\"footnotes\">\n\n<hr />\n\n<ol>\n<li id=\"fn:\">fn text</li>\n</ol>\n\n</div>\n",
 
 	"Some text.[^note1]\n\n[^note1]: fn1",
-	"<p>Some text.<sup class=\"footnote-ref\" id=\"fnref:note1\"><a rel=\"footnote\" href=\"#fn:note1\">1</a></sup></p>\n<div class=\"footnotes\">\n\n<hr />\n\n<ol>\n<li id=\"fn:note1\">fn1\n</li>\n</ol>\n</div>\n",
+	"<p>Some text.<sup class=\"footnote-ref\" id=\"fnref:note1\"><a href=\"#fn:note1\">1</a></sup></p>\n\n<div class=\"footnotes\">\n\n<hr />\n\n<ol>\n<li id=\"fn:note1\">fn1</li>\n</ol>\n\n</div>\n",
 
 	"Some text.[^note1][^note2]\n\n[^note1]: fn1\n[^note2]: fn2\n",
-	"<p>Some text.<sup class=\"footnote-ref\" id=\"fnref:note1\"><a rel=\"footnote\" href=\"#fn:note1\">1</a></sup><sup class=\"footnote-ref\" id=\"fnref:note2\"><a rel=\"footnote\" href=\"#fn:note2\">2</a></sup></p>\n<div class=\"footnotes\">\n\n<hr />\n\n<ol>\n<li id=\"fn:note1\">fn1\n</li>\n<li id=\"fn:note2\">fn2\n</li>\n</ol>\n</div>\n",
+	"<p>Some text.<sup class=\"footnote-ref\" id=\"fnref:note1\"><a href=\"#fn:note1\">1</a></sup><sup class=\"footnote-ref\" id=\"fnref:note2\"><a href=\"#fn:note2\">2</a></sup></p>\n\n<div class=\"footnotes\">\n\n<hr />\n\n<ol>\n<li id=\"fn:note1\">fn1</li>\n\n<li id=\"fn:note2\">fn2</li>\n</ol>\n\n</div>\n",
 
 	`Bla bla [^1] [WWW][w3]
 
@@ -958,15 +920,16 @@ what happens here
 
 [w3]: http://www.w3.org/
 `,
-	`<p>Bla bla <sup class="footnote-ref" id="fnref:1"><a rel="footnote" href="#fn:1">1</a></sup> <a href="http://www.w3.org/">WWW</a></p>
+	`<p>Bla bla <sup class="footnote-ref" id="fnref:1"><a href="#fn:1">1</a></sup> <a href="http://www.w3.org/">WWW</a></p>
+
 <div class="footnotes">
 
 <hr />
 
 <ol>
-<li id="fn:1">This is a footnote
-</li>
+<li id="fn:1">This is a footnote</li>
 </ol>
+
 </div>
 `,
 
@@ -974,24 +937,35 @@ what happens here
 
 [^fn1]: Fine print
 `,
-	`<p>This is exciting!<sup class="footnote-ref" id="fnref:fn1"><a rel="footnote" href="#fn:fn1">1</a></sup></p>
+	`<p>This is exciting!<sup class="footnote-ref" id="fnref:fn1"><a href="#fn:fn1">1</a></sup></p>
+
 <div class="footnotes">
 
 <hr />
 
 <ol>
-<li id="fn:fn1">Fine print
-</li>
+<li id="fn:fn1">Fine print</li>
 </ol>
+
 </div>
 `,
+
+	`This text does not reference a footnote.
+
+[^footnote]: But it has a footnote! And it gets omitted.
+`,
+	"<p>This text does not reference a footnote.</p>\n",
 }
 
 func TestFootnotes(t *testing.T) {
-	doTestsInlineParam(t, footnoteTests, Options{Extensions: EXTENSION_FOOTNOTES}, 0, HtmlRendererParameters{})
+	t.Parallel()
+	doTestsInlineParam(t, footnoteTests, TestParams{
+		extensions: Footnotes,
+	})
 }
 
 func TestFootnotesWithParameters(t *testing.T) {
+	t.Parallel()
 	tests := make([]string, len(footnoteTests))
 
 	prefix := "testPrefix"
@@ -1008,15 +982,20 @@ func TestFootnotesWithParameters(t *testing.T) {
 		tests[i] = test
 	}
 
-	params := HtmlRendererParameters{
+	params := HTMLRendererParameters{
 		FootnoteAnchorPrefix:       prefix,
 		FootnoteReturnLinkContents: returnText,
 	}
 
-	doTestsInlineParam(t, tests, Options{Extensions: EXTENSION_FOOTNOTES}, HTML_FOOTNOTE_RETURN_LINKS, params)
+	doTestsInlineParam(t, tests, TestParams{
+		extensions:             Footnotes,
+		HTMLFlags:              FootnoteReturnLinks,
+		HTMLRendererParameters: params,
+	})
 }
 
 func TestNestedFootnotes(t *testing.T) {
+	t.Parallel()
 	var tests = []string{
 		`Paragraph.[^fn1]
 
@@ -1025,25 +1004,26 @@ func TestNestedFootnotes(t *testing.T) {
 
 [^fn2]:
   Obelisk`,
-		`<p>Paragraph.<sup class="footnote-ref" id="fnref:fn1"><a rel="footnote" href="#fn:fn1">1</a></sup></p>
+		`<p>Paragraph.<sup class="footnote-ref" id="fnref:fn1"><a href="#fn:fn1">1</a></sup></p>
+
 <div class="footnotes">
 
 <hr />
 
 <ol>
-<li id="fn:fn1">Asterisk<sup class="footnote-ref" id="fnref:fn2"><a rel="footnote" href="#fn:fn2">2</a></sup>
-</li>
-<li id="fn:fn2">Obelisk
-</li>
+<li id="fn:fn1">Asterisk<sup class="footnote-ref" id="fnref:fn2"><a href="#fn:fn2">2</a></sup></li>
+
+<li id="fn:fn2">Obelisk</li>
 </ol>
+
 </div>
 `,
 	}
-	doTestsInlineParam(t, tests, Options{Extensions: EXTENSION_FOOTNOTES}, 0,
-		HtmlRendererParameters{})
+	doTestsInlineParam(t, tests, TestParams{extensions: Footnotes})
 }
 
 func TestInlineComments(t *testing.T) {
+	t.Parallel()
 	var tests = []string{
 		"Hello <!-- there ->\n",
 		"<p>Hello &lt;!&mdash; there &ndash;&gt;</p>\n",
@@ -1069,10 +1049,11 @@ func TestInlineComments(t *testing.T) {
 		"blahblah\n<!--- foo -->\nrhubarb\n",
 		"<p>blahblah\n<!--- foo -->\nrhubarb</p>\n",
 	}
-	doTestsInlineParam(t, tests, Options{}, HTML_USE_SMARTYPANTS|HTML_SMARTYPANTS_DASHES, HtmlRendererParameters{})
+	doTestsInlineParam(t, tests, TestParams{HTMLFlags: Smartypants | SmartypantsDashes})
 }
 
 func TestSmartDoubleQuotes(t *testing.T) {
+	t.Parallel()
 	var tests = []string{
 		"this should be normal \"quoted\" text.\n",
 		"<p>this should be normal &ldquo;quoted&rdquo; text.</p>\n",
@@ -1081,10 +1062,24 @@ func TestSmartDoubleQuotes(t *testing.T) {
 		"two pair of \"some\" quoted \"text\".\n",
 		"<p>two pair of &ldquo;some&rdquo; quoted &ldquo;text&rdquo;.</p>\n"}
 
-	doTestsInlineParam(t, tests, Options{}, HTML_USE_SMARTYPANTS, HtmlRendererParameters{})
+	doTestsInlineParam(t, tests, TestParams{HTMLFlags: Smartypants})
+}
+
+func TestSmartDoubleQuotesNBSP(t *testing.T) {
+	t.Parallel()
+	var tests = []string{
+		"this should be normal \"quoted\" text.\n",
+		"<p>this should be normal &ldquo;&nbsp;quoted&nbsp;&rdquo; text.</p>\n",
+		"this \" single double\n",
+		"<p>this &ldquo;&nbsp; single double</p>\n",
+		"two pair of \"some\" quoted \"text\".\n",
+		"<p>two pair of &ldquo;&nbsp;some&nbsp;&rdquo; quoted &ldquo;&nbsp;text&nbsp;&rdquo;.</p>\n"}
+
+	doTestsInlineParam(t, tests, TestParams{HTMLFlags: Smartypants | SmartypantsQuotesNBSP})
 }
 
 func TestSmartAngledDoubleQuotes(t *testing.T) {
+	t.Parallel()
 	var tests = []string{
 		"this should be angled \"quoted\" text.\n",
 		"<p>this should be angled &laquo;quoted&raquo; text.</p>\n",
@@ -1093,17 +1088,31 @@ func TestSmartAngledDoubleQuotes(t *testing.T) {
 		"two pair of \"some\" quoted \"text\".\n",
 		"<p>two pair of &laquo;some&raquo; quoted &laquo;text&raquo;.</p>\n"}
 
-	doTestsInlineParam(t, tests, Options{}, HTML_USE_SMARTYPANTS|HTML_SMARTYPANTS_ANGLED_QUOTES, HtmlRendererParameters{})
+	doTestsInlineParam(t, tests, TestParams{HTMLFlags: Smartypants | SmartypantsAngledQuotes})
+}
+
+func TestSmartAngledDoubleQuotesNBSP(t *testing.T) {
+	t.Parallel()
+	var tests = []string{
+		"this should be angled \"quoted\" text.\n",
+		"<p>this should be angled &laquo;&nbsp;quoted&nbsp;&raquo; text.</p>\n",
+		"this \" single double\n",
+		"<p>this &laquo;&nbsp; single double</p>\n",
+		"two pair of \"some\" quoted \"text\".\n",
+		"<p>two pair of &laquo;&nbsp;some&nbsp;&raquo; quoted &laquo;&nbsp;text&nbsp;&raquo;.</p>\n"}
+
+	doTestsInlineParam(t, tests, TestParams{HTMLFlags: Smartypants | SmartypantsAngledQuotes | SmartypantsQuotesNBSP})
 }
 
 func TestSmartFractions(t *testing.T) {
+	t.Parallel()
 	var tests = []string{
 		"1/2, 1/4 and 3/4; 1/4th and 3/4ths\n",
 		"<p>&frac12;, &frac14; and &frac34;; &frac14;th and &frac34;ths</p>\n",
 		"1/2/2015, 1/4/2015, 3/4/2015; 2015/1/2, 2015/1/4, 2015/3/4.\n",
 		"<p>1/2/2015, 1/4/2015, 3/4/2015; 2015/1/2, 2015/1/4, 2015/3/4.</p>\n"}
 
-	doTestsInlineParam(t, tests, Options{}, HTML_USE_SMARTYPANTS, HtmlRendererParameters{})
+	doTestsInlineParam(t, tests, TestParams{HTMLFlags: Smartypants})
 
 	tests = []string{
 		"1/2, 2/3, 81/100 and 1000000/1048576.\n",
@@ -1111,10 +1120,11 @@ func TestSmartFractions(t *testing.T) {
 		"1/2/2015, 1/4/2015, 3/4/2015; 2015/1/2, 2015/1/4, 2015/3/4.\n",
 		"<p>1/2/2015, 1/4/2015, 3/4/2015; 2015/1/2, 2015/1/4, 2015/3/4.</p>\n"}
 
-	doTestsInlineParam(t, tests, Options{}, HTML_USE_SMARTYPANTS|HTML_SMARTYPANTS_FRACTIONS, HtmlRendererParameters{})
+	doTestsInlineParam(t, tests, TestParams{HTMLFlags: Smartypants | SmartypantsFractions})
 }
 
 func TestDisableSmartDashes(t *testing.T) {
+	t.Parallel()
 	doTestsInlineParam(t, []string{
 		"foo - bar\n",
 		"<p>foo - bar</p>\n",
@@ -1122,7 +1132,7 @@ func TestDisableSmartDashes(t *testing.T) {
 		"<p>foo -- bar</p>\n",
 		"foo --- bar\n",
 		"<p>foo --- bar</p>\n",
-	}, Options{}, 0, HtmlRendererParameters{})
+	}, TestParams{})
 	doTestsInlineParam(t, []string{
 		"foo - bar\n",
 		"<p>foo &ndash; bar</p>\n",
@@ -1130,7 +1140,7 @@ func TestDisableSmartDashes(t *testing.T) {
 		"<p>foo &mdash; bar</p>\n",
 		"foo --- bar\n",
 		"<p>foo &mdash;&ndash; bar</p>\n",
-	}, Options{}, HTML_USE_SMARTYPANTS|HTML_SMARTYPANTS_DASHES, HtmlRendererParameters{})
+	}, TestParams{HTMLFlags: Smartypants | SmartypantsDashes})
 	doTestsInlineParam(t, []string{
 		"foo - bar\n",
 		"<p>foo - bar</p>\n",
@@ -1138,8 +1148,7 @@ func TestDisableSmartDashes(t *testing.T) {
 		"<p>foo &ndash; bar</p>\n",
 		"foo --- bar\n",
 		"<p>foo &mdash; bar</p>\n",
-	}, Options{}, HTML_USE_SMARTYPANTS|HTML_SMARTYPANTS_LATEX_DASHES|HTML_SMARTYPANTS_DASHES,
-		HtmlRendererParameters{})
+	}, TestParams{HTMLFlags: Smartypants | SmartypantsLatexDashes | SmartypantsDashes})
 	doTestsInlineParam(t, []string{
 		"foo - bar\n",
 		"<p>foo - bar</p>\n",
@@ -1147,7 +1156,61 @@ func TestDisableSmartDashes(t *testing.T) {
 		"<p>foo -- bar</p>\n",
 		"foo --- bar\n",
 		"<p>foo --- bar</p>\n",
-	}, Options{},
-		HTML_USE_SMARTYPANTS|HTML_SMARTYPANTS_LATEX_DASHES,
-		HtmlRendererParameters{})
+	}, TestParams{HTMLFlags: Smartypants | SmartypantsLatexDashes})
+}
+
+func TestSkipLinks(t *testing.T) {
+	t.Parallel()
+	doTestsInlineParam(t, []string{
+		"[foo](gopher://foo.bar)",
+		"<p><tt>foo</tt></p>\n",
+
+		"[foo](mailto://bar/)\n",
+		"<p><tt>foo</tt></p>\n",
+	}, TestParams{
+		HTMLFlags: SkipLinks,
+	})
+}
+
+func TestSkipImages(t *testing.T) {
+	t.Parallel()
+	doTestsInlineParam(t, []string{
+		"![foo](/bar/)\n",
+		"<p></p>\n",
+	}, TestParams{
+		HTMLFlags: SkipImages,
+	})
+}
+
+func TestUseXHTML(t *testing.T) {
+	t.Parallel()
+	doTestsParam(t, []string{
+		"---",
+		"<hr>\n",
+	}, TestParams{})
+	doTestsParam(t, []string{
+		"---",
+		"<hr />\n",
+	}, TestParams{HTMLFlags: UseXHTML})
+}
+
+func TestSkipHTML(t *testing.T) {
+	t.Parallel()
+	doTestsParam(t, []string{
+		"<div class=\"foo\"></div>\n\ntext\n\n<form>the form</form>",
+		"<p>text</p>\n\n<p>the form</p>\n",
+
+		"text <em>inline html</em> more text",
+		"<p>text inline html more text</p>\n",
+	}, TestParams{HTMLFlags: SkipHTML})
+}
+
+func BenchmarkSmartDoubleQuotes(b *testing.B) {
+	params := TestParams{HTMLFlags: Smartypants}
+	params.extensions |= Autolink | Strikethrough
+	params.HTMLFlags |= UseXHTML
+
+	for i := 0; i < b.N; i++ {
+		runMarkdown("this should be normal \"quoted\" text.\n", params)
+	}
 }

latex.go

@@ -1,332 +0,0 @@
-//
-// Blackfriday Markdown Processor
-// Available at http://github.com/russross/blackfriday
-//
-// Copyright © 2011 Russ Ross <russ@russross.com>.
-// Distributed under the Simplified BSD License.
-// See README.md for details.
-//
-
-//
-//
-// LaTeX rendering backend
-//
-//
-
-package blackfriday
-
-import (
-	"bytes"
-)
-
-// Latex is a type that implements the Renderer interface for LaTeX output.
-//
-// Do not create this directly, instead use the LatexRenderer function.
-type Latex struct {
-}
-
-// LatexRenderer creates and configures a Latex object, which
-// satisfies the Renderer interface.
-//
-// flags is a set of LATEX_* options ORed together (currently no such options
-// are defined).
-func LatexRenderer(flags int) Renderer {
-	return &Latex{}
-}
-
-func (options *Latex) GetFlags() int {
-	return 0
-}
-
-// render code chunks using verbatim, or listings if we have a language
-func (options *Latex) BlockCode(out *bytes.Buffer, text []byte, lang string) {
-	if lang == "" {
-		out.WriteString("\n\\begin{verbatim}\n")
-	} else {
-		out.WriteString("\n\\begin{lstlisting}[language=")
-		out.WriteString(lang)
-		out.WriteString("]\n")
-	}
-	out.Write(text)
-	if lang == "" {
-		out.WriteString("\n\\end{verbatim}\n")
-	} else {
-		out.WriteString("\n\\end{lstlisting}\n")
-	}
-}
-
-func (options *Latex) TitleBlock(out *bytes.Buffer, text []byte) {
-
-}
-
-func (options *Latex) BlockQuote(out *bytes.Buffer, text []byte) {
-	out.WriteString("\n\\begin{quotation}\n")
-	out.Write(text)
-	out.WriteString("\n\\end{quotation}\n")
-}
-
-func (options *Latex) BlockHtml(out *bytes.Buffer, text []byte) {
-	// a pretty lame thing to do...
-	out.WriteString("\n\\begin{verbatim}\n")
-	out.Write(text)
-	out.WriteString("\n\\end{verbatim}\n")
-}
-
-func (options *Latex) Header(out *bytes.Buffer, text func() bool, level int, id string) {
-	marker := out.Len()
-
-	switch level {
-	case 1:
-		out.WriteString("\n\\section{")
-	case 2:
-		out.WriteString("\n\\subsection{")
-	case 3:
-		out.WriteString("\n\\subsubsection{")
-	case 4:
-		out.WriteString("\n\\paragraph{")
-	case 5:
-		out.WriteString("\n\\subparagraph{")
-	case 6:
-		out.WriteString("\n\\textbf{")
-	}
-	if !text() {
-		out.Truncate(marker)
-		return
-	}
-	out.WriteString("}\n")
-}
-
-func (options *Latex) HRule(out *bytes.Buffer) {
-	out.WriteString("\n\\HRule\n")
-}
-
-func (options *Latex) List(out *bytes.Buffer, text func() bool, flags int) {
-	marker := out.Len()
-	if flags&LIST_TYPE_ORDERED != 0 {
-		out.WriteString("\n\\begin{enumerate}\n")
-	} else {
-		out.WriteString("\n\\begin{itemize}\n")
-	}
-	if !text() {
-		out.Truncate(marker)
-		return
-	}
-	if flags&LIST_TYPE_ORDERED != 0 {
-		out.WriteString("\n\\end{enumerate}\n")
-	} else {
-		out.WriteString("\n\\end{itemize}\n")
-	}
-}
-
-func (options *Latex) ListItem(out *bytes.Buffer, text []byte, flags int) {
-	out.WriteString("\n\\item ")
-	out.Write(text)
-}
-
-func (options *Latex) Paragraph(out *bytes.Buffer, text func() bool) {
-	marker := out.Len()
-	out.WriteString("\n")
-	if !text() {
-		out.Truncate(marker)
-		return
-	}
-	out.WriteString("\n")
-}
-
-func (options *Latex) Table(out *bytes.Buffer, header []byte, body []byte, columnData []int) {
-	out.WriteString("\n\\begin{tabular}{")
-	for _, elt := range columnData {
-		switch elt {
-		case TABLE_ALIGNMENT_LEFT:
-			out.WriteByte('l')
-		case TABLE_ALIGNMENT_RIGHT:
-			out.WriteByte('r')
-		default:
-			out.WriteByte('c')
-		}
-	}
-	out.WriteString("}\n")
-	out.Write(header)
-	out.WriteString(" \\\\\n\\hline\n")
-	out.Write(body)
-	out.WriteString("\n\\end{tabular}\n")
-}
-
-func (options *Latex) TableRow(out *bytes.Buffer, text []byte) {
-	if out.Len() > 0 {
-		out.WriteString(" \\\\\n")
-	}
-	out.Write(text)
-}
-
-func (options *Latex) TableHeaderCell(out *bytes.Buffer, text []byte, align int) {
-	if out.Len() > 0 {
-		out.WriteString(" & ")
-	}
-	out.Write(text)
-}
-
-func (options *Latex) TableCell(out *bytes.Buffer, text []byte, align int) {
-	if out.Len() > 0 {
-		out.WriteString(" & ")
-	}
-	out.Write(text)
-}
-
-// TODO: this
-func (options *Latex) Footnotes(out *bytes.Buffer, text func() bool) {
-
-}
-
-func (options *Latex) FootnoteItem(out *bytes.Buffer, name, text []byte, flags int) {
-
-}
-
-func (options *Latex) AutoLink(out *bytes.Buffer, link []byte, kind int) {
-	out.WriteString("\\href{")
-	if kind == LINK_TYPE_EMAIL {
-		out.WriteString("mailto:")
-	}
-	out.Write(link)
-	out.WriteString("}{")
-	out.Write(link)
-	out.WriteString("}")
-}
-
-func (options *Latex) CodeSpan(out *bytes.Buffer, text []byte) {
-	out.WriteString("\\texttt{")
-	escapeSpecialChars(out, text)
-	out.WriteString("}")
-}
-
-func (options *Latex) DoubleEmphasis(out *bytes.Buffer, text []byte) {
-	out.WriteString("\\textbf{")
-	out.Write(text)
-	out.WriteString("}")
-}
-
-func (options *Latex) Emphasis(out *bytes.Buffer, text []byte) {
-	out.WriteString("\\textit{")
-	out.Write(text)
-	out.WriteString("}")
-}
-
-func (options *Latex) Image(out *bytes.Buffer, link []byte, title []byte, alt []byte) {
-	if bytes.HasPrefix(link, []byte("http://")) || bytes.HasPrefix(link, []byte("https://")) {
-		// treat it like a link
-		out.WriteString("\\href{")
-		out.Write(link)
-		out.WriteString("}{")
-		out.Write(alt)
-		out.WriteString("}")
-	} else {
-		out.WriteString("\\includegraphics{")
-		out.Write(link)
-		out.WriteString("}")
-	}
-}
-
-func (options *Latex) LineBreak(out *bytes.Buffer) {
-	out.WriteString(" \\\\\n")
-}
-
-func (options *Latex) Link(out *bytes.Buffer, link []byte, title []byte, content []byte) {
-	out.WriteString("\\href{")
-	out.Write(link)
-	out.WriteString("}{")
-	out.Write(content)
-	out.WriteString("}")
-}
-
-func (options *Latex) RawHtmlTag(out *bytes.Buffer, tag []byte) {
-}
-
-func (options *Latex) TripleEmphasis(out *bytes.Buffer, text []byte) {
-	out.WriteString("\\textbf{\\textit{")
-	out.Write(text)
-	out.WriteString("}}")
-}
-
-func (options *Latex) StrikeThrough(out *bytes.Buffer, text []byte) {
-	out.WriteString("\\sout{")
-	out.Write(text)
-	out.WriteString("}")
-}
-
-// TODO: this
-func (options *Latex) FootnoteRef(out *bytes.Buffer, ref []byte, id int) {
-
-}
-
-func needsBackslash(c byte) bool {
-	for _, r := range []byte("_{}%$&\\~#") {
-		if c == r {
-			return true
-		}
-	}
-	return false
-}
-
-func escapeSpecialChars(out *bytes.Buffer, text []byte) {
-	for i := 0; i < len(text); i++ {
-		// directly copy normal characters
-		org := i
-
-		for i < len(text) && !needsBackslash(text[i]) {
-			i++
-		}
-		if i > org {
-			out.Write(text[org:i])
-		}
-
-		// escape a character
-		if i >= len(text) {
-			break
-		}
-		out.WriteByte('\\')
-		out.WriteByte(text[i])
-	}
-}
-
-func (options *Latex) Entity(out *bytes.Buffer, entity []byte) {
-	// TODO: convert this into a unicode character or something
-	out.Write(entity)
-}
-
-func (options *Latex) NormalText(out *bytes.Buffer, text []byte) {
-	escapeSpecialChars(out, text)
-}
-
-// header and footer
-func (options *Latex) DocumentHeader(out *bytes.Buffer) {
-	out.WriteString("\\documentclass{article}\n")
-	out.WriteString("\n")
-	out.WriteString("\\usepackage{graphicx}\n")
-	out.WriteString("\\usepackage{listings}\n")
-	out.WriteString("\\usepackage[margin=1in]{geometry}\n")
-	out.WriteString("\\usepackage[utf8]{inputenc}\n")
-	out.WriteString("\\usepackage{verbatim}\n")
-	out.WriteString("\\usepackage[normalem]{ulem}\n")
-	out.WriteString("\\usepackage{hyperref}\n")
-	out.WriteString("\n")
-	out.WriteString("\\hypersetup{colorlinks,%\n")
-	out.WriteString("  citecolor=black,%\n")
-	out.WriteString("  filecolor=black,%\n")
-	out.WriteString("  linkcolor=black,%\n")
-	out.WriteString("  urlcolor=black,%\n")
-	out.WriteString("  pdfstartview=FitH,%\n")
-	out.WriteString("  breaklinks=true,%\n")
-	out.WriteString("  pdfauthor={Blackfriday Markdown Processor v")
-	out.WriteString(VERSION)
-	out.WriteString("}}\n")
-	out.WriteString("\n")
-	out.WriteString("\\newcommand{\\HRule}{\\rule{\\linewidth}{0.5mm}}\n")
-	out.WriteString("\\addtolength{\\parskip}{0.5\\baselineskip}\n")
-	out.WriteString("\\parindent=0pt\n")
-	out.WriteString("\n")
-	out.WriteString("\\begin{document}\n")
-}
-
-func (options *Latex) DocumentFooter(out *bytes.Buffer) {
-	out.WriteString("\n\\end{document}\n")
-}

markdown.go

@@ -1,231 +1,200 @@
-//
 // Blackfriday Markdown Processor
 // Available at http://github.com/russross/blackfriday
 //
 // Copyright © 2011 Russ Ross <russ@russross.com>.
 // Distributed under the Simplified BSD License.
 // See README.md for details.
-//
 
-//
-//
-// Markdown parsing and processing
-//
-//
-
-// Blackfriday markdown processor.
-//
-// Translates plain text with simple formatting rules into HTML or LaTeX.
 package blackfriday
 
 import (
 	"bytes"
 	"fmt"
+	"io"
 	"strings"
 	"unicode/utf8"
 )
 
-const VERSION = "1.4"
+//
+// Markdown parsing and processing
+//
+
+// Version string of the package. Appears in the rendered document when
+// CompletePage flag is on.
+const Version = "2.0"
+
+// Extensions is a bitwise or'ed collection of enabled Blackfriday's
+// extensions.
+type Extensions int
 
 // These are the supported markdown parsing extensions.
 // OR these values together to select multiple extensions.
 const (
-	EXTENSION_NO_INTRA_EMPHASIS          = 1 << iota // ignore emphasis markers inside words
-	EXTENSION_TABLES                                 // render tables
-	EXTENSION_FENCED_CODE                            // render fenced code blocks
-	EXTENSION_AUTOLINK                               // detect embedded URLs that are not explicitly marked
-	EXTENSION_STRIKETHROUGH                          // strikethrough text using ~~test~~
-	EXTENSION_LAX_HTML_BLOCKS                        // loosen up HTML block parsing rules
-	EXTENSION_SPACE_HEADERS                          // be strict about prefix header rules
-	EXTENSION_HARD_LINE_BREAK                        // translate newlines into line breaks
-	EXTENSION_TAB_SIZE_EIGHT                         // expand tabs to eight spaces instead of four
-	EXTENSION_FOOTNOTES                              // Pandoc-style footnotes
-	EXTENSION_NO_EMPTY_LINE_BEFORE_BLOCK             // No need to insert an empty line to start a (code, quote, ordered list, unordered list) block
-	EXTENSION_HEADER_IDS                             // specify header IDs  with {#id}
-	EXTENSION_TITLEBLOCK                             // Titleblock ala pandoc
-	EXTENSION_AUTO_HEADER_IDS                        // Create the header ID from the text
-	EXTENSION_BACKSLASH_LINE_BREAK                   // translate trailing backslashes into line breaks
-	EXTENSION_DEFINITION_LISTS                       // render definition lists
+	NoExtensions           Extensions = 0
+	NoIntraEmphasis        Extensions = 1 << iota // Ignore emphasis markers inside words
+	Tables                                        // Render tables
+	FencedCode                                    // Render fenced code blocks
+	Autolink                                      // Detect embedded URLs that are not explicitly marked
+	Strikethrough                                 // Strikethrough text using ~~test~~
+	LaxHTMLBlocks                                 // Loosen up HTML block parsing rules
+	SpaceHeadings                                 // Be strict about prefix heading rules
+	HardLineBreak                                 // Translate newlines into line breaks
+	TabSizeEight                                  // Expand tabs to eight spaces instead of four
+	Footnotes                                     // Pandoc-style footnotes
+	NoEmptyLineBeforeBlock                        // No need to insert an empty line to start a (code, quote, ordered list, unordered list) block
+	HeadingIDs                                    // specify heading IDs  with {#id}
+	Titleblock                                    // Titleblock ala pandoc
+	AutoHeadingIDs                                // Create the heading ID from the text
+	BackslashLineBreak                            // Translate trailing backslashes into line breaks
+	DefinitionLists                               // Render definition lists
 
-	commonHtmlFlags = 0 |
-		HTML_USE_XHTML |
-		HTML_USE_SMARTYPANTS |
-		HTML_SMARTYPANTS_FRACTIONS |
-		HTML_SMARTYPANTS_DASHES |
-		HTML_SMARTYPANTS_LATEX_DASHES
+	CommonHTMLFlags HTMLFlags = UseXHTML | Smartypants |
+		SmartypantsFractions | SmartypantsDashes | SmartypantsLatexDashes
 
-	commonExtensions = 0 |
-		EXTENSION_NO_INTRA_EMPHASIS |
-		EXTENSION_TABLES |
-		EXTENSION_FENCED_CODE |
-		EXTENSION_AUTOLINK |
-		EXTENSION_STRIKETHROUGH |
-		EXTENSION_SPACE_HEADERS |
-		EXTENSION_HEADER_IDS |
-		EXTENSION_BACKSLASH_LINE_BREAK |
-		EXTENSION_DEFINITION_LISTS
+	CommonExtensions Extensions = NoIntraEmphasis | Tables | FencedCode |
+		Autolink | Strikethrough | SpaceHeadings | HeadingIDs |
+		BackslashLineBreak | DefinitionLists
 )
 
-// These are the possible flag values for the link renderer.
-// Only a single one of these values will be used; they are not ORed together.
-// These are mostly of interest if you are writing a new output format.
-const (
-	LINK_TYPE_NOT_AUTOLINK = iota
-	LINK_TYPE_NORMAL
-	LINK_TYPE_EMAIL
-)
+// ListType contains bitwise or'ed flags for list and list item objects.
+type ListType int
 
 // These are the possible flag values for the ListItem renderer.
 // Multiple flag values may be ORed together.
 // These are mostly of interest if you are writing a new output format.
 const (
-	LIST_TYPE_ORDERED = 1 << iota
-	LIST_TYPE_DEFINITION
-	LIST_TYPE_TERM
-	LIST_ITEM_CONTAINS_BLOCK
-	LIST_ITEM_BEGINNING_OF_LIST
-	LIST_ITEM_END_OF_LIST
+	ListTypeOrdered ListType = 1 << iota
+	ListTypeDefinition
+	ListTypeTerm
+
+	ListItemContainsBlock
+	ListItemBeginningOfList // TODO: figure out if this is of any use now
+	ListItemEndOfList
 )
 
+// CellAlignFlags holds a type of alignment in a table cell.
+type CellAlignFlags int
+
 // These are the possible flag values for the table cell renderer.
 // Only a single one of these values will be used; they are not ORed together.
 // These are mostly of interest if you are writing a new output format.
 const (
-	TABLE_ALIGNMENT_LEFT = 1 << iota
-	TABLE_ALIGNMENT_RIGHT
-	TABLE_ALIGNMENT_CENTER = (TABLE_ALIGNMENT_LEFT | TABLE_ALIGNMENT_RIGHT)
+	TableAlignmentLeft CellAlignFlags = 1 << iota
+	TableAlignmentRight
+	TableAlignmentCenter = (TableAlignmentLeft | TableAlignmentRight)
 )
 
 // The size of a tab stop.
 const (
-	TAB_SIZE_DEFAULT = 4
-	TAB_SIZE_EIGHT   = 8
+	TabSizeDefault = 4
+	TabSizeDouble  = 8
 )
 
 // blockTags is a set of tags that are recognized as HTML block tags.
 // Any of these can be included in markdown text without special escaping.
 var blockTags = map[string]struct{}{
-	"blockquote": struct{}{},
-	"del":        struct{}{},
-	"div":        struct{}{},
-	"dl":         struct{}{},
-	"fieldset":   struct{}{},
-	"form":       struct{}{},
-	"h1":         struct{}{},
-	"h2":         struct{}{},
-	"h3":         struct{}{},
-	"h4":         struct{}{},
-	"h5":         struct{}{},
-	"h6":         struct{}{},
-	"iframe":     struct{}{},
-	"ins":        struct{}{},
-	"math":       struct{}{},
-	"noscript":   struct{}{},
-	"ol":         struct{}{},
-	"pre":        struct{}{},
-	"p":          struct{}{},
-	"script":     struct{}{},
-	"style":      struct{}{},
-	"table":      struct{}{},
-	"ul":         struct{}{},
+	"blockquote": {},
+	"del":        {},
+	"div":        {},
+	"dl":         {},
+	"fieldset":   {},
+	"form":       {},
+	"h1":         {},
+	"h2":         {},
+	"h3":         {},
+	"h4":         {},
+	"h5":         {},
+	"h6":         {},
+	"iframe":     {},
+	"ins":        {},
+	"math":       {},
+	"noscript":   {},
+	"ol":         {},
+	"pre":        {},
+	"p":          {},
+	"script":     {},
+	"style":      {},
+	"table":      {},
+	"ul":         {},
 
 	// HTML5
-	"address":    struct{}{},
-	"article":    struct{}{},
-	"aside":      struct{}{},
-	"canvas":     struct{}{},
-	"figcaption": struct{}{},
-	"figure":     struct{}{},
-	"footer":     struct{}{},
-	"header":     struct{}{},
-	"hgroup":     struct{}{},
-	"main":       struct{}{},
-	"nav":        struct{}{},
-	"output":     struct{}{},
-	"progress":   struct{}{},
-	"section":    struct{}{},
-	"video":      struct{}{},
+	"address":    {},
+	"article":    {},
+	"aside":      {},
+	"canvas":     {},
+	"figcaption": {},
+	"figure":     {},
+	"footer":     {},
+	"header":     {},
+	"hgroup":     {},
+	"main":       {},
+	"nav":        {},
+	"output":     {},
+	"progress":   {},
+	"section":    {},
+	"video":      {},
 }
 
-// Renderer is the rendering interface.
-// This is mostly of interest if you are implementing a new rendering format.
+// Renderer is the rendering interface. This is mostly of interest if you are
+// implementing a new rendering format.
 //
-// When a byte slice is provided, it contains the (rendered) contents of the
-// element.
-//
-// When a callback is provided instead, it will write the contents of the
-// respective element directly to the output buffer and return true on success.
-// If the callback returns false, the rendering function should reset the
-// output buffer as though it had never been called.
-//
-// Currently Html and Latex implementations are provided
+// Only an HTML implementation is provided in this repository, see the README
+// for external implementations.
 type Renderer interface {
-	// block-level callbacks
-	BlockCode(out *bytes.Buffer, text []byte, lang string)
-	BlockQuote(out *bytes.Buffer, text []byte)
-	BlockHtml(out *bytes.Buffer, text []byte)
-	Header(out *bytes.Buffer, text func() bool, level int, id string)
-	HRule(out *bytes.Buffer)
-	List(out *bytes.Buffer, text func() bool, flags int)
-	ListItem(out *bytes.Buffer, text []byte, flags int)
-	Paragraph(out *bytes.Buffer, text func() bool)
-	Table(out *bytes.Buffer, header []byte, body []byte, columnData []int)
-	TableRow(out *bytes.Buffer, text []byte)
-	TableHeaderCell(out *bytes.Buffer, text []byte, flags int)
-	TableCell(out *bytes.Buffer, text []byte, flags int)
-	Footnotes(out *bytes.Buffer, text func() bool)
-	FootnoteItem(out *bytes.Buffer, name, text []byte, flags int)
-	TitleBlock(out *bytes.Buffer, text []byte)
+	// RenderNode is the main rendering method. It will be called once for
+	// every leaf node and twice for every non-leaf node (first with
+	// entering=true, then with entering=false). The method should write its
+	// rendition of the node to the supplied writer w.
+	RenderNode(w io.Writer, node *Node, entering bool) WalkStatus
 
-	// Span-level callbacks
-	AutoLink(out *bytes.Buffer, link []byte, kind int)
-	CodeSpan(out *bytes.Buffer, text []byte)
-	DoubleEmphasis(out *bytes.Buffer, text []byte)
-	Emphasis(out *bytes.Buffer, text []byte)
-	Image(out *bytes.Buffer, link []byte, title []byte, alt []byte)
-	LineBreak(out *bytes.Buffer)
-	Link(out *bytes.Buffer, link []byte, title []byte, content []byte)
-	RawHtmlTag(out *bytes.Buffer, tag []byte)
-	TripleEmphasis(out *bytes.Buffer, text []byte)
-	StrikeThrough(out *bytes.Buffer, text []byte)
-	FootnoteRef(out *bytes.Buffer, ref []byte, id int)
+	// RenderHeader is a method that allows the renderer to produce some
+	// content preceding the main body of the output document. The header is
+	// understood in the broad sense here. For example, the default HTML
+	// renderer will write not only the HTML document preamble, but also the
+	// table of contents if it was requested.
+	//
+	// The method will be passed an entire document tree, in case a particular
+	// implementation needs to inspect it to produce output.
+	//
+	// The output should be written to the supplied writer w. If your
+	// implementation has no header to write, supply an empty implementation.
+	RenderHeader(w io.Writer, ast *Node)
 
-	// Low-level callbacks
-	Entity(out *bytes.Buffer, entity []byte)
-	NormalText(out *bytes.Buffer, text []byte)
-
-	// Header and footer
-	DocumentHeader(out *bytes.Buffer)
-	DocumentFooter(out *bytes.Buffer)
-
-	GetFlags() int
+	// RenderFooter is a symmetric counterpart of RenderHeader.
+	RenderFooter(w io.Writer, ast *Node)
 }
 
 // Callback functions for inline parsing. One such function is defined
 // for each character that triggers a response when parsing inline data.
-type inlineParser func(p *parser, out *bytes.Buffer, data []byte, offset int) int
+type inlineParser func(p *Markdown, data []byte, offset int) (int, *Node)
 
-// Parser holds runtime state used by the parser.
-// This is constructed by the Markdown function.
-type parser struct {
-	r              Renderer
-	refOverride    ReferenceOverrideFunc
-	refs           map[string]*reference
-	inlineCallback [256]inlineParser
-	flags          int
-	nesting        int
-	maxNesting     int
-	insideLink     bool
+// Markdown is a type that holds extensions and the runtime state used by
+// Parse, and the renderer. You can not use it directly, construct it with New.
+type Markdown struct {
+	renderer          Renderer
+	referenceOverride ReferenceOverrideFunc
+	refs              map[string]*reference
+	inlineCallback    [256]inlineParser
+	extensions        Extensions
+	nesting           int
+	maxNesting        int
+	insideLink        bool
 
 	// Footnotes need to be ordered as well as available to quickly check for
 	// presence. If a ref is also a footnote, it's stored both in refs and here
 	// in notes. Slice is nil if footnotes not enabled.
 	notes []*reference
+
+	doc                  *Node
+	tip                  *Node // = doc
+	oldTip               *Node
+	lastMatchedContainer *Node // = doc
+	allClosed            bool
 }
 
-func (p *parser) getRef(refid string) (ref *reference, found bool) {
-	if p.refOverride != nil {
-		r, overridden := p.refOverride(refid)
+func (p *Markdown) getRef(refid string) (ref *reference, found bool) {
+	if p.referenceOverride != nil {
+		r, overridden := p.referenceOverride(refid)
 		if overridden {
 			if r == nil {
 				return nil, false
@@ -233,7 +202,7 @@ func (p *parser) getRef(refid string) (ref *reference, found bool) {
 			return &reference{
 				link:     []byte(r.Link),
 				title:    []byte(r.Title),
-				noteId:   0,
+				noteID:   0,
 				hasBlock: false,
 				text:     []byte(r.Text)}, true
 		}
@@ -243,6 +212,36 @@ func (p *parser) getRef(refid string) (ref *reference, found bool) {
 	return ref, found
 }
 
+func (p *Markdown) finalize(block *Node) {
+	above := block.Parent
+	block.open = false
+	p.tip = above
+}
+
+func (p *Markdown) addChild(node NodeType, offset uint32) *Node {
+	return p.addExistingChild(NewNode(node), offset)
+}
+
+func (p *Markdown) addExistingChild(node *Node, offset uint32) *Node {
+	for !p.tip.canContain(node.Type) {
+		p.finalize(p.tip)
+	}
+	p.tip.AppendChild(node)
+	p.tip = node
+	return node
+}
+
+func (p *Markdown) closeUnmatchedBlocks() {
+	if !p.allClosed {
+		for p.oldTip != p.lastMatchedContainer {
+			parent := p.oldTip.Parent
+			p.finalize(p.oldTip)
+			p.oldTip = parent
+		}
+		p.allClosed = true
+	}
+}
+
 //
 //
 // Public interface
@@ -267,102 +266,27 @@ type Reference struct {
 // See the documentation in Options for more details on use-case.
 type ReferenceOverrideFunc func(reference string) (ref *Reference, overridden bool)
 
-// Options represents configurable overrides and callbacks (in addition to the
-// extension flag set) for configuring a Markdown parse.
-type Options struct {
-	// Extensions is a flag set of bit-wise ORed extension bits. See the
-	// EXTENSION_* flags defined in this package.
-	Extensions int
-
-	// ReferenceOverride is an optional function callback that is called every
-	// time a reference is resolved.
-	//
-	// In Markdown, the link reference syntax can be made to resolve a link to
-	// a reference instead of an inline URL, in one of the following ways:
-	//
-	//  * [link text][refid]
-	//  * [refid][]
-	//
-	// Usually, the refid is defined at the bottom of the Markdown document. If
-	// this override function is provided, the refid is passed to the override
-	// function first, before consulting the defined refids at the bottom. If
-	// the override function indicates an override did not occur, the refids at
-	// the bottom will be used to fill in the link details.
-	ReferenceOverride ReferenceOverrideFunc
-}
-
-// MarkdownBasic is a convenience function for simple rendering.
-// It processes markdown input with no extensions enabled.
-func MarkdownBasic(input []byte) []byte {
-	// set up the HTML renderer
-	htmlFlags := HTML_USE_XHTML
-	renderer := HtmlRenderer(htmlFlags, "", "")
-
-	// set up the parser
-	return MarkdownOptions(input, renderer, Options{Extensions: 0})
-}
-
-// Call Markdown with most useful extensions enabled
-// MarkdownCommon is a convenience function for simple rendering.
-// It processes markdown input with common extensions enabled, including:
-//
-// * Smartypants processing with smart fractions and LaTeX dashes
-//
-// * Intra-word emphasis suppression
-//
-// * Tables
-//
-// * Fenced code blocks
-//
-// * Autolinking
-//
-// * Strikethrough support
-//
-// * Strict header parsing
-//
-// * Custom Header IDs
-func MarkdownCommon(input []byte) []byte {
-	// set up the HTML renderer
-	renderer := HtmlRenderer(commonHtmlFlags, "", "")
-	return MarkdownOptions(input, renderer, Options{
-		Extensions: commonExtensions})
-}
-
-// Markdown is the main rendering function.
-// It parses and renders a block of markdown-encoded text.
-// The supplied Renderer is used to format the output, and extensions dictates
-// which non-standard extensions are enabled.
-//
-// To use the supplied Html or LaTeX renderers, see HtmlRenderer and
-// LatexRenderer, respectively.
-func Markdown(input []byte, renderer Renderer, extensions int) []byte {
-	return MarkdownOptions(input, renderer, Options{
-		Extensions: extensions})
-}
-
-// MarkdownOptions is just like Markdown but takes additional options through
-// the Options struct.
-func MarkdownOptions(input []byte, renderer Renderer, opts Options) []byte {
-	// no point in parsing if we can't render
-	if renderer == nil {
-		return nil
+// New constructs a Markdown processor. You can use the same With* functions as
+// for Run() to customize parser's behavior and the renderer.
+func New(opts ...Option) *Markdown {
+	var p Markdown
+	for _, opt := range opts {
+		opt(&p)
 	}
-
-	extensions := opts.Extensions
-
-	// fill in the render structure
-	p := new(parser)
-	p.r = renderer
-	p.flags = extensions
-	p.refOverride = opts.ReferenceOverride
 	p.refs = make(map[string]*reference)
 	p.maxNesting = 16
 	p.insideLink = false
-
+	docNode := NewNode(Document)
+	p.doc = docNode
+	p.tip = docNode
+	p.oldTip = docNode
+	p.lastMatchedContainer = docNode
+	p.allClosed = true
 	// register inline parsers
+	p.inlineCallback[' '] = maybeLineBreak
 	p.inlineCallback['*'] = emphasis
 	p.inlineCallback['_'] = emphasis
-	if extensions&EXTENSION_STRIKETHROUGH != 0 {
+	if p.extensions&Strikethrough != 0 {
 		p.inlineCallback['~'] = emphasis
 	}
 	p.inlineCallback['`'] = codeSpan
@@ -371,115 +295,166 @@ func MarkdownOptions(input []byte, renderer Renderer, opts Options) []byte {
 	p.inlineCallback['<'] = leftAngle
 	p.inlineCallback['\\'] = escape
 	p.inlineCallback['&'] = entity
-
-	if extensions&EXTENSION_AUTOLINK != 0 {
-		p.inlineCallback[':'] = autoLink
+	p.inlineCallback['!'] = maybeImage
+	p.inlineCallback['^'] = maybeInlineFootnote
+	if p.extensions&Autolink != 0 {
+		p.inlineCallback['h'] = maybeAutoLink
+		p.inlineCallback['m'] = maybeAutoLink
+		p.inlineCallback['f'] = maybeAutoLink
+		p.inlineCallback['H'] = maybeAutoLink
+		p.inlineCallback['M'] = maybeAutoLink
+		p.inlineCallback['F'] = maybeAutoLink
 	}
-
-	if extensions&EXTENSION_FOOTNOTES != 0 {
+	if p.extensions&Footnotes != 0 {
 		p.notes = make([]*reference, 0)
 	}
-
-	first := firstPass(p, input)
-	second := secondPass(p, first)
-	return second
+	return &p
 }
 
-// first pass:
-// - extract references
-// - expand tabs
-// - normalize newlines
-// - copy everything else
-func firstPass(p *parser, input []byte) []byte {
-	var out bytes.Buffer
-	tabSize := TAB_SIZE_DEFAULT
-	if p.flags&EXTENSION_TAB_SIZE_EIGHT != 0 {
-		tabSize = TAB_SIZE_EIGHT
+// Option customizes the Markdown processor's default behavior.
+type Option func(*Markdown)
+
+// WithRenderer allows you to override the default renderer.
+func WithRenderer(r Renderer) Option {
+	return func(p *Markdown) {
+		p.renderer = r
 	}
-	beg, end := 0, 0
-	lastFencedCodeBlockEnd := 0
-	for beg < len(input) { // iterate over lines
-		if end = isReference(p, input[beg:], tabSize); end > 0 {
-			beg += end
-		} else { // skip to the next line
-			end = beg
-			for end < len(input) && input[end] != '\n' && input[end] != '\r' {
-				end++
-			}
-
-			if p.flags&EXTENSION_FENCED_CODE != 0 {
-				// track fenced code block boundaries to suppress tab expansion
-				// inside them:
-				if beg >= lastFencedCodeBlockEnd {
-					if i := p.fencedCode(&out, input[beg:], false); i > 0 {
-						lastFencedCodeBlockEnd = beg + i
-					}
-				}
-			}
-
-			// add the line body if present
-			if end > beg {
-				if end < lastFencedCodeBlockEnd { // Do not expand tabs while inside fenced code blocks.
-					out.Write(input[beg:end])
-				} else {
-					expandTabs(&out, input[beg:end], tabSize)
-				}
-			}
-			out.WriteByte('\n')
-
-			if end < len(input) && input[end] == '\r' {
-				end++
-			}
-			if end < len(input) && input[end] == '\n' {
-				end++
-			}
-
-			beg = end
-		}
-	}
-
-	// empty input?
-	if out.Len() == 0 {
-		out.WriteByte('\n')
-	}
-
-	return out.Bytes()
 }
 
-// second pass: actual rendering
-func secondPass(p *parser, input []byte) []byte {
-	var output bytes.Buffer
+// WithExtensions allows you to pick some of the many extensions provided by
+// Blackfriday. You can bitwise OR them.
+func WithExtensions(e Extensions) Option {
+	return func(p *Markdown) {
+		p.extensions = e
+	}
+}
 
-	p.r.DocumentHeader(&output)
-	p.block(&output, input)
-
-	if p.flags&EXTENSION_FOOTNOTES != 0 && len(p.notes) > 0 {
-		p.r.Footnotes(&output, func() bool {
-			flags := LIST_ITEM_BEGINNING_OF_LIST
-			for i := 0; i < len(p.notes); i += 1 {
-				ref := p.notes[i]
-				var buf bytes.Buffer
-				if ref.hasBlock {
-					flags |= LIST_ITEM_CONTAINS_BLOCK
-					p.block(&buf, ref.title)
-				} else {
-					p.inline(&buf, ref.title)
-				}
-				p.r.FootnoteItem(&output, ref.link, buf.Bytes(), flags)
-				flags &^= LIST_ITEM_BEGINNING_OF_LIST | LIST_ITEM_CONTAINS_BLOCK
-			}
-
-			return true
+// WithNoExtensions turns off all extensions and custom behavior.
+func WithNoExtensions() Option {
+	return func(p *Markdown) {
+		p.extensions = NoExtensions
+		p.renderer = NewHTMLRenderer(HTMLRendererParameters{
+			Flags: HTMLFlagsNone,
 		})
 	}
+}
 
-	p.r.DocumentFooter(&output)
-
-	if p.nesting != 0 {
-		panic("Nesting level did not end at zero")
+// WithRefOverride sets an optional function callback that is called every
+// time a reference is resolved.
+//
+// In Markdown, the link reference syntax can be made to resolve a link to
+// a reference instead of an inline URL, in one of the following ways:
+//
+//  * [link text][refid]
+//  * [refid][]
+//
+// Usually, the refid is defined at the bottom of the Markdown document. If
+// this override function is provided, the refid is passed to the override
+// function first, before consulting the defined refids at the bottom. If
+// the override function indicates an override did not occur, the refids at
+// the bottom will be used to fill in the link details.
+func WithRefOverride(o ReferenceOverrideFunc) Option {
+	return func(p *Markdown) {
+		p.referenceOverride = o
 	}
+}
 
-	return output.Bytes()
+// Run is the main entry point to Blackfriday. It parses and renders a
+// block of markdown-encoded text.
+//
+// The simplest invocation of Run takes one argument, input:
+//     output := Run(input)
+// This will parse the input with CommonExtensions enabled and render it with
+// the default HTMLRenderer (with CommonHTMLFlags).
+//
+// Variadic arguments opts can customize the default behavior. Since Markdown
+// type does not contain exported fields, you can not use it directly. Instead,
+// use the With* functions. For example, this will call the most basic
+// functionality, with no extensions:
+//     output := Run(input, WithNoExtensions())
+//
+// You can use any number of With* arguments, even contradicting ones. They
+// will be applied in order of appearance and the latter will override the
+// former:
+//     output := Run(input, WithNoExtensions(), WithExtensions(exts),
+//         WithRenderer(yourRenderer))
+func Run(input []byte, opts ...Option) []byte {
+	r := NewHTMLRenderer(HTMLRendererParameters{
+		Flags: CommonHTMLFlags,
+	})
+	optList := []Option{WithRenderer(r), WithExtensions(CommonExtensions)}
+	optList = append(optList, opts...)
+	parser := New(optList...)
+	ast := parser.Parse(input)
+	var buf bytes.Buffer
+	parser.renderer.RenderHeader(&buf, ast)
+	ast.Walk(func(node *Node, entering bool) WalkStatus {
+		return parser.renderer.RenderNode(&buf, node, entering)
+	})
+	parser.renderer.RenderFooter(&buf, ast)
+	return buf.Bytes()
+}
+
+// Parse is an entry point to the parsing part of Blackfriday. It takes an
+// input markdown document and produces a syntax tree for its contents. This
+// tree can then be rendered with a default or custom renderer, or
+// analyzed/transformed by the caller to whatever non-standard needs they have.
+// The return value is the root node of the syntax tree.
+func (p *Markdown) Parse(input []byte) *Node {
+	p.block(input)
+	// Walk the tree and finish up some of unfinished blocks
+	for p.tip != nil {
+		p.finalize(p.tip)
+	}
+	// Walk the tree again and process inline markdown in each block
+	p.doc.Walk(func(node *Node, entering bool) WalkStatus {
+		if node.Type == Paragraph || node.Type == Heading || node.Type == TableCell {
+			p.inline(node, node.content)
+			node.content = nil
+		}
+		return GoToNext
+	})
+	p.parseRefsToAST()
+	return p.doc
+}
+
+func (p *Markdown) parseRefsToAST() {
+	if p.extensions&Footnotes == 0 || len(p.notes) == 0 {
+		return
+	}
+	p.tip = p.doc
+	block := p.addBlock(List, nil)
+	block.IsFootnotesList = true
+	block.ListFlags = ListTypeOrdered
+	flags := ListItemBeginningOfList
+	// Note: this loop is intentionally explicit, not range-form. This is
+	// because the body of the loop will append nested footnotes to p.notes and
+	// we need to process those late additions. Range form would only walk over
+	// the fixed initial set.
+	for i := 0; i < len(p.notes); i++ {
+		ref := p.notes[i]
+		p.addExistingChild(ref.footnote, 0)
+		block := ref.footnote
+		block.ListFlags = flags | ListTypeOrdered
+		block.RefLink = ref.link
+		if ref.hasBlock {
+			flags |= ListItemContainsBlock
+			p.block(ref.title)
+		} else {
+			p.inline(block, ref.title)
+		}
+		flags &^= ListItemBeginningOfList | ListItemContainsBlock
+	}
+	above := block.Parent
+	finalizeList(block)
+	p.tip = above
+	block.Walk(func(node *Node, entering bool) WalkStatus {
+		if node.Type == Paragraph || node.Type == Heading {
+			p.inline(node, node.content)
+			node.content = nil
+		}
+		return GoToNext
+	})
 }
 
 //
@@ -505,24 +480,62 @@ func secondPass(p *parser, input []byte) []byte {
 //    [^note]: This is the explanation.
 //
 // Footnotes should be placed at the end of the document in an ordered list.
-// Inline footnotes such as:
+// Finally, there are inline footnotes such as:
 //
-//    Inline footnotes^[Not supported.] also exist.
+//    Inline footnotes^[Also supported.] provide a quick inline explanation,
+//    but are rendered at the bottom of the document.
 //
-// are not yet supported.
 
-// References are parsed and stored in this struct.
+// reference holds all information necessary for a reference-style links or
+// footnotes.
+//
+// Consider this markdown with reference-style links:
+//
+//     [link][ref]
+//
+//     [ref]: /url/ "tooltip title"
+//
+// It will be ultimately converted to this HTML:
+//
+//     <p><a href=\"/url/\" title=\"title\">link</a></p>
+//
+// And a reference structure will be populated as follows:
+//
+//     p.refs["ref"] = &reference{
+//         link: "/url/",
+//         title: "tooltip title",
+//     }
+//
+// Alternatively, reference can contain information about a footnote. Consider
+// this markdown:
+//
+//     Text needing a footnote.[^a]
+//
+//     [^a]: This is the note
+//
+// A reference structure will be populated as follows:
+//
+//     p.refs["a"] = &reference{
+//         link: "a",
+//         title: "This is the note",
+//         noteID: <some positive int>,
+//     }
+//
+// TODO: As you can see, it begs for splitting into two dedicated structures
+// for refs and for footnotes.
 type reference struct {
 	link     []byte
 	title    []byte
-	noteId   int // 0 if not a footnote ref
+	noteID   int // 0 if not a footnote ref
 	hasBlock bool
-	text     []byte
+	footnote *Node // a link to the Item node within a list of footnotes
+
+	text []byte // only gets populated by refOverride feature with Reference.Text
 }
 
 func (r *reference) String() string {
-	return fmt.Sprintf("{link: %q, title: %q, text: %q, noteId: %d, hasBlock: %v}",
-		r.link, r.title, r.text, r.noteId, r.hasBlock)
+	return fmt.Sprintf("{link: %q, title: %q, text: %q, noteID: %d, hasBlock: %v}",
+		r.link, r.title, r.text, r.noteID, r.hasBlock)
 }
 
 // Check whether or not data starts with a reference link.
@@ -530,7 +543,7 @@ func (r *reference) String() string {
 // (in the render struct).
 // Returns the number of bytes to skip to move past it,
 // or zero if the first line is not a reference.
-func isReference(p *parser, data []byte, tabSize int) int {
+func isReference(p *Markdown, data []byte, tabSize int) int {
 	// up to 3 optional leading spaces
 	if len(data) < 4 {
 		return 0
@@ -540,18 +553,18 @@ func isReference(p *parser, data []byte, tabSize int) int {
 		i++
 	}
 
-	noteId := 0
+	noteID := 0
 
 	// id part: anything but a newline between brackets
 	if data[i] != '[' {
 		return 0
 	}
 	i++
-	if p.flags&EXTENSION_FOOTNOTES != 0 {
+	if p.extensions&Footnotes != 0 {
 		if i < len(data) && data[i] == '^' {
 			// we can set it to anything here because the proper noteIds will
 			// be assigned later during the second pass. It just has to be != 0
-			noteId = 1
+			noteID = 1
 			i++
 		}
 	}
@@ -563,7 +576,11 @@ func isReference(p *parser, data []byte, tabSize int) int {
 		return 0
 	}
 	idEnd := i
-
+	// footnotes can have empty ID, like this: [^], but a reference can not be
+	// empty like this: []. Break early if it's not a footnote and there's no ID
+	if noteID == 0 && idOffset == idEnd {
+		return 0
+	}
 	// spacer: colon (space | tab)* newline? (space | tab)*
 	i++
 	if i >= len(data) || data[i] != ':' {
@@ -594,7 +611,7 @@ func isReference(p *parser, data []byte, tabSize int) int {
 		hasBlock              bool
 	)
 
-	if p.flags&EXTENSION_FOOTNOTES != 0 && noteId != 0 {
+	if p.extensions&Footnotes != 0 && noteID != 0 {
 		linkOffset, linkEnd, raw, hasBlock = scanFootnote(p, data, i, tabSize)
 		lineEnd = linkEnd
 	} else {
@@ -607,11 +624,11 @@ func isReference(p *parser, data []byte, tabSize int) int {
 	// a valid ref has been found
 
 	ref := &reference{
-		noteId:   noteId,
+		noteID:   noteID,
 		hasBlock: hasBlock,
 	}
 
-	if noteId > 0 {
+	if noteID > 0 {
 		// reusing the link field for the id since footnotes don't have links
 		ref.link = data[idOffset:idEnd]
 		// if footnote, it's not really a title, it's the contained text
@@ -629,7 +646,7 @@ func isReference(p *parser, data []byte, tabSize int) int {
 	return lineEnd
 }
 
-func scanLinkRef(p *parser, data []byte, i int) (linkOffset, linkEnd, titleOffset, titleEnd, lineEnd int) {
+func scanLinkRef(p *Markdown, data []byte, i int) (linkOffset, linkEnd, titleOffset, titleEnd, lineEnd int) {
 	// link: whitespace-free sequence, optionally between angle brackets
 	if data[i] == '<' {
 		i++
@@ -638,9 +655,6 @@ func scanLinkRef(p *parser, data []byte, i int) (linkOffset, linkEnd, titleOffse
 	for i < len(data) && data[i] != ' ' && data[i] != '\t' && data[i] != '\n' && data[i] != '\r' {
 		i++
 	}
-	if i == len(data) {
-		return
-	}
 	linkEnd = i
 	if data[linkOffset] == '<' && data[linkEnd-1] == '>' {
 		linkOffset++
@@ -700,13 +714,13 @@ func scanLinkRef(p *parser, data []byte, i int) (linkOffset, linkEnd, titleOffse
 	return
 }
 
-// The first bit of this logic is the same as (*parser).listItem, but the rest
+// The first bit of this logic is the same as Parser.listItem, but the rest
 // is much simpler. This function simply finds the entire block and shifts it
 // over by one tab if it is indeed a block (just returns the line if it's not).
 // blockEnd is the end of the section in the input buffer, and contents is the
 // extracted text that was shifted over one tab. It will need to be rendered at
 // the end of the document.
-func scanFootnote(p *parser, data []byte, i, indentSize int) (blockStart, blockEnd int, contents []byte, hasBlock bool) {
+func scanFootnote(p *Markdown, data []byte, i, indentSize int) (blockStart, blockEnd int, contents []byte, hasBlock bool) {
 	if i == 0 || len(data) == 0 {
 		return
 	}
@@ -799,7 +813,17 @@ func ispunct(c byte) bool {
 
 // Test if a character is a whitespace character.
 func isspace(c byte) bool {
-	return c == ' ' || c == '\t' || c == '\n' || c == '\r' || c == '\f' || c == '\v'
+	return ishorizontalspace(c) || isverticalspace(c)
+}
+
+// Test if a character is a horizontal whitespace character.
+func ishorizontalspace(c byte) bool {
+	return c == ' ' || c == '\t'
+}
+
+// Test if a character is a vertical character.
+func isverticalspace(c byte) bool {
+	return c == '\n' || c == '\r' || c == '\f' || c == '\v'
 }
 
 // Test if a character is letter.

markdown_test.go

@@ -0,0 +1,39 @@
+//
+// Blackfriday Markdown Processor
+// Available at http://github.com/russross/blackfriday
+//
+// Copyright © 2011 Russ Ross <russ@russross.com>.
+// Distributed under the Simplified BSD License.
+// See README.md for details.
+//
+
+//
+// Unit tests for full document parsing and rendering
+//
+
+package blackfriday
+
+import "testing"
+
+func TestDocument(t *testing.T) {
+	t.Parallel()
+	var tests = []string{
+		// Empty document.
+		"",
+		"",
+
+		" ",
+		"",
+
+		// This shouldn't panic.
+		// https://github.com/russross/blackfriday/issues/172
+		"[]:<",
+		"<p>[]:&lt;</p>\n",
+
+		// This shouldn't panic.
+		// https://github.com/russross/blackfriday/issues/173
+		"   [",
+		"<p>[</p>\n",
+	}
+	doTests(t, tests)
+}

node.go

@@ -0,0 +1,360 @@
+package blackfriday
+
+import (
+	"bytes"
+	"fmt"
+)
+
+// NodeType specifies a type of a single node of a syntax tree. Usually one
+// node (and its type) corresponds to a single markdown feature, e.g. emphasis
+// or code block.
+type NodeType int
+
+// Constants for identifying different types of nodes. See NodeType.
+const (
+	Document NodeType = iota
+	BlockQuote
+	List
+	Item
+	Paragraph
+	Heading
+	HorizontalRule
+	Emph
+	Strong
+	Del
+	Link
+	Image
+	Text
+	HTMLBlock
+	CodeBlock
+	Softbreak
+	Hardbreak
+	Code
+	HTMLSpan
+	Table
+	TableCell
+	TableHead
+	TableBody
+	TableRow
+)
+
+var nodeTypeNames = []string{
+	Document:       "Document",
+	BlockQuote:     "BlockQuote",
+	List:           "List",
+	Item:           "Item",
+	Paragraph:      "Paragraph",
+	Heading:        "Heading",
+	HorizontalRule: "HorizontalRule",
+	Emph:           "Emph",
+	Strong:         "Strong",
+	Del:            "Del",
+	Link:           "Link",
+	Image:          "Image",
+	Text:           "Text",
+	HTMLBlock:      "HTMLBlock",
+	CodeBlock:      "CodeBlock",
+	Softbreak:      "Softbreak",
+	Hardbreak:      "Hardbreak",
+	Code:           "Code",
+	HTMLSpan:       "HTMLSpan",
+	Table:          "Table",
+	TableCell:      "TableCell",
+	TableHead:      "TableHead",
+	TableBody:      "TableBody",
+	TableRow:       "TableRow",
+}
+
+func (t NodeType) String() string {
+	return nodeTypeNames[t]
+}
+
+// ListData contains fields relevant to a List and Item node type.
+type ListData struct {
+	ListFlags       ListType
+	Tight           bool   // Skip <p>s around list item data if true
+	BulletChar      byte   // '*', '+' or '-' in bullet lists
+	Delimiter       byte   // '.' or ')' after the number in ordered lists
+	RefLink         []byte // If not nil, turns this list item into a footnote item and triggers different rendering
+	IsFootnotesList bool   // This is a list of footnotes
+}
+
+// LinkData contains fields relevant to a Link node type.
+type LinkData struct {
+	Destination []byte // Destination is what goes into a href
+	Title       []byte // Title is the tooltip thing that goes in a title attribute
+	NoteID      int    // NoteID contains a serial number of a footnote, zero if it's not a footnote
+	Footnote    *Node  // If it's a footnote, this is a direct link to the footnote Node. Otherwise nil.
+}
+
+// CodeBlockData contains fields relevant to a CodeBlock node type.
+type CodeBlockData struct {
+	IsFenced    bool   // Specifies whether it's a fenced code block or an indented one
+	Info        []byte // This holds the info string
+	FenceChar   byte
+	FenceLength int
+	FenceOffset int
+}
+
+// TableCellData contains fields relevant to a TableCell node type.
+type TableCellData struct {
+	IsHeader bool           // This tells if it's under the header row
+	Align    CellAlignFlags // This holds the value for align attribute
+}
+
+// HeadingData contains fields relevant to a Heading node type.
+type HeadingData struct {
+	Level        int    // This holds the heading level number
+	HeadingID    string // This might hold heading ID, if present
+	IsTitleblock bool   // Specifies whether it's a title block
+}
+
+// Node is a single element in the abstract syntax tree of the parsed document.
+// It holds connections to the structurally neighboring nodes and, for certain
+// types of nodes, additional information that might be needed when rendering.
+type Node struct {
+	Type       NodeType // Determines the type of the node
+	Parent     *Node    // Points to the parent
+	FirstChild *Node    // Points to the first child, if any
+	LastChild  *Node    // Points to the last child, if any
+	Prev       *Node    // Previous sibling; nil if it's the first child
+	Next       *Node    // Next sibling; nil if it's the last child
+
+	Literal []byte // Text contents of the leaf nodes
+
+	HeadingData   // Populated if Type is Heading
+	ListData      // Populated if Type is List
+	CodeBlockData // Populated if Type is CodeBlock
+	LinkData      // Populated if Type is Link
+	TableCellData // Populated if Type is TableCell
+
+	content []byte // Markdown content of the block nodes
+	open    bool   // Specifies an open block node that has not been finished to process yet
+}
+
+// NewNode allocates a node of a specified type.
+func NewNode(typ NodeType) *Node {
+	return &Node{
+		Type: typ,
+		open: true,
+	}
+}
+
+func (n *Node) String() string {
+	ellipsis := ""
+	snippet := n.Literal
+	if len(snippet) > 16 {
+		snippet = snippet[:16]
+		ellipsis = "..."
+	}
+	return fmt.Sprintf("%s: '%s%s'", n.Type, snippet, ellipsis)
+}
+
+// Unlink removes node 'n' from the tree.
+// It panics if the node is nil.
+func (n *Node) Unlink() {
+	if n.Prev != nil {
+		n.Prev.Next = n.Next
+	} else if n.Parent != nil {
+		n.Parent.FirstChild = n.Next
+	}
+	if n.Next != nil {
+		n.Next.Prev = n.Prev
+	} else if n.Parent != nil {
+		n.Parent.LastChild = n.Prev
+	}
+	n.Parent = nil
+	n.Next = nil
+	n.Prev = nil
+}
+
+// AppendChild adds a node 'child' as a child of 'n'.
+// It panics if either node is nil.
+func (n *Node) AppendChild(child *Node) {
+	child.Unlink()
+	child.Parent = n
+	if n.LastChild != nil {
+		n.LastChild.Next = child
+		child.Prev = n.LastChild
+		n.LastChild = child
+	} else {
+		n.FirstChild = child
+		n.LastChild = child
+	}
+}
+
+// InsertBefore inserts 'sibling' immediately before 'n'.
+// It panics if either node is nil.
+func (n *Node) InsertBefore(sibling *Node) {
+	sibling.Unlink()
+	sibling.Prev = n.Prev
+	if sibling.Prev != nil {
+		sibling.Prev.Next = sibling
+	}
+	sibling.Next = n
+	n.Prev = sibling
+	sibling.Parent = n.Parent
+	if sibling.Prev == nil {
+		sibling.Parent.FirstChild = sibling
+	}
+}
+
+// IsContainer returns true if 'n' can contain children.
+func (n *Node) IsContainer() bool {
+	switch n.Type {
+	case Document:
+		fallthrough
+	case BlockQuote:
+		fallthrough
+	case List:
+		fallthrough
+	case Item:
+		fallthrough
+	case Paragraph:
+		fallthrough
+	case Heading:
+		fallthrough
+	case Emph:
+		fallthrough
+	case Strong:
+		fallthrough
+	case Del:
+		fallthrough
+	case Link:
+		fallthrough
+	case Image:
+		fallthrough
+	case Table:
+		fallthrough
+	case TableHead:
+		fallthrough
+	case TableBody:
+		fallthrough
+	case TableRow:
+		fallthrough
+	case TableCell:
+		return true
+	default:
+		return false
+	}
+}
+
+// IsLeaf returns true if 'n' is a leaf node.
+func (n *Node) IsLeaf() bool {
+	return !n.IsContainer()
+}
+
+func (n *Node) canContain(t NodeType) bool {
+	if n.Type == List {
+		return t == Item
+	}
+	if n.Type == Document || n.Type == BlockQuote || n.Type == Item {
+		return t != Item
+	}
+	if n.Type == Table {
+		return t == TableHead || t == TableBody
+	}
+	if n.Type == TableHead || n.Type == TableBody {
+		return t == TableRow
+	}
+	if n.Type == TableRow {
+		return t == TableCell
+	}
+	return false
+}
+
+// WalkStatus allows NodeVisitor to have some control over the tree traversal.
+// It is returned from NodeVisitor and different values allow Node.Walk to
+// decide which node to go to next.
+type WalkStatus int
+
+const (
+	// GoToNext is the default traversal of every node.
+	GoToNext WalkStatus = iota
+	// SkipChildren tells walker to skip all children of current node.
+	SkipChildren
+	// Terminate tells walker to terminate the traversal.
+	Terminate
+)
+
+// NodeVisitor is a callback to be called when traversing the syntax tree.
+// Called twice for every node: once with entering=true when the branch is
+// first visited, then with entering=false after all the children are done.
+type NodeVisitor func(node *Node, entering bool) WalkStatus
+
+// Walk is a convenience method that instantiates a walker and starts a
+// traversal of subtree rooted at n.
+func (n *Node) Walk(visitor NodeVisitor) {
+	w := newNodeWalker(n)
+	for w.current != nil {
+		status := visitor(w.current, w.entering)
+		switch status {
+		case GoToNext:
+			w.next()
+		case SkipChildren:
+			w.entering = false
+			w.next()
+		case Terminate:
+			return
+		}
+	}
+}
+
+type nodeWalker struct {
+	current  *Node
+	root     *Node
+	entering bool
+}
+
+func newNodeWalker(root *Node) *nodeWalker {
+	return &nodeWalker{
+		current:  root,
+		root:     root,
+		entering: true,
+	}
+}
+
+func (nw *nodeWalker) next() {
+	if (!nw.current.IsContainer() || !nw.entering) && nw.current == nw.root {
+		nw.current = nil
+		return
+	}
+	if nw.entering && nw.current.IsContainer() {
+		if nw.current.FirstChild != nil {
+			nw.current = nw.current.FirstChild
+			nw.entering = true
+		} else {
+			nw.entering = false
+		}
+	} else if nw.current.Next == nil {
+		nw.current = nw.current.Parent
+		nw.entering = false
+	} else {
+		nw.current = nw.current.Next
+		nw.entering = true
+	}
+}
+
+func dump(ast *Node) {
+	fmt.Println(dumpString(ast))
+}
+
+func dumpR(ast *Node, depth int) string {
+	if ast == nil {
+		return ""
+	}
+	indent := bytes.Repeat([]byte("\t"), depth)
+	content := ast.Literal
+	if content == nil {
+		content = ast.content
+	}
+	result := fmt.Sprintf("%s%s(%q)\n", indent, ast.Type, content)
+	for n := ast.FirstChild; n != nil; n = n.Next {
+		result += dumpR(n, depth+1)
+	}
+	return result
+}
+
+func dumpString(ast *Node) string {
+	return dumpR(ast, 0)
+}

ref_test.go

@@ -19,59 +19,8 @@ import (
 	"testing"
 )
 
-func runMarkdownReference(input string, flag int) string {
-	renderer := HtmlRenderer(0, "", "")
-	return string(Markdown([]byte(input), renderer, flag))
-}
-
-func doTestsReference(t *testing.T, files []string, flag int) {
-	// catch and report panics
-	var candidate string
-	defer func() {
-		if err := recover(); err != nil {
-			t.Errorf("\npanic while processing [%#v]\n", candidate)
-		}
-	}()
-
-	for _, basename := range files {
-		filename := filepath.Join("testdata", basename+".text")
-		inputBytes, err := ioutil.ReadFile(filename)
-		if err != nil {
-			t.Errorf("Couldn't open '%s', error: %v\n", filename, err)
-			continue
-		}
-		input := string(inputBytes)
-
-		filename = filepath.Join("testdata", basename+".html")
-		expectedBytes, err := ioutil.ReadFile(filename)
-		if err != nil {
-			t.Errorf("Couldn't open '%s', error: %v\n", filename, err)
-			continue
-		}
-		expected := string(expectedBytes)
-
-		// fmt.Fprintf(os.Stderr, "processing %s ...", filename)
-		actual := string(runMarkdownReference(input, flag))
-		if actual != expected {
-			t.Errorf("\n    [%#v]\nExpected[%#v]\nActual  [%#v]",
-				basename+".text", expected, actual)
-		}
-		// fmt.Fprintf(os.Stderr, " ok\n")
-
-		// now test every prefix of every input to check for
-		// bounds checking
-		if !testing.Short() {
-			start, max := 0, len(input)
-			for end := start + 1; end <= max; end++ {
-				candidate = input[start:end]
-				// fmt.Fprintf(os.Stderr, "  %s %d:%d/%d\n", filename, start, end, max)
-				_ = runMarkdownReference(candidate, flag)
-			}
-		}
-	}
-}
-
 func TestReference(t *testing.T) {
+	t.Parallel()
 	files := []string{
 		"Amps and angle encoding",
 		"Auto links",
@@ -100,6 +49,7 @@ func TestReference(t *testing.T) {
 }
 
 func TestReference_EXTENSION_NO_EMPTY_LINE_BEFORE_BLOCK(t *testing.T) {
+	t.Parallel()
 	files := []string{
 		"Amps and angle encoding",
 		"Auto links",
@@ -124,5 +74,53 @@ func TestReference_EXTENSION_NO_EMPTY_LINE_BEFORE_BLOCK(t *testing.T) {
 		"Tabs",
 		"Tidyness",
 	}
-	doTestsReference(t, files, EXTENSION_NO_EMPTY_LINE_BEFORE_BLOCK)
+	doTestsReference(t, files, NoEmptyLineBeforeBlock)
+}
+
+// benchResultAnchor is an anchor variable to store the result of a benchmarked
+// code so that compiler could never optimize away the call to runMarkdown()
+var benchResultAnchor string
+
+func BenchmarkReference(b *testing.B) {
+	params := TestParams{extensions: CommonExtensions}
+	files := []string{
+		"Amps and angle encoding",
+		"Auto links",
+		"Backslash escapes",
+		"Blockquotes with code blocks",
+		"Code Blocks",
+		"Code Spans",
+		"Hard-wrapped paragraphs with list-like lines",
+		"Horizontal rules",
+		"Inline HTML (Advanced)",
+		"Inline HTML (Simple)",
+		"Inline HTML comments",
+		"Links, inline style",
+		"Links, reference style",
+		"Links, shortcut references",
+		"Literal quotes in titles",
+		"Markdown Documentation - Basics",
+		"Markdown Documentation - Syntax",
+		"Nested blockquotes",
+		"Ordered and unordered lists",
+		"Strong and em together",
+		"Tabs",
+		"Tidyness",
+	}
+	var tests []string
+	for _, basename := range files {
+		filename := filepath.Join("testdata", basename+".text")
+		inputBytes, err := ioutil.ReadFile(filename)
+		if err != nil {
+			b.Errorf("Couldn't open '%s', error: %v\n", filename, err)
+			continue
+		}
+		tests = append(tests, string(inputBytes))
+	}
+	b.ResetTimer()
+	for n := 0; n < b.N; n++ {
+		for _, test := range tests {
+			benchResultAnchor = runMarkdown(test, params)
+		}
+	}
 }

smartypants.go

@@ -17,11 +17,14 @@ package blackfriday
 
 import (
 	"bytes"
+	"io"
 )
 
-type smartypantsData struct {
+// SPRenderer is a struct containing state of a Smartypants renderer.
+type SPRenderer struct {
 	inSingleQuote bool
 	inDoubleQuote bool
+	callbacks     [256]smartCallback
 }
 
 func wordBoundary(c byte) bool {
@@ -39,7 +42,7 @@ func isdigit(c byte) bool {
 	return c >= '0' && c <= '9'
 }
 
-func smartQuoteHelper(out *bytes.Buffer, previousChar byte, nextChar byte, quote byte, isOpen *bool) bool {
+func smartQuoteHelper(out *bytes.Buffer, previousChar byte, nextChar byte, quote byte, isOpen *bool, addNBSP bool) bool {
 	// edge of the buffer is likely to be a tag that we don't get to see,
 	// so we treat it like text sometimes
 
@@ -96,6 +99,12 @@ func smartQuoteHelper(out *bytes.Buffer, previousChar byte, nextChar byte, quote
 		*isOpen = false
 	}
 
+	// Note that with the limited lookahead, this non-breaking
+	// space will also be appended to single double quotes.
+	if addNBSP && !*isOpen {
+		out.WriteString("&nbsp;")
+	}
+
 	out.WriteByte('&')
 	if *isOpen {
 		out.WriteByte('l')
@@ -104,10 +113,15 @@ func smartQuoteHelper(out *bytes.Buffer, previousChar byte, nextChar byte, quote
 	}
 	out.WriteByte(quote)
 	out.WriteString("quo;")
+
+	if addNBSP && *isOpen {
+		out.WriteString("&nbsp;")
+	}
+
 	return true
 }
 
-func smartSingleQuote(out *bytes.Buffer, smrt *smartypantsData, previousChar byte, text []byte) int {
+func (r *SPRenderer) smartSingleQuote(out *bytes.Buffer, previousChar byte, text []byte) int {
 	if len(text) >= 2 {
 		t1 := tolower(text[1])
 
@@ -116,7 +130,7 @@ func smartSingleQuote(out *bytes.Buffer, smrt *smartypantsData, previousChar byt
 			if len(text) >= 3 {
 				nextChar = text[2]
 			}
-			if smartQuoteHelper(out, previousChar, nextChar, 'd', &smrt.inDoubleQuote) {
+			if smartQuoteHelper(out, previousChar, nextChar, 'd', &r.inDoubleQuote, false) {
 				return 1
 			}
 		}
@@ -141,7 +155,7 @@ func smartSingleQuote(out *bytes.Buffer, smrt *smartypantsData, previousChar byt
 	if len(text) > 1 {
 		nextChar = text[1]
 	}
-	if smartQuoteHelper(out, previousChar, nextChar, 's', &smrt.inSingleQuote) {
+	if smartQuoteHelper(out, previousChar, nextChar, 's', &r.inSingleQuote, false) {
 		return 0
 	}
 
@@ -149,7 +163,7 @@ func smartSingleQuote(out *bytes.Buffer, smrt *smartypantsData, previousChar byt
 	return 0
 }
 
-func smartParens(out *bytes.Buffer, smrt *smartypantsData, previousChar byte, text []byte) int {
+func (r *SPRenderer) smartParens(out *bytes.Buffer, previousChar byte, text []byte) int {
 	if len(text) >= 3 {
 		t1 := tolower(text[1])
 		t2 := tolower(text[2])
@@ -174,7 +188,7 @@ func smartParens(out *bytes.Buffer, smrt *smartypantsData, previousChar byte, te
 	return 0
 }
 
-func smartDash(out *bytes.Buffer, smrt *smartypantsData, previousChar byte, text []byte) int {
+func (r *SPRenderer) smartDash(out *bytes.Buffer, previousChar byte, text []byte) int {
 	if len(text) >= 2 {
 		if text[1] == '-' {
 			out.WriteString("&mdash;")
@@ -191,7 +205,7 @@ func smartDash(out *bytes.Buffer, smrt *smartypantsData, previousChar byte, text
 	return 0
 }
 
-func smartDashLatex(out *bytes.Buffer, smrt *smartypantsData, previousChar byte, text []byte) int {
+func (r *SPRenderer) smartDashLatex(out *bytes.Buffer, previousChar byte, text []byte) int {
 	if len(text) >= 3 && text[1] == '-' && text[2] == '-' {
 		out.WriteString("&mdash;")
 		return 2
@@ -205,13 +219,13 @@ func smartDashLatex(out *bytes.Buffer, smrt *smartypantsData, previousChar byte,
 	return 0
 }
 
-func smartAmpVariant(out *bytes.Buffer, smrt *smartypantsData, previousChar byte, text []byte, quote byte) int {
+func (r *SPRenderer) smartAmpVariant(out *bytes.Buffer, previousChar byte, text []byte, quote byte, addNBSP bool) int {
 	if bytes.HasPrefix(text, []byte("&quot;")) {
 		nextChar := byte(0)
 		if len(text) >= 7 {
 			nextChar = text[6]
 		}
-		if smartQuoteHelper(out, previousChar, nextChar, quote, &smrt.inDoubleQuote) {
+		if smartQuoteHelper(out, previousChar, nextChar, quote, &r.inDoubleQuote, addNBSP) {
 			return 5
 		}
 	}
@@ -224,15 +238,18 @@ func smartAmpVariant(out *bytes.Buffer, smrt *smartypantsData, previousChar byte
 	return 0
 }
 
-func smartAmp(out *bytes.Buffer, smrt *smartypantsData, previousChar byte, text []byte) int {
-	return smartAmpVariant(out, smrt, previousChar, text, 'd')
+func (r *SPRenderer) smartAmp(angledQuotes, addNBSP bool) func(*bytes.Buffer, byte, []byte) int {
+	var quote byte = 'd'
+	if angledQuotes {
+		quote = 'a'
+	}
+
+	return func(out *bytes.Buffer, previousChar byte, text []byte) int {
+		return r.smartAmpVariant(out, previousChar, text, quote, addNBSP)
+	}
 }
 
-func smartAmpAngledQuote(out *bytes.Buffer, smrt *smartypantsData, previousChar byte, text []byte) int {
-	return smartAmpVariant(out, smrt, previousChar, text, 'a')
-}
-
-func smartPeriod(out *bytes.Buffer, smrt *smartypantsData, previousChar byte, text []byte) int {
+func (r *SPRenderer) smartPeriod(out *bytes.Buffer, previousChar byte, text []byte) int {
 	if len(text) >= 3 && text[1] == '.' && text[2] == '.' {
 		out.WriteString("&hellip;")
 		return 2
@@ -247,13 +264,13 @@ func smartPeriod(out *bytes.Buffer, smrt *smartypantsData, previousChar byte, te
 	return 0
 }
 
-func smartBacktick(out *bytes.Buffer, smrt *smartypantsData, previousChar byte, text []byte) int {
+func (r *SPRenderer) smartBacktick(out *bytes.Buffer, previousChar byte, text []byte) int {
 	if len(text) >= 2 && text[1] == '`' {
 		nextChar := byte(0)
 		if len(text) >= 3 {
 			nextChar = text[2]
 		}
-		if smartQuoteHelper(out, previousChar, nextChar, 'd', &smrt.inDoubleQuote) {
+		if smartQuoteHelper(out, previousChar, nextChar, 'd', &r.inDoubleQuote, false) {
 			return 1
 		}
 	}
@@ -262,7 +279,7 @@ func smartBacktick(out *bytes.Buffer, smrt *smartypantsData, previousChar byte,
 	return 0
 }
 
-func smartNumberGeneric(out *bytes.Buffer, smrt *smartypantsData, previousChar byte, text []byte) int {
+func (r *SPRenderer) smartNumberGeneric(out *bytes.Buffer, previousChar byte, text []byte) int {
 	if wordBoundary(previousChar) && previousChar != '/' && len(text) >= 3 {
 		// is it of the form digits/digits(word boundary)?, i.e., \d+/\d+\b
 		// note: check for regular slash (/) or fraction slash (⁄, 0x2044, or 0xe2 81 84 in utf-8)
@@ -304,7 +321,7 @@ func smartNumberGeneric(out *bytes.Buffer, smrt *smartypantsData, previousChar b
 	return 0
 }
 
-func smartNumber(out *bytes.Buffer, smrt *smartypantsData, previousChar byte, text []byte) int {
+func (r *SPRenderer) smartNumber(out *bytes.Buffer, previousChar byte, text []byte) int {
 	if wordBoundary(previousChar) && previousChar != '/' && len(text) >= 3 {
 		if text[0] == '1' && text[1] == '/' && text[2] == '2' {
 			if len(text) < 4 || wordBoundary(text[3]) && text[3] != '/' {
@@ -332,27 +349,27 @@ func smartNumber(out *bytes.Buffer, smrt *smartypantsData, previousChar byte, te
 	return 0
 }
 
-func smartDoubleQuoteVariant(out *bytes.Buffer, smrt *smartypantsData, previousChar byte, text []byte, quote byte) int {
+func (r *SPRenderer) smartDoubleQuoteVariant(out *bytes.Buffer, previousChar byte, text []byte, quote byte) int {
 	nextChar := byte(0)
 	if len(text) > 1 {
 		nextChar = text[1]
 	}
-	if !smartQuoteHelper(out, previousChar, nextChar, quote, &smrt.inDoubleQuote) {
+	if !smartQuoteHelper(out, previousChar, nextChar, quote, &r.inDoubleQuote, false) {
 		out.WriteString("&quot;")
 	}
 
 	return 0
 }
 
-func smartDoubleQuote(out *bytes.Buffer, smrt *smartypantsData, previousChar byte, text []byte) int {
-	return smartDoubleQuoteVariant(out, smrt, previousChar, text, 'd')
+func (r *SPRenderer) smartDoubleQuote(out *bytes.Buffer, previousChar byte, text []byte) int {
+	return r.smartDoubleQuoteVariant(out, previousChar, text, 'd')
 }
 
-func smartAngledDoubleQuote(out *bytes.Buffer, smrt *smartypantsData, previousChar byte, text []byte) int {
-	return smartDoubleQuoteVariant(out, smrt, previousChar, text, 'a')
+func (r *SPRenderer) smartAngledDoubleQuote(out *bytes.Buffer, previousChar byte, text []byte) int {
+	return r.smartDoubleQuoteVariant(out, previousChar, text, 'a')
 }
 
-func smartLeftAngle(out *bytes.Buffer, smrt *smartypantsData, previousChar byte, text []byte) int {
+func (r *SPRenderer) smartLeftAngle(out *bytes.Buffer, previousChar byte, text []byte) int {
 	i := 0
 
 	for i < len(text) && text[i] != '>' {
@@ -363,38 +380,78 @@ func smartLeftAngle(out *bytes.Buffer, smrt *smartypantsData, previousChar byte,
 	return i
 }
 
-type smartCallback func(out *bytes.Buffer, smrt *smartypantsData, previousChar byte, text []byte) int
+type smartCallback func(out *bytes.Buffer, previousChar byte, text []byte) int
 
-type smartypantsRenderer [256]smartCallback
+// NewSmartypantsRenderer constructs a Smartypants renderer object.
+func NewSmartypantsRenderer(flags HTMLFlags) *SPRenderer {
+	var (
+		r SPRenderer
 
-func smartypants(flags int) *smartypantsRenderer {
-	r := new(smartypantsRenderer)
-	if flags&HTML_SMARTYPANTS_ANGLED_QUOTES == 0 {
-		r['"'] = smartDoubleQuote
-		r['&'] = smartAmp
-	} else {
-		r['"'] = smartAngledDoubleQuote
-		r['&'] = smartAmpAngledQuote
-	}
-	r['\''] = smartSingleQuote
-	r['('] = smartParens
-	if flags&HTML_SMARTYPANTS_DASHES != 0 {
-		if flags&HTML_SMARTYPANTS_LATEX_DASHES == 0 {
-			r['-'] = smartDash
+		smartAmpAngled      = r.smartAmp(true, false)
+		smartAmpAngledNBSP  = r.smartAmp(true, true)
+		smartAmpRegular     = r.smartAmp(false, false)
+		smartAmpRegularNBSP = r.smartAmp(false, true)
+
+		addNBSP = flags&SmartypantsQuotesNBSP != 0
+	)
+
+	if flags&SmartypantsAngledQuotes == 0 {
+		r.callbacks['"'] = r.smartDoubleQuote
+		if !addNBSP {
+			r.callbacks['&'] = smartAmpRegular
 		} else {
-			r['-'] = smartDashLatex
+			r.callbacks['&'] = smartAmpRegularNBSP
+		}
+	} else {
+		r.callbacks['"'] = r.smartAngledDoubleQuote
+		if !addNBSP {
+			r.callbacks['&'] = smartAmpAngled
+		} else {
+			r.callbacks['&'] = smartAmpAngledNBSP
 		}
 	}
-	r['.'] = smartPeriod
-	if flags&HTML_SMARTYPANTS_FRACTIONS == 0 {
-		r['1'] = smartNumber
-		r['3'] = smartNumber
+	r.callbacks['\''] = r.smartSingleQuote
+	r.callbacks['('] = r.smartParens
+	if flags&SmartypantsDashes != 0 {
+		if flags&SmartypantsLatexDashes == 0 {
+			r.callbacks['-'] = r.smartDash
+		} else {
+			r.callbacks['-'] = r.smartDashLatex
+		}
+	}
+	r.callbacks['.'] = r.smartPeriod
+	if flags&SmartypantsFractions == 0 {
+		r.callbacks['1'] = r.smartNumber
+		r.callbacks['3'] = r.smartNumber
 	} else {
 		for ch := '1'; ch <= '9'; ch++ {
-			r[ch] = smartNumberGeneric
+			r.callbacks[ch] = r.smartNumberGeneric
 		}
 	}
-	r['<'] = smartLeftAngle
-	r['`'] = smartBacktick
-	return r
+	r.callbacks['<'] = r.smartLeftAngle
+	r.callbacks['`'] = r.smartBacktick
+	return &r
+}
+
+// Process is the entry point of the Smartypants renderer.
+func (r *SPRenderer) Process(w io.Writer, text []byte) {
+	mark := 0
+	for i := 0; i < len(text); i++ {
+		if action := r.callbacks[text[i]]; action != nil {
+			if i > mark {
+				w.Write(text[mark:i])
+			}
+			previousChar := byte(0)
+			if i > 0 {
+				previousChar = text[i-1]
+			}
+			var tmp bytes.Buffer
+			i += action(&tmp, previousChar, text[i:])
+			w.Write(tmp.Bytes())
+			mark = i + 1
+		}
+	}
+	if mark < len(text) {
+		w.Write(text[mark:])
+	}
 }

testdata/Inline HTML (Simple).html

@@ -1,13 +1,13 @@
 <p>Here's a simple block:</p>
 
 <div>
-    foo
+	foo
 </div>
 
 <p>This should be a code block, though:</p>
 
 <pre><code>&lt;div&gt;
-    foo
+	foo
 &lt;/div&gt;
 </code></pre>
 
@@ -19,11 +19,11 @@
 <p>Now, nested:</p>
 
 <div>
-    <div>
-        <div>
-            foo
-        </div>
-    </div>
+	<div>
+		<div>
+			foo
+		</div>
+	</div>
 </div>
 
 <p>This should just be an HTML comment:</p>

testdata/Inline HTML comments.html

@@ -3,7 +3,7 @@
 <!-- This is a simple comment -->
 
 <!--
-    This is another comment.
+	This is another comment.
 -->
 
 <p>Paragraph two.</p>

testdata/Links, inline style.html

@@ -8,4 +8,6 @@
 
 <p><a href="/url/" title="title has spaces afterward">URL and title</a>.</p>
 
+<p><a href="/url/">URL with backslash\</a>.</p>
+
 <p>[Empty]().</p>

testdata/Links, inline style.text

@@ -8,5 +8,6 @@ Just a [URL](/url/).
 
 [URL and title](/url/ "title has spaces afterward"  ).
 
+[URL with backslash\\](/url/).
 
 [Empty]().

testdata/Markdown Documentation - Basics.html

@@ -244,6 +244,18 @@ numbers and spaces, but are <em>not</em> case sensitive:</p>
 &lt;a href=&quot;http://www.nytimes.com/&quot;&gt;The New York Times&lt;/a&gt;.&lt;/p&gt;
 </code></pre>
 
+<p>It is also common to find other protocols such as ftp used for links:</p>
+
+<p>Input:</p>
+
+<pre><code>For example one may test download speeds [here](ftp://speedtest.tele2.net/)
+</code></pre>
+
+<p>Output:</p>
+
+<pre><code>&lt;p&gt;For example one may test download speeds &lt;a href=&quot;ftp://speedtest.tele2.net/&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;
+</code></pre>
+
 <h3>Images</h3>
 
 <p>Image syntax is very much like link syntax.</p>

testdata/Markdown Documentation - Basics.text

@@ -239,6 +239,15 @@ Output:
     <p>I start my morning with a cup of coffee and
     <a href="http://www.nytimes.com/">The New York Times</a>.</p>
 
+It is also common to find other protocols such as ftp used for links:
+
+Input:
+
+    For example one may test download speeds [here](ftp://speedtest.tele2.net/)
+
+Output:
+
+    <p>For example one may test download speeds <a href="ftp://speedtest.tele2.net/">here</a>.</p>
 
 ### Images ###
 

testdata/Markdown Documentation - Syntax.html

@@ -939,8 +939,8 @@ _   underscore
 []  square brackets
 ()  parentheses
 #   hash mark
-+   plus sign
--   minus sign (hyphen)
++	plus sign
+-	minus sign (hyphen)
 .   dot
 !   exclamation mark
 </code></pre>

testdata/Tabs.html

@@ -13,13 +13,13 @@ indented with spaces</p></li>
 
 <p>And:</p>
 
-<pre><code>    this code block is indented by two tabs
+<pre><code>	this code block is indented by two tabs
 </code></pre>
 
 <p>And:</p>
 
-<pre><code>+   this is an example list item
-    indented with tabs
+<pre><code>+	this is an example list item
+	indented with tabs
 
 +   this is an example list item
     indented with spaces