SiteMaps.RobotsTxt (site_maps v0.1.0)

Renders a robots.txt pointing crawlers to the sitemaps.

SiteMaps.RobotsTxt.render(sitemaps: "https://acme.com/sitemap.xml.gz", extra: ["Disallow: /admin"])
#=> "User-agent: *\nAllow: /\nDisallow: /admin\nSitemap: https://acme.com/sitemap.xml.gz\n"

A sitemap may be given as a URL or as a SiteMaps.Config, e.g. SiteMaps.config!(MyApp.Sitemaps, ctx).

Summary

Functions

render(opts)

Renders the robots.txt content.

sitemap_directive(url)

The Sitemap: directive of a sitemap URL or configuration.

Functions

render(opts)

@spec render(keyword()) :: String.t()

Renders the robots.txt content.

Options

:sitemaps - one sitemap or a list of them (required)
:extra - lines added after Allow: /, e.g. ["Disallow: /admin"]

sitemap_directive(url)

@spec sitemap_directive(String.t() | SiteMaps.Config.t()) :: String.t()

The Sitemap: directive of a sitemap URL or configuration.